The Robots.txt file

One of the overlooked ways in which a website can get into trouble is by not fully utilizing the robots.txt file or even worse, not having one at all. Here's what a robots.txt file looks like- or you can simply visit ours at www.matrixmarketing.pl/robots.txt

The robots.txt file is there to tell web crawlers what to index and what they shouldn't index.

For example, let's say your web designer uses a folder for testing some new look and feel of your site called "v2". If the folder "v2" is a duplicate version of your website text in a new look, and is found by Googlebot, you will be slapped with a duplicate content penalty. And if you think Google will never find pages that aren't public or linked anywhere internally on your site, think again. Some of our clever competitors may know that you do temp work in the "v2" folder or the "temp" folder and PURPOSELY link to the site (via some free blog software perhaps), bookmark it, ping it, and voila! There it is, all of your duplicated content being indexed in Google- and you're left with the bill.

Another one of the several uses of the Robots.txt is to BLOCK bad robots from coming to your site, scraping your content, images, etc and stealing your bandwidth in the process. The Robots file is the first line of defense against the bad bots, and most webmasters have their front door wide open for all of these email and site scrapers to walk right in.

As Michael said to Dwight in The Office, "Don't be an idiot"...

Get a robots.txt file!!