I installed a translator plugin on one of my WordPress blogs but the plugin wasn't working properly so I disabled it but two days later I found out that my Google web master tools account was reporting about 1100 'Not Found' errors under the 'Web crawl errors' section. All the errors were from translated versions of my blog. I used the 'robots.txt' file to fix this issue. If you don't know what a 'robots.txt' file is, then read the article titled how to control access of the web crawlers or web robots to your site. Basically, add rules to your 'robots.txt' file to Disallow any spider from indexing the translated version of the pages. My 'robots.txt' file looks like the following Depending on your situation you might need to block more languages. Just look in the Google webmaster tools and see which languages are causing the error then add them to the Disallow rule. [lang=css] User-Agent: * # Language pages Disallow: /ar/* Disallow: /bg/* Disallow: /zh-hant/* Disallow: /ca/* Disallow: /cs/* Disallow: /da/* Disallow: /de/* Disallow: /el/* Disallow: /es/* Disallow: /fi/* Disallow: /fr/* Disallow: /he/* Disallow: /hi/* Disallow: /hr/* Disallow: /id/* Disallow: /it/* Disallow: /iw/* Disallow: /ja/* Disallow: /ko/* Disallow: /lt/* Disallow: /lv/* Disallow: /mr/* Disallow: /nl/* Disallow: /no/* Disallow: /pl/* Disallow: /pt-br/* Disallow: /pt/* Disallow: /ro/* Disallow: /ru/* Disallow: /sk/* Disallow: /sl/* Disallow: /sr/* Disallow: /sv/* Disallow: /tl/* Disallow: /tr/* Disallow: /uk/* Disallow: /vi/* Disallow: /zh-CN/* Allow: / [/lang] As far as I know, Google penalizes ...

There are numerous reasons as to why or when you should control the access of the web robots or web crawlers to your site.  As much as you want Googlebot to come to you site, you don't want the spam bots to come and collect private information from your site. Not to mention that when a robot crawls your site it uses the website's bandwidth too! In this post I have explained how you can control the access of the web robots to your site through the usage of a simple 'robots.txt' file. What are web robots or web spiders? Web Robots (also known as bots, web spiders, web crawlers, Ants) are programs that traverses the World Wide Web in an automated manner. Search engines (like Google, Yahoo etc.) use web crawlers to index the web pages to provide up to date data. Why use 'robots.txt' file? Gooble bot may be crawling your site to provide better search results but at the same time other spam bots may be collecting personal information such as email addresses for spamming purpose. If you want to control the access of the web crawlers on your site, you can do so by using the "robots.txt" file. How do I ...