How to Clean/Remove Not Found Errors from Google web master tools generated from translated versions

Categories: Troubleshooting

I installed a translator plugin on one of my WordPress blogs but the plugin wasn’t working properly so I disabled it but two days later I found out that my Google web master tools account was reporting about 1100 ‘Not Found’ errors under the ‘Web crawl errors’ section. All the errors were from translated versions of my blog. I used the ‘robots.txt’ file to fix this issue.

If you don’t know what a ‘robots.txt’ file is, then read the article titled how to control access of the web crawlers or web robots to your site.

Basically, add rules to your ‘robots.txt’ file to Disallow any spider from indexing the translated version of the pages. My ‘robots.txt’ file looks like the following Depending on your situation you might need to block more languages. Just look in the Google webmaster tools and see which languages are causing the error then add them to the Disallow rule.

User-Agent: * # Language pages Disallow: /ar/* Disallow: /bg/* Disallow: /zh-hant/* Disallow: /ca/* Disallow: /cs/* Disallow: /da/* Disallow: /de/* Disallow: /el/* Disallow: /es/* Disallow: /fi/* Disallow: /fr/* Disallow: /he/* Disallow: /hi/* Disallow: /hr/* Disallow: /id/* Disallow: /it/* Disallow: /iw/* Disallow: /ja/* Disallow: /ko/* Disallow: /lt/* Disallow: /lv/* Disallow: /mr/* Disallow: /nl/* Disallow: /no/* Disallow: /pl/* Disallow: /pt-br/* Disallow: /pt/* Disallow: /ro/* Disallow: /ru/* Disallow: /sk/* Disallow: /sl/* Disallow: /sr/* Disallow: /sv/* Disallow: /tl/* Disallow: /tr/* Disallow: /uk/* Disallow: /vi/* Disallow: /zh-CN/* Allow: /
As far as I know, Google penalizes for duplicate content. Translated version of your page is considered duplicate content so for SEO benefit it is best to use this method to block access to the translated version of a web page.

It took about two weeks for all the errors to go away from my Google webmaster tools account but the number of errors started to go down as soon as I updated my robots.txt file to block the spiders from crawling all the translated version of the site. Hope this helps.

Articles you may also like:

#1 by Miniclip on January 3, 2012 - 4:37 am

thanks a lot… you save me from this annoying not found errors..

#2 by admin on October 28, 2011 - 5:59 pm

If you use the Google webmaster tool it will only affect Google. The robot.txt is helpful for all search engine bots.

#3 by Hezy on October 28, 2011 - 4:40 am

Hi, thanks for this great tutorial. Is there any difference if I use robot.txt file for removing pages from index instead of using webmaster tools?

#4 by dualsim on September 14, 2011 - 7:30 am

thanks for your post its helpful…..

#5 by Jack on September 12, 2011 - 10:59 pm

Thank you for sharing. That’s good for me.

#6 by New Movie Download on August 22, 2011 - 2:21 pm

i face this problem & this tips is so helpful for me.
thanks a lot.

#7 by nike air max store on April 9, 2011 - 5:34 am

Thanks for your sharing the information!

#8 by NikeShoesClearance on March 23, 2011 - 11:49 pm

Good share, thanks a lot

#9 by replica orologi on March 16, 2011 - 1:14 am

Thanks for your sharing.They are very useful.

#10 by admin on February 19, 2011 - 10:56 pm

@Watson, I have cleaned up the code a bit in this post. asterisk (*) is a wild card witch mean everything under that directory. So you are telling the bots that ignore everything under that language directory.

Yes, it is important to have the following two lines in the robots.txt file as the first line says who this rule applies to (again asterisk mean everyone) and the 2nd line is saying allow access to everything else under the root (it won’t access the stuff that you specified in the disallow command)

User-Agent: *
Allow: /

#11 by Watson on February 19, 2011 - 5:09 am

Is it crucial to have that code above and below the Disallows, and is it important to have the asterisk ” * ” after each Disallow?

Just want to double check to make sure I don’t mess it up.

#12 by Watson on February 18, 2011 - 10:47 pm

Terrific. Thanks for the help!!!!!

A million thanks.

#13 by admin on February 18, 2011 - 10:38 pm

@Watson, You don’t need to delete the robots.txt file as it is a standard file for controlling access of the web crawlers. You can read more about this here:

http://www.tipsandtricks-hq.com/how-to-control-access-of-the-web-crawlers-or-web-robots-to-your-site-166

No, the “robots.txt” file will not affect your human visitors to the site.

#14 by Watson on February 18, 2011 - 6:33 am

I have had the same problem. Since the plug in was active I have thousands of errors.

Thanks for laying out the details about the robots.txt file. Hope it helps.

Questions:
1) I assume the robots.txt file will fix the current errors (along with deleting the WP-google translator app). How long should the robots.txt file exist before it is deleted?
2) Will the robots.txt file effect visitors from using their own translator tools to read content?

Thanks,
Jay

#15 by Togrul on January 18, 2011 - 9:23 am

Thanks for sharing,

I was looking for these tips.

Cheers,
Togrul
.-= Togrul´s last blog ..How to get more people to follow you on Twitter =-.

#16 by Michelle on January 7, 2011 - 3:26 am

Thanks for your sharing the information. They are very useful.

#17 by John Gamings on January 3, 2011 - 11:06 am

Thanks for these tips. I’ve always had trouble remembering to play with my robots.txt file when I make new sites, but fortunately this post helped remind me

#18 by mcse on December 20, 2010 - 3:44 am

souds good ,thanks a lot

#19 by Jim on December 14, 2010 - 10:36 pm

I had no idea that the duplicate content penalty was affected by different language versions of a site. Seems unfair, especially as having multiple language versions shows that you are actually presenting a more robust site.

#20 by shegaoxia on October 20, 2010 - 1:09 am

Happy to see your blog as it is just what I’ve looking for and excited to read all the posts. I am looking forward to another great article from you. After skimming through your website.

#21 by Emily on September 28, 2010 - 2:18 am

Thanks for sharing this informative article.My website recently ran into this problem. Your info is very helpful.
Bookmarked and I’ll back to see you updates. Thanks again.

#22 by Generator on hire on September 22, 2010 - 5:40 am

I really apprieciate this article for it will do good to me in the future

#23 by jerry on September 15, 2010 - 2:49 am

quite useful info, thanks. I will try it.
.-= jerry´s last blog ..Internet Marketing Tips updated Sun Sep 12 2010 9-24 pm CDT =-.

#24 by Myron on September 11, 2010 - 4:39 am

It helps a lot.

Many thanks,

robots.txt is very useful!

#25 by gucci on August 20, 2010 - 12:15 pm

Happy to see your blog as it is just what I’ve looking for and excited to read all the posts. I am looking forward to another great article from you. After skimming through your website.
.-= gucci´s last blog ..Gucci Handbag Hobos 232955 FAFXG 9761 =-.

#26 by christian louboutin on August 16, 2010 - 10:42 pm

Well, personally, I use the plugin called all in one seo. It is very effective and I recommend everyone to use it.

#27 by sunny on August 16, 2010 - 5:06 am

Great blog,thank you for your sharing!!!
.-= sunny´s last blog ..PUMA DESIGNER WOMENS SANDELS-IPY005140 =-.

#28 by Richard Chidike | Motivatory on August 15, 2010 - 5:32 am

Thank God i am able to find this post. I have been searching for months on how to fix not found errors on my blog and this post has come to my aid.
.-= Richard Chidike | Motivatory´s last blog ..How To Create Wealth With Domain Names =-.

#29 by baofeng on August 11, 2010 - 10:53 pm

that good article,thanks! i will back soon
.-= baofeng´s last blog ..Well-known domestic enterprises which have enjoyed the service =-.

#30 by Hieu Martin@Blog Tips on August 7, 2010 - 12:07 pm

Thanks for share this tip for .htta
.-= Hieu Martin@Blog Tips´s last blog ..Four Easy Tips For Blog Marketing Online =-.

#31 by Electrosurgical Pad on August 3, 2010 - 2:40 am

Thank you for the reminder, It is certainly a lot useful.I’ll record it

#32 by headphones on August 1, 2010 - 11:14 pm

Thanks a lot for sharing the tip. It is certainly a lot useful. Most of the times I used to fix it manually but now I will use this tip

#33 by fengcheng on July 15, 2010 - 4:59 am

very great idea thanks to share ,Very useful icon sets, tweeted and saved on Delicious.

#34 by Kate on June 16, 2010 - 3:36 am

Thank you for sharing these tips.
.-= Kate´s last undefined ..If you register your site for free at =-.

#35 by Neil Young on June 15, 2010 - 3:12 am

You are a geniu for this idea .

#36 by custom promotional on June 14, 2010 - 9:30 pm

#37 by Bearings on May 11, 2010 - 6:09 am

Actually, someone will use other language article by google translation tool, if the content is not in one website, it’s hard for google to find it.
.-= Bearings´s last undefined ..Response cached until Wed 12 @ 11:02 GMT (Refreshes in 23.91 Hours) =-.

#38 by fitted hat on May 6, 2010 - 2:56 am

Thank you for sharing such a good passage

#39 by admin1 on May 4, 2010 - 2:48 am

we i have read all the articles. Very useful information was written. Thanks ….
.-= admin1´s last blog ..Macromedia dreamweaver cs3 free download =-.

#40 by admin on March 22, 2010 - 8:39 pm

If you have translated content then it gets treated as duplicate content as it is the same content in different language. It’s best to minimize duplicate content.

#41 by Noter on March 22, 2010 - 9:28 am

is it true that google hate translated content? because I use auto translate plugin on all of my blogs
.-= Noter´s last blog ..Kindle for BlackBerry Free Software =-.

#42 by hookah on March 7, 2010 - 2:49 am

I didn’t know they don’t like sites that have the same content that’s translate. I’ll have to use that robot file.

#43 by girisim on February 5, 2010 - 12:12 pm

I have read all the articles. Very useful information was written. Thanks

#44 by Raul Gonzelous on January 30, 2010 - 5:20 am

Thanks for the great article it is really useful you should also deny access to all inside folders

#45 by Forum Indonesia on January 11, 2010 - 2:14 pm

#46 by turisuna on January 10, 2010 - 10:43 am

Hmmm it sounds little bit complicated. Sometimes I found some error reports from google webmaster tools but not too much, so I usually fix it manually and doesn’t take too much time. But thanks for this information