Not Only Is Bing’s Crawler (MSNBot) Slow, But Also Stupid
By on January 16th, 2010

microsoft-bingLast week, a Program Manager at Bing Webmaster Center had admitted that MSNBot is fairly slow at indexing. However, that is not the only problem that plagues the bot utilized by Bing to index the web. It also happens to be quite stupid.

MSNBot was single handedly responsible for knocking out the CPAN Testers server. In a weird demonstration of incompetence, Microsoft Bing unleashed 20-30 bots every few seconds, which pretty much amounted to a denial of service (DOS) attack. In order to avoid this kind of unwanted problems, most other search engines (including Google) have a policy of allowing only one bot to access a site at a time.

The webmaster also alleges that MSNBot ignored the rules specified in robots.txt. If true, it is a particularly troubling issue, since robots.txt was created in the first place to control various automated bots. Recently, GitHub also experienced similar problems with the MSNBot.

MSNBot seems to be in a desperate need for an upgrade. If Microsoft is serious about challenging Google’s dominance, they should begin paying more attention to their crawler. It’s not a good practice to allow your bot to crash webservers.

Tags: ,
Author: Pallab De Google Profile for Pallab De
Pallab De is a blogger from India who has a soft spot for anything techie. He loves trying out new software and spends most of his day breaking and fixing his PC. Pallab loves participating in the social web; he has been active in technology forums since he was a teenager and is an active user of both twitter (@indyan) and facebook .

Pallab De has written and can be contacted at pallab@techie-buzz.com.
  • http://www.fundybuzz.com Williams

    I hate bing. They do all late work. They index my site very slowly.

    They have to improve , or else they get bad reputations.

  • Vishwas

    there are a numerous issues reported about Bing's bots and SEO too. Its a known bug that Bing doesn't recognize nofollow rules and indexes stuff even if you mark it as nofollow.
    source: http://technofriends.in/2009/08/06/bing-is-indexi

  • MikeKallis

    Bing didnt crawl my website http://www.addenda.us , pages are not updated in search result, website is regularly modified

 
Copyright 2006-2012 Techie Buzz. All Rights Reserved. Our content may not be reproduced on other websites. Content Delivery by MaxCDN