Not Only Is Bing’s Crawler (MSNBot) Slow, But Also Stupid
By on January 16th, 2010

microsoft-bingLast week, a Program Manager at Bing Webmaster Center had admitted that MSNBot is fairly slow at indexing. However, that is not the only problem that plagues the bot utilized by Bing to index the web. It also happens to be quite stupid.

MSNBot was single handedly responsible for knocking out the CPAN Testers server. In a weird demonstration of incompetence, Microsoft Bing unleashed 20-30 bots every few seconds, which pretty much amounted to a denial of service (DOS) attack. In order to avoid this kind of unwanted problems, most other search engines (including Google) have a policy of allowing only one bot to access a site at a time.

The webmaster also alleges that MSNBot ignored the rules specified in robots.txt. If true, it is a particularly troubling issue, since robots.txt was created in the first place to control various automated bots. Recently, GitHub also experienced similar problems with the MSNBot.

MSNBot seems to be in a desperate need for an upgrade. If Microsoft is serious about challenging Google’s dominance, they should begin paying more attention to their crawler. It’s not a good practice to allow your bot to crash webservers.

Tags: ,
Author: Pallab De Google Profile for Pallab De
Pallab De is a blogger from India who has a soft spot for anything techie. He loves trying out new software and spends most of his day breaking and fixing his PC. Pallab loves participating in the social web; he has been active in technology forums since he was a teenager and is an active user of both twitter (@indyan) and facebook .

Pallab De has written and can be contacted at pallab@techie-buzz.com.

Leave a Reply

Name (required)

Website (optional)

 
    Warning: call_user_func() expects parameter 1 to be a valid callback, function 'advanced_comment' not found or invalid function name in /home/keith/techie-buzz.com/htdocs/wp-includes/comment-template.php on line 1694
 
Copyright 2006-2012 Techie Buzz. All Rights Reserved. Our content may not be reproduced on other websites. Content Delivery by MaxCDN