Not Only Is Bing’s Crawler (MSNBot) Slow, But Also Stupid

microsoft-bingLast week, a Program Manager at Bing Webmaster Center had admitted that MSNBot is fairly slow at indexing. However, that is not the only problem that plagues the bot utilized by Bing to index the web. It also happens to be quite stupid.

MSNBot was single handedly responsible for knocking out the CPAN Testers server. In a weird demonstration of incompetence, Microsoft Bing unleashed 20-30 bots every few seconds, which pretty much amounted to a denial of service (DOS) attack. In order to avoid this kind of unwanted problems, most other search engines (including Google) have a policy of allowing only one bot to access a site at a time.

The webmaster also alleges that MSNBot ignored the rules specified in robots.txt. If true, it is a particularly troubling issue, since robots.txt was created in the first place to control various automated bots. Recently, GitHub also experienced similar problems with the MSNBot.

MSNBot seems to be in a desperate need for an upgrade. If Microsoft is serious about challenging Google’s dominance, they should begin paying more attention to their crawler. It’s not a good practice to allow your bot to crash webservers.

Published by

Pallab De

Pallab De is a blogger from India who has a soft spot for anything techie. He loves trying out new software and spends most of his day breaking and fixing his PC. Pallab loves participating in the social web; he has been active in technology forums since he was a teenager and is an active user of both twitter (@indyan) and facebook .