Last week, a Program Manager at Bing Webmaster Center had admitted that MSNBot is fairly slow at indexing. However, that is not the only problem that plagues the bot utilized by Bing to index the web. It also happens to be quite stupid.
MSNBot was single handedly responsible for knocking out the CPAN Testers server. In a weird demonstration of incompetence, Microsoft Bing unleashed 20-30 bots every few seconds, which pretty much amounted to a denial of service (DOS) attack. In order to avoid this kind of unwanted problems, most other search engines (including Google) have a policy of allowing only one bot to access a site at a time.
The webmaster also alleges that MSNBot ignored the rules specified in robots.txt. If true, it is a particularly troubling issue, since robots.txt was created in the first place to control various automated bots. Recently, GitHub also experienced similar problems with the MSNBot.
MSNBot seems to be in a desperate need for an upgrade. If Microsoft is serious about challenging Google’s dominance, they should begin paying more attention to their crawler. It’s not a good practice to allow your bot to crash webservers.