Not Only Is Bing’s Crawler (MSNBot) Slow, But Also Stupid

microsoft-bingLast week, a Program Manager at Bing Webmaster Center had admitted that MSNBot is fairly slow at indexing. However, that is not the only problem that plagues the bot utilized by Bing to index the web. It also happens to be quite stupid.

MSNBot was single handedly responsible for knocking out the CPAN Testers server. In a weird demonstration of incompetence, Microsoft Bing unleashed 20-30 bots every few seconds, which pretty much amounted to a denial of service (DOS) attack. In order to avoid this kind of unwanted problems, most other search engines (including Google) have a policy of allowing only one bot to access a site at a time.

The webmaster also alleges that MSNBot ignored the rules specified in robots.txt. If true, it is a particularly troubling issue, since robots.txt was created in the first place to control various automated bots. Recently, GitHub also experienced similar problems with the MSNBot.

MSNBot seems to be in a desperate need for an upgrade. If Microsoft is serious about challenging Google’s dominance, they should begin paying more attention to their crawler. It’s not a good practice to allow your bot to crash webservers.

Published by

Pallab De

Pallab De is a blogger from India who has a soft spot for anything techie. He loves trying out new software and spends most of his day breaking and fixing his PC. Pallab loves participating in the social web; he has been active in technology forums since he was a teenager and is an active user of both twitter (@indyan) and facebook .

  • I hate bing. They do all late work. They index my site very slowly.

    They have to improve , or else they get bad reputations.

  • Vishwas

    there are a numerous issues reported about Bing's bots and SEO too. Its a known bug that Bing doesn't recognize nofollow rules and indexes stuff even if you mark it as nofollow.

  • MikeKallis

    Bing didnt crawl my website , pages are not updated in search result, website is regularly modified

  • Brenda Bell

    They lie. Just this morning, my evasion tactics had to disable requests because they were doing a GET for /robots.txt 5 times per second. The rules simply don’t apply to them.