Microsoft Sucks Bandwidth
I’ve seen a trend in my server logs this month (day, week, month averages):
1199 10199 32944 msnbot/1.0 (+http://search.msn.com/msnbot.htm)
Microsoft’s MSN search engine crawling is making requests for pages and resources 1.2k times a day on my site, every day. That’s about four times more requests than the total number of resources on my site, or 1/4 of my current request volume (32k/120k raw hits a month). Compared to GoogleBot (and all other bots combined), MSNBot sucks ten times more resources five times more frequently.
What the hell are they doing? And why do their search results still suck?
If my results are at all consistent with other sites, it would appear as if the MSNBot sucks at least 15 times more bandwidth than all other webcrawling combined. I guess it’s no surprise that Microsoft sucks more than most. I only wonder if there is a point to crawling sites so frequently.