Comments on: Microsoft Sucks Bandwidth http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/ A technical tumblelog of links and articles on programming, design, and other geek interests Wed, 02 Jun 2010 20:28:53 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Matt Cutts: Gadgets, Google, and SEO » Crawl caching proxy http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/comment-page-1/#comment-1329 Matt Cutts: Gadgets, Google, and SEO » Crawl caching proxy Sun, 23 Apr 2006 21:07:38 +0000 http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/#comment-1329 <p>[...] As part of the Bigdaddy infrastructure switchover, Google has been working on frameworks for smarter crawling, improved canonicalization, and better indexing. On the smarter crawling front, one of the things we’ve been working on is bandwidth reduction. For example, the pre-Bigdaddy webcrawl Googlebot with user-agent “Googlebot/2.1 (+http://www.google.com/bot.html)” would sometimes allow gzipped encoding. The newer Bigdaddy Googlebots with user-agent “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” are much more likely to support gzip encoding. That reduces Googlebot’s bandwidth usage for site owners and webmasters. From my conversations with the crawl/index team, it sounds like there’s a lot of head-room for webmasters to reduce their bandwith by turning on gzip encoding. [...]</p> As part of the Bigdaddy infrastructure switchover, Google has been working on frameworks for smarter crawling, improved canonicalization, and better indexing. On the smarter crawling front, one of the things we’ve been working on is bandwidth reduction. For example, the pre-Bigdaddy webcrawl Googlebot with user-agent “Googlebot/2.1 (+http://www.google.com/bot.html)” would sometimes allow gzipped encoding. The newer Bigdaddy Googlebots with user-agent “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” are much more likely to support gzip encoding. That reduces Googlebot’s bandwidth usage for site owners and webmasters. From my conversations with the crawl/index team, it sounds like there’s a lot of head-room for webmasters to reduce their bandwith by turning on gzip encoding.

]]> By: ChipCuccio.US http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/comment-page-1/#comment-327 ChipCuccio.US Wed, 31 Aug 2005 23:27:34 +0000 http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/#comment-327 <p><strong>No more MSNbot</strong></p> <pre class='prettyprint'><code>Today I added the following rule/exclusion from my robots.txt; User-agent: msnbot Disallow: / Why? Because MSN’s spiders request too many resources, too frequently. MSN spiders don’t play nice, and I’ve had my eye on them... </code></pre> No more MSNbot

Today I added the following rule/exclusion from my robots.txt;
     User-agent: msnbot
 Disallow: /
Why? Because MSN&#8217;s spiders request too many resources, too frequently. MSN spiders don&#8217;t play nice, and I&#8217;ve had my eye on them...
]]> By: mx http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/comment-page-1/#comment-326 mx Fri, 26 Aug 2005 17:25:38 +0000 http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/#comment-326 <p>It also looks like MSNBot isn't caching things very well. If it's re-grabbing all the pages on a daily basis, then they're not looking at the last-modified header. It also looks like the bot is re-reading resources (images) that are on each page. Looks like a very beta-version, or maybe they just don't care.</p> It also looks like MSNBot isn’t caching things very well. If it’s re-grabbing all the pages on a daily basis, then they’re not looking at the last-modified header. It also looks like the bot is re-reading resources (images) that are on each page. Looks like a very beta-version, or maybe they just don’t care.

]]> By: Steven Fisher http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/comment-page-1/#comment-325 Steven Fisher Fri, 26 Aug 2005 17:00:30 +0000 http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/#comment-325 <p>Well, you know me well enough to know I'm not a Microsoft apologist. Still, I'd like to believe that they could get something like a spider correct.</p> <p>The evidence seems to be against it, though. Thanks for the link. Just to be safe, I've added a line to robots.txt excluding msnbot... hopefully their spider works at least that well by now.</p> Well, you know me well enough to know I’m not a Microsoft apologist. Still, I’d like to believe that they could get something like a spider correct.

The evidence seems to be against it, though. Thanks for the link. Just to be safe, I’ve added a line to robots.txt excluding msnbot… hopefully their spider works at least that well by now.

]]> By: mx http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/comment-page-1/#comment-324 mx Fri, 26 Aug 2005 16:47:49 +0000 http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/#comment-324 <p>I can find quite a few complaints from Google too: [google: MSNBot hits greedy].</p> I can find quite a few complaints from Google too: .

]]> By: mx http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/comment-page-1/#comment-323 mx Fri, 26 Aug 2005 16:46:22 +0000 http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/#comment-323 <p>From what I can see the hits are all from a single group of IPs that reverse-map to Microsoft (sampling a few daily logs). I've read elsewhere that the MSNBot is aggressive, but only this month has it eclipsed GoogleBot on my site. It may be possible that the MSNBot is buggy, or that someone has hijacked some machines in their IP range (though that seems unlikely).</p> <p>I would exclude it from crawling my site, except that the exposure is probably not a bad thing. It just seems that they're beeing a bit greedy (to the point of insanity) with their crawling.</p> From what I can see the hits are all from a single group of IPs that reverse-map to Microsoft (sampling a few daily logs). I’ve read elsewhere that the MSNBot is aggressive, but only this month has it eclipsed GoogleBot on my site. It may be possible that the MSNBot is buggy, or that someone has hijacked some machines in their IP range (though that seems unlikely).

I would exclude it from crawling my site, except that the exposure is probably not a bad thing. It just seems that they’re beeing a bit greedy (to the point of insanity) with their crawling.

]]> By: Steven Fisher http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/comment-page-1/#comment-322 Steven Fisher Fri, 26 Aug 2005 15:58:40 +0000 http://warpedvisions.org/2005/08/26/microsoft-sucks-bandwidth/#comment-322 <p>Is that based strictly on the user agent? If so, it is probably spammers crawling with the same user agent string.</p> Is that based strictly on the user agent? If so, it is probably spammers crawling with the same user agent string.

]]>