It’s a funny story. I’ve been testing out various styles of fun articles on the site to see if there’s any money to be made as a B-list obscure blogger. I still post the thinker-links and the occasional serious bit, but I tend to be funnier than I am poignant, and the funny stuff is fun to write¹.

Anyway, my server has been crushed several times this year², and I’ve found a few simple ways of coping. But first, what does it look like to crush a Dreamhost server?

$ uptime
... load average: 198.75, 201.04, 143.88

That’s a 50x overloaded server, not considering DB or I/O load³. But today’s traffic threw it right off the map, as the box was seeing more than 100 connections per second (compared to the previous max of 10/second).

So how do you pull yourself out from under a good server-kicking?

Step 1 - Pull the plug

It’s simple:

mv site.com site.com.off

You’ll lose a few thousand visitors, but that’s the price you pay for not planning ahead.

Step 2 - Find and cache

Create one directory per busy article, based on the path setup for your site. Dig through your browser cache⁴, and save each busy page at their full path as static html.

So if your hosed page is http://site.com/2007/10/31/funny-ha-ha/, you mkdir -p 2007/10/31/funny-ha-ha/ and add a static html file with the content in it. Disable any other CGI calls on the page (like comments), and add some ads if they’re not there already.

Step 3 - Re-enable site

By this point your server has stopped bleeding processes and has a chance of sustaining some static traffic. Move the disabled site back to its original name, and run top while tailing your server logs looking for errors in your static pages.

Total time to hack-cache-resurrect your site should be no more than 2-3 minutes. And your server should look something like:

... load average: 11.41, 12.50, 24.68

It’s not perfect, but it’s functional.

Step 4 - Clean up

Don’t forget to clean up your static pages when the storm subsides. Remove the static pages, and consider better planning for the next time.

But what about wp-caching?

I’ve used wp-cache (both the plugin and the built-in object cache), and found that they aren’t as effective as simple, static pages. The plugin form of the cache still hits the DB on each page load, and the built-in object cache doesn’t seem to improve things against heavy loads⁵.

(Note: I’m going to give the updated wp-cache plugin another whirl, see if it has improved.)

1. Duh ↩
2. Today is the first time I’ve had to intervene, a 10x increase from the previous largest crushing ↩
3. The server is a capable 4-way system, with a few other semi-busy sites on it, so a normal storm doesn’t phase it ↩
4. You cached the pages, right? ↩
5. I need to measure this, and verify that I’m not missing some configuration ↩

Rewriting Reddit. The story of Reddit’s rewrite from Lisp to Python.

A reddit-like site about YCombinator news, staring mostly startup and web2.0 stuff.

An Ask Reddit about people’s favorite Firefox extensions.

The new reddit of quotations, bigcite.