Hartley Brody

Surviving the Front Page of HN

Last night, before I went to bed, I wrote a short article about Google’s awesome online Python classes, posted the link on Hacker News, and then fell asleep.

When I woke up this morning, I was excited to find that the link had made it all the way to the front page, with hundreds of upvotes!

But as I read the comments on the HN thread, my excitement quickly turned to terror. People were having trouble accessing my site – the huge wave of traffic had knocked my server offline, and it had been down for a few hours!

It was so backed up that I couldn’t even SSH in, but eventually I was able to restart everything and get it all back online. I kept a nervous eye on my analytics as I tried to figure out what went wrong, and how to prevent this from happening again.


Monitoring Issues

I purchased the Rackspace server about a year ago, and have used it to host a number of small projects, including this blog. I had also setup a Pingdom test and configured Cloudflare protection to help keep an eye on things for me.

However, I had apparently misconfigured my Pingdom check, so I didn’t receive any emails or push alerts to my phone (they have an app!) to know that something was amiss.

I had setup a TCP Port Check instead of an HTTP Check. And while my server was running out of memory and apache was returning 5xx response codes out the wazoo, TCP was humming along just fine, so Pingdom thought everything was a-okay. I made sure to change that.

Cloudflare actually held up pretty well, which made it a bit hard to tell if my server was actually down at first. From what I observed, Cloudflare served static, cached copies of my article about half the time, even though the server was non-responsive. I thought that was pretty cool.


While I love Wordpress as a blogging engine, it can be a bit of a memory hog.

From what I could tell, Wordpress was using about 40MB - 50MB per request, which means my 1GB of RAM could only support about 20 concurrent users before it started using swap space. While this was enough to cover me for most situations, it didn’t last long when the hits started pouring in.

After looking at a number of caching plugins, I decided to go with Really Static. It’s a pretty sweet plugin which generates, caches and then serves static HTML files for every page of my site. This means requests don’t hit the database or execute any PHP, unless a post is updated.

Theoretically, this provides enormous memory savings. From my tests so far, it seems like requests are using less than 10MB of memory each, which should support approximately 100 concurrent users. Much better.

The Hardware

The server I have been using is the standard “1GB RAM, 40GB disk” flavor which runs about $40 a month from Rackspace.

And while that sized server wasn’t able to withstand the onslaught of traffic from HN, I’ve actually decided to downsize it following this whole dilemma.

Why? I’ll use the extra cash to put a load balancer in front of it.

I spent some time researching the ideal base configuration for a website that can scale quickly and cheaply and it looks something like this:

  1. A load balancer that receives incoming traffic
  2. One or more "web heads" that run Apache and PHP (if you're trying to scale a LAMP stack)
  3. A database server that fetches data for the web heads

Experienced IT-Ops people will know that this is pretty basic and can be expanded on in a number of ways, like adding read-only DB slave backups and all of that jazz.

For now, I’m going to stick with just the load balancer and a single small web head. If I’m feeling adventurous this weekend, I might split the database out onto its own server.

If I downsize everything correctly, I won’t end up paying any more than I already am, but I’ll have a solution that’ll scale to handle traffic spikes much more easily in the future.

Sticking with Rackspace over AWS

AWS seems to be the “cool” way to host a website these days – it’s certainly pretty cheap – and I’ve been thinking of trying them out for awhile. I poked around their docs this morning, but ultimately decided to stick with Rackspace.

In a nutshell, here’s why: Amazon sells services, but Rackspace sells servers + service. See what I did there?

I’m the kind of crazy person who wants to know how everything works, and set it up myself, even if I really have no idea what I’m doing. If you just want compute power, or a database service, and don’t care about how they work or why, Amazon is probably the way to go.

But Rackspace is much better if you’re a tinkerer like me. Plus, if you screw everything up, you always have their fanatical support to back you up.

I called Rackspace this afternoon to get some advice on how to scale my site in the future, and ended up spending over an hour on the phone with a support engineer as he showed me all the different ways I could check on my server and all of the ways I should expect it to fail. He was brilliant.

I’m not an IT-Ops guru yet, but those guys definitely are. Having them on-call 24/7/365 is a service I’d gladly shell out a little extra for.

In case I hit the front page of HN again ;)