Lately I've gotten fed up with Lighttpd. There's been outstanding bugs that are so familiar they've acquired names. The project's lead, Jan Kneche, seems more interested in schmoozing up to the Rails crowd than providing a decent product (which is ironic given that Mongrel aims to make Lighttpd irrelevant in the Rails world).

Anyway, a discussion today on the TurboGears list brought up an alternative.

Bob Ippolito replied to a discussion between myself and another person about whether to use Pound or Lighttpd as a reverse-proxy in front of TurboGears applications. I held that Pound is the correct solution as it is a proxy, whereas Lighttpd is a web server that can act as one. Further, I expressed my frustration regarding the state of Lighttpd development and the unmaintainability of its config files.

Bob offered up the following information:

One problem with Lighty is that it leaks memory like a sieve [1]. I audited it for a little bit and I gave up, it's a mess. I'd steer clear of it, it will quickly ruin your day if you throw a lot of traffic at it.

The only solution I know of that's extremely high performance that offers all of the features that you want is nginx [2], but its documentation is largely in Russian. I can't read Russian, but I was able to figure it out (the configuration language isn't Russian, neither is C source). I currently have nginx doing reverse proxy of over tens of millions of HTTP requests per day (thats a few hundred per second) on a single server. At peak load it uses about 15MB RAM and 10% CPU on my particular configuration (FreeBSD 6).

Under the same kind of load, apache falls over (after using 1000 or so processes and god knows how much RAM), pound falls over (too many threads, and using 400MB+ of RAM for all the thread stacks), and lighty leaks more than 20MB per hour (and uses more CPU, but not significantly more).

[1] http://trac.lighttpd.net/trac/ticket/758

[2] http://sysoev.ru/en/

I found this interesting, first off because I know that Bob has used Pound in the past and because, well, it's Bob. Also, someone on the #cherokee channel had suggested Nginx as an option, but since the docs were in Russian I was reluctant to commit to it.

Well after Bob's email, I started searching and it turns out that while the official docs are in Russian, there's a bit of English documentation on the web, and apparently some happy users as well:

• http://blog.kovyrin.net/category/web-tech/nginx/
• http://zh.stikipad.com/notes/show/nginx
• http://brainspl.at/articles/2006/08/23/nginx-my-new-favorite-front-end-for-mongrel-cluster

Also of interest is that Nginx happens to live in Gentoo's portage.

I'm an enthusiast. For what? Apparently whatever looks really promising at the moment. That being said, upon hearing that Bob Ippolito has been successfully running Nginx as a high-traffic proxy, I decided to dive right in.

My current standard setup has been Pound proxy (which frankly I've been quite pleased with) in front of various web servers and services, the most common case being in front of Lighttpd and TurboGears. TurboGears doesn't require a separate web server, but CherryPy is suboptimal (to say the least) for serving static content and I host many sites that don't use TurboGears, so there you have it.

Anyway, I'm an enthusiast, but I'm also inclined to be my own crash-test dummy. I don't recommend things until I've at least taken them around the block a few times. So it was time to convert my own sites to Nginx and test the waters.

I started with develix.com as it's pretty lightweight and Nginx's job would consist mainly of serving static files and proxying to the TurboGears app develix.com runs on. Also the fact that develix.com is on it's own IP address simplified the migration. I could leave Pound in place for the other IP addresses and only tinker with develix.com.

I installed Nginx, edited the sample configuration file that came with it, commented out the develix.com entries (quite a few actually, since there are many subdomains), restarted Pound, then fired up Nginx. After a couple minor missteps, develix was happily chugging away on Nginx. Too easy.

Next I decided to migrate a customer's dedicated server. I figured if the experiment were a failure, it would be easy to switch back to his original configuration (Pound proxying to several Lighttpd instances, all running Joomla, Mambo, Sitedepth and Coppermine, i.e. a buttload of crappy, overly complicated PHP code). Further, because this server is mostly about serving large images and videos (yes, it hosts a few of those sites), I figured it would be both a good test of Nginx and a possible performance benefit to my customer.

I notified him of what I wanted to do, and since we'd been having issues with Lighty on that box, he was happy to let me have a go at it.

Well, it wasn't as easy as I'd hoped. Most of the issues were minor, but there were many (remember, Joomla . With lots of modules). Most of these were just a matter of figuring out what the hell was going on, making some rewrite rules, etc. Typical stuff when setting up a website.

One issue however, almost became a showstopper: it seems Nginx doesn't do CGI.

That's right. Of all things. Who would have thought? I mean, after static content, CGI is the simplest thing on the face of the earth. Additionally, it hadn't even occurred to me that he used CGI (only two scripts as it turns out). Anyway, long story short, I ended up writing a small Perl wrapper for the Perl CGI he had that let it be served via FastCGI and then running that from Lighttpd's spawn-fcgi program. This took a surprisingly long time to figure out how to do (think long, painful hours). You'd think there'd be a ready-made Perl script for doing this, but if there is, it's buried under tons of mirrors of the Perl documentation. By the time I finished, the script was quite short, on the order of a dozen lines, but all that meant was I was averaging two lines of code per hour. I won't say that Perl sucks, but I'm sure thinking it.

Anyway, once that was resolved, there was a bit of minor cleanup with file permissions, yada yada, but so far so good.