With the recent announcement that Apple's CalDAV server was built on Twisted and the following announcement that Apple had donated an Xserve to the Twisted project, it's become clear that Apple has some interest in Twisted.

Twisted isn't exactly the most popular framework on the planet, so I found this a little surprising but chalked it up to mysterious forces.

This morning I was showering and I was thinking about how lame it was to get an Xserve. I'd seen benchmarks that demonstrated that OSX simply is not there as a server OS. The blame is apparently to be placed on the overhead associated with spawning threads and processes on OSX under Mach. The performance of server processes like Apache and MySQL is terrible compared to Linux on the same hardware.

Then it dawned on me: this is undoubtedly the root of Apple's interest in Twisted: it's asyncronous, hence, no threads. I've done no benchmarks or even heard of any, but if Twisted is twice as slow as Apache on Linux, and Apache on OSX is five times as slow as Apache on Linux, then it stands to reason that Twisted on OSX is at least twice as fast as Apache on OSX.

Pure speculation of course, since I've neither seen nor done any type of benchmarks, but it's interesting speculation.

So I've had it in mind to use Twisted for doing various ugly networking tasks that CherryPy doesn't seem appropriate for (since it's a threaded server and I don't want to tie up threads doing things that take an unknown amount of time).

So far it's working well. I've implemented two different services (an RSS aggregator and an IRC proxy) that simply do their tasks, then serve up the data as an XML RPC service on localhost.

I did the RSS aggregator a week or so ago, and tonight hacked together the IRC proxy (the idea being a way to offer customer support via IRC without requiring them to install an IRC client).

The single ugliness in the whole thing (from my POV anyway) is having to poll the server from Javascript every n seconds to see if new messages are posted. I'd like to find a way around that. It's not the Javascript itself that's ugly (MochiKit makes it quite simple), rather it's the concept of needlessly polling that bugs me.

I'm wondering about utilizing a timeout (i.e. the server doesn't respond until it actually has data) to this end. If the Javascript gets data within the timeout, it displays it and runs again. If it doesn't, it runs again. This would reduce the latency (no n second delay between updates) and the number of hits on the server. On the downside, it would increase the number of open sockets, but I doubt we're looking at such a huge load of IRC users that this would be an issue.

I'm developing a TurboGears app that I want to aggregate some blogs onto the side of. Because CherryPy works from a thread pool, it seemed unwise to tie up a thread (or more) trying to download and parse RSS feeds from potentially unreliable sources.

Enter Twisted.

Twisted seemed a much better solution for fetching and parsing the feeds. Even more so since there's a recipe in the Python Cookbook on ASPN for doing just this. I used the recipe and added an XML-RPC publisher to it so that I could query the results from TurboGears. Overall I added perhaps a half dozen lines to the recipe and was finished.

I'm always amazed by Twisted and I really, really would like to move completely to it for all my server-side development. Someday I will. Someday, after Twisted-Web has a 'twisted-admin' command that builds me a skeleton application like tg-admin does. Someday when Nevow feels more coherent. Someday when I've got the time and deadlines aren't looming. Someday when I can perhaps implement a couple of these things myself.

Someday...

DynamicSite

From the sAsync site:

"sAsync offers asynchronous access to the outstanding SQLAlchemy package by Michael Bayer. It also provides SQLAlchemy enhancements including persistent dictionaries, text indexing and searching, and an access broker for conveniently managing database access, table setup, and transactions. Everything can be run in an asynchronous fashion using the Twisted framework and its deferred processing capabilities."