So I've settled on a name for my new template engine: Breve. Formerly known as espresso with half-and-half, Breve is now a Stan-like template engine written in pure Python with no external dependencies.

I've also written a Buffet plugin for Breve and it's been tested with Pylons and TurboGears. TurboGears testing was pretty limited (tested that a basic template rendered), but I sat down tonight and reimplemented the Pylons QuickWiki tutorial completely in Breve without issue.

I have to say that Pylons seems pretty straightforward. It's a little more upfront work than TurboGears (mostly due to Routes), but after that it's quite similar in its learning curve. One thing I like better right away is the error messages. Sometimes TurboGears exceptions can be downright obscure (and worse, occasionally being raised far away from the original exception), whereas Pylons gives almost too much information. I know which case I'd rather deal with.

Anyway, I did the Pylons test before the TurboGears test for one reason: the TurboGears site was down all day on the day I wanted to test the Buffet adapter. That meant I couldn't install TurboGears on the laptop I'm currently using. That meant no testing. That meant at least mild aggravation that TurboGears is still failing to recognize its own importance to its community. So far TurboGears has changed hosting services at least twice and still doesn't have any infrastructure to ensure availability (mirrors, anyone?). I've mentioned this to Kevin Dangoor before and I get the feeling he simply doesn't have the time to deal with it. Regardless, I'm starting to feel that simple availability might become the achilles heel of TurboGears. It depends on a couple dozen sub-projects, at least a few of which I've seen become unavailable for download at various times. The sheer number of subprojects that comprise TurboGears makes the likelihood of one of them being unavailable for install far too high. If TurboGears is going to depend on third-party packages, then it needs to do something to ensure not only its own availability, but the availability of all the sub-packages.

Anyway, I'm currently in the middle of a rather largish TurboGears application, so I won't be switching just yet, but I see Pylons in my future.

I decided to check once again if there remained a bug with specifying Unix domain sockets in the TurboGears configuration (we're on our third configuration system, so it seemed reasonable to think a bug or two might have gotten shaken out at some point ;-)

And apparently it did (or perhaps the note I make at the end of this article is why I've gotten it to work this time and failed previously).

Nginx configuration:

upstream mycluster {
    server unix:/path/to/socket-1;
    server unix:/path/to/socket-2;
    # ...
}

server {
    # ...
    location / {
        proxy_pass http://mycluster;
    }
}

TurboGears configuration:

server.socket_port=""
server.socket_file="/path/to/socket-1"

Repeat the above configuration in separate config files for each backend you want.

Note that it's necessary to set server.socket_port="" or else someone (TurboGears), somewhere sets it to a default value of 8080. This wouldn't matter except it's how CherryPy determines whether or not we want a domain socket vs a TCP port.

Humberto Diógenes has added Zeroconf support for TurboGears:

View Trac ticket and patch

Pretty interesting idea.

Fixed a major issue with how Breve dealt with template paths. I'm not 100% happy with the solution, but it's an issue that the frameworks (Pylons and TurboGears, in this case) tell the template engine nothing about the root path to the templates.

This is an issue for Breve because all Breve paths are relative to a static root path. I've found that this makes it far easier to have fragments in subdirectories that can still inherit from templates in higher-level directories that might then include templates from subdirectories. I tried using things like "inherits ( '../index.stan' )" with TurboStan and it turned into an unmanageable mess real quick. However, TurboStan was only really meant to support TurboGears so I cheated and called out the the TG config file to get a root directory. Supporting both Pylons and TurboGears makes that solution unusable.

Anyway, like it or not, it's been tested with Pylons 0.94 and seems to work. Note that you must specify:

breve_opts = { 'root': 'myproject.templates' }

in your config/middleware.py file.

I decided to test out Nginx's load-balancing capabilities. Turns out to be remarkably easy.

CherryPy is not a fast HTTP server (CP3 is supposedly at least twice as fast, but not released yet, and either way, TurboGears is currently tied to CP2). That's okay, but there are some things you can do to make sure that it doesn't become a bottleneck for your TurboGears website.

The first and obvious thing is to serve static files (images, javascript, css, etc) from a regular webserver. Lots of people use Apache or Lighttpd for this purpose. A few others (including myself) prefer to use Nginx.

This solves a lot of the performance issues right off the bat. However there's another technique that can help quite a bit as well: load balancing.

Nginx provides a simple and elegant way to load balance across two or more backend servers. Here's how you do it with TurboGears.

The TurboGears Setup Edit your prod.cfg file and add/change the following lines to look like:

server.socket_host="127.0.0.1"
server.socket_port=8000

Next, copy your prod.cfg file to a file called prod2.cfg and edit it with the following:

server.socket_host="127.0.0.1"
server.socket_port=8001

Next, start two TurboGears instances:

./start-myproject.py prod.cfg&
./start-myproject.py prod2.cfg&

The Nginx Configuration The next step is to edit your Nginx configuration and add the following:

upstream myproject {
    server 127.0.0.1:8000;
    server 127.0.0.1:8001;
}

server {
    listen: 80;
    server_name: www.domain.com;
    location / {
        proxy_pass http://myproject;
    }
}

Now restart Nginx.

Unbelievably, that's it. If you want to see the effects (and convince yourself it works), you can change your dev.cfg files in the same fashion (probably from two separate terminals) and click through your site like mad (or wget it) and watch as each TurboGears backend takes part of the load.

A couple things to note:

• Sessions appear to work fine. I've used this technique with Identity and it doesn't appear to matter which backend the request goes to.
• This probably won't work with SQLite. SQLite is not meant for concurrent access and running two TG servers against the same database is going to be a problem. Besides, if you are doing load balancing, why are you using a toy database anyway? ;-)

In my previous article , I showed that it's pretty easy to have TurboGears generate static HTML files. However, unless they get served somehow, it's pretty useless. In my quest to figure out Nginx, I decided to make this work.

The Nginx Configuration:

location / {
    default_type text/html;
    types { text/html html; }
    root /var/www/public_html/cache;

    if (-f $request_filename.html) {
        rewrite (.*) $request_uri.html last;
    }

    proxy_pass 127.0.0.1:8000;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

The TurboGears Code

You'll need to add a directory under project/static for storing the cached HTML files. I call mine "cache". I think it's a good name so you should too =)

Add the following to your project's controllers.py:

class StaticOutFilter ( BaseFilter ):
    def before_finalize ( self ):
        if cherrypy.response.status is None:
            request = filter ( None, cherrypy.request.path.split ( '/' ) )
            path = os.path.join ( 'var/www/public_html/cache/', *tuple ( request ) ) + '.html'
            try:
                os.makedirs ( os.path.dirname ( path ) )
            except ( OSError, IOError ):
                pass # should log this
            file ( path, 'w' ).writelines ( cherrypy.response.body )

class Root ( controllers.RootController ):
    _cp_filters = [ StaticOutFilter ( ) ]
    # rest of root controller follows ...

At this point, whenever Nginx receives a request, it appends ".html" to the end of the URL and checks that it exists in the cache directory. If it does, it serves the static HTML file, if it doesn't it passes the request on to TurboGears which will generate the file.

Now clearly this simple setup is rife with limitations:

• Changes made in the TurboGears app won't be reflected unless you manually remove the relevant HTML file(s).
• Even if you do remove the files, truly dynamic content (such as displaying a username) won't be reflected.

The first problem is fairly trivial to solve. Assuming you provide an interface for editing content on your TurboGears site, simply add some code that cleans out the relevant cache file.

The second problem is much trickier. There are lots of ways to address it, depending upon your application's needs, but some ideas that come to mind are:

• Add a decorator that indicates whethor or not a page is cacheable and have the filter check this attribute prior to trying to cache it.
• Use AJAX to manage dynamic items.
• For the specific example given of the username (and user-customizable items), it might make sense to have a separate view of the site for authenticated users that bypasses Nginx and goes straight to TurboGears (i.e. myaccount.domain.com rather than www.domain.com). If most of your traffic consists of anonymous users then this might be ideal. Serve static content to anonymous users and dynamic content to authenticated users.

Anyway, this aspect is left as an exercise for the reader (or perhaps myself at a later date). If you have suggestions, feel free to make a comment.

So I've had it in mind to use Twisted for doing various ugly networking tasks that CherryPy doesn't seem appropriate for (since it's a threaded server and I don't want to tie up threads doing things that take an unknown amount of time).

So far it's working well. I've implemented two different services (an RSS aggregator and an IRC proxy) that simply do their tasks, then serve up the data as an XML RPC service on localhost.

I did the RSS aggregator a week or so ago, and tonight hacked together the IRC proxy (the idea being a way to offer customer support via IRC without requiring them to install an IRC client).

The single ugliness in the whole thing (from my POV anyway) is having to poll the server from Javascript every n seconds to see if new messages are posted. I'd like to find a way around that. It's not the Javascript itself that's ugly (MochiKit makes it quite simple), rather it's the concept of needlessly polling that bugs me.

I'm wondering about utilizing a timeout (i.e. the server doesn't respond until it actually has data) to this end. If the Javascript gets data within the timeout, it displays it and runs again. If it doesn't, it runs again. This would reduce the latency (no n second delay between updates) and the number of hits on the server. On the downside, it would increase the number of open sockets, but I doubt we're looking at such a huge load of IRC users that this would be an issue.

I'm exploring caching options for use with TurboGears, and of course, the penultimate cache is when you are serving static HTML pages from a webserver (like Lighty or Apache) rather than dynamic content from a framework.

The key to generating the static HTML files is to use what is known as a filter. CherryPy lets you attach filters at several points in the request processing chain. In this case, I wanted to intercept the request right before it's returned to the browser and dump it to a file:

from cherrypy.filters.basefilter import BaseFilter
class StaticOutFilter ( BaseFilter ):
    def before_finalize ( self ):
        if cherrypy.response.status is None:
            path = os.path.join ( 'path/to/static/cache',
                                  *tuple ( cherrypy.request.path.split ( '/' ) ) ) + '.html'
            try:
                os.makedirs ( os.path.dirname ( path ) )
            except OSError:
                pass
            file ( path, 'w' ).writelines ( cherrypy.response.body )

class Root ( controllers.RootController ):
    _cp_filters = [ StaticOutFilter ( ) ]

That's it! Now the content is stored as static HTML files in a directory hierarchy that mirrors the URL hierarchy in my application. There are many caveats and limitations (and probable dangers - I'll need to analyze the writer code to make sure no path manipulation can be done), but it's a start.

Read Part 2 of this Article

I'm developing a TurboGears app that I want to aggregate some blogs onto the side of. Because CherryPy works from a thread pool, it seemed unwise to tie up a thread (or more) trying to download and parse RSS feeds from potentially unreliable sources.

Enter Twisted.

Twisted seemed a much better solution for fetching and parsing the feeds. Even more so since there's a recipe in the Python Cookbook on ASPN for doing just this. I used the recipe and added an XML-RPC publisher to it so that I could query the results from TurboGears. Overall I added perhaps a half dozen lines to the recipe and was finished.

I'm always amazed by Twisted and I really, really would like to move completely to it for all my server-side development. Someday I will. Someday, after Twisted-Web has a 'twisted-admin' command that builds me a skeleton application like tg-admin does. Someday when Nevow feels more coherent. Someday when I've got the time and deadlines aren't looming. Someday when I can perhaps implement a couple of these things myself.

Someday...

TurboX2 a TurboGears front-end to XMMS2.