When I first published TurboStan, I admit I was a bit taken aback by the apparent lack of interest from the majority of TurboGears users. Stan's beauty and clarity seemed so obvious to me that it baffled me that people would choose what I perceived as a lesser PHP over something as clearly elegant as Stan. I tallied the possible reasons for TurboStan's lack of fanfare and came up with the following:

1. Not the default engine for TurboGears. While not a good technical reason, there are reasons for sticking close to the mainstream, especially with an unstable target like TurboGears was in the early days.
2. Dependencies on Twisted, Nevow and zope.interfaces. At least one user mentioned that he was unable to find a version of zope.interfaces that would both install and was compatible with the other two dependencies. I figured that if one person mentioned this problem, then there were probably a dozen more who gave up in silence.
3. Lack of documentation. This plagued me in my early days using Nevow and I didn't improve the situation for others any by extending Stan with even more features that were only documented briefly in my blog (and I changed blogs about as often as my socks in those days).
4. Not really seeing the elegance of Stan.

Since the release of Breve, I've seen a bit more interest than I did with TurboStan (I even received a patch from someone, something that TurboStan never did), but still the warm reception isn't quite what I'd like it to be.

Now I admit that when I first decided to try Nevow (way back before TurboGears was a glimmer in Kevin's eye), I took a look at the two options for templating (Nevow's XML engine and Stan) and went with the XML option. Stan looked... odd. I didn't really see it for what it was. Plus I had some odd sense that XML was a better choice because it was more "standard". I'm not sure when the lights came on and I was able to appreciate the elegance of Stan, but at some point it happened and I'm now loathe to touch the angle brackets on my keyboard. I became so attached to Stan that I refused to try Django, Pylons, CleverHarold or any other framework that I couldn't use Stan with. The main reason I used TurboGears was so I could use TurboStan. Now, I'm sure at least part of this love came from the fact that I was playing with my own creation (not Stan, but the extensions I added) and my ability to direct its future. There's always a lot of fun in that and I can't deny that is part of it.

Still, as I watch TurboGears (and a few other frameworks) adopt Genshi as their default engine, I can't help but wonder at the attraction people find to these sorts of engines. I can't help but notice with a bit of smugness that there's a good chunk of helper functions in Pylons (and Rails) to alleviate the need to spell out HTML, which seems to confirm my belief that their default template engines don't really help as much as they might when it comes to HTML generation (an odd weakness for a web framework to have). When I was first starting on the web (and Python web frameworks were pretty ugly), I was forced to program in PHP/Smarty at my job. Once I got over the novelty of programming amongst the web's various vagaries and oddnesses, I bemoaned these tools and wished for a Python framework that would allow me to escape the ugliness I perceived in them. So what's my point? Well my point is that template engines such as Genshi, Kid, Myghty, et al all smell of PHP to me. Is some of that just cosmetic? You bet. In fact, probably most of it is just cosmetic. There's no doubt in my mind that Genshi is a powerful too and probably full of beautiful Python code under the hood. That doesn't change how it looks to me from the outside. It isn't Python. I find it mildly ironic that Pythonistas are renowned for their loathing of Perl's typographical perversions and then turn around and create equally ugly things to describe web pages with.

Anyway, this isn't so much about how much I dislike those types of engines as it is curiosity as to what reasoning (whether it be founded in technical or taste) other Python programmers have for selecting their preferred template engine. Going along with that, I'm curious why DSL's like Stan, Breve, XIST and others don't generate more enthusiasm amongst Pythonistas.

So if you've got an opinion on this, I'd be interested in hearing it. I'm not looking for an argument, just enlightenment (right now I feel like I'm the enlightened one looking out on the unwashed masses, but there's always the remote possibility I just don't get it, and if that's the case, someone should be kind enough to show me the light) . Also I'm interested to see if there is an actual perceived shortcoming in Breve so that I can perhaps address it.

So I've settled on a name for my new template engine: Breve. Formerly known as espresso with half-and-half, Breve is now a Stan-like template engine written in pure Python with no external dependencies.

I've also written a Buffet plugin for Breve and it's been tested with Pylons and TurboGears. TurboGears testing was pretty limited (tested that a basic template rendered), but I sat down tonight and reimplemented the Pylons QuickWiki tutorial completely in Breve without issue.

I have to say that Pylons seems pretty straightforward. It's a little more upfront work than TurboGears (mostly due to Routes), but after that it's quite similar in its learning curve. One thing I like better right away is the error messages. Sometimes TurboGears exceptions can be downright obscure (and worse, occasionally being raised far away from the original exception), whereas Pylons gives almost too much information. I know which case I'd rather deal with.

Anyway, I did the Pylons test before the TurboGears test for one reason: the TurboGears site was down all day on the day I wanted to test the Buffet adapter. That meant I couldn't install TurboGears on the laptop I'm currently using. That meant no testing. That meant at least mild aggravation that TurboGears is still failing to recognize its own importance to its community. So far TurboGears has changed hosting services at least twice and still doesn't have any infrastructure to ensure availability (mirrors, anyone?). I've mentioned this to Kevin Dangoor before and I get the feeling he simply doesn't have the time to deal with it. Regardless, I'm starting to feel that simple availability might become the achilles heel of TurboGears. It depends on a couple dozen sub-projects, at least a few of which I've seen become unavailable for download at various times. The sheer number of subprojects that comprise TurboGears makes the likelihood of one of them being unavailable for install far too high. If TurboGears is going to depend on third-party packages, then it needs to do something to ensure not only its own availability, but the availability of all the sub-packages.

Anyway, I'm currently in the middle of a rather largish TurboGears application, so I won't be switching just yet, but I see Pylons in my future.

Over at about:cmlenz, Chris discusses his thoughts on my previous article wherein I compared Genshi (and other XMLish template engines) to PHP. To be fair, the comparison was a bit tongue-in-cheek since most of my complaints about PHP relate to its use as a programming language, not as a templating engine, but Chris makes a key point that I think is worth addressing:

At least with Genshi and Kid, that “ugly thing to describe web pages with” is pretty much just XHTML. And that, of course, is a feature. For a web developer, reading and writing HTML (as well as CSS and Javascript) is at the core of any front-end work. I don't want a template system that tries to hide the HTML from me; rather, it should enable me to just take some HTML and hook it up to server-side application code.

This seems like a reasonable enough position. However this brings me back to a simple point: a document is independent of the particular syntax used to describe it. Whether XHTML, Latex, PDF, Postscript or whatever, it's the document that matters, not the particular syntax. XHTML is simply one more in a long line of machine formats for describing a document. And this is key. XHTML is not the document. It's a description of a document. Further, XHTML is not special. The only thing that sets XHTML apart from, say a binary Word document is the fact that it's reasonably parsable in its raw form by humans. This hardly qualifies it as an optimal representation for hand manipulation. The most compelling reason for using XHTML today is that everyone else does it (and in some cases, this is in fact a valid argument).

The argument that "I don't want a template engine that tries to hide the XHTML from me" reminds me of back when people argued against high level programming languages in favor of assembly language because HLL's moved them "too far from the machine". For most programmers, the machine doesn't matter. It's the end result that they care about. I feel much the same way about document-generation on the web. The fact that XHTML happens to be the language that browsers currently speak means little to me aside from the fact that I need to somehow feed it to them in order to retrieve my desired document. The plethora of tools for assisting with this suggests that I'm not alone in this desire. Some people actually use tools such as Dreamweaver exclusively for this task and never bother to learn the ins and outs of XHTML. Of course, as a programmer, this option isn't really available to me since I do need to integrate with a backend application (and seeing the output of some of this software discourages using it seriously anyway), but that doesn't mean I have to do it the absolute hardest way possible simply because the easiest way doesn't yet exist.

Also, to be accurate, Breve doesn't "hide" the XHTML, it simply gives you a shorter, cleaner way of writing it. What's the difference between having a vi macro or autocomplete in your editor or having what amounts to the same thing in your template engine? Breve is a shortcut for writing XHTML just as much as having a macro in your editor is.

Talk to anyone who does much XHTML editing and you'll undoubtedly get a laundry list of tools and macros they've written or collected to ease the pain of writing markup. Ask yourself this: if it weren't reasonable to write Python code without extensive help from editing macros and validators, would you still be inclined to use it? Pylons and Rails both include a bunch of "web helpers" to help reduce the need to write markup. My question is, why not just follow this to its logical conclusion and make your entire template engine a "web helper"?

My point in comparing XML-oriented template engines with PHP was not about how PHP is abused, but rather how it mixes two unrelated syntaxes together in a eye-straining mess. XML-based template engines are no different. XHTML is ugly all on its own. There is absolutely no way to make it prettier by stirring in another syntax.

Once I've made a maintenance 1.1.8 release this weekend, Breve 2.0 will be next on stage.

The big version jump is due to the fact that Breve 2.0 will have some backwards-incompatible changes (although I'll try to make a legacy API available). These changes won't affect Breve templates, only how they are invoked.

Join us on the list and discuss it! Your ideas will be welcomed with the arms of condescension and your fears gently patronized.

Io happens to include a nice little HTTP server. The HTTP protocol parsing is handled by Zed Shaw's Ragel-generated C parser (same as Mongrel uses), but the rest is in Io. On top of that, I've used an improved version of the Breve-like DSL I outlined a couple of days ago for generating dynamic content (which I am dubbing Iota for the time being).

Io is not a particularly fast language, but it excels at concurrency, as I'll demonstrate here. First, the complete code:

Iota := Object clone do (
    stack := nil
    output := nil

    init := method (
        stack = List clone
        output = List clone
    )

    squareBrackets := method (
        for ( arg, 0, call argCount - 1,
            output append ( call evalArgAt ( arg ) )
        )
        output append ( stack pop )
        ""
    )

    forward := method (
        name := call message name
        attrs := call message arguments map ( arguments ) map ( pair,
            k := pair at ( 0 ) asString slice ( 1, -1 )
            v := pair at ( 1 )
            " " .. k .. "=" .. v
        ) join

        if ( call message next ?name == "squareBrackets",
            output append ( "<" .. name .. attrs .. ">" )
            stack push ( "</" .. name .. ">" )
            self

        , // else
            output append ( "<" .. name .. attrs .. " />" )
            ""
        )
    )
)

server := HTTPServer do (
    setPort ( 5000 )
    setHost ( "localhost" )

    streamResponse := method ( socket, request,
        HTTPResponse withSocket ( socket ) setBody (
            createBody
        ) send
    )

    createBody := method (
        Iota clone do (
            html [
                head [
                    title [ "the title" ]
                ]

                body [
                    div ( class="foo", id="bar" ) [
                        "hello, world" , br
                        "this is a test"
                    ]
                ]
            ]
        ) output join
    )
)

server start

First I'll make clear I'm benchmarking on my laptop (1.5GHz Core2 Duo, 2.5GB of RAM, Fedora 9, 32 bit) which isn't the fastest thing around. Also ab is running on the same system, so we have a fast pipe but some of the CPU is going to ab. Not an ideal setup for benchmarking, but adequate for my purposes.

Now, the above code isn't particularly fast (the HTTPServer on its own gets around 339req/s serving a the same amount of static HTML). However, the requests per second barely dip as the concurrency increases (ab output is trimmed):

$ ab -c 10 -n 10000 http://localhost:5000/

Document Length:        149 bytes
Concurrency Level:      10
Time taken for tests:   49.915 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Non-2xx responses:      10000
Total transferred:      1870000 bytes
HTML transferred:       1490000 bytes
Requests per second:    200.34 [#/sec] (mean)
Time per request:       49.915 [ms] (mean)
Time per request:       4.992 [ms] (mean, across all concurrent requests)
Transfer rate:          36.59 [Kbytes/sec] received

So we're seeing around 200req/s at a concurrency level of 10.

Now let's push it:

$ ab -c 1000 -n 10000 http://localhost:5000/

Document Length:        149 bytes
Concurrency Level:      1000
Time taken for tests:   63.247 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Non-2xx responses:      10000
Total transferred:      1870000 bytes
HTML transferred:       1490000 bytes
Requests per second:    158.11 [#/sec] (mean)
Time per request:       6324.657 [ms] (mean)
Time per request:       6.325 [ms] (mean, across all concurrent requests)
Transfer rate:          28.87 [Kbytes/sec] received

At 1000 concurrent connections, we drop to 158req/s. That's a 100-fold increase in concurrency with only a 21% drop in performance. Not too shabby.

Next up I'll be visiting Io's PostgreSQL library so we can verify that we have the tools for a full web framework stack.

Lisp has long tended to use DSLs for HTML generation, and Clojure is no different in this regard. In fact, Breve was inspired by Stan which in turn was inspired by these Lisp DSLs.

However, I've tended to like the Breve syntax just a little better than the true s-expression DSLs:

# breve example
html [
    head [
        title [ 'hello, world' ]
    ],
    body [
        div ( id='main' ) [
            'hello, world'
        ]
    ]
]

Clojure provides it's own approach (and Compojure provides a very similar DSL):

;; clojure prxml example
(prxml
   [:html
     [:head
       [:title "hello, world"]
     ]
     [:body
       [:div {:id "main"}
         "hello, world"
       ]
     ]
   ]
 )

Overall, they are quite similar in approach, but the Lisp-like bracket placement continues to disturb me slightly, and I suspect it will disturb my designer (whom I got to eventually love Breve) even more.

Today I was researching Clojure stuff built around Ring and came across Cascade. Cascade includes its own variant of this type of DSL and it's a nice cross between the traditional Lispy s-expression approach and that used by Breve:

(defview index
  [env]
  :html [
    :head [
      :title [ "hello, world" ]
    ]
    :body [
      :div { :id "main" } [
        "hello, world"
      ]
    ]
  ]
)

This is pretty subjective, but I find the bracket placement a bit easier on the eyes and brain.

My work on Breve has revealed more dark corners of the web than I thought I ever wanted to know.

Breve is, at its heart, an XML-generation engine. The fact that it can output HTML and that doing so is its primary purpose is almost incidental, really. What this means is that Breve happily outputs things like <div /> which is logically sound, but technically incorrect.

Now, in a previous, happier life I was blissfully unaware that not all elements can be self-closing in XHTML, especially since such tags will even validate in the W3C's own validator. I ran across the issue when Breve output something similar to the following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
           Strict//EN" "DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
  </head>
  <body>
    <div />
    <div>test</div>
    <div />
  </body>
</html>

Now on the surface, this looks fine. However, this is how Firefox rendered it:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
           Strict//EN"  "DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
  </head>
  <body>
    <div>
      <div>test</div>
      <div></div>
    </div>
  </body>
</html>

I spent about an hour trying to figure out what I was doing wrong, changing doctypes, adding meta tags, etc. As it turns out, I was missing a key element: the HTTP header "text/xml". Now, I didn't think I was missing it, since I'd tried setting that via a meta tag to no avail. Luckily someone on #firefox pointed me to this bug report.

The short story is that there is no practical way to specify that an XML document is, in fact, XML from within the document itself. You must configure the web server to output the proper headers.

The longer story is that I modified Breve to only output self-closing tags for a small subset of XHTML and all is well (except for my faith in people who write RFC's).

Fixed a major issue with how Breve dealt with template paths. I'm not 100% happy with the solution, but it's an issue that the frameworks (Pylons and TurboGears, in this case) tell the template engine nothing about the root path to the templates.

This is an issue for Breve because all Breve paths are relative to a static root path. I've found that this makes it far easier to have fragments in subdirectories that can still inherit from templates in higher-level directories that might then include templates from subdirectories. I tried using things like "inherits ( '../index.stan' )" with TurboStan and it turned into an unmanageable mess real quick. However, TurboStan was only really meant to support TurboGears so I cheated and called out the the TG config file to get a root directory. Supporting both Pylons and TurboGears makes that solution unusable.

Anyway, like it or not, it's been tested with Pylons 0.94 and seems to work. Note that you must specify:

breve_opts = { 'root': 'myproject.templates' }

in your config/middleware.py file.

So it's finally usable. Breve 1.0 is up and out there. Kick the tires (but not too hard) and let me know what you think.

http://breve.twisty-industries.com

I've been meaning to tackle some of the problems inherent in TurboStan for a while. One of the biggest issues is the dependencies on Twisted, Nevow and zope.interfaces. That's a pretty heavy dependency tree for a template engine to have.

I started digging in to the code for nevow.stan to see if I could extract the core of Stan from the jungle, but finally changed my mind and decided on a clean rewrite. Stan has lots of features I don't need and lacks features I want to add. Since the basic concept behind Stan is simple to implement it seemed more reasonable to start from scratch.

I had the basic syntax and flatteners working within a couple hours and moved from there to adding features (inheritance, includes, loadable tag definitions, etc). It took a couple days of hacking but I now have the core of a new template engine. I'm quite pleased as the entire code base is not much larger than the original TurboStan plugin (which was mostly just glue code tying Stan and TurboGears together).

I've been using the working name "sexml" (s-expression markup language) but I've also considered "stanly" (i.e. stan-like) and "sexpress" (s-expressions). I guess it's true men think of sex every 10 seconds.

Anyway, I've got to write some documentation and a Buffet adapter (and test it as a TurboStan replacement on a live site), but expect to see a release before the first of the year (i.e. this week).

Oh, and the only external dependency is the os module from the standard Python library.

Okay, maybe it doesn't. But I do. Fed up with Bitakora, I cracked down and started a blog from scratch (well, building on Splee's SimpleBlog tutorial - mostly for the SQLAlchemy tips).

What you see here is the first basic incarnation of it. It's built using Pylons, Breve and SQLAlchemy. Not that it's required, but it also happens to have PostgreSQL living underneath of it.

This is also my second small Pylons application (the first being the Breve site). This also makes it the second live Breve site in existence (that I'm aware of). This is actually much of the reason I'm doing it: I need to learn Pylons and I need to test Breve on real applications (I'm already seeing features that would make life a bit simpler).

Anyway, I've got a lot to do but my basic goals are as follows:

1. Very basic blog core - no Javascript. No web 2.0. Nothing but posts, comments and tags (and a few themes).
2. Plugins (for adding all the stuff explicitly excluded in #1).
3. One-off pages (e.g. "About", "My Projects")
4. Admin interface.
5. Multiuser

Of these, #1 is around 90% finished. #2 is in exploratory testing. #3 will take all of an hour. #4 and #5 are pipe-dreams for now.

Some of the first plugins I have in mind are:

1. Threaded comment system.
2. Gallery

Oh, and as you can see, the blog is named "Pentropy". I'll have a site/trac/svn up for it soonish.

I've had this idea for a while (it's been on the Breve to-do page as a speculative feature), but I also saw that Genshi supports such a feature and that there's a new template engine "Art" (not released yet) that will also have such a feature.

At first I thought of it as a cool, but not really important feature. I mean, you could do things like embed your last.fm playlist easily but other than that it seemed like fluff, so it was low on my priority list.

However, on the Pylons list, the author of Art mentioned how he planned to leverage this concept to easily support plugins. Duh! It's brilliantly simple (just how I like it). I was thinking in terms of pulling in XML data from other sites, but actually it could be used to pull a fragment from another controller on the same server. This would allow plugins to be nicely encapsulated and easily allow a page to be built from fragments generated by multiple controllers (something not currently possible with Pylons that I'm aware of), all from within the template.

I quickly hacked up an xinclude tag in Breve (mostly a thin wrapper around urllib2.urlopen() and it worked fine for pulling from remote sites. However, when I pointed it to another controller on the same Pylons app, things fell apart. The rendered template was incomplete. Further testing revealed that it only broke if the controller rendered another template (i.e. returning a simple string worked fine). My initial guess (without further testing) is that somehow the subrequest is happening withing the same context as the outer request or the template subsystem has an ugly global somewhere (i.e. the subrequest tries to reuse the existing Template instance). There's also the possibility that it's a bug in Breve, but this seems unlikely at this point (Breve is pretty simple).

My next phase is to narrow down where it's happening with the following tests:

1. Have a Pylons controller directly call another Pylons controller using urlopen and see what we get. Note that it must utilize the template subsystem since simple strings already appear to work.
2. If this works without issue, then work up a simple Breve app (outside of Pylons) and try to replicate the issue there.
3. If that works, then file a bug report on Pylons since I suspect it will be outside my abilities to track down.

Anyway, this feature seems so killer as to be well-worth the effort. I was casting about for ways to support plugins for this blog and this is clearly it.

Follow up:

Lo and behold, the bug was in Breve, and it was a pretty serious one. breve.Template was using register_flattener on a method named __slot which I had expected to not collide with other Template instances but it didn't work. What did work was making it an internal class, but what worked even better was avoiding register_flattener altogether and simply adding a __str__ method to the slot class. 1.0.32 is now the recommended release as it addresses this bug, a memory-leak issue (related to the first bug) and provides an initial (read: testing only) implementation of xinclude (although I think I'll rename it at some point).

The tag cloud on the right side is now a plugin. I utilized the new xinclude feature of Breve to create a component-based page.

The main blog controller:

class PostController ( BaseController ):

    # ...

    @jsonify
    api_tags ( self ):
        tag_index = Tag.select ( order_by = [ asc ( Tag.c._name ) ] )
        tags = [ ( t.name,
                   ( h.url_for ( controller='post', action='by_tag', id=t.id ),
                     len ( t.posts ) ) )
                 for t in tag_index ]
        return dict ( tags )

the plugin controller:

class TagcloudController ( BaseController ):
    def cloud ( self ):
        body = urlopen ( 'http://pentropy.twisty-industries.com/post/api_tags' ).read ( )
        c.tags = simplejson.loads ( body ).items ( )
        c.max = float ( max ( zip ( *zip ( *c.tags )[ 1 ] )[ 1 ] ) )
        return render_response ( 'tagcloud/cloud?fragment=1' )

the plugin's template:

div [
    ol ( class_ = 'tag-list' ) [ [
        li [
            span ( class_ = 'tag-context' ) [ '%d posts are tagged %s' % ( _posts, _tag ) ],

            a ( href = _link, class_ = 'tag',
                style = 'font-size: %0.2fem;' % ( 0.8 + _posts / c.max ) +
                        'color: rgb(%d,100,120);' % ( _posts / c.max * 180 ) +
                        'padding: %0.2fem;' % ( c.max / ( _posts or 1 ) / 10 ) )
            [ _tag ]

        ] for _tag, ( _link, _posts ) in c.tags
    ] ]
]

and finally, the index template looks like this:

# index.b
html [
    body [
        ...
        xinclude ( 'http://pentropy.twisty-industries.com/tagcloud/cloud' )
        ...
    ]
]

The basic sequence of events goes like this: a request is made for index.b (or rather a child template that inherits index, but that's not relevant here), index.b requests the plugin via the xinclude directive. The controller at the url passed to xinclude accesses the published API available in the main controller to get the JSON data describing the available tags (a list of ( tagname, ( link, article_count ) ) ). It then renders its own template (cloud.b) and returns the XHTML output back to the main template which injects it into its own final output.

It seems like a lot of steps for something that could have been (and originally was) simply an included template. However what it provides in return is the ability to define completely encapsulated plugins that have no knowledge of the internal workings of the main application. All they require is a defined JSON (or XMLRPC) interface to retrieve their data from.

The next step is to define a caching mechanism to reduce the number of times this complete sequence must be traversed. In the case of the tag cloud, it's clear we can cache until a new post is made (tags are never created independently of posts in Pentropy). To this end I'll probably add a generic cache directive that will be utilized something like this:

html [
    body [
        cache ( expires = c.tags_changed ) [
            h1 [ 'Tags' ],
            xinclude ( 'http://pentropy.twisty-industries.com/tagcloud/cloud' )
        ]
    ]
]

Having a generic cache directive will allow for static HTML fragments to be stored based upon specific criteria. More on this later.

Had some time to hack on Pentropy last night and managed to get RSS 2.0 working (remarkably easy using xsd2breve), one-off pages (i.e. the "about" link) and the beginnings of an admin interface.

To give a quick example of how easy it was to create an RSS feed in Breve, here's the steps I followed:

First I needed to obtain an xsd file for RSS (this took some Googling as apparently there isn't an actual formal standard). I ended up using the one found here.

The next step is to create a test RSS feed. To this end I copied an example off some random site and edited it by hand:

rss (version="2.0") [
    channel [
        title [ 'Google Jobs' ],
        link [ 'http://www.google.com/support/jobs/' ],
        description [ 'Information about job openings at Google Inc.' ],
        item [
            title [ 'HR Analyst - Mountain View' ],
            link [ 'http://www.google.com/support/jobs/bin/topic.py?dep_id=1077&loc_id=1116 ],
            description [ '''
                We have an immediate need for an experienced
                analytical HR professional.
                The ideal candidate has a proven record of developing
                analytical frameworks to make fact-based decisions.
            ''' ]
        ]
    ]
]

I saved this template in templates/feeds/rss20.b. This gives us a bit of test data to verify our custom tags and template are working.

Next we need to create our custom tags and put them in a place where Breve can find them:

$ mkdir custom_tags
$ cd custom_tags
$ xsd2breve http://www.westinkriebel.com/Public/RSS20.xsd > rss20.py

Now we need to add our custom_tags directory to our Python path and tell Breve to actually use them:

# a Pylons controller
import sys; sys.path.append ( 'pentropy/custom_tags' )

class FeedController ( BaseController ):
    def rss20 ( self ):
        return render_response ( 'feeds/rss20?format=rss20' )

Now I can visit '/feeds/rss20' and see the result (a fantastic RSS feed).

The argument format=rss20 tells Breve to import tags from a module with that name.

Next is to put some actual data into the feed. Modify the controller to return the actual data:

def rss20 ( self ):
     c.posts = Post.select (
         order_by = [ desc ( Post.c.post_date ) ],
         limit = 50
     )
     return render_response ( 'feeds/rss20?format=rss20' )

and in our template:

rss ( version="2.0" ) [
    channel [
        title [ 'Pentropy' ],
        link [ 'http://pentropy.twisty-industries.com/' ],
        description [ 'Random Stuff' ],
        [ item [
              title [ _p.title ],
              link [ 'http://pentropy.twisty-industries.com/%s' % _p.slug ],
              description [ cdata ( _p.html_content ) ]
        ] for _p in c.posts ]
    ]
]

Now clearly, at some point I'll want to pull some of that data from the database or a config file, but for now it actually works.

I've been neglecting my projects for a while, so it felt pretty good to get Breve's Subversion repository back online, and actually commit a pretty important fix all in one night (include directive suddenly does the right thing).

I think I'll put out Breve 1.1.8 later this week.

Nevow Stan, which Breve is inspired by, had a feature called "patterns". More or less, this feature let you repeat blocks of the DOM, substituting values into the repeated block.

I've been wanting to add a similar feature to Breve, but didn't feel the Stan way of doing it fit well (it could be a little hard to understand).

Anyway, I think I finally found a couple of approaches that I like (and implemented one of them).

If you're interested, I discuss it here.

My work on getting macros to work in Breve forced me to figure something out that's going to drastically improve Breve.

The fundamental problem is that when Python calls a function, that function gets a local namespace. This is usually a good thing, but it's not always what you'd want in a specialized environment such as a template engine. For example, Breve has an include ( template ) feature that lets you include fragments of Breve code into another template. This worked okay, except that there were all sorts of limitations on what you could or couldn't do in an include file. Further, it required building a dictionary to use as the included file's namespace to make sure that user-supplied variables would be available to it.

Enter sys._getframe. This little-utilized function accepts an integer and retrieves the frame that many levels up the execution stack. For example sys._getframe ( 1 ) returns your current frame of execution. This can be used, for example, to retrieve the name of the function you're currently in (useful for debuggers and the like).

What's interesting to me, however, is that sys._getframe ( 2 ) returns the frame of the function that called the current function. This allows you to retrieve the global namespace of that function and put things back into it. In short, you can make the current function change or create variables in the outer scope:

import sys

def caller ( ):
    return sys._getframe ( 2 )

def f ( ):
    g ( )
    print v

def g ( ):
    frame = caller ( )
    print frame.f_code.co_name
    frame.f_globals [ 'v' ] = 2

f ( )

Running this code will output:

f
2

What this means to Breve is that now when a template calls include(), now the included file will be executed in the caller's namespace. It's actually even simpler than what's outlined above because Breve templates are passed to eval, so all that I need to do is pass caller().f_globals as the globals parameter to eval and the magic is done:

co = compile ( 'fragment.b', 'fragment.b', 'eval' )
return eval ( co, caller().f_globals, { } )

This will provide a much smoother interface. I also expect to apply this to template inheritance.

You can find out more about this topic here.

I've been resisting testing frameworks for a long time. I tend to test by using (and apparently being adept at writing relatively bug-free code), but one place tests provide true value is when modifying code.

I'm reworking the internals of aspects of Breve and I realized that I needed to do more than run visually inspecting the output of the included examples (I also test by deploying trunk onto the breve site and this blog prior to releases, but since I don't necessarily exercise all aspects of Breve, this is only a feel-good test).

Anyway, part of the daunting task of testing a template engine is that you must test against literal chunks of HTML. I decide to see what Genshi does, since whether I like that engine or not, it is the 900lb gorilla of Python template engines, and I know Chris Lenz and his cohorts have put a lot of effort into having a solid development process.

I took a look at Genshi's test suite and it was enough to get me started, although I wasn't thrilled about the tests having the expected output embedded directly in the tests. It seemed a bit cluttered and I knew it would get more cluttered as I implemented tests for corner cases (nesting of directives, inheritance, and combining features to check for side-effects).

I hacked about for a bit and came up with a system that seems to be working pretty well.

If you've been avoiding Breve because you're loathe to use software that doesn't include a test suite, you are now encouraged to look again.

My unit testing paid off (found one bug and a lot of confidence) so 1.2 is slated for release tomorrow.

This blog and the Breve site are already running on it =)

I was going to wait until later today (it's 2:30a.m.), but I couldn't see a reason to withhold the release.

Here's a short summary of what's new:

• Tag multiplication - you can multiply a tag by a list of dictionaries and have the dictionaries values propagated as attributes and data.
• Macros - Define a function inside a template.
• Assign - Create new variables inside a template.
• Tag.walk - traverse a tag and all its children, performing an action on each tag.
• Improved include function - included templates are now evaluated within the scope of the caller and several former limitations are removed.
• Template code cleanup.
• Unit tests for major API features.
• Improved documentation.
• Tools (soup2breve, xsd2breve, html2breve) are now properly installed.

Be sure to check out the documentation and review the tests to get an idea of what Breve is about.

Continuing my tradition of abusing whatever language is under my keyboard (and in the interest of stress-testing Breve), here's a template that traverses its own DOM and extracts out CSS selectors:

assign ( 'selectors', [ ] ),

macro ( 'get_selectors', lambda tag, is_tag: (
    macro ( 'css_sep', lambda attr:
        '.' if attr == 'class' else '#'
    ),
    selectors.extend ( [
        "%s%s%s { }" % ( tag.name, css_sep ( _k.strip ( '_' ) ), _v )
        for _k, _v in tag.attrs.items ( )
        if _k.strip ( '_' ) in ( 'id', 'class' )
    ] )
) ),
macro ( 'walk_dom', lambda tag:
    tag.walk ( get_selectors, True ) and tag
),
macro ( 'walk_results', lambda selectors:
    pre [ '\n'.join ( selectors ) ]
),

html [
    head [ title [ 'macro madness' ] ],
    body [ walk_dom (
        div ( class_='text', id='main-content' ) [
            img ( src='/images/breve-logo.png', alt='breve logo' ),
            br,
            span ( class_='bold' ) [ "Hello from Breve!" ]
        ]
    ), walk_results ( selectors ) ]
]

This outputs the original page with the CSS selectors at the end like so:

<html>
  <head>
    <title>macro madness</title>
  </head>
  <body>
    <div class="text" id="main-content">
      <img src="/images/breve-logo.png" alt="breve logo"></img>
      <br />
      <span class="bold">Hello from Breve!</span>
    </div>
    <pre>
      div.text { }
      div#main-content { }
      span.bold { }
    </pre>
  </body>
</html>

Don't try this at home, professional driver on closed-course, etc.

Also, this requires Breve 1.2.1 or greater (it required Tag.walk to be changed to return self rather than None).

Exactly what I was thinking.

[Update: made a few improvements to the code]

I'm trying to understand the mechanisms for creating internal DSLs (embedded languages) in Io. Here's my first attempt at a Breve-like template language:

Tag := Object clone do (
    output := nil
    stack := nil

    init := method (
        output = List clone
        stack = List clone
    )

    squareBrackets := method (
        attrs := stack pop map ( pair,
            k := pair at ( 0 ) asString slice ( 1, -1 )
            v := pair at ( 1 )
            "#{k}=#{v}" interpolate
        ) join ( " " )
        name := stack pop

        output append ( "<#{name} #{attrs}>" interpolate )
        output append (
            call evalArgs select (
                result, result isKindOf ( List ) not
            ) join ( "\n" )
        )
        output append ( "</#{name}>" interpolate )
    )

    forward := method (
        stack push ( call message name )
        stack push ( call message arguments map ( v, v arguments ) )
        self
    )
)

Tag clone do (

    html [
        head [
            title [ "the title" ]
        ]

        body [
            div ( class="foo", id="bar" ) [
                "hello, world",
                "this is a test"
            ]
        ]
    ]

) output join println

Note that it's not very Breve-like except in general appearance (it's not a directed graph of objects that can be traversed). It's also not feature-complete (although many Breve features would be redundant in Io). Still, it's a nice proof-of-concept and not bad for a couple hours of floundering around by a complete Io novice.

In any case, Breve's syntax was largely bound by Python's and I'm thinking that I'll be ditching this syntax in favor of something even better.