When I first published TurboStan, I admit I was a bit taken aback by the apparent lack of interest from the majority of TurboGears users. Stan's beauty and clarity seemed so obvious to me that it baffled me that people would choose what I perceived as a lesser PHP over something as clearly elegant as Stan. I tallied the possible reasons for TurboStan's lack of fanfare and came up with the following:

1. Not the default engine for TurboGears. While not a good technical reason, there are reasons for sticking close to the mainstream, especially with an unstable target like TurboGears was in the early days.
2. Dependencies on Twisted, Nevow and zope.interfaces. At least one user mentioned that he was unable to find a version of zope.interfaces that would both install and was compatible with the other two dependencies. I figured that if one person mentioned this problem, then there were probably a dozen more who gave up in silence.
3. Lack of documentation. This plagued me in my early days using Nevow and I didn't improve the situation for others any by extending Stan with even more features that were only documented briefly in my blog (and I changed blogs about as often as my socks in those days).
4. Not really seeing the elegance of Stan.

Since the release of Breve, I've seen a bit more interest than I did with TurboStan (I even received a patch from someone, something that TurboStan never did), but still the warm reception isn't quite what I'd like it to be.

Now I admit that when I first decided to try Nevow (way back before TurboGears was a glimmer in Kevin's eye), I took a look at the two options for templating (Nevow's XML engine and Stan) and went with the XML option. Stan looked... odd. I didn't really see it for what it was. Plus I had some odd sense that XML was a better choice because it was more "standard". I'm not sure when the lights came on and I was able to appreciate the elegance of Stan, but at some point it happened and I'm now loathe to touch the angle brackets on my keyboard. I became so attached to Stan that I refused to try Django, Pylons, CleverHarold or any other framework that I couldn't use Stan with. The main reason I used TurboGears was so I could use TurboStan. Now, I'm sure at least part of this love came from the fact that I was playing with my own creation (not Stan, but the extensions I added) and my ability to direct its future. There's always a lot of fun in that and I can't deny that is part of it.

Still, as I watch TurboGears (and a few other frameworks) adopt Genshi as their default engine, I can't help but wonder at the attraction people find to these sorts of engines. I can't help but notice with a bit of smugness that there's a good chunk of helper functions in Pylons (and Rails) to alleviate the need to spell out HTML, which seems to confirm my belief that their default template engines don't really help as much as they might when it comes to HTML generation (an odd weakness for a web framework to have). When I was first starting on the web (and Python web frameworks were pretty ugly), I was forced to program in PHP/Smarty at my job. Once I got over the novelty of programming amongst the web's various vagaries and oddnesses, I bemoaned these tools and wished for a Python framework that would allow me to escape the ugliness I perceived in them. So what's my point? Well my point is that template engines such as Genshi, Kid, Myghty, et al all smell of PHP to me. Is some of that just cosmetic? You bet. In fact, probably most of it is just cosmetic. There's no doubt in my mind that Genshi is a powerful too and probably full of beautiful Python code under the hood. That doesn't change how it looks to me from the outside. It isn't Python. I find it mildly ironic that Pythonistas are renowned for their loathing of Perl's typographical perversions and then turn around and create equally ugly things to describe web pages with.

Anyway, this isn't so much about how much I dislike those types of engines as it is curiosity as to what reasoning (whether it be founded in technical or taste) other Python programmers have for selecting their preferred template engine. Going along with that, I'm curious why DSL's like Stan, Breve, XIST and others don't generate more enthusiasm amongst Pythonistas.

So if you've got an opinion on this, I'd be interested in hearing it. I'm not looking for an argument, just enlightenment (right now I feel like I'm the enlightened one looking out on the unwashed masses, but there's always the remote possibility I just don't get it, and if that's the case, someone should be kind enough to show me the light) . Also I'm interested to see if there is an actual perceived shortcoming in Breve so that I can perhaps address it.

As it turns out, in Python 2.4, eval ( ) was changed to allow any sort of mapping object (not just a plain Python dictionary) as the locals namespace. What this means is that we are free to pass in an object that can pretend it has all the variables in the world:

class Proxy ( dict ):
    def __getattr__ ( self, attr ):
        return Proxy ( )
    def __getitem__ ( self, key ):
        return Proxy ( )
    def __call__ ( self, *args, **kw ):
        return Proxy ( )
    def __str__ ( self ):
        return ''

eval ( 'x.y ( ).z [ 3 ]', { }, Proxy ( ) )

Despite the fact that none of the variables (x, y, z) exist in the code fragment we're evaluating, we get no exceptions. In fact we don't get a peep of output.

Clearly you'd need to flesh this class out a bit (like handling mathematical operations) and you'd probably want to actually have some values in it, so you'd need to define __setitem__ and __setattr__ behaviour, but that's left as an exercise for someone else.

I've been struggling for the last few weeks to find a way to explain why I'm leaving Python. I could give the technical nitty-gritty (rejects functional programming, lacks lazy evaluation, grows syntactical features like cancer to overcome the first two design decisions), but at the end of the day, even with this intentional crippling of the language, it's still quite possible to write almost anything you want and actually be quite productive in Python.

So, in the face of this, what's my reason for leaving? Well, while it's possible to accomplish nearly any end result you like, you are constrained in how you get there. The language imposes an imperative style through its distinction between statements and expressions. The only way to achieve lazy evaluation is by using lambda and lambda is crippled by the presence of statements (which can't be used inside of a lambda).

Most Python fans claim (arguably with some validity) that these restrictions are in place to enforce code-readability.

Even if I grant that these restrictions do, in fact, help enforce a type of readability, I don't think anyone would disagree that it limits what a programmer is able to express. To me this argument is akin to demanding that authors limit themselves to 10th grade English so as to not confuse readers. While arguably effective, it also places a hard restriction on how concisely, elegantly, and accurately the writer can convey a concept. Worse, eventually the author will be crippled by this restriction as he or she will lose fluency in more esoteric corners of the language. His vocabulary will shrink from disuse.

In short, I think that such restrictions cater to the average (and below average) programmer. This is quite pragmatic, but hardly attractive if you want to advance your art. Poetry may be quite difficult to read at first, but is vastly rewarding once you've learned to appreciate it.

Lately I've been keeping my eye on alternatives to Python. I've gotten tired of waiting for features in Python that, to put it in GvR's words "ain't gonna happen". Most of these features aren't strictly essential to day-to-day programming, and the lack of them doesn't make Python a bad language, but they are features that give that warm feeling known as "elegance".

What are these features? A very short list (for me) includes:

1. Expression-based syntax
2. Closures
3. Macros (Lisp-style, not C-style)

To be honest, I'd settle for #1, but if I'm looking for an alternative, I may as well look for all three.

You might be wondering why Ruby isn't an option, since it supports at least the first two items and it currently has a thriving community (which is always good to have around a language you plan to use for more than hobbies). Ruby has two major drawbacks to me: first of all the performance is absolutely dismal. Until Ruby has a bytecode compiler or leverages an existing VM such as .NET/Mono, this is unlikely to change (I know work is being done here, but I'm not holding my breath. By the time a Ruby 1.x VM is done, Ruby 2.x may well be released, which would seem be a setback for one or the other). The second reason is I simply don't like the look of Ruby code. Too much syntax. This is more critical than even the performance issues. The performance issues will certainly be fixed at some point, but the language isn't likely to change much cosmetically (lest it become a different language). While Ruby may have the heart of Smalltalk, it has the face of Perl. I may like the ugly girl with the heart of gold, but I'm not going to marry her, thanks.

Anyway, I'm not bashing Ruby here, for all of you who are going to get upset about it. I think Ruby embodies some great ideas that have been languishing in the programming world. The deaths of Smalltalk and Lisp pretty much set the programming world back 20 years, IMO, and the fact that Ruby has brought back to life many of the fundamental concepts embodied in those languages is deeply appreciated. Ruby just isn't for me.

Anyway, I'm old and lazy these days, so whenever I see a new language with a syntax I'm readily familiar with, I perk up. Two languages currently sit on my radar as candidates, Boo and Logix (see bottom of article for references).

Of the two, Logix is the most appealing. Logix isn't so much a language as a metalanguage, with a Python-like syntax definition. It also runs on the Python VM and is, in fact, implemented as a Python module that you import into the Python interpreter. This means you can use the Python standard library which is a big win. It meets every requirement I've outlined above (and then some). My major concern with it is that it appears to be dead or at least stagnating and never appeared to have a community to start with. It's so great, I've considered opening a Trac and trying to jump-start a community, but I doubt my own ability to take the lead on development (language design isn't my area of expertise).

The second language I've recently discovered is Boo. I saw the announcement for Boo on c.l.py a couple years ago where it was met with some bit of scorn by the Python community, but recently I stumbled across it again and see that it's grown into a real project in the interim. Boo isn't as easy a jump as Logix (it's a completely new language, with a Python-like syntax, but some major incompatibilities). Also, it runs on .NET/Mono, not the Python VM. I don't consider this a bad thing, but that VM isn't quite as stable as the Python VM on Linux, at least in my experience. Of course, Boo hasn't hit 1.0 yet, so I expect that by the time it does, the CLR will be production-ready across platforms.

Update: Boo also scores major points for having an MVC web framework called Webbness. Documentation looks pretty thin, but it's a start. I'm guessing (but don't know) that Logix could work with some existing Python framework such as Pylons or TurboGears. That remains to be tested.

Lots of people like to use virtualenv, but personally I like using the method outlined on the easy_install section of the PEAK guide. I've shortened the instructions here to make them specific to Python 2.5 and POSIX systems (I have no idea how well this works on Windows, nor do I care much).

First, create an altinstall.pth file in Python's site-packages directory, containing the following:

import os, site
site.addsitedir(os.path.expanduser('~/lib/python2.5'))

This will automatically add each user's ~/lib/python2.5 directory to sys.path (if it exists), and it will process any .pth files in that directory.

The next step is to create or modify distutils.cfg in the distutils directory of your Python library. The correct directory will be something like /usr/lib/python2.5/distutils on most Posix systems:

[install]
install_lib = ~/lib/python2.5
install_scripts = ~/bin

This will configure the distutils and EasyInstall to install packages to the user's home directory by default.

The next step is to prevent these changes from affecting the root user (since usually root will be installing packages system-wide).

Create a file named .pydistutils.cfg in root's home directory (usually /root):

[install]
prefix=/usr
install_lib=/usr/lib/python2.5/site-packages
install_scripts=/usr/bin

[easy_install]
site-dirs=/usr/lib/python2.5/site-packages

Over at On the Edge, Greg Black has written an article explaining why he's dumping Python.

No, it isn't for Ruby, this time it's for AWK.While I can understand Greg's complaint about changing (even undocumented) behaviour of a standard library function in an incompatible way, Greg's conclusion fails to convince me that this was anything more than a tantrum. To be fair, I have tantrums all the time about software, so I can sympathize. However, Greg also wrote a follow-up article clarifying and solidifying his stance. This takes it beyond a tantrum (which tends to be a temporary thing that you get over rather quickly) into the realm of actual opinion. Therefore I feel compelled to address this opinion with some of my own.

What I found most striking is the deep and fundamental contradiction plainly visible in Greg's stance:

Greg wants to use a dead language, but Python 2.3 isn't going to cut it. Why not? Python 2.3 is certainly as dead as Awk by any measure. If Greg wants an unchanging Python he's found it and yet still complains. This is what I find contradictory. He wants both a living language (i.e. one that is updated) and a dead language (one that doesn't have changes). He's even stumbled upon the perfect solution (locking to 2.3), but is now heart-set upon finding some mythical language that never breaks backwards compatibility (don't try Ruby Greg! They can't even get a book out the door before it's obsolete) but still ships new versions that anyone cares about.

To further the irony, I've taken the title of this article directly from Greg's page where it's presented as some sort of motto above a picture of a BMW taking a corner on two wheels. Apparently life-or-death for a thrill is okay by Greg, but changing some parameters (or filing a bug report) in exchange for a cleaner library or language is a bit too "on the edge."

Go figure.

Something has been eating at me for a while. There is a fundamental inconsistency in Python, something that has kept me from writing code how I'd like to write it, but I haven't been able to put my finger on it.

I think I finally have.When I first saw anonymous code blocks in Ruby, I thought "aha, that's it", and I was close. But really anonymous blocks aren't it. They just remind me of it. In fact they are a special case of "it". "It" happens to be expressions. Python makes a distinction between statements and expressions and because my background has been with procedural languages (Pascal, C), this distinction seemed natural. But Python's flexibility hints at something more, something that it has not quite provided but I couldn't describe. I liked the presence of functional tools in Python (map, filter, reduce, list comprehensions, etc), but bemoaned lambda's crippled self. I wanted a ternary operator. I've finally realized these things are all just ways of using statements as expressions, and I've also realized that statements really could be expressions, except that they are intentionally crippled in Python.

As anyone who reads this blog knows, I've been doing lots of work with Stan. Stan is an s-expression style DSL for generating XHTML from Python. I decided early on when creating the TurboStan plugin for TurboGears that I would keep Stan templates as pure expressions, even though I realized this would reduce their power to some extent. My feeling was that too much code in the template was wrong. I still feel that way. However, I don't feel that way about my program itself. I feel the language I use should not restrict me in fundamental ways. Many problem domains lend themselves to a functional style of programming and not having language support for that idiom can be painful. I do acknowledge that functional programs can be more difficult to take in at first glance, but they also tend to be more elegant when properly applied. The same argument can be made about many other concepts, such as recursion. Until you are familiar with the concept, recursive code can be like looking into a hall of mirrors. That certainly isn't a reason to not support it in a language. I don't think anyone would take a language seriously that didn't support recursion. Or object-orientation, for that matter. So why does functional-style programming have such a bad rap with the Python devs?

I admire Phillip Eby's work and find his articles thoughtful, so I emailed him about it (he probably thinks I'm a stalker). Anyway he suggested that having statements as expressions leads to nested code and flat is better than nested. This may be true. However so does nested functions and classes. Further, just because something can be abused doesn't mean it's necessary to ban it from the language, especially if having it easily doubles the expressive power of the language itself. Many hacks that have been added to the Python syntax to support special cases of functional programming would no longer be necessary. The syntax itself would be more self-consistent. I certainly don't find Lisp to be an attractive language visually, but I do find the concept it embodies quite elegant and tempting.

In short, I think Python needs to dump the artificial distinction between statements and expressions.

In the meantime, Phillip was kind enough to point me to Logix, which is more or less exactly what I'm asking for. It even generates Python bytecode.

So provided I'm able to actually leverage Logix for use in my day-to-day work, it may be goodbye Python (sort of) and hello, Logix.

CleverHarold.

Like any language, Python has misfeatures. Some of them are huge (like the distinction between statements and expressions), some are minor unexpected behaviours. This one definitely falls into the latter category. A long-standing wart in Python has been the fact that list.sort() doesn't return a reference to the list. It makes sense that it doesn't because sort() is an in-place sort (for efficiency reasons). However, it's different than most of the other list methods in this respect. I think this makes it a wart. The fact that it's an in-place sort is an implementation detail that I shouldn't need to worry about.

Today I got caught out by something similar to this (although more complicated):

>>> import random
>>> l = range ( 10 )
>>> random.shuffle ( l )
>>> l
[4, 6, 3, 2, 0, 8, 1, 7, 5, 9]
>>> list ( l ).sort ( )
>>> l
[4, 6, 3, 2, 0, 8, 1, 7, 5, 9]
>>>

Looks like sort() didn't do anything, doesn't it? Well something got sorted, it just wasn't what you might expect at first glance. Casting to list() created a new, anonymous list that gots sorted, then immediately discarded.

Now if it was expected behaviour in Python were expected to return a new list (or even a reference to the original list), I'd have probably written

l = list ( l ).sort ( )

and all would have been well.

and so much more:

Schevo

Nufox: XUL from Python

This looks amazingly interesting. It makes me wonder how difficult it would be to say, reimplement CherryPy's wsgiserver using this library rather than Python's native threads.

My work on getting macros to work in Breve forced me to figure something out that's going to drastically improve Breve.

The fundamental problem is that when Python calls a function, that function gets a local namespace. This is usually a good thing, but it's not always what you'd want in a specialized environment such as a template engine. For example, Breve has an include ( template ) feature that lets you include fragments of Breve code into another template. This worked okay, except that there were all sorts of limitations on what you could or couldn't do in an include file. Further, it required building a dictionary to use as the included file's namespace to make sure that user-supplied variables would be available to it.

Enter sys._getframe. This little-utilized function accepts an integer and retrieves the frame that many levels up the execution stack. For example sys._getframe ( 1 ) returns your current frame of execution. This can be used, for example, to retrieve the name of the function you're currently in (useful for debuggers and the like).

What's interesting to me, however, is that sys._getframe ( 2 ) returns the frame of the function that called the current function. This allows you to retrieve the global namespace of that function and put things back into it. In short, you can make the current function change or create variables in the outer scope:

import sys

def caller ( ):
    return sys._getframe ( 2 )

def f ( ):
    g ( )
    print v

def g ( ):
    frame = caller ( )
    print frame.f_code.co_name
    frame.f_globals [ 'v' ] = 2

f ( )

Running this code will output:

f
2

What this means to Breve is that now when a template calls include(), now the included file will be executed in the caller's namespace. It's actually even simpler than what's outlined above because Breve templates are passed to eval, so all that I need to do is pass caller().f_globals as the globals parameter to eval and the magic is done:

co = compile ( 'fragment.b', 'fragment.b', 'eval' )
return eval ( co, caller().f_globals, { } )

This will provide a much smoother interface. I also expect to apply this to template inheritance.

You can find out more about this topic here.

I've been resisting testing frameworks for a long time. I tend to test by using (and apparently being adept at writing relatively bug-free code), but one place tests provide true value is when modifying code.

I'm reworking the internals of aspects of Breve and I realized that I needed to do more than run visually inspecting the output of the included examples (I also test by deploying trunk onto the breve site and this blog prior to releases, but since I don't necessarily exercise all aspects of Breve, this is only a feel-good test).

Anyway, part of the daunting task of testing a template engine is that you must test against literal chunks of HTML. I decide to see what Genshi does, since whether I like that engine or not, it is the 900lb gorilla of Python template engines, and I know Chris Lenz and his cohorts have put a lot of effort into having a solid development process.

I took a look at Genshi's test suite and it was enough to get me started, although I wasn't thrilled about the tests having the expected output embedded directly in the tests. It seemed a bit cluttered and I knew it would get more cluttered as I implemented tests for corner cases (nesting of directives, inheritance, and combining features to check for side-effects).

I hacked about for a bit and came up with a system that seems to be working pretty well.

If you've been avoiding Breve because you're loathe to use software that doesn't include a test suite, you are now encouraged to look again.

The evil spirit of Python? I knew it.

I'm thinking of stealing satanpythonface as an IRC handle.

I've been keeping half an eye on web2py for a while, since it seems to fill a previously untapped niche in the web framework world (TTW development without a lot of complexity).

I decided to check up tonight and I have to say I'm amazed. I'm mildly leery of the fact that it uses a custom ORM rather than SQLAlchemy (SA seems sufficient and is rapidly becoming the de facto Python ORM), but overall, web2py seems to offer a rather appealing way to manage web applications.

I was especially impressed by the number and quality of appliances built on top of it (and the fact that many of them are from third parties is encouraging).

I was also pleased to see that it supports alternative template engines (so I could use Breve with it if I wanted to [and I do]).

Yes, I've said it before, but the final hurdle has been crossed (I swear!).

After a long and fruitless debate on python-ideas, I've realized that Python is what it is and isn't going to change. In fact, most Pythonistas appear completely oblivious to the fact there is a deep and fundamental flaw that prevents Python from becoming what it could be.

Anyway, as a side-effect, I am actively seeking a maintainer to take over Breve as I will no longer be developing it (although I will get out at least one more release which contains some bug fixes).

[EDIT] I need to clarify this: I will not abandon Breve (I have too many projects that depend on it), but I won't be adding new features. I'll continue to accept patches and provide bugfixes where I'm able.

I was reading Lambda the Ultimate and came across a link to this interview with Guido van Rossum, the creator of Python.

Where do you see computer programming languages heading in the near future?

I hope that at some point computers will have sufficient power that we don't need separate functional, dynamic, and statically typed languages, and instead can use a single language that combines the benefits of all three paradigms.

Given that Python firmly rejects functional programming (indeed any paradigm outside procedural and object-oriented), I can only assume that Python's BDFL doesn't see a long-term future for Python.

Make that two of us.

MetaPython provides a macro and code quoting facility for Python. This looks pretty brilliant. I'll have to try it out soon. Project is hosted on Google Code.