Thought I'd follow up on a previous post regarding recursive queries in PostgreSQL. I had opportunity to use this particular query again, and I thought I'd share a couple small enhancements:

CREATE FUNCTION subaccounts(INTEGER) RETURNS SETOF accounts AS $$

DECLARE
  row accounts%rowtype;

BEGIN
  FOR row IN (
    WITH RECURSIVE subs(id) AS (
      SELECT * FROM accounts WHERE id = $1
    UNION ALL
      SELECT a.* FROM accounts a, subs
      WHERE a.accounts_id = subs.id
    )
    SELECT *
    FROM subs
  ) LOOP RETURN NEXT row;
  END LOOP;
END;

$$ LANGUAGE 'plpgsql';

The key features here are

1. Refactored into a function, so you can pass it an id. This allows you to select any branch from the tree.
2. Included the parent record in the result set. This turns out to be more useful to me.

Here's an example usage scenario:

SELECT * FROM accounts;

 id | accounts_id |   name
----+-------------+-----------
  1 |             | Account 1
  2 |             | Account 2
  3 |           1 | Account 3
  4 |           3 | Account 4
(4 rows)

SELECT * FROM subaccounts(1);

 id | accounts_id |   name
----+-------------+-----------
  1 |             | Account 1
  3 |           1 | Account 3
(2 rows)

from "man tmpwatch":

Make the decision about deleting a file based
on the file's atime (access time). This is the default.

Combine this with a common trick of turning atime off on filesystems and you have a recipe for breaking things that expect their temporary files and directories to be around when they need them. I configure Nginx to put its temp files under /var/tmp/nginx, but that directory kept disappearing. I added code to build the directory tree when Nginx is started, but they would disappear while it was running. Turns out that noatime is the culprit, and tmpwatch is the unwitting accomplice.

There's a few ways to fix this issue:

1. Put your temp files somewhere else.
2. Use relatime rather than noatime. Unfortunately relatime is only available in newer kernels (2.6.18 doesn't have it).
3. Change tmpwatch to use --ctime. There's a nice writeup on this here.

In the case of Nginx, I actually decided to move the temp files to /var/run/nginx. Since the files in question are actually temporary buffers that Nginx manages on its own, this seemed the right solution.

I'm not a big fan of screen or tmux because they have a lot of features I don't use and get in the way some things I actually do use (namely requiring bizarre key chords for scrollback and copy/paste). Instead I prefer dtach.

However, dtach requires passing too many arguments for my taste and level of laziness. Here's my shell function (placed in ~/.bashrc) for simplifying my dtach session management. It's nothing special but just one of those little things that makes daily life a bit simpler:

dtach () {
   case "$1" in
       -ls)
           # ls -1A ~/.dtach/
           ls -lrt1A ~/.dtach/ | awk '{print $9, "--", $6, $7, $8}'
           ;;

       *)
           name=$*
           if [ "$name" == "" ]; then name="default"; fi
           mkdir -p ~/.dtach/
           /usr/bin/dtach -A ~/.dtach/"$name" -z /bin/bash
           ;;
   esac
}

Now I can arbitrarily name my sessions, "dtach -ls" shows me a list of sessions, and plain old "dtach" starts or attaches to the "default" session.

I saw the disclaimer starting with "Dear visitors from Google..." about halfway down this page and my brain refused to process what the disclaimer appeared to be telling me. People couldn't really be this stupid...

Then I read the comments.

I have tears in my eyes, but I don't know if it's from laughing or because the world is full of idiots. And they have internet access.

I just want to thank all the inconsiderate shitheads who remembered that I have been under a rock, surely without Internet, for the past decade and THANK GOD, remembered me and sent me a fucking reminder that I'm missing out on LinkedIn where I can... well, I don't know, do some shit or perhaps meet other dumb-asses like yourself.

Seriously people. Everyone and their grandmother is fully aware of Facebook, LinkedIn, MySpace etc. There is no fucking reason on earth to send another fucking reminder to the people unfortunate enough to be your friends. If they aren't on there by now, chances are it's because they don't want to be, and there's at least a slight chance that they weren't just waiting for that special invitation from your goofy ass.

I signed up with 24 Hour Fitness back in December. I noticed on their website today that they will be implementing cardless checkin. "How on earth will they manage that?" I wondered to myself. Well, wonder no more: PIN plus fingerprints.

You gotta be fucking joking me. I'll tell you, if people in this country are retarded (and lazy) enough to give their fingerprints to some fucking gym so that they don't have to pull a card out of their fat fucking pants, then I guess we are all going to get what we deserve.

On a similar note, Facebook founder Mark Zuckerberg confirms that Facebook users are dumbfucks for trusting him with their personal info. But then, we already knew that.

I just discovered this morning that Google Mail sucks. There's so many annoying and stupid things on this page that it gives me a slight eye twitch.

I discovered it because I tried to send a tgz of a directory under /etc to someone with a Gmail address. I'm fairly certain there were no .exe's there, so I can only assume that they are filtering #!shebang files as well. Personally, I feel that if they are going to be this stringent in their big-brothering, they ought to block images (buffer-overflow anyone?) and most especially PDF, cause hey, that's executable code. Oh, and let's not forget URLs, since email often contains links to malware.

Anyway, I guess I should take it for what it is: a reminder that free email hasn't changed much in the last decade: suckage with a bigger inbox.

So, 1.5 years after purchasing Laura an EEE PC, the primary 8GB SSD has begun to fail with intermittent I/O errors. While somewhat disappointing (under two years isn't too hot for any long-term storage medium, IMO), I didn't consider it a big deal. "I'll just replace the two SSD's with a single SATA drive", I think to myself. Yeah, not so much. While most other Asus Netbooks use standard IDE or SATA interfaces for their SSDs, the 1000 has no standard storage interface. That's right: NONE. It has only a pair of PCE-e slots. This means to replace the failed SSD, you have to replace it with another SSD at around 1/2 to 2/3 the price of a complete replacement netbook.

I also noticed, while backing up the netbook to an external 1.8" 60GB USB drive (which is years older than the SSD's in the system), that the second 32GB SSD has at least one bad block on it as well.

So at the end of the day, SSD is permanently out as a storage solution in this house and Asus just lost a future customer (we currently own two) as a result of the retarded decision to omit any sort of IDE/SATA connector inside the EEE 1000. Also their decision to make the master/slave IDE settings configurable via soldering wasn't thrilling either.

I've gotten two bounced emails in the last two days from two different hosting companies using Barracuda appliances. I was a little baffled when I read the bounce report:

<address@domain.com>: host apollo.utilicomnet.com[209.4.188.122]
   said: 554 Service unavailable; Client host [mail.develix.com]
   blocked using Barracuda Reputation;
   http://www.barracudanetworks.com/reputation/?r=1&ip=74.92.173.86
   (in reply to end of DATA command)

Now what's interesting is that 74.92.173.86 is not the IP address of mail.develix.com:

$ host mail.develix.com
mail.develix.com has address 69.168.53.29

Also we can verify that Barracuda doesn't really block mail.develix.com by checking mxtoolbox.

This lead me to inspect the mail headers, where it became obvious that Barracuda is checking the IP address of the MUA, not the IP address of the MTA. I think this qualifies as completely busted. It also appears to be an upstream issue since I received two bounces from two separate ISP's in two different countries. My guess is that Barracuda pushed an update out a few days ago and now millions of emails will be incorrectly bounced.

In any case, while this is nice case of fail, it gets better when I tried to report the issue to Barracuda: "Sorry, but you'll have to report it to the ISP". Um, yeah, I should worry about your customers? Don't think so champ. In any case, I did notify the ISP, cause I'm a nice guy, but I suspect that they will be clueless (otherwise why did they buy a Barracuda?) so I'm not hopeful.

...please feature the link to the ONLINE bug tracker a little more prominently and provide perhaps one or two less fucking hoops to jump through. After spending an hour or so tracking down a bug (one that prevents logging, no less) the last thing one wants to do is play footsie with some shitty user-interface nightmare purporting to be a website.

Also try to realize that not every bug occurs on the machine that the bug report is being filed from, so using the cute GNOME tool may not actually be the appropriate way of doing it and wading through pages of useless text gets to be annoying. I get it: you'd rather we use that tool. And if we don't want to, well you'd just rather.

PostgreSQL now supports recursive queries. This is handy for a few things, most notably recursing graphs.

For instance, assume we create a table accounts that looks something like this:

CREATE TABLE accounts (
  id SERIAL PRIMARY KEY NOT NULL,
  accounts_id INTEGER REFERENCES accounts,
  name TEXT
);

INSERT INTO accounts (accounts_id, name)
VALUES (NULL, 'Account 1');
INSERT INTO accounts (accounts_id, name)
VALUES (NULL, 'Account 2');
INSERT INTO accounts (accounts_id, name)
VALUES (
  (SELECT id FROM accounts WHERE name='Account 1'),
  'Account 3'
);
INSERT INTO accounts (accounts_id, name)
VALUES (
  (SELECT id FROM accounts WHERE name='Account 3'),
  'Account 4'
);


SELECT * FROM accounts;

 id | accounts_id |   name
----+-------------+-----------
  1 |             | Account 1
  2 |             | Account 2
  3 |           1 | Account 3
  4 |           3 | Account 4
(4 rows)

This lets us have accounts that in turn have subaccounts. This type of organization has many applications, such as resellers. The problem is, how to find out things like "what accounts are subaccounts of account x?"

Usually you'd resort to traversal in a procedural language, but with recursive queries you don't have to:

WITH RECURSIVE subaccounts (id) AS (
    SELECT * FROM accounts WHERE accounts_id = 1
  UNION ALL
    SELECT a.* FROM accounts a, subaccounts
    WHERE a.accounts_id = subaccounts.id
)
SELECT id, name, accounts_id
FROM subaccounts
ORDER BY accounts_id, name;

 id |   name    | accounts_id
----+-----------+-------------
  3 | Account 3 |           1
  4 | Account 4 |           3
(2 rows)

UPDATE: it appears that disabling sync from the UI doesn't actually disable it. See here for more info.

Do not, I repeat, do not utilize the offline sync feature in Thunderbird 3.0.x. It is horribly broken. It will suck down your bandwidth, consume your transfer limits and get your accounts terminated. It may also have sex with your dog, but since I don't own any dogs I am unable to confirm this.

This graph shows bandwidth utilization for a single instance of TB 3.0.3 syncing the last 30 days a 3.3GB mailbox (probably a few MB total to sync). I'd like to point out that this is day two of the sync:

You can see the dropoff about halfway through when I killed Thunderbird.

Best of all, it's a known bug and apparently no one cares since it's several months old and still not addressed.

I've set up SPF on my Postfix install recently and added SPF records for my own domains. In my research I've come across a few articles that claim that SPF is not a good solution (SPF considered harmful ). From what I've seen, they center around the proposition that SPF requires a change in user behavior and that this cannot be accomplished. Further they claim that SPF isn't effective anyway and point out that most spammers actually have SPF records.

I disagree and I'll address both of these points:

1. Requires changes in user behavior. Yes and no. What's actually required is changes in service provider's behavior. The example that is thrown around as being unsolvable without the end user learning the vagaries of SPF is email forwarding. The argument is that if a user decides to use an email forwarding service, then they'll have to know to add this information to their SPF record. This much is true. However, it's not as if they need to learn SPF to do so. The email forwarding service can easily query the domain's SPF record themselves and notify the user. If the forwarding service also happens to be the DNS provider (fairly common) then they can just make the changes themselves. If they aren't the DNS provider, they can at least offer a copy of the recommended SPF string. Further, the DNS provider themselves should provide an easy way to make the changes that don't involve knowledge of how SPF works.
2. SPF doesn't block spam, and in fact, spammers have adopted SPF in an attempt to look legitimate. I think this is a marked victory for SPF. The purpose of SPF is not to block spam and to claim it fails at it is a straw-man argument. The point of SPF is to verify that the From header is allowed from the particular host that sent it. In short, it's a form of identity. So now spammers must use their own validated domains or validated forwarding services to send mail. This makes it much easier for end users and service providers to determine who the spammer really is. SPF is not spam prevention, it's identity-theft prevention.

SPF does have some silliness that ought to be removed (e.g. SOFTFAIL), but overall it is a decent idea that belongs in every service provider's toolbox.

I was thinking today about how most modern web frameworks go to great lengths to shield the framework user from SQL and database-specifics and then don't apply the same logic to the presentation layer. "SQL strings embedded in my code are ugly" they say, but then turn around and embed code in HTML (or vice versa), which is equally ugly.

Here's my take: if you use a DSL to write SQL, then you should use a DSL to write HTML, for all the same reasons. If you write raw HTML with programming code embedded in it, then you ought to write raw SQL.

"My designer won't understand a DSL" you argue. Maybe. I'll also bet that your DBA won't understand your HLL database expressions either. Worse I'll bet that unless you use a toy database like MySQL or SQLite you are giving up a ton of speed and power in favor of an abstraction that you then throw away on the other end (the presentation layer).

I guess I don't think that one approach is the Right Thing, but I do think that mixing them and then trying to defend one position or the other is wildly inconsistent and demonstrates that maybe you haven't thought it through too thoroughly (I'm guilty here as well, using a DSL on the presentation layer then preferring raw SQL on the backend).

LAMP (Linux, Apache, MySQL, Perl/PHP) continues to retain a large following of developers.

Here's the classic pieces and a good alternative to each:

• Linux: Okay, Linux is still great, but FreeBSD has some great features (ZFS, pf). Each has strengths and weaknesses. You should at least make a decision rather than simply defaulting to one or the other.
• Apache: Unless you have a need for some esoteric Apache module or CGI, use Nginx instead. It's no contest. Nginx is faster, lighter, more scalable, and simpler to manage.
• MySQL: Use PostgreSQL. There's literally only one reason to use MySQL these days: ignorance of the alternatives.
• Perl/PHP: Ruby, Clojure, Io... something else. Perl and PHP are a bit long in the tooth, and frankly were rather sloppy languages to start with (and I'm being really generous in that assessment).

Not going to get excessively ranty here. This is just a gentle reminder to folks that just as LAMP has been a great alternative to Windows servers, there are great alternatives to LAMP as well, and they are just as free and technically superior in many ways.

I've been pondering a way to efficiently extract sequences of consecutive values from a table without resorting to iteration in an external programming language. My specific problem is a database of thousands of phone numbers (technically ANI's), in varying block sizes:

SELECT ani
FROM phone_numbers
ORDER BY ani;

    ani
------------
 5551230000
 5551230001
 5551230002
 5551230003
 5551230004
 ...
 5554560000
 5554560001
 5554560002
 5554560003
 5554560004
 ...
 5559990000
 5559990001
 5559990002
 5559990003
 5559990004
 ...

Now generally we buy ANI's in blocks of 100 (or multiples of) from CLEC's, but not always. We have some stray blocks with 10, 20, or 50 ANIs. Anyway, sometimes we need to get an inventory of numbers and it's much nicer to have a list of ranges like:

5551230000 - 5551230099
5554560000 - 5554560099
5559990000 - 5559990099

rather than a giant list containing each individual number. The problem is, how to extract these consecutive lists of ANI's when you can't iterate? Well, first we need to figure out how to group these numbers. If the block sizes were uniform, we could perhaps group by some prefix such as:

SELECT substr(ani, 1, 8) AS prefix, count(*)
FROM phone_numbers
GROUP BY prefix
ORDER BY prefix;

Unfortunately this assumes that there are 100 ANI's in each block, something that simply isn't true.

What we really need is a way to somehow index the row number of the query. Oracle has an extension called ROWNUM that contains the index of the row in the current result set:

rownum |    ani
-------+-----------
     0 | 5551230000
     1 | 5551230001
     2 | 5551230002
     ...

etc.

Now, if we had this, we could easily group our results using the difference of the ANI and the ROWNUM (ani - rownum = delta where delta remains constant over each sequence):

rownum |    ani      | delta
-------+-------------+-----------
     0 | 5551230000  | 5551230000
     1 | 5551230001  | 5551230000
     2 | 5551230002  | 5551230000
    ...
    10 | 5554560000  | 5554559990
    11 | 5554560001  | 5554559990
    ...

At this point it would be a simple matter of grouping by delta.

Unfortunately PostgreSQL doesn't have this feature. Well, not exactly.

PostgreSQL supports a feature called window functions and these will be just what we need, since this provides us with a function row_number() that is basically the equivalent of Oracle's ROWNUM (except it can only be used in the context of a window).

Here's an example:

SELECT ani, row_number() OVER block
FROM phone_numbers
WINDOW block AS (ORDER BY ani);

   ani     | row_number
-----------+-----------
5551230000 |          0
5551230001 |          1
5551230002 |          2
5551230003 |          3
5551230004 |          4
...

That's what we need! Now let's finish it out:

WITH indexed AS (
  SELECT
    ani,
    ani::numeric - row_number() OVER block AS delta
  FROM phone_numbers
  WINDOW block AS (ORDER BY ani)
)
SELECT
  min(ani),
  max(ani),
  count(*)
FROM indexed
GROUP BY delta
ORDER BY min;

    min     |    max     | count
------------+------------+-------
 5551230000 | 5551230099 |   100
 5554560000 | 5554560009 |    10
 5558880000 | 5558880024 |    25
 5559990000 | 5559990009 |    10
 ...

Looks good! It also happens to be very fast for the amount of data I'm dealing with (returns instantly with around 5000 ANI's). I should note that my actual query was slightly more complicated since I also needed to partition based on CLEC, but I didn't want to cloud the logic with those details.

As an aside, I'm simply using the WITH clause as a convenience method for describing a subselect, although it can do much, much more (see WITH RECURSIVE).

The credit for this approach (at least, the first I've seen it) belongs to Rob Farley from this thread.

I hate OS X.

For years I've heard Debian folks brag about how superior .deb is to .rpm. Whatever. I've been using some Debian and Ubuntu boxes for several months now and I can assure you that the package management tools on those platforms are as primitive as any other distro, and in many ways they are far worse.

Here's a quick example. From the command line, locate and install OpenSSL development files:

Ubuntu 9.10:

# apt-cache search openssl
erlang-crypto - Erlang/OTP cryprographic modules
libcrypt-openssl-bignum-perl - Access OpenSSL multiprecision integer arithmetic libraries
libcrypt-openssl-random-perl - Access to the OpenSSL pseudo-random number generator
libcrypt-openssl-rsa-perl - Perl module providing basic RSA functionality
libcurl3 - Multi-protocol file transfer library (OpenSSL)
libcurl3-dbg - libcurl compiled with debug symbols
libcurl4-openssl-dev - Development files and documentation for libcurl (OpenSSL)
libneon27 - An HTTP and WebDAV client library
libopenssl-ruby - OpenSSL interface for Ruby
libqca2-plugin-ossl - QCA OSSL plugin for libqca2
openssl-blacklist - list of blacklisted OpenSSL RSA keys
openssl-blacklist-extra - list of non-default blacklisted OpenSSL RSA keys
openvpn - virtual private network daemon
python-openssl - Python wrapper around the OpenSSL library
python-openssl-dbg - Python wrapper around the OpenSSL library (debug extension)
python-openssl-doc - Python wrapper around the OpenSSL library (documentation package)
ssl-cert - simple debconf wrapper for OpenSSL
aolserver4-nsopenssl - AOLserver 4 module: module for SSL mode
cl-plus-ssl - A simple Common Lisp interface to OpenSSL
cl-ssl - Common Lisp interface to OpenSSL package
cryptmount - Management of encrypted file systems
cryptonit - A client side PKI (X.509) cryptographic tool
ebox-ca - eBox - Certification Authority
elfsign - ELF binary signing and verification utilities
etpan-ng - console mail user agent based on libEtPan!
fp-units-net - Free Pascal - networking units
gjots2 - A simple jotter (outline processor) for X11/gtk-gnome
globus-openssl-progs - Globus Toolkit - Openssl Library Programs
libace-ssl-5.6.3 - ACE secure socket layer library
libapache2-mod-gnutls - Apache module for SSL and TLS encryption with GnuTLS
libcrypt-openssl-dsa-perl - module which implements the DSA signature verification system
libcrypt-openssl-x509-perl - Perl extension to OpenSSL's X509 API
libcrypt-ssleay-perl - Support for https protocol in LWP
libengine-pkcs11-openssl - OpenSSL engine for PKCS#11 modules
libengine-tpm-openssl - OpenSSL engine for TPM modules
libglobus-gsi-callback0 - Globus Toolkit - Globus GSI Callback Library
libglobus-gsi-openssl-error-dev - Globus Toolkit - Globus OpenSSL Error  Handling Development Files
libglobus-gsi-openssl-error-doc - Globus Toolkit - Globus OpenSSL Error Handling Documentation Files
libglobus-gsi-openssl-error0 - Globus Toolkit - Globus OpenSSL Error Handling
libglobus-gsi-proxy-ssl1 - Globus Toolkit - Globus GSI Proxy SSL Library
libglobus-openssl - Globus Toolkit - Openssl Library
libglobus-openssl-dev - Globus Toolkit - Openssl Library Development Files
libglobus-openssl-module-dev - Globus Toolkit - Globus OpenSSL Module Wrapper Development Files
libglobus-openssl-module-doc - Globus Toolkit - Globus OpenSSL Module Wrapper Documentation Files
libglobus-openssl-module0 - Globus Toolkit - Globus OpenSSL Module Wrapper
libneon26 - An HTTP and WebDAV client library
libnewpki2 - PKI based on the OpenSSL low-level API (core library)
libopenssl-ruby1.9.1 - OpenSSL interface for Ruby 1.9.1
libpathfinder-dev - Development files for pathfinder
libpathfinder-openssl-1 - Pathfinder integration Library for OpenSSL
libpoconetssl6 - The C++ Portable Components Network library with SSL
libpoconetssl6-dbg - The C++ Portable Components Network library with SSL, dbg version
libruby-extras - a bundle of additional libraries for Ruby
libruby1.8-extras - a bundle of additional libraries for Ruby 1.8
libssl-ocaml - OCaml bindings for OpenSSL
libssl-ocaml-dev - OCaml bindings for OpenSSL
libtcnative-1 - Tomcat native library using the apache portable runtime
libxmlsec1-openssl - Openssl engine for the XML security library
mcrypt - Replacement for old unix crypt(1)
newpki-client - PKI based on the OpenSSL low-level API (client package)
newpki-server - PKI based on the OpenSSL low-level API (server package)
python-ncrypt - python wrapper for OpenSSL
somaplayer - player audio for the soma suite
stone - TCP/IP packet repeater in the application layer
tcl-tls - the TLS OpenSSL extension to Tcl
tinyca - simple graphical program for certification authority management
zope-externaleditor - Zope External Editor
zopeedit - Helper Application for Zope External Editor
libapache2-mod-php5 - server-side, HTML-embedded scripting language (Apache 2 module)
libopenssl-ruby1.8 - OpenSSL interface for Ruby 1.8
libssl-dev - SSL development libraries, header files and documentation
libssl0.9.8 - SSL shared libraries
libssl0.9.8-dbg - Symbol tables for libssl and libcrypto
openssl - Secure Socket Layer (SSL) binary and related cryptographic tools
openssl-doc - Secure Socket Layer (SSL) documentation
php5-cgi - server-side, HTML-embedded scripting language (CGI binary)
php5-cli - command-line interpreter for the php5 scripting language
ruby1.8 - Interpreter of object-oriented scripting language Ruby 1.8
libapache2-mod-php5filter - server-side, HTML-embedded scripting  language (apache 2 filter module)
libopenssl-ruby1.9 - OpenSSL interface for Ruby 1.9

Pretty much anything that utilizes OpenSSL is listed. Not terribly useful, if you ask me.

Now here's CentOS 5.4:

# yum search openssl
globus-gsi-openssl-error.i386 : Globus Toolkit - Globus OpenSSL Error Handling
globus-gsi-openssl-error-devel.i386 : Globus Toolkit - Globus OpenSSL Error Handling Development Files
globus-gsi-openssl-error-doc.i386 : Globus Toolkit - Globus OpenSSL Error Handling Documentation Files
globus-openssl.i386 : Globus Toolkit - Openssl Library
globus-openssl-devel.i386 : Globus Toolkit - Openssl Library Development Files
globus-openssl-module.i386 : Globus Toolkit - Globus OpenSSL Module Wrapper
globus-openssl-module-devel.i386 : Globus Toolkit - Globus OpenSSL Module Wrapper Development Files
globus-openssl-module-doc.i386 : Globus Toolkit - Globus OpenSSL Module Wrapper Documentation Files
globus-openssl-progs.i386 : Globus Toolkit - Openssl Library Programs
m2crypto.i386 : Support for using OpenSSL in python scripts
mingw32-openssl.noarch : MinGW port of the OpenSSL toolkit
openscada-Transport-SSL.i386 : Open SCADA transports
openssl.i386 : The OpenSSL toolkit
openssl.i686 : The OpenSSL toolkit
openssl-devel.i386 : Files for development of applications which will use OpenSSL
openssl-perl.i386 : Perl scripts provided with OpenSSL
openssl097a.i386 : The OpenSSL toolkit
openvpn.i386 : A full-featured SSL VPN solution
perl-Crypt-OpenSSL-Bignum.i386 : Perl interface to OpenSSL for Bignum
perl-Crypt-OpenSSL-RSA.i386 : Perl interface to OpenSSL for RSA
perl-Crypt-OpenSSL-Random.i386 : Perl interface to OpenSSL for Random
perl-Crypt-SSLeay.i386 : Crypt::SSLeay - OpenSSL glue that provides LWP https support
perl-Net-SSLeay.i386 : Perl extension for using OpenSSL
pkcs11-helper.i386 : A library for using PKCS#11 providers
pyOpenSSL.i386 : Python wrapper module around the OpenSSL library
tomcat-native.i386 : Tomcat native library
tomcatjss.noarch : JSSE implementation using JSS for Tomcat
xmlsec1.i386 : Library providing support for "XML Signature" and "XML Encryption" standards
xmlsec1-openssl.i386 : OpenSSL crypto plugin for XML Security Library
xmlsec1-openssl-devel.i386 : OpenSSL crypto plugin for XML Security Library

We won't even complain about the fact that there is ONE utility "yum" to both search and install packages vs Debians two ("apt-get" and "apt-cache"), but the search results speak for themselves.

Even better, let's look at Gentoo:

# esearch openssl
[ Results for search key : openssl ]
[ Applications found : 8 ]

*  app-crypt/openssl-blacklist
     Latest version available: 0.4.1
     Latest version installed: [ Not Installed ]
     Size of downloaded files: [no/bad digest]
     Homepage:    https://launchpad.net/ubuntu/+source/openssl-blacklist/
     Description: Detection of weak ssl keys produced by certain debian versions between 2006 and 2008
     License:     GPL-2

*  app-crypt/openssl-tpm-engine [ Masked ]
     Latest version available: 0.4.1
     Latest version installed: [ Not Installed ]
     Size of downloaded files: [no/bad digest]
     Homepage:    http://trousers.sourceforge.net
     Description: This provides a OpenSSL engine that uses private keys stored in TPM hardware
     License:     GPL-2

*  dev-libs/openssl
     Latest version available: 0.9.8h-r1
     Latest version installed: 0.9.8h-r1
     Size of downloaded files: [no/bad digest]
     Homepage:    http://www.openssl.org/
     Description: Toolkit for SSL v2/v3 and TLS v1
     License:     openssl

*  dev-perl/Crypt-OpenSSL-Bignum
     Latest version available: 0.04
     Latest version installed: [ Not Installed ]
     Size of downloaded files: [no/bad digest]
     Homepage:    http://search.cpan.org/~iroberts/
     Description: OpenSSL's multiprecision integer arithmetic
     License:     Artistic

*  dev-perl/Crypt-OpenSSL-RSA
     Latest version available: 0.25
     Latest version installed: [ Not Installed ]
     Size of downloaded files: [no/bad digest]
     Homepage:    http://search.cpan.org/~iroberts/Crypt-OpenSSL-RSA-0.25/
     Description: Crypt::OpenSSL::RSA module for perl
     License:     || ( Artistic GPL-2 )

*  dev-perl/Crypt-OpenSSL-Random
     Latest version available: 0.04
     Latest version installed: [ Not Installed ]
     Size of downloaded files: [no/bad digest]
     Homepage:    http://search.cpan.org/~iroberts/
     Description: Crypt::OpenSSL::Random module for perl
     License:     || ( Artistic GPL-2 )

*  dev-perl/OpenCA-OpenSSL
     Latest version available: 0.9.91-r1
     Latest version installed: [ Not Installed ]
     Size of downloaded files: [no/bad digest]
     Homepage:    http://search.cpan.org/~madwolf/
     Description: The perl OpenCA::SSL Module
     License:     || ( Artistic GPL-2 )

*  dev-python/pyopenssl
     Latest version available: 0.6
     Latest version installed: [ Not Installed ]
     Size of downloaded files: [no/bad digest]
     Homepage:    http://pyopenssl.sourceforge.net/
     Description: Python interface to the OpenSSL library
     License:     LGPL-2.1

That's pretty ideal. Now to be clear, I do like a lot of stuff about Ubuntu, but frankly the package system isn't one of them. I think the Debian/Ubuntu communities would do well to stop inviting comparisons of this aspect of their favorite distro until it stacks up a little better.

One thing I've noticed is that Ubuntu and Debian seems to have spent more time ensuring that packages are correct and they lack good tools for dealing with the inevitable package that is bad (broken dependencies, etc) whereas the Red Hat side tend to have more bad packages and better tools for dealing with them. Clearly neither approach is anywhere near ideal, but I'd be hard-pressed to say which is worse.

Effing awesome.

Xronos is a DLR-based Lisp variant. It is a Clojure-like language for .NET. Xronos shares the same syntax and base library as Clojure (but not any source code).

I usually bitch mostly about software and services that suck. However there's a large amount of suckage to be found in the physical world as well, and every so often I find it worth mentioning in case someone who cares stumbles across it.

About a year ago I purchased a new sink for the kitchen, and while I was at it, decided to replace the faucet as well. I found a nice Peerless brand faucet for around $70 at True Value. It was absolutely beautiful. Unfortunately, time has proven that this beauty was chrome-deep.

First of all, once it was installed, I noticed that the spout itself was a bit... wiggly. This hasn't caused any problems to date, but was my first indication that the quality was perhaps a bit sub-par. The second indication came about 6 months ago when my girlfriend accidentally hit the side of the spout with a pan. It left a visible dent in the side of the spout. Now, I imagine it's possible to dent almost anything metal given enough force, but it turns out that this faucet is paper-thin. The next indicator of poor quality came about two weeks ago when the hand-sprayer started leaking water from the handle. Better yet, if you actually tried to use the handle, water would literally spray out the back of it, usually all over your shirt and pants. Now, this is only a $10 replacement part, but really, the damn thing is less than a year old and the spray handle only gets used on occasion. There's no way it should have failed already.

In any case, I won't buy Peerless again and if you are doing some home project, I'd look at other brands.

MetaPython provides a macro and code quoting facility for Python. This looks pretty brilliant. I'll have to try it out soon. Project is hosted on Google Code.

If you've got an Intel wireless card (especially 802.11n adapters), I highly recommend using the latest drivers from the kernel wireless project.

I have an Intel 5100 and was experiencing lots of lost connections, especially under any type of load. Not only that, but it would typically max out at 60Mbps. Since updating to the latest stable drivers, I now get around 130Mbps and haven't had a single disconnect, even when pushing a total of 2.4GB at 9.9MB/s to a server on my LAN. It also seems to associate much quicker, but that might just be my new love for the card.

As a note, these same drivers are available in Ubuntu via the backports package, but I don't recommend using this as it includes a lot of other drivers, and for me, I experienced kernel panics when using them. I also note that uninstalling them doesn't remove them from /lib/modules/2.6.x/updates, so you get to have fun figuring that out on your own.

I finally got my issues resolved by completely removing all files from /lib/modules/2.6.x/updates and following the build/install directions for the new drivers.

If you're already using 9.04, I'd suggest waiting. 9.10 has way too many minor but annoying changes and overall, reminds me of why I left Fedora. I just wasted a few hours upgrading today (can the Ubuntu mirrors be any slower?) and now I'll be wasting more time reinstalling 9.04.

I don't program in Ruby. I don't write Rails apps. Still, now and again I see apps written in Ruby that look useful so I think "I know, I'll give this a try". To date, this has been an utter loss.

1. Typo. 300MB of RAM for an absolutely minimal blog framework? Seriously? Wordpress has approximately 1434 times the features and runs in a fraction of the memory. It's undoubtedly faster too, but I haven't bothered to measure.
2. PowerDNS on Rails. Installation instructions are remarkably simple. Turns out this is because they are sorely incomplete. But this turns out to be moot because it simply doesn't work in any case. I filed a bug report but I'm not holding my breath as most of the bugs listed are months old.

Admittedly, two apps isn't a representative sample by any means. Still, 0 for 2 isn't a good start and doesn't encourage me to invest the time in trying others.

As an aside, I learned a few months ago that "rails" is slang for lines of cocaine (yes, I'm not that up on the drug subculture these days). In any case, I'm suddenly enlightened as to the origins of RoR and DHH's oddly self-aggrandizing attitude.

If you're trying to use pam_pgsql 1.0 and keep getting this message:

PAM_pgsql[652]: the database, table and user_column options are required.

The problem is that the message is an outright lie. Those columns don't exist anymore in 1.0. What it really means to say is:

the connectionstring, getpassword, and changepw options are required.

Apparently there was a renaming at some point but someone failed to update the error message. This can be really confusing as there's a lot of conflicting information on the web. In short, the README is correct, but fails to note which fields are required (I was omitting changepw as I didn't need it).

I'd love to report this bug as it wasted a fair amount of my time and ultimately required that I finally read the source code to figure it out (not too bad, it's only a small C file), but unfortunately the authors are undocumented as well.

Anyone know where can I get one of these in the states?

If you're looking for colocation in Portland, I highly recommend SpireTech. I've been with them since February of this year and I couldn't be happier. They are a good bunch of guys, their prices are excellent, their support is excellent, and their uptime is great too. I may be an easy sell given my previous horrible experience with Infinity Internet, but really I have no complaints with SpireTech.

I will say that Spire's facility doesn't look as high tech as some of the others you'll find, but frankly, I'd rather have the high-quality service than cosmetics. By contrast, Infinity Internet looks great, but then, I guess they have to, since their customers will be there an awful lot due to the outages (they ought to put in a couple of cots and maybe a full lounge).

BUG is a modular, open source system for building devices.

All I can say is "damn, I want one."

Scriptjure is a Clojure library for generating Javascript.

In the last few weeks, Chromium has made leaps and bounds in functionality (with only a couple of regressions that lasted a day or two):

1. Printing now works.
2. Plugin support is acceptable. I've got both Flash and Java working.

Here's the quick and easy for getting Java working:

# apt-get install sun-java6-plugin
# cd /usr/lib/chromium-browser/plugins
# ln -s /usr/lib/jvm/java-6-sun-1.6.0.16/jre/lib/amd64/libnpjp2.so .

Restart Chromium and you should have a working Java plugin.

Lisp has long tended to use DSLs for HTML generation, and Clojure is no different in this regard. In fact, Breve was inspired by Stan which in turn was inspired by these Lisp DSLs.

However, I've tended to like the Breve syntax just a little better than the true s-expression DSLs:

# breve example
html [
    head [
        title [ 'hello, world' ]
    ],
    body [
        div ( id='main' ) [
            'hello, world'
        ]
    ]
]

Clojure provides it's own approach (and Compojure provides a very similar DSL):

;; clojure prxml example
(prxml
   [:html
     [:head
       [:title "hello, world"]
     ]
     [:body
       [:div {:id "main"}
         "hello, world"
       ]
     ]
   ]
 )

Overall, they are quite similar in approach, but the Lisp-like bracket placement continues to disturb me slightly, and I suspect it will disturb my designer (whom I got to eventually love Breve) even more.

Today I was researching Clojure stuff built around Ring and came across Cascade. Cascade includes its own variant of this type of DSL and it's a nice cross between the traditional Lispy s-expression approach and that used by Breve:

(defview index
  [env]
  :html [
    :head [
      :title [ "hello, world" ]
    ]
    :body [
      :div { :id "main" } [
        "hello, world"
      ]
    ]
  ]
)

This is pretty subjective, but I find the bracket placement a bit easier on the eyes and brain.

You learn something new every day, and today was no different for me.

First off, it turns out Emacs has a package management system (at least since 22). It's called ELPA and it makes it much easier to install Emacs extensions. This is the feature we'll use to install clojure-mode.

First, get ELPA installed: copy and paste the following elisp code from the ELPA site into your *scratch* buffer:

(let ((buffer (url-retrieve-synchronously
              "http://tromey.com/elpa/package-install.el")))
 (save-excursion
   (set-buffer buffer)
   (goto-char (point-min))
   (re-search-forward "^$" nil 'move)
   (eval-region (point) (point-max))
   (kill-buffer (current-buffer))))

Next, eval this code by putting your cursor after the last parentheses and pressing Control+j (C-j). This will install ELPA and create a default ~/.emacs file for you.

Next, we tell the ELPA package to download and install clojure-mode. In emacs, press M-x package-install (Alt+x package-install) and enter clojure-mode when prompted. This will download the package and install it for you so that it's automatically loaded when emacs starts.

That's it! Now whenever you load a .clj file, emacs will switch to clojure-mode.

Various links to interesting Clojure stuff.

More will be added as I discover them.

1. Compojure: Web framework similar to web.py or Sinatra.
2. Webjure: Web programming.
3. ClojureSQL: SQL abstraction layer.
4. Ring: Web framework similar to WSGI or Rack.
5. JDBC examples: reference for using JDBC with Clojure.
6. cljrecord: similar to ActiveRecord
7. Using PostgreSQL with Compojure

Clojure is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system. Clojure is predominantly a functional programming language, and features a rich set of immutable, persistent data structures. Clojure runs on the JVM.

To get satisfactory results on Ubuntu, I suggest the following setup:

1. Remove all the java-gcj stuff that Ubuntu probably has installed:

$ sudo aptitude remove java-gcj*

2. Install the Sun Java binaries:

$ sudo aptitude install libjline-java sun-java6-jdk sun-java6-jre ant

3. Download and install git head of Clojure:

$ git clone git://github.com/richhickey/clojure.git
$ cd clojure
$ ant
$ java -cp clojure.jar clojure.main

If the last line presents you with the Clojure REPL (Read-Evaluate-Print-Loop) prompt ("user=>"), then you are have Clojure successfully running.

Next, we want to move Clojure to a more sane place (I chose /usr/local/lib/clojure):

$ cd ..
$ sudo mv clojure /usr/local/lib

Next, we want Jeffrey Chu's Clojure Extras:

$ wget http://github.com/jochu/clojure-extra/zipball/master -O clojure-extra.zip
$ unzip -p clojure-extra.zip "*clojure.conf.sample" > .clojure.conf
$ sudo unzip -p clojure-extra.zip "*clojure" > /usr/local/bin/clojure
$ sudo chmod 755 /usr/local/bin/clojure

Next, edit ~/.clojure.conf and change/uncomment the following lines:

clj=/usr/local/lib/clojure/clojure.jar
clj_ext=~/.clojure
clj_wrapper=jline.ConsoleRunner

Although I don't understand why you should need to, I had to link the jline jar file into ~/.clojure, otherwise java couldn't find it:

$ ln -s /usr/share/java/jline.jar ~/.clojure

Next, install clojure-contrib:

$ git clone git://github.com/richhickey/clojure-contrib.git
$ cd clojure-contrib
$ ant
$ cp clojure-contrib.jar ~/.clojure

At this point you should have a fully functional Clojure install. Just type "clojure" at the command prompt.

I've been testing Chrome off-and-on for a while. It's quite fast, but has lacked basic features that make it usable day-to-day.

That's all changed recently. While still light on features, it actually has enough of a core to present a usable daily browser.

1. Plugin support. Only 32-bit plugins work (I'm using AMD64), but those are easily installed. 32-bit Flash seems to work as well as it ever has (here's one place 64-bit would have really helped).
2. Themes. Not a lot, but they are there.
3. Fonts. Still not ideal, but much improved over previous alphas.
4. Stability. I'm actually fairly impressed. Even failures on youtube (you get a cute sad-mac sort of face) don't crash the browser and a refresh seems to get you rolling again.

In any case, I've set Chrome to be my default browser for the near future. I'm clearly going to miss lots of the Firefox plugins (web developer and firebug especially), but I don't use those as much as I used to anyway.

Firefox 3.5 on Linux has turned into a huge disappointment. It's just plain not fast. In fact, it seems quite noticeably slower than 3.0. I'm sure the Mozilla team knows what's best for their browser, but I'm not sure it's what's best for my Linux laptop. So here's to hoping Chrome is the future of the Linux browser.

As Americans, we are mislead by a common phrase that is fundamental to our criminal legal system: "innocent until proven guilty". We feel entitled to a level of protection that simply does not exist. In fact, the phrase is literally self-contradictory in the context of any sort of criminal proceeding. If you are presumed innocent, then how could you even be arrested and tried?

What the phrase really implies is that the burden of proof is on the accuser rather than the accused. That is, if you are accused of a crime, then you are not required to defend yourself (in fact, you have the right to remain silent for the entirety of the proceedings - see Miranda and the 5th amendment). The burden of proof lies completely on the accuser.

Now, this seems pretty obvious. The problem is not so much that this system is wrong, it's the false sense of security that it bestows on the citizenry. People believe that if they've done nothing wrong (or illegal) then they will be presumed innocent and terrible things will not happen to them at the hands of our government. Of course, this assumption relies on the idea that a criminal conviction is the only terrible thing that can happen to a person. This is most emphatically not the case. You might be detained indefinitely without trial. You might have your reputation destroyed. You might have your children taken. You might be financially ruined defending yourself (simply being detained can ruin most people financially - it's difficult to earn a living from within a cell).

Our current climate of fear has elevated the likelihood of this happening to anyone. I've heard it claimed by a top-level government official that anyone on the no-fly list is "not part of the American family" and should not have the right to bear arms. Never mind that the no-fly list is not a list of convicted terrorists (nor even too specific about why such individuals have been placed there). It's the modern equivalent of McCarthy's list of Communists. Remember this: we have absolutely no idea why most of the people on that list have been placed there, nor can we even hope to find out. These people have been "convicted" completely outside of our legal system. I'd be willing to be that at this point every single person in the U.S. knows at least one person on the no-fly list. I personally know two (one because he has a common name, the other because she was in an animal-rights group during college a decade ago). Of course, since most people don't fly that often, most of them may not even realize they are on that list.

Maoist China had a similar list of people who were tormented under that regime. The motivations for putting people there were varied, but it was quite well known that the "Gang of Four" used this list for their own political ends. Anyone critical of their regime was likely to find themselves imprisoned repeatedly without any specific cause nor with any expectation of a trial.

As Americans, we suffer from a sense that we are somehow superior to these sorts of things. Those things only happen "over there" or "back then". In our arrogance we think that it won't happen here.

To bad it already has.

I was researching versioning filesystems and came across Tux2, which eventually led me to a discussion on LKML with this tidbit:

Lawyers built the patent system. Tim O'Rielly once asked a patent lawyer how he would feel if other lawyers could patent legal arguments and charge him money to use those arguments in court. Though he tried to twist out of answering that one, eventually he had to admit that he had no answer. This lawyer IIRC is the director of the U.S. Trade and Patent office.

—Daniel Phillips

I think that legal arguments are not too different from any other "business process" (i.e. Amazon's infamous "one-click checkout"), so this argument has a lot of validity.

EDIT: I managed to find a link to the discussion referenced. The lawyer (Patent Office Director Q. Todd Dickinson) wasn't nearly as befuddled as Daniel seems to recall, although he didn't really address the question adequately (or more importantly, the potential problems such a system would allow).

I'm not a fan of social networking in general (Orkut was the first to give me a sense of neo-McCarthyistic potential for abuse). This article seems to indicate that the paranoid fears I developed weren't too far off base.

Results 1 - 10 of about 125,000 for vonage sucks. (0.13 seconds)

—Google

I signed up with Vonage as I figured I'd give them a shot in my search for reasonably priced telephony. Overall, the service isn't terrible, although it's hardly stellar either. About one out of every 20 calls gets one-way voice, and it's a bit choppy. Really on-par for what you'd expect from a cheap VOIP service. Problem is, Vonage isn't really a cheap VOIP service. $25/month is cheap for a decent phone service, but it's pretty spendy in the world of VOIP.

As a side-note, I should add that the hardware device they provided is placed between your internet device (cable modem or whatever) and your router so as to bypass any firewall-related issues. A good idea, except the device has a tendency to die under load. Torrent transfers will cause it to reset.

In any case, I was disappointed with Vonage, but not terribly unhappy with them. In the end I decided they simply weren't what I was looking for, so I decided to cancel and try another service. Today I called them to cancel, and after jumping through a number of hoops to get to an agent, I was set to terminate my service on friendly terms.

Unfortunately, Vonage has apparently decided that customers who terminate probably won't be back (I'd agree with them in this assessment, by the way), so they might as well gouge them for what they can before they get out the door.

I was informed that because I terminated before a year's worth of service (I've been using it since February), I'd need to pay $79.99 for the hardware device. When I protested that they'd sold me a reconditioned device which was advertised on the site as costing $10, they told me that this cost was "shipping charges" and that I was given a "rebate" for the full cost of a new device that required that I as well as an early termination fee of $39.99 that makes the total cost come to around $120.

Now, the early termination fee is indeed in their terms of service. I did scan the terms when I signed up, but apparently I didn't read the pages of boilerplate closely enough. Fine. I resent them for not being more upfront about it, but I'm willing to accept blame for being duped.

What is pissing me off is the $80 they want to charge me for the hardware device. When I signed up, I had a choice of getting a new device or a reconditioned device. The new device cost $79.99. The reconditioned device cost $9.99. I've never had many issues with reconditioned devices and the price was right, so I chose the latter. Now they inform me that the $9.99 was for shipping (it wasn't) and that I was given a $79.99 rebate that was contingent on my using their service for a full year. This simply isn't true. This would mean that the new device and the reconditioned device essentially cost the same. When I spoke to the manager, she insisted that this was acceptable because the reconditioned device was "as good as a new one". Huh? I'm sure it is. That isn't the point. The point is that it isn't a new one. The second point is that I was charged $9.99 for it, not $79.99. Further adding to my aggravation here is that they no longer offer this particular device as reconditioned so I cannot confirm it on the website.

In any case, Vonage is crap service for the price they charge and I just wanted to save some other poor sucker from getting sucked into their little scam.

Design... is an attempt to make a contribution through change. When no contribution is made or can be made, the only process available for giving the illusion of change is 'styling.'

—George Nelson

I read this in the intro to the Revised Report on the Kernel Programming Language:

Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary.

It seems obvious, but so few languages in common use today give more than lip service to this philosophy.

As a corollary, I'd add that if you feel especially clever about some twisting of data structures to solve a programming problem, chances are good that you've uncovered a weakness in your programming language.

This is the gut feeling I left Python with. The Python object model has become disproportionately powerful (with many complex subtleties and subtle complexities) to help overcome weaknesses elsewhere in the language. Interesting problems invariably end up being solved via class magic as there are few other powerful constructs in the language.

If you're considering an Asterisk installation, take an extra few minutes to consider Freeswitch instead. You won't be sorry.

I decided to release the new Nginx wiki early as it is working great and I'm tired of trying to manage spam in MoinMoin (must every action take a dozen clicks and page loads to complete?)

I shouldn't have been surprised, but MediaWiki is quite small and fast. The new wiki only uses around ~96MB of the 256MB RAM available in the VPS (that includes all userspace processes: Nginx, PostgreSQL, PHP/MediaWiki, etc). That's a refreshing change from the hundreds of megabytes MoinMoin would suck down (although I blame much of that on Python 2.4).

Anyway, the new wiki is here. Enjoy!

I'm almost done with converting our former MoinMoin-based wiki to a MediaWiki instance.

Overall I'm pretty pleased. Page loads are much faster, deployment is much simpler, and the management seems (so far) to be much less cumbersome in general.

You can see the work-in-progress here. I'm still hacking on the theme a bit, but it's pretty close to what I want.

I spent the better part of the evening converting several sites belonging to a friend of mine from Apache to Nginx. Previously the front page of the biggest site (written in Joomla) took anywhere from 5 to 10 seconds to load. If request load got even moderately heavy, Apache would quickly exhaust the 256MB of memory available in the VPS and the site would become permanently unavailable as processes were killed by the kernel, requiring the VPS to be restarted.

After this happened three times over a period of a week, I decided to take some action. I tried the "easy method", tuning Apache by following guides I found that purported to make Apache suitable to a VPS, but nothing really helped. At best they made the site slower and delayed the inevitable crash. Finally I bit the bullet and converted them all to Nginx (I was reluctant to do so because I'm not terribly familiar with Joomla and there aren't many examples around for running Joomla on anything but Apache - and previous trials a few years ago with Nginx + Mambo turned out to be a huge pain).

At the end of it all, the Nginx configuration turned out to be remarkably simple, and page loads are down to under 3 seconds (the site is very image-heavy and Joomla isn't terribly efficient). And of course, Nginx's memory utilization never exceeds a few megabytes, regardless of load.

To be fair, I'm anything but an Apache expert, so I'm curious if there really is a solution to running something like Joomla in a VPS with 256MB of RAM under Apache. If you think you have the answer, please post it in the comments.

I love top-esque utilites, so here's htop.

Progress Ohio has an article and video.

If you have a recent Linux kernel, iotop is a new utility that makes detecting I/O hogs a snap.

I often use multiple SSH sessions into the same server. I also do things like port-forward remote ports to local ones (for accessing managed switches, databases, and other things that might not have a public IP address).

A trick I've used before is having ssh setup a "master" socket, so that all ssh sessions to a particular host use a shared connection. This also prevents the harmless but annoying messages about being unable to forward ports that have already been forwarded by another ssh session. However, the method I'd been using previously required that I start the first ssh session differently than subsequent ssh sessions.

Tonight I stumbed across this article in Linux Journal which gives a really great setup that gives all the benefits of my old method and makes the whole process completely transparent.

This weekend we moved all our colocated servers from Infinity Internet over to SpireTech. At the same time I consolidated some systems into VPS' and did some general house-cleaning.

Infinity Internet, of course, made their typical contribution in the form of shutting off the network to my cabinet 7 hours early, causing outages for a couple of servers that I had planned on moving after hours. Maybe they didn't feel that getting over twice the bandwidth at roughly half the cost was a good enough reason for me to leave, so they wanted to reassure me that I'd made the right choice by reminding me that they offer shitty customer service as well.

Well played, sirs, well played.

Royce thought it was smooth sailing, until Monica Pleshette, a well-endowed, flesh-eating female zombie and Cliff Wells, her psychopathic boyfriend turned Hollywood vampire, invade Royce's smug life and threaten his life and family.

I usually like Comcast. The only issue I've ever had with them is the typical clueless people they have in "tech support". For instance, getting new service installed unless you have a Windows PC can be a pain in the ass (in fact a couple years ago I tried with a Mac and they insisted I download an .exe file and try to run it - fucking geniuses). However, I've found workarounds for that (call back over and over until I get someone who knows how to remotely register the cable modem).

Anyway, today I got an email with this:

Dear Comcast High-Speed Internet Subscriber:

Action Taken: In an effort to help prevent spam and ensure the security of our network and customers, Comcast has modified your modem’s settings to prevent the sending of email on port 25. That is the default port email programs such as Outlook Express use to send email. We’ve taken this action because we may have detected virus-like activity from your modem or received reports from other email providers that mail from your modem generated complaints from their users. Please read this message to understand how this action may impact your ability to send email and what you should do next.

Now this is annoying on many levels. I'll outline the annoyances in order of severity:

1. They sent the notice after they blocked port 25. For me, this meant I had to not only reconfigure my email client (not a big deal), but also reconfigure the firewall and Postfix settings on my mail servers at the beginning of a business day to accept SMTP on port 587. Nothing I enjoy more than interrupting service for a bunch of people so that I can send mail again.
2. The suggestion that I may be sending spam. I don't have any fucking Windows machines. The real problem is that Comcast's "engineers" are too stupid to setup SPF records to prevent Russian spammers from hijacking comcast.net email addresses. "Our engineers can't spell SPF, so we'll just blame the customer."
3. When I called Comcast to inquire about whether or not they actually believed I was personally sending spam or if this was a network-wide policy change. They informed me it was network-wide. So why must they accuse their customers of being spammers? Oh yeah, they can't figure out how to Google for "SPF".

Anyway, I hate to do it, but I think I may switch to Qwest for a while. Comcast really crossed the line on this one.

I'm looking to replace MoinMoin as the platform for the Nginx wiki. I've tried Confluence, which is pretty flashy but way too heavy (~1GB of RAM before any pages are added).

I'm considering MediaWiki, after all, it's well tested and it should be familiar to anyone who's used wikipedia.org (i.e. everyone). I used to be biased against it because it's written in PHP, but I'm over that.

I've also considered just moving the whole thing onto Google Sites (and that's still an option).

Nevertheless, if anyone has any suggestions I'd love to hear them.

It came up the other day on the Io mailing list that managing multiple Io VM's (each with an HTTP server) was a bit too complicated. Since I'd done a little testing in this area, I decided to see what I could come up with.

First off, we need a bootstrap to launch our VM's. Again I'm utilizing OpenMP to this end:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <string.h>
#include <getopt.h>
#include <omp.h>
#include "IoState.h"

void usage ( char *program ) {
  fprintf ( stderr, "Usage: %s [-n threads] <script>\n", program );
  exit ( EXIT_FAILURE );
}

char *mmap_src ( char *script ) {
  int src;
  struct stat finfo;
  char *buffer;

  src = open ( script, O_RDONLY );
  fstat ( src, &finfo );
  buffer = mmap ( 0, finfo.st_size, PROT_READ, MAP_SHARED, src, 0 );
  close ( src );

  return buffer;
}

void spawn_vm ( char *buffer ) {
  IoState *self;

  self = IoState_new ( );
  IoState_init ( self );
  IoState_doCString_ ( self, buffer );
  IoState_free ( self );
}

int main ( int argc, char **argv ) {
  int threadcount = 0;
  char *script = NULL, *script_src;
  int opt;

  while ( ( opt = getopt ( argc, argv, "n:" ) ) != -1 ) {
    switch ( opt ) {
    case 'n':
      threadcount = atoi ( optarg );
      break;
    default:
      usage ( argv [ 0 ] );
    }
  }

  if ( ! ( script = argv [ optind ] ) )
    usage ( argv [ 0 ] );

  script_src = mmap_src ( script );

  if ( threadcount < 1 )
    threadcount = omp_get_num_procs ( );
  threadcount++;
  omp_set_num_threads ( threadcount );

#pragma omp parallel
  {
#pragma omp single nowait
    spawn_vm ( mmap_src ( "do_server.io" ) );

    spawn_vm ( script_src );
  }

  return 0;
}

This is called mio.c and is compiled with:

gcc -fopenmp -Wall -I/usr/local/include/io \
      -L/usr/local/lib/io -liovmall -lgomp -o mio mio.c

Next, we need a DistributedObjects server so we can centralize our configuration (and maybe use it for other stuff later on). We'll name this file do_server.io:

Shared := Object clone do (
    vars := Map clone
    ip_addr := "127.0.0.1"
    ip_port := 5000

    put := method ( identifier, value,
        vars atPut ( identifier, value )
        return value
    )

    get := method ( identifier,
        return vars at ( identifier )
    )

    addr := method ( return ip_addr )

    port := method (
        ip_port = ip_port + 1
        return ip_port
    )
)

doServer := DOServer clone do (
    setRootObject ( Shared clone )
    setPort ( 8456 )
    writeln ( "Starting DistributedObject Server at 8456" )
)

doServer start

And finally, our HTTP servers:

loop ( // until we have a connection to the DOServer
    System sleep ( 0.5 )
    shared := nil
    do_conn := DOConnection clone setHost ( "127.0.0.1" ) setPort ( 8456 ) connect
    e := try (
        shared = do_conn serverObject
    )
    if ( shared type == "DOProxy", break )
)

server := HttpServer clone do (
    addr := shared addr
    port := shared port
    setHost ( addr )
    setPort ( port )

    writeln ( "Starting HttpServer at #{addr}:#{port}" interpolate )

    renderResponse := method ( request, response,
        response body = (
            "<html><body>hello, world from #{addr}:#{port}</body></html>" interpolate
        )
    )
)

server start

We'll name this file runserver.io.

Now we can run our servers with something like:

./mio -n 4 runserver.io

Of course, it's not much use to run four separate servers on four separate ports, so we need a proxy. As usual, I'll utilize Nginx, which is extremely fast and lightweight and has built-in load-balancing features:

upstream io_server {
    server 127.0.0.1:5001;
    server 127.0.0.1:5002;
    server 127.0.0.1:5003;
    server 127.0.0.1:5004;
}

server {
    listen       80;
    server_name  _;

    location / {
        proxy_pass http://io_server;
    }
}

(obviously this config is just showing the relevant bits to our application).

I've been having a problem for a while that seems to span whatever Linux distro I'm using. It seems to affect both Gentoo and Fedora in any case.

If I ssh into a server, there's an interminably long delay. Running "ssh -v host" outputs something like:

debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

This delay is really long. For example, to ssh to my media server in the basement (802.11n to gigabit) can take over 30 seconds before it asks for a password.

Here's the fix I found: set the following in /etc/ssh/sshd_config:

# GSSAPI options
GSSAPIAuthentication no
GSSAPICleanupCredentials no

Frankly I have no idea what this is. I use key-based authentication almost exclusively and that continues to work, so I don't think I care.

In any case, if someone knows if this opens a potential security exploit or why it would cause the connection to take so long, drop a note here.

I've been wanting to get an actual dedicated media center for the house for a while. I've tried lots of home-brew solutions that ended up being suboptimal for various reasons. Note that I'm not looking for a DVR, just a way to stream my huge collection of amazingly legal media I keep on a Linux box in my basement.

To date I've tried MythTV, Freevo, Elisa and a few others I can't recall. MythTV and Freevo were nice except they felt like they were primarily DVR software with things like music added as an afterthought (and this is actually the case). Elisa would have been pretty awesome except it crashed frequently and consistently kept the CPU utilization of my dual Xeons at 100%. Also, I wasn't entirely happy with my hardware. I had a Shuttle for a while, but ended up giving it away and most of the replacements I came up with were too bulky and loud for the living room.

Then I learned about XBMC. This is a great bit of software. It's pure media center with no pretentions of being a DVR. It has a nice interface. I also learned that it runs on Linux, Windows and OSX. And in fact, it can run on an Apple TV.

When I learned that the Apple TV was only $229, I decided to shell out for one. After all, it comes with TV-out (HDMI and component), a remote, 802.11n and a very nice form-factor. It does lack a couple of things (a DVD drive being an obvious one), but I wouldn't use it anyway (I'd either use my plain ole Panasonic DVD player or just rip the DVD on my laptop anyway). It does have one baffling quirk: no power button of any sort - you have to yank the power cord (which is somewhat resistant to being removed).

First off, the Apple TV software by itself is basically just a glorified front-end to iTunes Store. The only free content you can easily get to is YouTube. Not very compelling. I'm fairly certain the reason Apple prices it so low is that they hope to make their money back from content sales.

Anyway, installing XBMC is a snap (the trickiest part is coming up with a USB thumbdrive that is compatible... I had two that weren't and two that were). You also need a box running OSX (I borrowed my neighbor's iBook).

XBMC doesn't replace Frontrow (or whatever Apple calls it), rather it just adds itself as a menu item (although it appears that any updates from Apple will also uninstall it).

Once XBMC is in place, you can easily stream video and music to it from a UPnP server (I use mediatomb). It's got a really nice interface and only crashes occasionally (and predictably on particular files).

Overall, I find the combination to be the best price/performance/feature package I've used to date.

FOLLOWUP:

Ok, I'm going to completely reverse my previous position. After having this thing for several months, I'd recommend buying something else, notably something with a fan. For another $100 you can get something with twice the horsepower and better cooling.

Here's my complaints:

1. The Apple TV constantly overheats and reboots without warning.
2. Apple updates must be disabled or Apple will thoughtfully wipe out all your customizations.
3. It's not fast enough for HD content.
4. Apple remote has a weak signal and short battery life.

1. buy an iPhone 3G
2. let AT&T drop your calls

Eventually people will simply stop calling, and voila, you can start saving on cellular bills and go back to using the iPhone for what it excels at: showing off the touchscreen.

Manuals, manuals, and more manuals.

Laura got her EEE today, so here's my impromptu review.

First of all, I have to mention the very worst thing about it, because it's something that will definitely affect everyone who owns one: the touchpad buttons are remarkably stiff. Did I say "remarkably"? I meant "ridiculously". It's like, hold the machine steady with one hand and press hard with the other. Or maybe press with both hands. Overall, I'd say they are more-or-less unusable. This means that you will need to leave touchpad tapping enabled or stick to the keyboard.

On the bright side, the touchpad itself works nicely. Two-finger scroll, clever click-to-drag features and some limited gestures in certain applications (OpenOffice and image viewers, mostly). If you can live with touchpad tapping then you won't have an issue with the buttons.

We spent about an hour with the default Linux install before overwriting it with the Ubuntu-EEE distro. I have to say this is a worthwhile upgrade. The Ubuntu-EEE project has done a stellar job with the interface. It's not your typical desktop (it's highly optimized for the small screen), it's more like a sophisticated device (e.g. cell phone or PDA) GUI. It's quite attractive and usable, even without any customization. The only real wart I noticed is that because it runs all windows full-screen, some dialogs can look funny (in particular I noticed a gFTP progress dialog that was pretty bizarre).

Speed-wise the EEE is pretty respectable for such a small machine. I immediately upgraded it to 2GB (yes, I happened to have a 2GB DDR2 SODIMM lying around), but even with 1GB it never seemed to bog down when launching applications. Even compiz works fine (although not with the default EEE GUI, only the traditional desktop).

It's got a nice list of features: 802.11n (although it seems to be connecting at 54Mbps on my home network, I've not yet tracked down why), bluetooth, webcam, two SSD drives (8GB and 32GB), VGA port, 3 USB ports, and an SD reader. Battery life seems on target at around 4.5 hours. It's only got one SODIMM slot, so keep that in mind if you decide to upgrade the RAM. The keyboard is decent. It's slightly cramped, but I had no problems typing, aside from having to adjust for the quirky location of a few lesser-used keys. Laura has small hands so she finds it perfect. The speakers are surprisingly good.

Overall we feel it was a good value ($449 from NewEgg after a $50 discount), although I remain baffled how the touchpad buttons ever made it past QA.

I've finally come to the understanding (somewhat belatedly) that there isn't a lot of difference between an object and a closure. Once again, I'm forced to admit that the Lisp/Scheme people were right all along.

I came across a message from Guy L. Steele regarding the distinction (or lack of) where he states:

A closure is an object that supports exactly one method: "apply".

But of course, this is actually true of objects as well. If the object itself is considered a closure, and each of the object's methods is also a closure, then what you are left with is nested closures, each with one method: "apply". The distinction seems more semantic (i.e. how you choose to view it) than fundamental.

Brian Ford, knowing that I'm experimenting like a freshman college girl after two beers, pointed me to Nu, which looks to be an interesting language. Superficially, it resembles Lisp, but also incorporates ideas from Smalltalk (objects, messages). I'm a little concerned about the Mac-orientedness of many of the examples, but it does run on Linux so I'll probably give it a try.

This post won't interest you much unless you live in Oregon.

I've been seeing advertisements lately accusing Bill Sizemore of racketeering and fraud, which appear to be true. I also see that once again a number of ballot measures are Sizemore projects. This seems incongruous to me. It occurred to me that perhaps what's needed is a ballot measure banning any measures sponsored by Sizemore or any Sizemore-affiliate.

But then, a better initiative occurred to me: an idea I'm now naming the Bill Sizemore Handjob Fund. This would be a ballot initiative that declares that any time a Sizemore measure is shown to have forged signatures, a number of handjobs equal to the number of forgeries is put into a "handjob pool". This pool would be available to people who don't often receive handjobs, namely the homeless and the homely. Vouchers could be issued in soup lines, homeless shelters, cat shows, and other places where the poor and ugly congregate. Whenever a voucher is presented to a policeman, that policeman could escort the voucher-holder to Bill Sizemore's place of residence to verify that the holder receive the handjob in question.

This seems like an all-around win for everyone involved. The otherwise handjobless are blessed with expert handjobs, the voters are protected against fraud, and Bill Sizemore keeps doing what he does best: political whoring.

A recent security "fix" for Adobe Flash causes it to consistently crash Firefox, so I decided to try one of the competitors: swfdec and gnash. I used to simply not install Flash, but I watch enough videos on YouTube these days to find that undesireable now (although I'd much prefer YouTube switch to a non-proprietary format, I'm not holding my breath).

swfdec is kind of nice in that it doesn't automatically start playing. This is a great feature as I consider most Flash content to represent major suckage. If I'm not watching a movie then I don't really need the modern equivalent of an animated GIF wasting my time.

Anyway, swfdec seems to work well, at least in my limited testing (youtube). One site I found that didn't work was voodoopc.com, which happens to work under gnash. Given that, I'm using gnash for the time being, but I consider them both viable as neither of them has crashed Firefox yet.

They both also nicely eliminate the very last bit of non-open source software from my laptop. If you're running Linux, I highly recommend you try one of these alternate players.

I've know about Komodo IDE for a long time, but I only recently discovered Komodo Edit which is a free (as in beer) programmer's editor. I also just discovered that there's an Io language plugin for it (which is actually how I stumbed upon Edit).

I'm not a big IDE fan, but I've been wanting something a bit more modern than my usual editor Emacs for a while. Komodo Edit provides that plus features such as remote opening of files over SFTP.

I've just committed my first changes to Io. Nothing spectacular but it's nice to be on board.

Today I decided to tackle the Io build system. It's a bit of a mess, consisting of several hand-rolled Makefiles with lots of platform-specific definitions and conditions.

I decided I had three main options:

1. Do a cleanup on the existing Makefiles
2. Try to get Automake working
3. Switch to a newer build system such as Rake or SCons

Steve Devortke told me he'd prefer keeping the existing system unless it could be shown that using an alternative was less work.

I decided to give SCons a try, just to get a feel. A few hours later, I've got a working io_static. I seriously cannot believe how easy SCons is to use. I haven't written a Makefile that's over a dozen lines in years and I was dreading it. SCons makes it effing easy.

I understand the desire to stick to known toolchains as much as anyone, but if you don't give SCons a try you are seriously shorting yourself. I didn't try Rake, but if it's nearly as good as SCons you can't go wrong either way.

I upgraded to the bleeding edge from git of Io, and plenty of stuff has changed, notably the Volcano addon, which is the HTTP server I used in my previous post. It's changed enough that the old code won't work under the latest version, so here's an update.

server := HttpServer clone do (
    setPort ( 5000 )
    setHost ( "localhost" )

    renderResponse := method ( request, response,
        response body = (
            """<html><body>hello, world</body></html>"""
        )
    )
)

server start

It's left as an exercise for the reader to update the above to use something besides static HTML (such as Iota).

As a side-note, slice has been deprecated in the latest version of Io and Volcano needs a small fix (just change all occurrences of "slice" with "exclusiveSlice" in addons/Volcano/io/StatusCodes - there's two of them on the very last line).

[UPDATE: this is fixed in latest git ]

Io's concurrency is implemented as async cooperative coroutines. This gives great performance, but doesn't take advantage of multiple cores. Luckily the Io VM is threadsafe, so it shouldn't be too hard to write a program that can run multiple VM's, each in its own thread.

I decided to try OpenMP to implement this as it provides a nice way to do this sort of thing.

This example doesn't quite work just yet. It does sometimes, and other times it segfaults. I'm not sure if the problem is my code, Io, or OpenMP at this point (although I'm suspecting OpenMP due to some messages I've seen in forums).

[UPDATE: It turns out it was a static variable introduced into Io's coroutine library. It's fixed in git - thanks Steve!]

In any case, I'm posting it here in the hopes that it might inspire someone or maybe solicit some assistance.

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <getopt.h>
#include <omp.h>
#include "io/IoState.h"

void usage ( char *program ) {
  fprintf ( stderr, "Usage: %s [-n threads] <script>\n", program );
  exit ( EXIT_FAILURE );
}

void do_io_state ( char *scriptname ) {
#define CMD "doFile(\"%s\")"
  char *cmd;
  IoState *self;

  cmd = (char *) alloca ( strlen ( scriptname ) + strlen ( CMD ) - 1 );
  sprintf ( cmd, CMD, scriptname );

  self = IoState_new ( );
  IoState_init ( self );
  IoState_doCString_ ( self, cmd );
  IoState_free ( self );
}

int main ( int argc, char **argv ) {
  int threadcount = 1;
  char *scriptname = NULL;
  int opt;

  while ( ( opt = getopt ( argc, argv, "n:" ) ) != -1 ) {
    switch ( opt ) {
    case 'n':
       threadcount = atoi ( optarg );
       break;
    default:
      usage ( argv [ 0 ] );
    }
  }

  if ( ! ( scriptname = argv [ optind ] ) )
    usage ( argv [ 0 ] );

  omp_set_num_threads ( threadcount );

#pragma omp parallel
  do_io_state ( scriptname );

  return 0;
}

To compile it, run this:

$ gcc -fopenmp test.c -o test -I/usr/local/include/io -lgomp -liovmall

You'll need to have the Io source headers in /usr/local/include/io (you have to copy them by hand from the source tree).

Then you can run it like so:

$ ./test -n 4 test.io

where test.io has a really useful bit of code such as:

"hello, world" println

You should then get the amazing output:

hello, world
hello, world
hello, world
hello, world

I haven't programmed in C for many years, so don't be too disappointed =)

I've got a vague notion about web development that I'm jotting down here mostly to help me clarify some thoughts and as something to revisit sometime down the road. The question mark in the title reflects my uncertainty about some of what I'm positing below.

First of all, I don't think that MVC as it's applied to the web is the same MVC first described in Smalltalk. Mark Ramm gives a nice summary of what MVC meant in the Smalltalk environment and it's clear that it isn't very close to what we're doing today. A notable difference is what the Controller meant to Smalltalk apps vs what it means (or doesn't mean) to web applications.

Here's a short list of things I see wrong with MVC as it's currently applied to the web:

1. "Designer-friendly" template engines are counter-productive. Designers ought to be writing CSS, not HTML. Programmers ought to be generating HTML and designers styling it. HTML is the output of the program and should have as little to do as possible with how it's presented.
2. The Model, as it is commonly used, is a merely a language-specific repetition of what is already defined elsewhere (the database).
3. The Controller, having lost its purpose as a place to control flow, has been relegated to an ill-defined (but required!) place to shove anything that isn't strictly Model or View.
4. The View exists solely to insulate designers from code and the programmers from markup.

To be clear, I'm not suggesting that MVC is bad, just that it's misapplied to the web. The goal of separation of concerns is a good one, but I think the lines between concerns are poorly drawn for most web applications.

Django redefined MVC into what they term Model-Template-View (MTV), which at one level is just shuffling terms around, but I think it reflects a goal of eliminating the Controller. I originally didn't really get the point of this, but now I'm thinking they didn't go quite far enough (although had they gone the whole distance, it might have hurt Django's acceptance).

So here's some general thoughts about what should be done in place of MVC.

1. Make HTML generation programmer-friendly, not designer-friendly. This means making HTML look like code and HTML generation natural in the context of regular code. I'm thinking in terms of the Lisp philosophy of growing the language toward the program, in other words, by using an embedded language/internal DSL for generating markup.
2. Code that previously got lumped into the Controller becomes part of the Model or is expressed directly in the View. This revives an older paradigm where work was done "in the database" (i.e. in stored procedures). Stored procedures have a lot of advantages (they function outside of the framework, can take advantage of transactions, etc). The main downside to stored procedures is that most databases don't provide nice tools for dealing with them and they often require a separate programming language than what is used in the main application. This can be addressed by building those tools and by using databases that support programming stored procedures in multiple languages (e.g. PostgreSQL). Alternatively, an ORM layer could be used, although this approach has drawbacks as well (poor performance, added complexity, increasing distance between programmer request and actual function).
3. Designers write CSS only. This means the framework should help programmers generate markup with minimal fuss.
4. The application is the Controller. In Smalltalk, the Controller is where things like input loops occurred and program flow was defined. On the web this task falls to the web server event loop and the framework's dispatch mechanism.

At the end of it all, we're left with what seems to amount to Model-View-Style. To summarize:

• Model: the database and related procedures, the sum of which equates to business logic
• View: presentation logic, including markup generation. The Model is directly accessible from here in the form of objects and methods.
• Style: CSS

To rephrase this in terms of a familiar framework, consider that Django is the Controller, the Model is the same, and the View subsumes the Template in their MTV model. If we acknowledge that there's little reason to create a line between the View and the Template in this model, then we simply merge the two and are very close to what I'm talking about (with the notable exception that Django template code isn't appropriate to the solution).

I'm still a bit doubtful about using stored procedures in spite of their obvious advantages. It's a very limiting decision in some ways (reduces choices of databases and languages, raises deployment concerns), but I think it's worth considering. It would be more appealing if things like ORMs (which are slow and complicated) could be rendered moot by having stored procedures return native objects, but that brings it's own concerns and complications. At the end of the day, a more traditional (non-DRY) Model could be used without affecting the overall architecture significantly.

(03:30:00 PM) JSLinuxMan: Ænima
(03:30:02 PM) JSLinuxMan: nice
(03:30:12 PM) JSLinuxMan: Just discovered that on mah keyboard.
(03:39:26 PM) cliffwells18: why would you need an "Ænima" key?
(03:39:41 PM) JSLinuxMan: jahahahaah
(03:39:47 PM) cliffwells18: now, an "enema" key... I could see mac
users needing one of those
(03:39:48 PM) JSLinuxMan: I meant the Æ
[Image] (03:39:52 PM) JSLinuxMan has ended his/her private
conversation with you; you should do the same.

Io happens to include a nice little HTTP server. The HTTP protocol parsing is handled by Zed Shaw's Ragel-generated C parser (same as Mongrel uses), but the rest is in Io. On top of that, I've used an improved version of the Breve-like DSL I outlined a couple of days ago for generating dynamic content (which I am dubbing Iota for the time being).

Io is not a particularly fast language, but it excels at concurrency, as I'll demonstrate here. First, the complete code:

Iota := Object clone do (
    stack := nil
    output := nil

    init := method (
        stack = List clone
        output = List clone
    )

    squareBrackets := method (
        for ( arg, 0, call argCount - 1,
            output append ( call evalArgAt ( arg ) )
        )
        output append ( stack pop )
        ""
    )

    forward := method (
        name := call message name
        attrs := call message arguments map ( arguments ) map ( pair,
            k := pair at ( 0 ) asString slice ( 1, -1 )
            v := pair at ( 1 )
            " " .. k .. "=" .. v
        ) join

        if ( call message next ?name == "squareBrackets",
            output append ( "<" .. name .. attrs .. ">" )
            stack push ( "</" .. name .. ">" )
            self

        , // else
            output append ( "<" .. name .. attrs .. " />" )
            ""
        )
    )
)

server := HTTPServer do (
    setPort ( 5000 )
    setHost ( "localhost" )

    streamResponse := method ( socket, request,
        HTTPResponse withSocket ( socket ) setBody (
            createBody
        ) send
    )

    createBody := method (
        Iota clone do (
            html [
                head [
                    title [ "the title" ]
                ]

                body [
                    div ( class="foo", id="bar" ) [
                        "hello, world" , br
                        "this is a test"
                    ]
                ]
            ]
        ) output join
    )
)

server start

First I'll make clear I'm benchmarking on my laptop (1.5GHz Core2 Duo, 2.5GB of RAM, Fedora 9, 32 bit) which isn't the fastest thing around. Also ab is running on the same system, so we have a fast pipe but some of the CPU is going to ab. Not an ideal setup for benchmarking, but adequate for my purposes.

Now, the above code isn't particularly fast (the HTTPServer on its own gets around 339req/s serving a the same amount of static HTML). However, the requests per second barely dip as the concurrency increases (ab output is trimmed):

$ ab -c 10 -n 10000 http://localhost:5000/

Document Length:        149 bytes
Concurrency Level:      10
Time taken for tests:   49.915 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Non-2xx responses:      10000
Total transferred:      1870000 bytes
HTML transferred:       1490000 bytes
Requests per second:    200.34 [#/sec] (mean)
Time per request:       49.915 [ms] (mean)
Time per request:       4.992 [ms] (mean, across all concurrent requests)
Transfer rate:          36.59 [Kbytes/sec] received

So we're seeing around 200req/s at a concurrency level of 10.

Now let's push it:

$ ab -c 1000 -n 10000 http://localhost:5000/

Document Length:        149 bytes
Concurrency Level:      1000
Time taken for tests:   63.247 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Non-2xx responses:      10000
Total transferred:      1870000 bytes
HTML transferred:       1490000 bytes
Requests per second:    158.11 [#/sec] (mean)
Time per request:       6324.657 [ms] (mean)
Time per request:       6.325 [ms] (mean, across all concurrent requests)
Transfer rate:          28.87 [Kbytes/sec] received

At 1000 concurrent connections, we drop to 158req/s. That's a 100-fold increase in concurrency with only a 21% drop in performance. Not too shabby.

Next up I'll be visiting Io's PostgreSQL library so we can verify that we have the tools for a full web framework stack.

[Update: made a few improvements to the code]

I'm trying to understand the mechanisms for creating internal DSLs (embedded languages) in Io. Here's my first attempt at a Breve-like template language:

Tag := Object clone do (
    output := nil
    stack := nil

    init := method (
        output = List clone
        stack = List clone
    )

    squareBrackets := method (
        attrs := stack pop map ( pair,
            k := pair at ( 0 ) asString slice ( 1, -1 )
            v := pair at ( 1 )
            "#{k}=#{v}" interpolate
        ) join ( " " )
        name := stack pop

        output append ( "<#{name} #{attrs}>" interpolate )
        output append (
            call evalArgs select (
                result, result isKindOf ( List ) not
            ) join ( "\n" )
        )
        output append ( "</#{name}>" interpolate )
    )

    forward := method (
        stack push ( call message name )
        stack push ( call message arguments map ( v, v arguments ) )
        self
    )
)

Tag clone do (

    html [
        head [
            title [ "the title" ]
        ]

        body [
            div ( class="foo", id="bar" ) [
                "hello, world",
                "this is a test"
            ]
        ]
    ]

) output join println

Note that it's not very Breve-like except in general appearance (it's not a directed graph of objects that can be traversed). It's also not feature-complete (although many Breve features would be redundant in Io). Still, it's a nice proof-of-concept and not bad for a couple hours of floundering around by a complete Io novice.

In any case, Breve's syntax was largely bound by Python's and I'm thinking that I'll be ditching this syntax in favor of something even better.

Jeez. I feel like I've been living in a hole for the last decade. Apparently there is no shortage of fascinating programming languages.

Here's the new one for today: Reia

Enjoy.

I'm encouraged to see an alternate implementation of Io for .NET

Unless you've never used MySQL, I don't think this needs much justification.

I've been struggling for the last few weeks to find a way to explain why I'm leaving Python. I could give the technical nitty-gritty (rejects functional programming, lacks lazy evaluation, grows syntactical features like cancer to overcome the first two design decisions), but at the end of the day, even with this intentional crippling of the language, it's still quite possible to write almost anything you want and actually be quite productive in Python.

So, in the face of this, what's my reason for leaving? Well, while it's possible to accomplish nearly any end result you like, you are constrained in how you get there. The language imposes an imperative style through its distinction between statements and expressions. The only way to achieve lazy evaluation is by using lambda and lambda is crippled by the presence of statements (which can't be used inside of a lambda).

Most Python fans claim (arguably with some validity) that these restrictions are in place to enforce code-readability.

Even if I grant that these restrictions do, in fact, help enforce a type of readability, I don't think anyone would disagree that it limits what a programmer is able to express. To me this argument is akin to demanding that authors limit themselves to 10th grade English so as to not confuse readers. While arguably effective, it also places a hard restriction on how concisely, elegantly, and accurately the writer can convey a concept. Worse, eventually the author will be crippled by this restriction as he or she will lose fluency in more esoteric corners of the language. His vocabulary will shrink from disuse.

In short, I think that such restrictions cater to the average (and below average) programmer. This is quite pragmatic, but hardly attractive if you want to advance your art. Poetry may be quite difficult to read at first, but is vastly rewarding once you've learned to appreciate it.

From an article about Microsoft's new D language:

"Today’s developer world is insanely object-oriented,” Lucco said. “Every last piece of data is encapsulated in an object that is a Turing machine, so you have no way to analyze what’s going on with it. Then that’s in stark contrast with the SQL world, where there’s a ton of things you can do with your data, and, over time, different programs can hit the same data and get something out of it. So we were like, ‘Well, how can we make mainstream programming more like SQL programming, without making it harder, like SQL programming is often considered to be?

This is a pretty interesting observation. I still wonder if the issue couldn't be resolved neatly within the OO paradigm by defining "query" interfaces on objects, perhaps not unlike the interface ORMs present. Of course, in another part of the article, he also mentions that D borrows Lisp's "code as data" paradigm, so there may be more to it than meets the eye, assuming he's using the word "data" to mean the same thing in both contexts.

Of course, at some level, it sounds sort of like this might all add up to introspection/reflection, something that's already been pioneered elsewhere.

I've made another attempt to implement Bayesian filtering on this blog using divmod's Reverend. If you have trouble posting a comment, drop me a note (cliff at this domain).

In my recent quests to find a replacement for Python, I've found a few languages that look very appealing. Io is one I'm playing with and another I'm taking a long look at is Scala. Scala, while not quite as fascinating as Io, is probably a lot more practical for doing "real work". It runs on the JVM and can access Java libraries, there's at least two usable web frameworks for it (sweet and lift), and performance-wise it's not as fast as Java, but much faster than your typical dynamic language.

Scala is interesting in that it's strongly typed (and can accept static type declarations), but they aren't strictly required (it seems to do type inference). It also supports the actor-model of concurrency, anonymous blocks and functional programming.

I was reading Lambda the Ultimate and came across a link to this interview with Guido van Rossum, the creator of Python.

Where do you see computer programming languages heading in the near future?

I hope that at some point computers will have sufficient power that we don't need separate functional, dynamic, and statically typed languages, and instead can use a single language that combines the benefits of all three paradigms.

Given that Python firmly rejects functional programming (indeed any paradigm outside procedural and object-oriented), I can only assume that Python's BDFL doesn't see a long-term future for Python.

Make that two of us.

So I think I've settled on a replacement for Python: Io. I find several things highly appealing about it:

1. expression-based
2. almost no syntax
3. actor-based concurrency
4. prototype-based object model
5. lazy evaluation

When I say Io has almost no syntax, I mean it. Here's the complete BNF that defines Io's grammar:

exp        ::= { message | terminator }
message    ::= symbol [arguments]
arguments  ::= "(" [exp [ { "," exp } ]] ")"
symbol     ::= identifier | number | string
terminator ::= "\n" | ";"

Contrast this with (for example) the BNF defining Python's grammar or Ruby's (Perl apparently cannot even be described in a BNF). Having a complicated grammar isn't necessarily a bad thing, but I find smaller grammars and less syntax highly appealing (especially when they actually result in a more expressive language).

Io's guiding design principle is simplicity and power through conceptual unification:

blocks with assignable scope => functions, methods, closures
prototypes => objects, classes, namespaces, locals
messages => operators, calls, assignment, variable accesses

Io still lacks in many areas outside the language itself. The documentation leaves a lot to be desired, libraries are few, the community is small and as such it's a long way away from being a productive language. But, these are things that can be addressed and fixed by people outside the core devs, which I find promising.

Yes, I've said it before, but the final hurdle has been crossed (I swear!).

After a long and fruitless debate on python-ideas, I've realized that Python is what it is and isn't going to change. In fact, most Pythonistas appear completely oblivious to the fact there is a deep and fundamental flaw that prevents Python from becoming what it could be.

Anyway, as a side-effect, I am actively seeking a maintainer to take over Breve as I will no longer be developing it (although I will get out at least one more release which contains some bug fixes).

[EDIT] I need to clarify this: I will not abandon Breve (I have too many projects that depend on it), but I won't be adding new features. I'll continue to accept patches and provide bugfixes where I'm able.

You gotta love Jello Biafra:

We need a new law that owners of SUVs are automatically in the military reserve. Then they can go get their own goddamn oil.

Disclaimer: I'm not an economist... not even an armchair economist (I lack both training and an armchair). Nevertheless I'm going to forward an idea that I see as interesting fodder for thought. Please feel free to disassemble it in whatever fashion you choose.

I'm going to just jump right in and lay the idea on the table with no preparation whatsoever:

Eliminate the concept of bulk discount.

There. I said it. And I mean it: no more buying 1000 items and saving a percentage per item over buying one. When Compaq buys 10,000 processors from Intel, they pay the same price per unit as I do when ordering just one. A 16oz cup of coffee costs twice as much as an 8oz cup of coffee. Supersizing a milkshake will supersize the cost proportionately.

So what are the benefits? I see several right away:

1. It would significantly level the playing field between large corporations and small ones.
2. It discourages over-buying and stockpiling (and hence helps prevent both shortages and dumping).
3. It has no effect on economy of scale, which I consider a cornerstone of progress.
4. It protects against a whole realm of anti-competitive actions (i.e. Microsoft's shafting of Gateway over Windows licenses).
5. It encourages competition based on quality rather than quantity.
6. It cuts down on global waste (because it encourages the practice of buying what is needed rather than trying to achieve the best price per unit).
7. It provides a check against the unrestrained growth of large corporations.

How would it be enforced? The same way antitrust and anticompetitive laws currently are, except it would be much simpler as it's no longer important to prove motive (one of the more difficult aspects of any legal action).

What about downsides? I'm sure there are some (not least of which would be the disruption as the change was enforced).

Some things I don't think would be a result of implementing such an idea:

1. Reduced profit margins would negatively impact the job market. Companies hire people because they need people, not because they have extra money lying around. This was one of the fundamental flaws in the reasoning of Reaganomics. Undoubtedly companies might take a second look at whether they actually need some people, but this would only serve to make the company more productive on the whole. This would also be offset by the increased ability of small companies and startups to compete.
2. Economy of scale would be damaged. It would still cost less to produce a large number of items, and hence they would still be sold for less. What's eliminated is favoritism toward large companies who purchase the items.
3. Made-to-order would be impossible. Made-to-order items are always cheaper in bulk because the setup time is amortized across many items. All that's needed is to separate out the setup/breakdown time. The item itself would cost the same.

And to clarify, some things I'm not talking about:

1. Retail markup.
2. Wholesale pricing. What I am opposing is tiered pricing.

Anyway, this is a rather poorly thought-out idea (in that I've thought on it very little rather than it's a bad idea) but I'd be interested in hearing other people's thoughts on it.

I came across this post which, quite frankly, I consider to be complete and utter FUD. The author clearly at least tried Nginx or perused the documentation since he's got at least a passing familiarity with the software, but he seems to have missed the boat overall. I'm going to address his points one-by-one (at least as much as possible).

[Nginx] behaves in inexplicable ways for different browsers.

This complete and utter FUD. Browsers behave in inexplicable and different ways given the same content, but all Nginx does is serve content. It doesn't create content, so this claim is patently ridiculous.

The primary documentation is in Russian.

More FUD. It is true that Igor (the author of Nginx) is Russian and writes his documentation in Russian (surprise!). However, every bit of that documentation has been translated to English (and has ongoing efforts in several other languages as well, but I cannot attest to their completeness). This documentation has existed for over two years (and is linked to from the "primary" documentation), so there's little excuse to not be aware of it.

Nginx does not support .htaccess files.

Absolutely true. It's considered a feature. .htaccess files are certainly convenient, but they also introduce a serious performance penalty (if you enable them, Apache must check every directory on every page request to see if it has an .htaccess file and if it has changed since the last time it read it). One of Nginx' claims to fame is that it is much faster than Apache. One of the ways it achieves this is by not doing things that add lots of overhead to every request when there are faster ways of achieving the same thing. Further, .htaccess files are a potential attack vector for hackers, although this isn't the main reason Nginx doesn't support them.

Nginx requires you to have apache support tools lying around to do stuff.

Sort of true, but ultimately false. It's true that Nginx does not ship with its own version of htpasswd. It could (and such a program is trivial to write) but for whatever reason it doesn't (perhaps the assumption is that most people have Apache installed and so it would be redundant). But of course the web is full of such tools so you don't really need to install Apache just for this one tool. Also, the author says "tools", implying that anything except htpasswd is missing, but of course this is highly misleading. There's nothing else missing.

Nginx doesn't actually do anything beyond serve static HTML and binary assets... which is to say, it doesn't run php or perl or any of the other P's that you might find in the LAMP stack. What it does is take requests and proxies them to other servers that do know how to execute that code.

Once again, sort of true but highly misleading. The author suggests that all Nginx can do is proxy requests to another HTTP server. This is patently false. Nginx can use HTTP proxying or FastCGI to talk to backend applications. What it does not do is embed languages into the webserver or support CGI. There is a project to embed Python into Nginx (mod_wsgi) but it's not widely used as most Nginx users consider this separation of concerns a good thing. With mod_php or mod_perl, it can be a real pain to debug things like memory leaks because the programming language's interpreter runs within the Apache process. Apache's mod_language tools make for easy deployment but painful debugging. Also, unless you want to restart the entire web server anytime you make a change, you must use .htaccess files (whose drawbacks were outlined above). CGI isn't supported simply because, as Igor says "if you use CGI then you don't care about performance and you should just use Apache in that case". In any case, the difference between how Nginx handles dynamic content and how Apache does it boils down to how they pass the request along to another process (binary API vs HTTP/FastCGI).

The author sums up with this:

Finally, I am left with the question why? The ostensible reason is that it's faster and can therefore handle more requests. Even if we accept that as true (grumble, grumble), it only accomplishes that speed by passing the buck off to other servers. When you find a non-responsive site it's not because the static assets like images and HTML text are being served slowly... it's because the dynamic content generated by php/perl/python/ruby/whatever and the underly database from which the data is drawn cannot keep up. Nginx suffers that same failing... while requiring just as many resources because you now have to run so many different servers for each of the languages you want to code it.

Again, the author displays his complete and utter lack of understanding of a web stack and the difference between how Apache handles a request and how Nginx does it. Apache is a process-based server (threads being a type of process), whereas Nginx is asyncronous. Threaded programs can (but usually don't) perform as well as async programs if you are only measuring, say requests per second under a small to average load. What they don't do nearly as well is called scaling. The reason threaded programs don't scale as well is because they typically launch a separate thread for each incoming request. Not only is there latency introduced as the thread is spawned (this can be addressed to some degree with thread pools, but that introduces other issues), but the worst part is that a thread consumes a significant amount of RAM. On 32 bit Linux, this amount is typically 2MB per thread (the default stack size). That's 2MB per concurrent request, even if you are serving a 2K page of static HTML or a small image. The thread can also allocate more RAM if it needs to, but let's assume it doesn't for simplicity.

Let's say your server has 1GB of RAM and we ignore the RAM used by the kernel, Apache itself, PHP/Perl/Python/etc and all the other system processes. How many concurrent requests can you handle? Pretty easy: 1024MB/2MB = 512. Now for most sites, that's a pretty decent amount of traffic. Of course, we've consumed all the memory to do this, so if we get 513 requests we're now forced into swap. In reality, due to the other software running on the system, we'd certainly be forced into swap much, much sooner and we'd have much less available RAM to start with. Realistically, a 1GB Apache server could probably only handle around 100-200 simultaneous requests (much depends on the size of the response). Note that this is not the same as requests per second, which, if you have sub-second response times, 100-200 requests per second might only amount to 10-20 simultaneous requests.

The author's claim that site slowness isn't caused by static files is probably true for 90% of the web. Of course for sites that serve large files this is absolutely false. The longer a request takes to process, the more likely it is we'll start seeing a higher number of concurrent requests (since new requests come in faster than we can respond to them). This is why sites like youtube.com and torrentspy.com have switched to Nginx for serving static content. A single Nginx server can easily handle as many concurrent connections as imposed by the operating system limits (rather than memory limits). You don't need to be youtube.com to bring an Apache server to its knees if you serve many files that are more than a a few kilobytes each. A single Nginx server can easily replace a dozen similarly configured Apache servers for handling static content.

So what if you aren't serving big files? What difference does this make to you? Well, as a result of its asynchronous approach, Nginx also has the benefit of utilizing far less CPU. It doesn't take much traffic to drive Apache up into the 70-100% CPU utilization range. If you can manage to drive Nginx past 20% CPU then you don't need to read this as you certainly already have a team of scalability experts who can tell you the pitfalls of threading already.

The claim that Nginx is faster because it "hands off" processing to some other server is simply ludicrous. Apache does the same thing. The only difference is the protocol they use to communicate with that other process. Apache uses a binary API to pass information to PHP, Perl, Python, etc. Nginx uses HTTP or FastCGI. The processing of dynamic content is still done by the other process, not Apache or Nginx. Further, because Nginx uses less CPU and RAM, there are more resources available for that other process to use to get its job done. Frankly, this assertion left me flabbergasted. It's actually really difficult for me to believe that someone could have such a poor understanding of such a simple software stack.

Finally, I'm going to address the author's grudging allowance that Nginx might be faster ("grumble, grumble") for static content by simply asserting that Nginx stomps an absolute mudhole in Apache when it comes to overall performance. We aren't discussing a few KB/s faster or a few requests/second faster, we are discussing hundreds to thousands of requests per second faster for static content (depending on your hardware), all while using a fraction of the CPU and RAM Apache does before it fails.

Anyway. I'm going to try to forget I read this pile of rubbish. If the author wants to believe that Apache is better, so be it. I just find it unfortunate that a certain percentage of clueless people will find the article informative.

Get the shirt that makes Patrick Swayze cry.

$20USD via PayPal to sales(at)develix.com. Be sure to include your shirt size and shipping info (overseas is fine).

If you'd like more information, you can email me directly cliff(at)develix.com.

Today I stumbled across Spawning, a high-performance WSGI server written by Donovan Preston. It's based on eventlet, which is a coroutine-based server framework for Python.

Initial reports deploying a Django application behind it appear promising, so I decided to give it a try.

For Pylons, the configuration required only a couple of changes to the Paste .ini file:

[server:main]
use = egg:Spawning
num_processes = 2
host = 127.0.0.1
port = 8000

I've deployed it on both this blog and on the Breve site. I'm planning on getting a proper benchmarking environment up later tonight so I can compare it to Paste#http.

First of all, let me note that I tried to install Foresight 2.0 on this laptop only to find the process too annoying to continue. Fedora, for all its shortcomings, remains one of the best distros for dealing with laptops and less-common hardware.

As expected, the Fedora install went flawlessly. I used the defaults except for filesystem layout (sorry, but I will not use ext3 on my machines).

After installing Fedora, the next item on the list was to change /etc/fstab and add noatime to all the mount options. One day the distros will wake up to what a bad idea access timestamps are. Until then this is requisite, especially on laptops, which tend to have slower disks than servers or desktop machines.

Next, replace Fedora's pathetic package management utilities with SmartPM. You must use the bazaar checkout in order to have Fedora's new mirror system supported. Smart has few dependencies and I can't recall any that aren't included with the base Fedora install.

At this point you have a functional system, but there's still a few things missing:

1. suspend, hibernate and resume
2. Vaio function keys (volume up/down, lcd brightness, etc)

Pretty much everything else should already work, so these two will be the focus for the rest of this article.

The very first thing you'll want to verify is that the sony_laptop kernel module is loaded. Most likely it already is, but you should verify this:

# lsmod | grep sony
sony_laptop            30684  0

If it isn't loaded, you'll need to load it by hand using:

# modprobe sony_laptop

and also add an appropriate line to /etc/modprobe.d/sony.conf or something similar so that it's loaded on boot.

At the time I installed this laptop, gnome-power-manager was at version 2.22. This version will not work properly on a Vaio. You can find the 2.23 version in Rawhide that should work properly.

Next, we need to enable HAL quirks for our model. You can get your exact model string with the following command:

# lshal | grep -i system.hardware.product
system.hardware.product = 'VGN-NR160E'  (string)

You'll need this string when you edit quirks files.

Next, edit /usr/share/hal/fdi/information/10freedesktop/30-keymap-module-sony-laptop.fdi and add a section like this:

<match key="/org/freedesktop/Hal/devices/computer:system.hardware.product" string_outof="VGN-NR160E">
  <append key="input.keymap.data" type="strlist">0x06:mute</append>
  <append key="input.keymap.data" type="strlist">0x07:volumedown</append>
  <append key="input.keymap.data" type="strlist">0x08:volumeup</append>
  <append key="input.keymap.data" type="strlist">0x09:brightnessdown</append>
  <append key="input.keymap.data" type="strlist">0x0a:brightnessup</append>
  <append key="input.keymap.data" type="strlist">0x0b:switchvideomode</append>
  <append key="input.keymap.data" type="strlist">0x10:suspend</append>
  <append key="info.capabilities" type="strlist">input.keymap</append>
</match>

You also need to edit /etc/X11/xorg.conf and add the following:

Section "ServerLayout"
       ...
       InputDevice    "Vaio keys" "SendCoreEvents"
EndSection

and:

Section "InputDevice"
       Identifier  "Vaio keys"
       Driver      "evdev"
       Option      "Name" "Sony Vaio Keys"
       Option      "XkbLayout" "jp"
       Option      "XkbModel" "jp106"
EndSection

Add this as a new section, do not replace the existing InputDevice section.

This will enable the Vaio function keys. It also enables the lid closed sensor (which, like the function keys, is an ACPI button). They won't actually work until you've restarted the computer, but don't do that yet.

Next up, we need to set the HAL quirks for the Intel G965 video so that the laptop will resume properly from suspend or hibernate.

Edit the file /usr/share/hal/fdi/information/10freedesktop/20-video-quirk-pm-sony.fdi and add the following section:

<match key="system.hardware.product" string_outof="VGN-NR160E">
  <merge key="power_management.quirk.s3_bios" type="bool">true</merge>
  <merge key="power_management.quirk.s3_mode" type="bool">true</merge>
</match>

Finally, you'll most likely discover that even though the brightness up/down function key invokes the little onscreen indicator in GNOME, nothing actually happens. To fix this, you must run this command:

$ xrandr --output LVDS --set BACKLIGHT_CONTROL native

Note that this must be run every time X is restarted, so you'll probably want to add it to your session (System->Preferences->Session).

You'll probably need to restart your laptop (I've had mixed results restarting hal-daemon), but at this point you should have a fully functional system.

I'd like to add that the integrated Intel G965 video provides more than adequate performance for a non-gamer and works fine with compiz (I haven't tried compiz-fusion because, well... I just don't care).

References:

1. http://www.pihhan.info/sony/sony-hotkeys.html
2. http://ubuntuforums.org/showthread.php?t=626406&page=2
3. http://www.linux.it/~malattia/wiki/index.php/Vaio_VGN-SZ72B

LinuxHater is an effective man. His blog is like a car-wreck you can't peel your horrified eyes away from despite your desire to maintain a sense of decency and the terrible knowledge that you will be haunted by disgust for days afterward. It's damn hard to stop reading his crap.

Anyway, in much the same way that watching the Special Olympics makes us ponder our own humanity, LinuxHater has compelled me to think about what makes FOSS fundamentally different than proprietary software.

First of all, ask yourself, why the hell would anyone write software for free? I mean, writing software is hard in general and you can actually get paid to do it. What kind of moron gives it away? Here's a suggestion: ask your wife/girlfriend why she doesn't charge you for sex. After all, there's clearly a market and, regardless of what you might like to believe, having sex with you isn't likely the pinnacle of sexual experience for a woman. Still, she does it. There's some inexplicable driving force that causes an otherwise sane person to want to get sweaty with your fat ass. We refer to this phenomenon as "love", probably for lack of a more rational explanation.

Free software is driven by a similar force. Software developers, like women, come in two handy flavors: the ones who do it for love of the task, and Professionals. Now, I have nothing against professionals, but there is a very, very small intersection of people who are both paid to do something and love what they are doing. This means that most of the people who write "professional" software hate their jobs. In contrast, I'd be amazed to find the open source contributor who doesn't love his job. This doesn't mean that the the open source developer doesn't face the same frustrations as the professional developer, only that they are differently motivated and one's expectations of them should be different than a professional. To illustrate this point, consider again your wife or girlfriend (or "life partner" for you fruity-rainbow Mac folks). There are certain, um "tasks" that are almost exclusively the realm of professionals. So when a free software developer tells you in no uncertain terms that he's not going to bend over for you, perhaps you should take it with the same disappointed silence you do when your wife declines to do that thing you saw on while browsing thumbnail sites or maybe, just maybe you might help taking take out the trash now and again you lazy bastard.

You might just get lucky.

Somehow I missed that this guy has a pretty interesting blog, even if it's a bit difficult for a Linux user to read due to the apparent vitrol he extols with every other word. It's hard to read because so many of his bitches about Linux and FOSS are dead-on and it hurts a bit to have the weaknesses of Linux pulled from under the rock and cooked with a magnifying glass. It's also hard to read because the truths are so insidiously intertwined with utter crap that your mind has to do somersaults to properly separate and digest it.

LinuxHater lives in a world of hypocrisy. Why? Because he simultaneously rips into FOSS and then turns and compares OSX in a positive light. Maybe he should remove all FOSS software from OSX and then tell me how much he likes it: remove Mach, BSD, Safari, Adium, bash, etc and then tell me how great it is. Apple has done a wonderful job of polishing these FOSS turds with their millions of dollars. Quite an amazing feat. Take free software, spend tons of money cleaning it up, and then call it your own. Brilliant, if your name is Montgomery Burns.

Secondly, OSX runs on a very limited amount of hardware. I don't necessarily think this is a bad thing, but I expect running OSX on, say, a cheap Dell laptop would not be a much happier experience than with Linux (in fact, probably worse). Would suspend work? How about the sound card or 3D acceleration? Maybe, but your odds would be probably much better with Linux. He complains that Ubuntu crashes on him frequently. As anyone will tell you, this is almost certainly caused by hardware problems (either broken hardware or poorly-supported hardware). Any computer with unsupported or broken hardware is almost guaranteed to cause serious stability issues regardless of the OS you use. Next time, buy a Linux-certified PC (they do exist!).

Of course, any time I mention Apple's limited hardware selection, I'm forced (by the buildup of bile) to point out that Apple's computers tend to be among the most expensive. Sure, you can argue that the price difference will be made up by time save by not futzing around with cheaper alternatives. We could also just solve the world's energy crisis by insisting everyone buy a new electric vehicle. I know that in our precious sense of self-entitlement we often forget that some people actually have more time than money, but let us take a moment to remind ourselves that not everyone is skinny because they are rich anorexics, or that they buy clothes at second-hand stores simply because they have no sense of fashion. Linux fills a niche for those not rich enough to get on the proprietary-software wagon.

And yes, I know many of you will point to the $1200 wonders that Apple sells as low-end models. $1200 isn't horrible for a laptop, but those particular laptops tend to be end-of-life models that would normally appear on overstock.com or in the vendor's closeout section if they were PC's. I just purchased a Sony Vaio for $620 from overstock.com (core2 duo, 15" widescreen). I decided to check what a similar Mac would cost, but I couldn't. You either pay $1100 for a 13" Mac, or jump straight to $2000 for the Pro series. Inexpensive 14" and 15" Macs are a thing of the past, apparently. I guess sometimes choice is a good thing.

Anyway, back to the main topic.

Windows has the same problem Linux has in regard to hardware (it must support a wide variety of hardware), but it gets around the lion's share of the problem by getting support from hardware vendors. They don't have to write 435,485,308 drivers, just charge vendors to have the driver certified. Frankly, there is no way to force vendors to do this from the Linux world. Linux must rely on good will from vendors (e.g. nVidia, Intel, AMD) and the work of volunteers (many of whom have other jobs that actually pay money). Of course, even this actual advantage has failed to make Windows any more stable. After 5 or 6 years, XP was actually usable (if mind-numbingly boring), but Vista is the most god-awful piece of crap I've ever had to pinch my nose over while downloading a Fedora DVD to replace it. Microsoft has had exactly two successful operating systems (from a non-suck perspective): NT 3.1 and 2000. Neither of them were very interesting but at least didn't remind you every 3.5 seconds that there was an OS there who's primary function was to prevent you from working.

At the end of the day, I can't help but wonder what the hell LinuxHater's problem is. If he hates Linux and has the money to buy a Mac, then fucking do it. No one is forcing him to accept what amounts to a free gift and certainly no one is asking for his whining about it (even if he's right). If he's forced to use it at work, then maybe I can sympathize a bit (I bitch endlessly when forced to use Windows or Mac for more than a few minutes), but I doubt this is the case. It sounds to me like a case of someone with too much time on their hands (and too much disposable income as well). Nothing like being rich and bored to get that endearing sense of entitlement and superiority going. At the end of the day, it sounds like LinuxHater is probably technically capable enough (and certainly loud-mouthed enough) to make splash in the FOSS world and help address some of the issues he finds so annoying, so the fact that he doesn't simply leaves me believing he needs his diaper changed.

Some people seem to believe that he's making a valuable contribution by pointing out the shortcomings in Linux. If you also consider punching someone in the mouth when they buy you socks for Christmas as valid criticism, then I guess you might agree.

LinuxHater: we get the message. Please take your ball and go the fuck home.

I've been outed.

I've been keeping half an eye on web2py for a while, since it seems to fill a previously untapped niche in the web framework world (TTW development without a lot of complexity).

I decided to check up tonight and I have to say I'm amazed. I'm mildly leery of the fact that it uses a custom ORM rather than SQLAlchemy (SA seems sufficient and is rapidly becoming the de facto Python ORM), but overall, web2py seems to offer a rather appealing way to manage web applications.

I was especially impressed by the number and quality of appliances built on top of it (and the fact that many of them are from third parties is encouraging).

I was also pleased to see that it supports alternative template engines (so I could use Breve with it if I wanted to [and I do]).

SQL doesn't naturally lend itself to nested data. However, there are several clever solutions to managing tree-like data in SQL, but most of them involve one of the following:

• adding columns to your tables that explicitly map the current nesting (and triggers to keep them sync'd)
• a huge brain (google for "sql nested sets")

If you use PostgreSQL, you can avoid both of these drawbacks (well, a huge brain isn't necessarily a drawback if you have one available - unfortunately for me, most of the huge brains I know belong to evil robots). Actually, any database with a decent procedural language will work, but I only know of one in the open source world: PostgreSQL.

Let's assume we have a table that was created like this:

CREATE TABLE accounts (
  id SERIAL PRIMARY KEY NOT NULL,
  accounts_id FOREIGN KEY (accounts),
  account_name TEXT
);

And it's got records like this:

    account_name     | id  | accounts_id
---------------------+-----+-------------
 account             |  78 |
 sub account         | 102 |          78
 sub sub account     | 151 |         102

As you can see, this table represents a tree graph, where an account can have one or more subaccounts.

This is a really simple and easy-to-maintain system. Unfortunately, it's not so easy to ask questions like "given an account, what are all the subaccounts of that account?" Obviously if there's only one level of subaccounts, the answer is easy, but if you have multiple levels, then it isn't quite so simple.

Here's a fairly simple PL/PGSQL function that can answer that particular question for you. Given the id of an account, it returns a list of rows informing you of all the children of that account.

CREATE FUNCTION get_subaccounts ( parent_id INTEGER )
RETURNS SETOF accounts AS $$
  DECLARE
    child accounts;
    schild accounts;

  BEGIN
    FOR child IN SELECT * FROM accounts WHERE accounts_id=parent_id LOOP
      RETURN NEXT child;
      FOR schild IN SELECT * FROM get_subaccounts ( child.id ) LOOP
        RETURN NEXT schild;
      END LOOP;
    END LOOP;
    RETURN;
  END;
$$ LANGUAGE 'plpgsql';

You can now do a query like this:

db=# select id, accounts_id from get_subaccounts ( 78 );
 id  | accounts_id
-----+-------------
 102 |          78
 151 |         102
(2 rows)

I wanted it back, so there you have it.

After reading this article about how the OLPC initiative is going to embrace Windows on the XO, I was struck about how sometimes being focused on a single solution can be detrimental to your goals.

Negroponte said he was mainly concerned with putting as many laptops as possible in children's hands.

On the surface, this is laudable. However this is a silly goal unless there's a corollary. In the OLPC project, education is the corollary. Without the education aspect, we may as well expect the XO's to be used mainly for browsing MySpace and internet porn. So simply "getting laptops... in children's hands" isn't sufficient. They don't need laptops so much as they need learning tools. If the laptop isn't a learning tool then it's just more strain on the environment and a distraction from real solutions.

This is where Negroponte's narrow vision begins to fail his goal.

Proprietary software, such as Windows, OSX, or Flash, isn't viable as a learning environment unless you are able to spend money to make it into one. The entire point of OLPC is to get learning tools into the hands of children who have no money. Providing them with proprietary software is providing them with a dead-end solution. Will Microsoft continue to provide them with free upgrades that are available over low-bandwidth connections? What about development tools and API's? Can we expect that in ten years the XO will be as up-to-date as Windows 98 is today? Microsoft doesn't have a good track record of supporting legacy software and interfaces nor for providing an inexpensive upgrade path.

I consider providing Windows-based XO's the equivalent of selling African farmers crop seeds that contain the so-called "death gene". It solves their immediate problem today, but gives them no way to become self-sufficient. In fact, all it guarantees is that they will become locked into the corporate food-chain and another revenue source for Microsoft (or more likely, another abandoned project, once the PR has faded).

Exactly what I was thinking.

Continuing my tradition of abusing whatever language is under my keyboard (and in the interest of stress-testing Breve), here's a template that traverses its own DOM and extracts out CSS selectors:

assign ( 'selectors', [ ] ),

macro ( 'get_selectors', lambda tag, is_tag: (
    macro ( 'css_sep', lambda attr:
        '.' if attr == 'class' else '#'
    ),
    selectors.extend ( [
        "%s%s%s { }" % ( tag.name, css_sep ( _k.strip ( '_' ) ), _v )
        for _k, _v in tag.attrs.items ( )
        if _k.strip ( '_' ) in ( 'id', 'class' )
    ] )
) ),
macro ( 'walk_dom', lambda tag:
    tag.walk ( get_selectors, True ) and tag
),
macro ( 'walk_results', lambda selectors:
    pre [ '\n'.join ( selectors ) ]
),

html [
    head [ title [ 'macro madness' ] ],
    body [ walk_dom (
        div ( class_='text', id='main-content' ) [
            img ( src='/images/breve-logo.png', alt='breve logo' ),
            br,
            span ( class_='bold' ) [ "Hello from Breve!" ]
        ]
    ), walk_results ( selectors ) ]
]

This outputs the original page with the CSS selectors at the end like so:

<html>
  <head>
    <title>macro madness</title>
  </head>
  <body>
    <div class="text" id="main-content">
      <img src="/images/breve-logo.png" alt="breve logo"></img>
      <br />
      <span class="bold">Hello from Breve!</span>
    </div>
    <pre>
      div.text { }
      div#main-content { }
      span.bold { }
    </pre>
  </body>
</html>

Don't try this at home, professional driver on closed-course, etc.

Also, this requires Breve 1.2.1 or greater (it required Tag.walk to be changed to return self rather than None).

I was going to wait until later today (it's 2:30a.m.), but I couldn't see a reason to withhold the release.

Here's a short summary of what's new:

• Tag multiplication - you can multiply a tag by a list of dictionaries and have the dictionaries values propagated as attributes and data.
• Macros - Define a function inside a template.
• Assign - Create new variables inside a template.
• Tag.walk - traverse a tag and all its children, performing an action on each tag.
• Improved include function - included templates are now evaluated within the scope of the caller and several former limitations are removed.
• Template code cleanup.
• Unit tests for major API features.
• Improved documentation.
• Tools (soup2breve, xsd2breve, html2breve) are now properly installed.

Be sure to check out the documentation and review the tests to get an idea of what Breve is about.

The evil spirit of Python? I knew it.

I'm thinking of stealing satanpythonface as an IRC handle.

My unit testing paid off (found one bug and a lot of confidence) so 1.2 is slated for release tomorrow.

This blog and the Breve site are already running on it =)

I've been resisting testing frameworks for a long time. I tend to test by using (and apparently being adept at writing relatively bug-free code), but one place tests provide true value is when modifying code.

I'm reworking the internals of aspects of Breve and I realized that I needed to do more than run visually inspecting the output of the included examples (I also test by deploying trunk onto the breve site and this blog prior to releases, but since I don't necessarily exercise all aspects of Breve, this is only a feel-good test).

Anyway, part of the daunting task of testing a template engine is that you must test against literal chunks of HTML. I decide to see what Genshi does, since whether I like that engine or not, it is the 900lb gorilla of Python template engines, and I know Chris Lenz and his cohorts have put a lot of effort into having a solid development process.

I took a look at Genshi's test suite and it was enough to get me started, although I wasn't thrilled about the tests having the expected output embedded directly in the tests. It seemed a bit cluttered and I knew it would get more cluttered as I implemented tests for corner cases (nesting of directives, inheritance, and combining features to check for side-effects).

I hacked about for a bit and came up with a system that seems to be working pretty well.

If you've been avoiding Breve because you're loathe to use software that doesn't include a test suite, you are now encouraged to look again.

My work on getting macros to work in Breve forced me to figure something out that's going to drastically improve Breve.

The fundamental problem is that when Python calls a function, that function gets a local namespace. This is usually a good thing, but it's not always what you'd want in a specialized environment such as a template engine. For example, Breve has an include ( template ) feature that lets you include fragments of Breve code into another template. This worked okay, except that there were all sorts of limitations on what you could or couldn't do in an include file. Further, it required building a dictionary to use as the included file's namespace to make sure that user-supplied variables would be available to it.

Enter sys._getframe. This little-utilized function accepts an integer and retrieves the frame that many levels up the execution stack. For example sys._getframe ( 1 ) returns your current frame of execution. This can be used, for example, to retrieve the name of the function you're currently in (useful for debuggers and the like).

What's interesting to me, however, is that sys._getframe ( 2 ) returns the frame of the function that called the current function. This allows you to retrieve the global namespace of that function and put things back into it. In short, you can make the current function change or create variables in the outer scope:

import sys

def caller ( ):
    return sys._getframe ( 2 )

def f ( ):
    g ( )
    print v

def g ( ):
    frame = caller ( )
    print frame.f_code.co_name
    frame.f_globals [ 'v' ] = 2

f ( )

Running this code will output:

f
2

What this means to Breve is that now when a template calls include(), now the included file will be executed in the caller's namespace. It's actually even simpler than what's outlined above because Breve templates are passed to eval, so all that I need to do is pass caller().f_globals as the globals parameter to eval and the magic is done:

co = compile ( 'fragment.b', 'fragment.b', 'eval' )
return eval ( co, caller().f_globals, { } )

This will provide a much smoother interface. I also expect to apply this to template inheritance.

You can find out more about this topic here.

As it turns out, in Python 2.4, eval ( ) was changed to allow any sort of mapping object (not just a plain Python dictionary) as the locals namespace. What this means is that we are free to pass in an object that can pretend it has all the variables in the world:

class Proxy ( dict ):
    def __getattr__ ( self, attr ):
        return Proxy ( )
    def __getitem__ ( self, key ):
        return Proxy ( )
    def __call__ ( self, *args, **kw ):
        return Proxy ( )
    def __str__ ( self ):
        return ''

eval ( 'x.y ( ).z [ 3 ]', { }, Proxy ( ) )

Despite the fact that none of the variables (x, y, z) exist in the code fragment we're evaluating, we get no exceptions. In fact we don't get a peep of output.

Clearly you'd need to flesh this class out a bit (like handling mathematical operations) and you'd probably want to actually have some values in it, so you'd need to define __setitem__ and __setattr__ behaviour, but that's left as an exercise for someone else.

Once I've made a maintenance 1.1.8 release this weekend, Breve 2.0 will be next on stage.

The big version jump is due to the fact that Breve 2.0 will have some backwards-incompatible changes (although I'll try to make a legacy API available). These changes won't affect Breve templates, only how they are invoked.

Join us on the list and discuss it! Your ideas will be welcomed with the arms of condescension and your fears gently patronized.

Nginx broke the 1 million domain mark this month.

Nevow Stan, which Breve is inspired by, had a feature called "patterns". More or less, this feature let you repeat blocks of the DOM, substituting values into the repeated block.

I've been wanting to add a similar feature to Breve, but didn't feel the Stan way of doing it fit well (it could be a little hard to understand).

Anyway, I think I finally found a couple of approaches that I like (and implemented one of them).

If you're interested, I discuss it here.

I've been neglecting my projects for a while, so it felt pretty good to get Breve's Subversion repository back online, and actually commit a pretty important fix all in one night (include directive suddenly does the right thing).

I think I'll put out Breve 1.1.8 later this week.

I've become rather disillusioned with the "wiki way". We've been using MoinMoin for maintaining the Nginx documentation, but we get an ever-increasing amount of spam. I know that Moin 1.6 adds some new features (such as captchas) to help prevent spam, but I consider these sort of techniques the equivalent of using SpamAssassin to prevent spam (an ever-escalating battle with no end in sight and the spammers always on the winning side).

Moin also has features (albeit horribly cumbersome) for reverting unwanted changes. The problem with these is that not only are they awkward and annoying to use, they allow spam to hit the web and sit there until someone notices and reverts it.

It's occurred to me that what's really needed is a built-in review process (i.e. workflow) for controlling what makes it to the front of the site. Plone has just this thing and seems like it would be a pretty ideal fit for us. Unfortunately Plone is... well, Plone. It seems to me that the only people who successfully use Plone are people who are paid to use Plone (they can spend all day tinkering).

The amount of workflow we'd need is actually fairly minimal: we'd need a few levels of control (anonymous, editor, publisher). New users (who have registered) can make changes as they please, but these changes will never see the light of day until a publisher approves those changes. This seems counter to maintaining a wiki-like set of documents, but we can address this by promoting people to publishers once they've established some small amount of credibility (i.e. they write anything legitimate).

It really does seem like Plone would be perfect here, and in fact, I installed Plone 3 in the hopes that the rumours of massive improvements I've been hearing for the last year or so are true. Unfortunately, I wasn't able to determine if Plone is really improved or not as it continues to suffer a dearth of documentation on the fundamentals. After installing Plone, I spent several hours on the Plone site, searching Google, etc trying to figure out how I might begin changing the design (I did manage to change the logo, but all that did was make the page look odd). The one bit of documentation I found warned that it wasn't updated for Plone 3. Worse, I found that there appear to be not one, but two ways to skin Plone 3 and while neither of them are well-documented, the recommended way wasn't discussed except as a footnote on the deprecated way.

Bottom line Plone guys: no documentation means no users. Seriously. If you want to know why TurboGears, Django, Pylons, roll-your-own are beating out Plone and Zope, look no further than your new user docs. They suck.

I'm not a huge fan of widgets (TSR's anyone?), but I know some people are, so this project appears to be quite interesting for those people.

Just one more reminder why I refuse to use MySQL for anything:

$ ssh user@domain.com -L8888:localhost:3306

then, in another shell:

$ mysql --port=8888 --host=localhost somedatabase
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

Why do people use this turd?

This looks amazingly interesting. It makes me wonder how difficult it would be to say, reimplement CherryPy's wsgiserver using this library rather than Python's native threads.

I decided I would write a quick Python script to move old articles from my ex-ex-ex blog into this one. Turned out to be not too quick.

Anyway, I still need to reformat the comments, but the articles are all there in their fabulous glory.

Here to help us find out is one npseo.com. These idiots sent me a spam this morning. Here are the contents:

Dear Website Owner,

If I could get you five times the RELEVANT traffic at a substantially reduced cost would you be interested? National Positions can place your website on top of the Natural Listings on Google, Yahoo and MSN. Our Search Engine Optimization team delivers more top rankings than anyone else and we can prove it. We do not use "link farms" or "black hat" methods that Google and the other search engines frown upon and can use to de-list or ban your site. The techniques are proprietary, involving some valuable closely held trade secrets. Our prices are less than half of what other companies charge.

I would be happy to send you a proposal using the top search phrases for your area of expertise. Please contact me at your convenience so I can start saving you some money. Please do not hesitate to email or call me if you would like further information.

Sincerely,
Ron Carvel
Executive Vice President

National Positions
26500 W. Agoura Road
Suite 102-547
Calabasas, California 91302

Phone – 866-669-8789
Toll Free – 866.446.2885

Profitable Internet Marketing

I'm seeing several things here:

1. They are clearly scraping WHOIS data (in violation of ICANN rules)
2. They are spammers
3. Apparently their "valuable closely held trade secrets" amount to sending spam.

To add insult to injury, these goat-felchers are using a private registration through another set of fellow animal-lovers who go by the name PrivacyProtect.org. That's right. These scum don't want their fellow scum to scrape WHOIS and spam them with offers of SEO. But who can blame them? I mean really, who wants to get a fresh pile of bullshit from some former door-to-door types who hired a couple minimum-wage high-school VB hacks to scrape contact info from WHOIS? Apparently NO ONE. This must be what they mean by "Profitable Internet Marketing".

This is, of course, the event that triggered my anti-private-registration rant earlier.

Well, I'm continuing to complain to privacyprotect.org (who so far sees "no evidence of abuse" - I can only assume due to a pair of horse balls obscuring their vision) and I've also filed a complaint with ICANN. Not that they ever do anything but avoid useful work, but hey, we all have to do our part.

UPDATE PrivacyProtect.org has notified me that after requesting two copies of the original email from me (their email copier is apparently broked), they agree that a violation has occurred. This is, of course, a huge victory for all of mankind, or at the very least, some of SE Portland, the concentration of victory being densest around my keyboard. Anyway, I'm curious to see what they actually do about it.

UPDATE PrivacyProtect.org has apparently unmasked them =) WHOIS now shows the registrant of npseo.com as:

Registrant:
   RoyalTechNet
   Deepak Bansal        (sales@royaltechnet.com)
   A-3, Shree Balaji C.G.H.S.
   Sector-6, Plot # 37, Dwarka
   Delhi
   Delhi,110045
   IN
   Tel. +91.9416075968

Creation Date: 13-Sep-2007
Expiration Date: 13-Sep-2008

Domain servers in listed order:
   ns4.rover-host.com
   ns3.rover-host.com

I'm actually fairly impressed that PrivacyProtect.org took action in a reasonable time frame, all animal-loving aside. Go PETA.

Still, lest all seem swimmingly georgeous, WHOIS also reveals the following:

$ whois royaltechnet.com
Registrant:
   PrivacyProtect.org
   Domain Admin        (contact@privacyprotect.org)
   P.O. Box 97
   All Postal Mails Rejected, visit Privacyprotect.org
   Moergestel
   null,5066 ZH
   NL
   Tel. +45.36946676

Creation Date: 10-Jun-2002
Expiration Date: 10-Jun-2008

Domain servers in listed order:
   ns2.messagecontact.com
   ns1.messagecontact.com

Sigh. Private registrations remain evil.

[UPDATE (the victory)] Today I was cleaning up some blog spam and decided to check the status of npseo.com once more. Here's what I now get:

[whois.PublicDomainRegistry.com]
Domain Name: NPSEO.COM

Registrant:
    Directi False Whois Suspended Account
    Directi False Whois Suspended Account (inaccuratewhois@suspended-domain.com)
    This Domain is Suspended
    Due to inaccurate Whois
    Contact Support Desk
    null,00000,0000
    US
    Tel. +00.0000

Woohoo. One small victory =)

I hate private registrations. They defeat the entire purpose of whois and mostly protect spammers against compliants.

As it turns out, it also hurts registrants who use it.

I already knew this as I've had to help clients who had allowed ignorant web developers to set it up for them and then lost the information (godaddy.com pushes it as if it will save your social security number from getting stolen). If you have private registration and don't know your way around the registration process, you'll end up paying someone to solve your problem.

Personally, I'm glad. If you use private registration unwittingly, then you ought to complain to your registrar for pushing this dung heap in your face. If you use it knowingly, then I have only one suggestion: FUCK YOU. If you're a web developer who foists this crap on your unwitting clients, please hang up your copy of DreamWeaver or whatever lame crap you use and go out and shoot yourself.

He's right.

Lots of people like to use virtualenv, but personally I like using the method outlined on the easy_install section of the PEAK guide. I've shortened the instructions here to make them specific to Python 2.5 and POSIX systems (I have no idea how well this works on Windows, nor do I care much).

First, create an altinstall.pth file in Python's site-packages directory, containing the following:

import os, site
site.addsitedir(os.path.expanduser('~/lib/python2.5'))

This will automatically add each user's ~/lib/python2.5 directory to sys.path (if it exists), and it will process any .pth files in that directory.

The next step is to create or modify distutils.cfg in the distutils directory of your Python library. The correct directory will be something like /usr/lib/python2.5/distutils on most Posix systems:

[install]
install_lib = ~/lib/python2.5
install_scripts = ~/bin

This will configure the distutils and EasyInstall to install packages to the user's home directory by default.

The next step is to prevent these changes from affecting the root user (since usually root will be installing packages system-wide).

Create a file named .pydistutils.cfg in root's home directory (usually /root):

[install]
prefix=/usr
install_lib=/usr/lib/python2.5/site-packages
install_scripts=/usr/bin

[easy_install]
site-dirs=/usr/lib/python2.5/site-packages

I saw this article and it made me think of this one.

After all the work I'd put into getting suspend to work, I thought I was done at last. However, in grand fresh-Linux-install tradition, I discovered that my backlight dimming was no longer working. This seemed odd as it's always worked without any special attention from me.

Lots of googling later, I found a solution, even if it wasn't exactly what was working before:

# smart install fnfx
# service fnfxd start
# chkconfig --levels 345 fnfxd on

Not only does this let me use the Toshiba function keys, it also allows me to reprogram them =)

You can read the fine manual for fnfx here.

In FC7 and earlier, it appeared that HAL/pm-utils was managing the LCD backlight (GNOME brightness-applet worked, Toshiba function keys caused a little dialog to appear showing the brightness level, etc). This is no longer the case (applet doesn't do anything, no info dialog, etc), but it works, so I'm not going to complain too much.

As an aside, it appears to me that what has changed since FC7 is that HAL no longer recognizes my display (it shows up as "generic LCD" rather than the specific model and "lshal" doesn't list it at all). Not sure how to fix this. I may look into it more later, since having it properly supported would be good. For now just working is satisfactory. Still, if you have any hints, please feel free to drop them here =)

UPDATE: I did an update tonight, and after a reboot things are back to normal. Not sure which package it was that fixed it. I do notice that HAL is once again seeing my lcd though:

# lshal | grep panel
info.capabilities = {'laptop_panel'} (string list)
info.category = 'laptop_panel'  (string)
laptop_panel.access_method = 'general'  (string)
laptop_panel.num_levels = 8  (0x8)  (int)

Is it just me and my gutter-condemned mind, or does this remind you of a psychedelic crotch?

Tonight I decided two things would happen: I would upgrade to Fedora 8 and I would get my damn suspend to work (it hasn't worked since FC5).

The upgrade to FC8 went smoothly as expected (I was on FC7). I decided to dump my GNOME settings in order to not interfere with any new goodness that might get bestowed on me. More or less it was a fresh install, but like any sane person, I keep /home on a separate partition so no backup/restore was required. All in all it was about an hour from start to finish (that included reinstalling any missing apps and basically getting it back into working shape). Pretty acceptable.

After the novelty of the newish looking desktop had worn off a bit, I decided to delve in and make my suspend work (suspend-to-RAM that is, I consider hibernate or suspend-to-disk to be a waste of time).

First thing was to assess what the current state of affairs was. So without further ado, I selected "Suspend" from the GNOME power icon. They system suspended properly down to a single little blinking light on the front. So far so good. Next comes resuming. I press the power button, things come back to life, I get a tiny bit of text that says "Linu" and... nothing. Black screen. I try switching virtual terminals, ctrl-alt-backspace, etc, all to no avail. The system isn't locked up (I can see disk light activity when I killed X) but there's no screen. Ctrl-alt-F1 and ctrl-alt-del reboots me. I do some research and finally come upon this page. I've seen it before but felt daunted by the absolute denial that my configuration was supported (binary nVidia driver) and that my laptop model invoked pages of not mentioning. I decide to forge ahead anyway.

My process is going to basically be trial-and-error. There are only so many quirks to try. Because I don't want to screw up my brand new GNOME settings (Gconf has a tendency to bork your settings if you kill it enough), I create a user called "pm" for testing with. I begin trying the most promising settings first (--quirk-vbe-post, etc). Nothing, nothing... nothing. Lots of reboots and little to show for it.

But then. Finally:

$ pm-suspend --quirk-vbe-post --quirk-dpms-on

and voila. I'm right back in GNOME where I left off. Joy.

Next step is to make this permanent. Following the docs, I edit:

/usr/share/hal/fdi/information/10freedesktop/20-video-quirk-pm-toshiba.fdi

and add the following section:

<!-- Toshiba Tecra M3 -->
<match key="system.hardware.product" string="TECRA M3">
  <merge key="power_management.quirk.vbe_post" type="bool">true</merge>
  <merge key="power_management.quirk.dpms_on" type="bool">true</merge>
</match>

Next I restart HAL with:

# service haldaemon restart

and finally the grand final test: select Suspend from the GNOME power icon.

Works perfectly.

Anyway, I hope this helps someone else. I'll post on the nVidia forums and Fedora forums.

Firefox 3 really rocks. It's sooo much faster than FF2, it's like a different browser.

However, one thing that really sucks is that my favorite and most useful extensions (Firebug, Web Developer) haven't yet been ported.

At least one of those now has. There's an unofficial build of Firebug for FF3. Enjoy.

I've always hated putting CD-ROM drives in my rackmount equipment. They waste space, have a high failure rate and require that I keep a bunch of rescue CD's around.

I decided to investigate using a compact flash (CF) card instead. My idea was to use an IDE-to-CF converter and simply leave a small (128MB) CF card permanently installed inside the server case.

Turns out that it's not only cheaper to use CF, it's also remarkably easy.

I bought the CF->IDE converters from tekgems.com. I liked these because:

1. they were bootable
2. they were cheap
3. simple design leaves little room for mistakes
4. positive user reviews from Linux users

I ordered 10 of them for a grand total of $32USD. One thing I'd recommend is either buying a very short IDE cable (i.e. one or two inches) or an IDE gender-changer. Due to the layout of these adapters they are somewhat difficult to mount inside the case (IDE connector is on one side, floppy power connector on the other).

I happened to have a couple old 128MB SD cards lying around, so I chose to use one of them, but if I were to buy one I'd probably get a 1GB since the price difference is negligible and you'd have a bit more room to customize.

I wasn't really looking to have a full Linux live environment available. Rather I wanted the equivalent of a rescue/install CD (something to let me do repairs and/or reinstall when needed).

I'm in the process of building a new server (Opteron) for running OpenVZ on, so this was a good candidate for my test. I chose Gentoo for the host OS since I wanted to minimize downtime when upgrading the host OS.

I looked at a variety of projects for small and embedded Linux distros, but none of them really met my goals. I realized I was going the wrong direction and instead investigated simply putting the Gentoo Minimal/Install CD on the CF card.

Turns out this was not only remarkably easy (far easier than dealing with some of the micro-distros) but only took around 10 minutes.

I followed these instructions (a CF card on an IDE adapter appears to be a normal disk). I would recommend changing at least one parameter: the author puts a timeout of 150 seconds. This is way too long. I'd recommend 30 seconds top. Also, if you want to get a bit fancier, you could mount the squashfs image, customize some of the parameters (IP address, add a root password, start sshd, etc).

In a perfect world, I'd like to figure out a way of having a server failure cause it to boot from the CF card with sshd started so I could remotely access it in case of disaster (versus having the server try to boot normally and fail to complete). It's occurred to me to have the server always boot from the CF, but it's not obvious how I'd have it select whether to boot normally from that point or enter rescue mode. Too bad GRUB isn't scriptable.

In conclusion, a CF solution with a 1GB card will run you well under $20. That's cheap enough to permanently install one in every server (cheaper than a no-name CD-ROM drive and much cheaper than a slimline CD-ROM as is often found in rackmount equipment). Also, it's much easier to build a customized boot disk than with a CD (think of it in terms of ease of fixing mistakes).

Nginx 0.6.20 introduced the ability to use variables with the proxy_pass directive. This seems like a rather small feature by itself, but it allows us to utilize the powerful map directive to better organize virtual hosting.

In our shared hosting configuration, we typically assign an entire internal IP address (127.0.0.1, 127.0.0.2, etc) to each client. This allows clients to map multiple ports to various applications (web apps, databases, etc) without too much concern about other users.

However, this can be difficult to keep in order since these addresses tend to get scattered across dozens of virtual host entries and include files in the Nginx configuration.

This is where map and using variables with the proxy_pass directive come in handy.

Here's a short example:

http {
    resolver 127.0.0.1; # dns server
    map_hash_max_size 4096; # needed if your hash table gets big

    map $host $internal_ip {
        hostnames;
        .develix.com            127.0.0.2;
        .twisty-industries.com  127.0.0.3;
        .codemongers.com        127.0.0.4;
    }

    server {
        server_name develix.com www.develix.com;
        listen 198.145.247.210:80;
        location / {
            proxy_pass http://$internal_ip:8000$request_uri;
            include /etc/nginx/proxy.conf;
        }
    }

    server {
        server_name twisty-industries.com www.twisty-industries.com;
        listen 198.145.247.210:80;
        location / {
            proxy_pass http://$internal_ip:8000$request_uri;
            include /etc/nginx/proxy.conf;
        }
    }

    server {
        server_name codemongers.com www.codemongers.com;
        listen 198.145.247.210:80;
        location / {
            proxy_pass http://$internal_ip:8000$request_uri;
            include /etc/nginx/proxy.conf;
        }
    }

    server {
        server_name wiki.codemongers.com;
        listen 198.145.247.210:80;
        location / {
            proxy_pass http://$internal_ip:8001$request_uri;
            include /etc/nginx/proxy.conf;
        }
    }
}

Note that while I could have put the port in the variable as well, this detracts from the flexibility, since each server section could conceivably utilize more than one port. It also tends to make the mapping less readable.

As you can see, the advantage of this approach is the ability to keep all the mappings in a single location, which is far easier to maintain.

Caveat: as of 0.6.20, the fastcgi_pass and upstream directives do not support variables, although I expect this will change soon.

My mother-in-law brought over a couple watercolors done by a friend of hers. Both of them feature seagulls, otherwise known as the rats of the sea. One of them doubles up the foulness by having a seagull perched on a pair of McDonald's golden arches. I can't help but imagine the picture captures the glowing moment right before the seagull swoops down and starts picking at a half-eaten Big Mac in the dumpster.

Apparently the intent is that they be displayed in my daughter's bedroom.

What's next? A watercolor of a rat chewing on a cat skull?

New rule: we don't accept "art" as gifts.

Over at hacknot, there's an article criticizing dynamic language "weenies". In general I find the author's criticisms valid when it comes to many of the breathless claims of fans of DL's, but disagree with his conclusion that since it can't be proven that DL's are more productive that they aren't.

I have no empirical evidence to add to this debate, but lack of evidence is not a counter-example. That is to say, while the DL weenies may not have empirical evidence that DL's are more productive than statically typed languages, there is also no counter evidence demonstrating they are wrong. However there is a large body of anecdotal evidence that seems to support their position.

I'll add my own anecdote as I think it's quite expository:

I had been programming almost exclusively in C for over a decade when I undertook a task of writing a custom instant messaging system for an employer (this was in the 90's and there wasn't the plethora of IM software and protocols that there are today, plus they had specific requirements regarding logging, encryption, etc). The task seemed interesting so another employee and myself undertook to write the system using C. A week into the project, we'd managed to hack out a protocol, some low-level libraries and began work on server process. At this point we began getting bogged down. Neither of us had done much network-specific programming at such a low level and the details of process management were a real problem.

At some point I recalled a Linux Journal article about Python I'd seen a year or so prior. Out of curiosity, I decided to start a side project mirroring our system using Python rather than C to see if there were any benefits to be had.

This idea turned out to be a miserable failure. Within two days I'd finished my server process, implemented our required feature set, had a basic client GUI, written some stress tests and was starting to add new features. I showed this to my colleague. We immediately abandoned the C version and continued work on the Python version. At leat some of this was due to Python having better libraries than my C compiler, but the fact is, I mostly wrote low-level code, so I would attribute most of the productivity to Python's language features rather than libraries.

Now I want to reiterate: a decade of C experience. Zero Python experience. Also, I prided myself on my C skills. I'd written things like a cross-platform GUI (widgets and window manager) and a 3D surface modeler in the past. I had absolutely no problems with C. In fact I loved it. But I couldn't deny how productive I'd become almost overnight with Python.

I also want to nitpick one more little bit from the article. The author states:

As an aside, let me point out that the removal of explicit typing is not without its disadvantages, even though it may reduce code volume to some extent. Consider that when you remove type qualification from the source code, you're also removing some documentation of your intent. Given the paucity of documentation in many code bases, it seems unwise to start removing type-related documentation as well. Consider also — if it's good form to give your identifiers meaningful names to aid in comprehension, isn't it equally good form to explicitly assign them types, for exactly the same reason?

First of all, the author badly confuses "type" with "static". Python has types. Further, they are explicit. Consider, for example:

x = 6

Is that not explicit? 6 is clearly an integer. Do I need to spell it out? Well in some languages you do, but I don't think that it necessarily makes it any more explicit. Is the following really any clearer?:

int x;
x = 6;

If you must spell everything out, then why not:

var x = int 6

I'll tell you why: because there's a difference between explicit and redundant. The assignment makes the type abundantly clear. All the type declaration does is add a useless line of code that helps no one except the compiler.

Frankly, if you think the word "int" adds any clarification to your intent, then you are badly deluded. If you have read a section of code to determine the purpose of a variable and are still unsure of its type, then no amount of documentation is going to help you.

"Sure," I hear you say, "but I'm not referring to scalar types, but user-defined types, such as records." The same argument holds for more complex types as well. Let's have some real code to see how Python is a bit more terse than C:

class Person ( object ):
    def __init__ ( self, name, email ):
        self.name = name
        self.email = email

 p = Person ( 'John Doe', 'john@jdoe.com' )

Now in C:

typedef struct Person {
    char *name;
    char *email;
} *personPtr;

personPtr create_person ( char *name, char *email ) {
    personPtr;

    p = ( personPtr ) malloc ( sizeof ( struct Person ) );
    p->name = malloc ( strlen ( name ) );
    p->email = malloc ( strlen ( email ) );
    strcpy ( name, p->name );
    strcpy ( email, p->email );
    return p;
}

void destroy_person ( personPtr *p ) {
    free ( *p->name );
    free ( *p->email );
    free ( *p );
    *p = NULL;
}

personPtr p;
p = create_person ( 'John Doe', 'john@jdoe.com' );
destroy_person ( &p );

It's been a while since I've done much C and I don't feel like testing the above, so please forgive any errors.

You might think that the C example is a bit overkill, but you're wrong. If you think about how such code might be used in a real application, you'll quickly realize those two pieces of code are functionally equivalent. They both allow for an arbitrary number of instances of Person records, and both allow for arbitrary length of name and email.

Finally I'm going to call "foul" when proponents of statically-typed languages claim that explicit type declarations (I should call them "separate" rather than "explicit", but whatever) buy any sort of help when it comes to debugging. Write very much C and you'll soon find that your most flexible code is filled with void pointers. A linked list of arbitrary structures? void pointers. Almost all the interesting code I wrote in C involved void pointers and type casting to a large degree and frankly that's the main thing dynamic languages do for you: they encapsulate and provide support for using arbitrary objects without the nonsense of declaring a type that you aren't going to use anyway.

I'll grant that C is hardly the epitome of statically-typed languages, but it's what I'm most familiar with and makes a striking example of how static typing leads to overly verbose and redundant code.

I was thinking the other day about how any action against the U.S. government is now considered "terrorism". I was watching Stephen Colbert who made some tongue-in-cheek comment about the number of terrorist attacks and amongst his list of recent attacks was the attack against the U.S.S. Cole. I realized that most people probably consider this attack a terrorist attack. This is incorrect.

The attack against the Cole may have been tragic, but it was not a terrorist attack. It was guerilla warfare. What's the difference? It was waged against a military target, not civilians.

If we are to believe the Bush administration and the media, then any and all actions against an established government are "terrorist" attacks. But of course, by that definition, the Revolutionary War was a (successful) series of terrorist attacks against the British.

The casualties in the war on words are piling up.

Zed Shaw wrote an article that I find to be both highly informative and deeply annoying. Zed discusses the failures of most programmers to properly analyze performance data. His insight into the matter is really quite enlightening. Of course, that's if you can stand to listen to him past the first paragraph.

Zed spends a lot of time talking about fucking and killing, but his girlish whine only leads me to believe he hasn't done a lot of either.

He gives an example of how understanding statistics has helped him save the day:

Then I hit the DB2 database and about crapped my pants. Almost all of the queries performed great, except one query that had sub-second response on average, but a 60 second standard deviation! This was the query. I made a chart of all the different queries, marched into a meeting, slapped them on the table and said, “It’s not the database, it’s IBM’s DB2 configuration. Here’s the time measurements to prove it.”

The next day we had IBM fixing the problem (turned out to be a single update index command) and we all kept our jobs. That’s what a proper analysis method can do for you.

Talk about confounding. Here's my analysis of what Zed did right and what Zed did that was wrong (or at least pointless):

Right:

1. Looked at the database logs and located the slow query.

Wrong:

1. Wasted time and paper making graphs and charts that did anything except sort the execution time of queries in descending order.
2. Slapped this handful of toilet paper on the table as if he'd done anything more than his job.

I mean seriously. Anyone who's administered a database (or any other bit of server software for that matter) will tell you that the very first thing you do when something goes wrong is check the logs. It doesn't take a PhD in rocket surgery to parse a log file looking for anomalies.

I also can't help but notice Zed didn't actually fix the problem. He just found someone to blame. I wonder if the IBM tech who actually fixed it wrote a big rant in his blog threatening to kill all programmers who can't add "a single index" to their databases.

A request that pops up on the Nginx mailing list every so often is the ability for Nginx to restrict backend servers in a clustered proxy to each receiving only one request at a time. This request invariably comes from someone using Rails who has discovered that Rails isn't thread-safe and can therefore only handle one request at a time.

For some reason I was under the impression that Mongrel was intended to help solve (or at least work around) this issue, so I decided to get a bit more info from the Mongrel site.

It was there that I found this gem:

Rails is not thread safe so if you have long running actions then you can potentially block the server. Keep in mind that this is the case for any server, it’s just more pronounced for Rails.

Okay, seriously. There is so much wrong with this statement that the question isn't whether or not to laugh at it, but whether or not to find the author's phone number, call him up, and laugh in his face.

First off, while I'm less than certain the author intended to say this, the awkward structure of the paragraph could lead some to believe that no servers are thread-safe. Fact is, with the exception of potential programming errors, all servers are thread-safe (either fully reentrant or protected with locks to force serialization at critical points). Rails and perhaps a few homework assignments from mediocre junior college students being the exceptions.

While I doubt the author truly meant to assert the above, I am certain he meant to convey the fact that all servers are potentially susceptible to dropping requests when there aren't enough resources to handle them all. Obviously this assertion is true at some level. Where the falsehood lies is where the resource constraint comes from. With Rails, there is a single shared resource that is constantly exhausted whenever there's a request: it's Rails itself. With other servers, you'd have to exhaust system resources such as RAM, threads, TCP sockets, CPU time, etc. Other servers don't have bottlenecks designed into them. Claiming all servers have this problem is akin to defending the Corvair because all cars can explode under the right circumstances.

This shortcoming in Rails is so ridiculous that it literally leaves me agog that anyone would even consider using it for anything but the simplest of toys.

One of the things I find tremendously funny is how the Python community is always criticized for having too many web frameworks. The core concepts of Rails has been reimplemented in Python at least a dozen times. Apparently it just isn't that difficult to do. Apparently it's not difficult to write a threaded or asynchronous server either, since all of these servers were able to handle multiple simultaneous requests just fine.

Seriously, if you love Ruby, I can understand. But Rails? How far must someone put their head up their own ass in order to be able to overlook how fundamentally flawed it is?

You know what? I don't want to know. I only hope that Igor doesn't put too much work into adding this retarded "feature" to Nginx.

There's a lot of talk (okay, usually arguing and whining) about whether particular technologies are "enterprise-ready". Most programmers who favor stodgy technologies like Java and Oracle crow about "enterprise-ready" in terms of supporting thousands of concurrent users (i.e. scalability), but usually fail to make a real case when it comes to showing where competing products fall short. On the other side, fans of newer languages and frameworks (especially Web 2.0-ish stuff) like to disparage the term as meaning "expensive and hard to integrate".

Wikipedia defines it as... oh wait. It doesn't. Apparently even the venerable Wikipedia is afraid to weigh in on the matter. Probably out of fear that the constant stream of competing edits would demonstrate that Wikipedia isn't enterprise-ready.

So which is the real definition of "enterprise-ready"?

None of the above. I hereby define "enterprise-ready" to mean it's ready for the Enterprise. The USS Enterprise. When someone asks if a product is "enterprise-ready", what they really want to know is would James T. Kirk entrust his crew to it?

With this in mind, let's review a few popular technologies that often have this question leveled at them:

1. Ruby on Rails The Enterprise is under attack by Klingons. Kirk orders shields up and fire photon torpedoes. Scotty replies "we cannot process more than one request at a time, sir!". Verdict: Not ready.
2. Java The Enterprise still isn't built and Klingons have overrun the galaxy. Meanwhile there's plans on the board for Java Next Generation which will address these issues. Meanwhile everyone switches to .NET. Verdict: Not ready.
3. PHP A 14-year-old school student does a SQL injection attack on Starfleet's payroll system, forcing most of the crew out of work and Kirk to take a job shilling cheaper hotel rates. Verdict: Not ready, but eerily familiar.
4. Python Every single member of the crew writes their own framework to support the Enterprise. Before any of them are fully documented or complete, they are labeled "deprecated" as the authors start on new, cooler versions that never materialize. They also write some cool metaclasses that no one else can use. Meanwhile the Romulans install Ruby on Rails and very, very, very slowly, destroy the Starfleet ships, one at a time. Verdict: Not ready.
5. Linux The warp drive system lacks a driver and the vendor won't release specs. The developers argue that the warp drive system violates RFC 453446349 anyway and that Starfleet should get rid of the Enterprise and only buy spaceships without proprietary hardware in them. Verdict: Ready real soon now.
6. Windows Every time the Enterprise engages in battle, the ship slows to a crawl and must be restarted. Pressing the intercom button too fast can cause the entire ship to completely lock up. Often, enemies can remotely control the Enterprise and make it send badly spelled messages back to everyone on base and attack other Starfleet vessels. Also, it turns out what everyone thought was warp 5 was just a screensaver. Verdict: Not ready, but funny to watch.
7. OSX The Enterprise destroys the Klingon Empire by dragging it onto the trash icon. Then they spend the next 5 episodes bragging about it until the entire universe hates them. They claim every sci-fi show written since 1966 is a knock-off. Most of the crew grow sideburns which look ridiculous. Verdict: Ready as soon as the crew is done admiring themselves in the mirror.
8. Oracle This is the Enterprise, not the Death Star. Verdict: Vader chokes Kirk who fruitlessly beats on Vader's back with both fists. Baffling.
9. MySQL Spent years denying that the Enterprise existed. Later tries to sneak in the back door in a poorly-fitted uniform that rips in embarrassing places when stressed. Verdict: Ready if your Enterprise is the sort made by Mattel.
10. PostgreSQL If only I'd stored some PostgreSQL jokes in PostgreSQL. Then I'd still have them. Verdict: Ready when you are.
11. .NET The Borg. Verdict: Ready if you want your ass assimilated.
12. Visual Basic The crew sets their phasers on stun and shoot each other over and over and over until you get so bored you switch channels to The Dukes of Hazzard. Verdict: I still have the hots for Daisy.
13. ASP In episode 7, Spock struggles to maintain control of his emotions, at times sobbing uncontrollably. This was caused by ASP. Verdict: Seriously, it makes Mr. Spock cry.

Sulu and Uhura look crazy sexy after a hot bout of ASP coding. Mostly crazy though.

As many of you know, I do hosting for web, mail and various other internet services. For the last three years or so, I've leased a cabinet from Infinity Internet in a colocation facility in downtown Portland. I chose them because they offered by far the best rates for the services I required.

Anyway, a few months ago I noticed my bandwidth had become pretty drastically reduced (a download from kernel.org that previously came down at 2-3MB/s now comes down at 100-200Kb/s). They have been expanding (doubling the number of cabinets in the colocation) so I wrote it off as growing pains and figured it would shake itself out sooner or later.

Recently, however I became more concerned as I have acquired a customer who is hosting a VOIP server in my cabinet. He's had some jitter and dropouts on his calls so in an attempt to help him isolate where the problem is, I started investigating my bandwidth issues a bit deeper. After talking to a couple techs at Infinity, one of them reveals to me that my port is capped at 5Mbps. This sounds pretty low to me, so I inform him that my contract specifies 2000GB/month transfer. "Oh, that's 5Mbps" he assures me. Well, I figure, what has happened is they've finally tightened down on their bandwidth usage, but this is too low for serious hosting (congestion will be a noticeable issue, especially for the VOIP guy), so I go ahead and call one of the sales people to see what it's going to cost to get myself up around 10Mbps. Turns out my bill will just around double. The salesman informs me that I've been getting a "sweet deal" that they don't offer any more and in fact he's surprised no one has contacted me to try to renegotiate my contract.

I'm surprised too.

Anyway, I decide I should shop around and compare rates to see if Infinity remains the best deal in town. The problem is that my 2000GB/month of transfer isn't a option most facilities offer, so I turn to a bandwidth calculator to give me a more reasonable comparison. Interestingly, it turns out that 5Mbps is not 2000GB/month. 6.1Mbps is the absolute minimum. Mildly annoyed, I call the tech at Infinity and inform him of my discovery. 6Mbps isn't super hot either, but it's a 16% difference which is significant and I figure it will help until I work out a permanent deal (either with Infinity or somewhere else). The tech informs me there's nothing he can do, but he can refer me to someone who can help. That person turns out to be Infinity's other salesperson. I inform her of my issue, asking only that I have my bandwidth capped at 6Mbps rather than 5Mbps. She tells me she'll check and call me back. About an hour later she calls me back and informs me they won't change it. I'm a bit taken aback, as I didn't think my request was that outrageous. I really had expected her to tell me "no problem, we'll take care of it." Once my initial shock wore off, I ask her if this means they were going to void my contract, so I'd be free to shop around. "No." she informs me. I ask her how she expects to violate my contract and at the same time expect me to uphold my end of it. "Well," she explains, "the contract allows for us to change our rates at any time." I consider this for a moment and then ask her "without notifying me?" "Yes." she asserts.

I hang up and decide I'd better dig up my contract. These sound like pretty untenable claims. I have doubts about whether such a collection of terms would even amount to a contract in Oregon courts (but I'm not a lawyer, etc).

Anyway I find my contract, and low and behold, not only does it explicitly require that they give 60 days written notice prior to any changes to it, it also has attached the service order which states that I'm entitled to 2000GB/Month burstable to 10Mbps. Translated, this means 6Mbps burstable to 10Mbps. What they have been giving me for the last few months is 5Mbps non-burstable.

I'm making attempts to move higher up the management chain at Infinity, but quite frankly, after the shoddy way I've been treated I don't think I'll remain there under any circumstances.

I was reading the feature list for Fedora Core 8 when I discovered that there would be a shift toward an "online desktop". Apparently it's been decided that the future of the desktop is web-centric applications. While I understand the appeal of this paradigm, I think it's dangerous and undesirable for many reasons.

First off, the fundamental principal of Big Board is flawed. The page describing Big Board states the following:

When a user logs into their Desktop for the first time they are greeted with almost a completely empty space. There are a few items hanging around, but we don't give any clues to help people get started.

An interesting perspective exercises is to look at our desktop as if it were a web page. Would you build a web application with the layout of our desktop? Mostly empty space, drop down items for tools to get you started? Probably not.

Why is this flawed? Well first of all comparing a web application to a desktop is just plain silly. Why not compare a desktop application to a web application? Quite frankly, the first thing I do when installing an OS is clean up the desktop. I turn off desktop icons, cut the number of panels to one, and then trim the contents of that panel down to the minimum. The last thing I want is a bunch of social networking site crap cluttering up my desktop (and social networking appears to be the current focus for Big Board and Mugshot). I can understand that Red Hat wants to push Mugshot. And I also know that Microsoft is trying the same tack with "Windows Live". The key difference here is that Microsoft are known to be assholes with only their own best interest in mind. I expect more from Red Hat.

User experience aside, there's a real and serious concern over privacy. This push to put your entire life online is perhaps the most ill-conceived trend in civilized history. Granted things aren't quite like they were in Victorian days when all but the most genteel of personal details were kept to oneself, but I hardly think that putting everything online is a wise choice. Are you absolutely certain you want strangers to be able to Google your wife's menstrual cycle? I'm not. And we don't even use the rhythm method (there, now you know).

There's a disturbing trend in both the willingness of people to volunteer their personal information to web sites and the ability of search engines to harvest that information.

I know that most people think that the stuff they reveal online is harmless and who cares if anyone knows. Of course, any single fact you might consider would probably seem harmless in its own right. But now take two or three or a dozen or a thousand of these seemingly innocuous facts and start combining them into a picture. You were searching for what book? What's your opinion about the war? You voted for who? Next consider how much can be gleaned and inferred about anything given a large enough database (last.fm applies this technique to music with amazing effect). You might think that you are so unique that no realistic picture of you could be gleaned from a heap of factoids. Of course you are dead wrong. Even if you weren't wrong, the problem is that anyone analyzing that data wouldn't care if they were wrong or not: they'd believe they had drawn an accurate enough picture of you and that picture would be who you are to them. And if them is the FBI, the NSA, your bank, a credit agency, your boss or anyone else who might be able to have some real impact on your life, then quite frankly you are screwed (or if you place your hopes in lottery-scale odds, insanely lucky).

Welcome to the world wide web. "Web" appears to be evolving into a far more appropriate term than anyone could have expected.

Here's my suggestion: pretend that every bit of your personal information is likely to indict you for a felony and treat it accordingly. Reject the notion that the convenience of having your information available at any given moment from any internet-connected device is worth the huge sacrifice in privacy you must undertake to obtain it.

You can’t usually do the right thing without inconveniencing yourself. Bruce Byfield reckons free software is a rare opportunity.

Long ago on comp.lang.python, there was an off-topic discussion regarding the use of lethal force, armed citizens and government in general. Alex Martelli explained the choices brilliantly and I want to preserve it for posterity (full context and reference):

"[S]aying that we do NOT want the government to have the monopoly of lethal force is exactly equivalent to saying we do not want effective government (Hobbes would surely argue that way) -- we prefer deliberately-hobbled government to government that is maximally effective. In this day and age it's hard to make a case for deliberately inefficient arrangements, although it IS possible to do so [...]. People who don't want ID cards to exist, don't want government DB's to be cross-linked, etc, plead much the same case -- they prefer inefficient government (whose inefficiencies may help terrorists and other criminals) to efficient government (whose efficiency might allow more effective oppression)."

—Alex Martelli, on comp.lang.python

I'm going to lay off contract programming for a while and take the time to work on projects that I've had in mind for a long time. For my own future reference I'm going to list them here:

1. A PostgreSQL GUI. I want something that's PostgreSQL-specific. It should give me easy control of every feature of PostgreSQL. It should completely replace psql and pgadmin. I want to be able to edit stored procedures with code highlighting. I want to be able to backup/restore/export data. You name it.
2. A hosting panel that doesn't suck or turn your server configuration into a black box you're afraid to touch.
3. Finish Pentropy so it can be released.
4. Invent new, amazing ways of templating for Breve.

I'll add more here as I think of them.

I must be in a particularly anti-Mac-enthusiast mood today. Somehow the web is conspiring to keep me in that mood because I ran across this ancient article and it irked me, particularly this section:

But perception matters. A small but vocal class of very-small-weinered[sic] self-proclaimed PC experts has been attempting to delegitimize the Mac ever since it came out. Surely you know at least one of these guys (and they are all guys):

• In the 1980s, they declared that graphical user interfaces were for sissies, dummies, and artists (and frequently insinuated that the three terms were synonymous). The Mac was overpriced and Apple was going out of business.
• In the early 1990s, they declared that Windows was just as good as a Mac, although experts would never be satisfied with less than a pure command-line interface. Discordantly, they declared the Mac “a toy”, but bemoaned that there weren’t any good games available for it.
• In the late 1990s, after everyone, including them, had switched to desktop-metaphor GUIs, they conveniently dropped their “real men stick with the command line” mantra, and instead proclaimed that Windows wasn’t merely just as good as the Mac, it was better. Or at least that it had better games. And Apple was going out of business.

Now, for reasons of full-disclosure, I'll state upfront that I was one of those people who thought GUIs were for wimps. Despite this, I happen have a huge wiener. Also, I've never thought Windows was better than anything except perhaps herpes.

But on to my response:

The reason GUI's were "for sissies" back in the 80's is because they didn't help you work more efficiently. Was it a problem with the GUI metaphor? Obviously not. The problem was that hardware was slow, memory was limited and quite frankly, very few applications from that era benefited much from a GUI. So the choices were: use a fast, efficient CLI or a slow, less-flexible GUI. Who would choose a GUI in this case? One group would be desktop publishers and graphic artists, but these folks were the minority. The other group was simply people too lazy, too stupid or too unconcerned to learn the most efficient way to accomplish a task.

Now, today, all this has changed. A CLI is still the most efficient for some tasks (i.e. remote server management), but for the most part, the GUI has caught up to the CLI in functionality, and more importantly, the hardware makes the computational inefficiency of a GUI practically unnoticable. Also the fact that all the commodity operating systems can multitask more-or-less successfully means having multiple windows is useful (I remember using Windows 3.1 solely because it allowed me to run multiple DOS windows simutaneously).

So, I'd argue it's not that "the GUI people were right all along and now the CLI people are tasting crow" so much as the way we compute has changed drastically and what was once a dumb idea now makes sense.

No doubt Steve Jobs is a visionary. But by definition, visionaries are ahead of their time and following them can make you look downright stupid.

Incidentally, remote server management is in this same boat today: stupid, lazy admins use remote desktops to manage their servers. Smart guys use SSH. Will that change? Probably, when 100Mb internet connections are commonplace and all the CLI tools have GUI equivalents. Will that make the stupid, lazy admins appear smarter or more efficient in retrospect? Only to them.

1. Tickless kernel. My processor in my laptop used to get up to 95c when watching movies or doing anything remotely CPU-intensive. Since upgrading to FC7 which includes the tickless kernel and powertop, it rarely gets over 60c. My old battery died, so I can't really compare battery life, but I can't see how it wouldn't improve.
2. Satchmo. A Django project that lets you setup an online store in a matter of hours (well, hours if you include styling the site, creating products and taking a shower and a nap). It's still alpha but already impressive.
3. ExtJS. IMO the best Javascript library out there, bar none.
4. YUI Grids. Making CSS-based layout as easy and predictable as table-based layout, without all the cruft and maintenance problems tables bring.

I'm a bit tired of hearing Mac fans complain about Apple's ideas being "ripped off" by others (whether it's Vista, Linux or whatever).

First of all, most of OSX is derived from the actual work of others (Mach, BSD). Apple did a fair amount of work in the GUI but the core of the system consists of millions of lines of code written outside of Apple. This is all fine. What I find aggravating is when Mac fans bitch about people "stealing" Apple's ideas. First of all, let's be clear: they are just ideas. Since Apple doesn't open source code, anyone who "copies" them must do so from scratch (contrast this with Apple using actual code from other people).

Secondly, many, if not most, of the ideas that Apple has "invented" actually have prior art that predates Apple's particular implementation by several years.

Let's cover a few of the currently popular ones Mac users seem certain were ripped off from Apple:

1. The GUI (including all the key elements: mouse, window, menu, icon, etc). Invented by Xerox PARC and shown to Steve Jobs who immediately wanted to bring it to market.
2. The taskbar. It's part of the CDE spec. If you don't know what CDE is, it's because you're too young. Even KDE had a taskbar long before Mac. Apple's main contribution in this regard was to make it more annoying with flashy animations.
3. The desktop cube. I used 3Ddesktop back around 2002 on Linux. Pretty cool but used too many resources back then. I'm sure there's been several implementations and variations prior to that. It's not an original idea.
4. The iPod. Sorry, MP3 players have been in existence almost as long as the MP3. Apple's big contribution was to take another technology invented by someone else (IBM) called the microdrive and apply it in a patently obvious way. Oh, and they got to market first with this incremental improvement. I also seem to recall a product called a "Walkman" which used different technology but embodied many of the same ideas.
5. Podcasting. We used to call it "streaming" in the old, less-branded days.
6. White. Apple's marketing has made white the new black. Let's not forget that other companies (Gateway) had white (not beige) PC's and laptops back in the early 90's. Back then we called them "ugly". I'll grant that Apple does make slick hardware and if I packed a Powerbook around all day I'd probably have hairy palms too.
7. Arrogant, condescending users. Sorry, Unix had those years before Jobs and Woz had stopped doing laundry at their mom's house.
8. Marketers who claim their product somehow makes you sexy, young and hip. Apple's only real innovation here is having customers who actually believe it.

I'm not claiming Apple hasn't contributed to the modern PC at all. In fact, they have made huge contributions (much like Microsoft has) to bringing inexpensive hardware and software to the masses. They've also made improvements to existing ideas (like most of the ones I've outlined above). We'd be far worse off without them. I'm just tired of hearing Mac fans pretend that the contributions are a one-way street. Apple borrows as much from the rest of the world as the world does from them. That's a good thing.

Of course it would be nice if they gave back a little more than ideas (i.e. real source code), but ideas are valuable too. Just stop acting like they're somehow worth more than actual man-hours of programming.

[Update]

9. Safari/WebCore: I plain forgot this one. Safari is basically a port of Konqueror to OSX. It also serves as a fine example of Apple's interaction with the FOSS communities it rapes for source code: http://www.kdedevelopers.org/node/1001

Today I stumbed upon this. It's been badly neglected since we hammered out the csv module in the Python standard library. I've meant to port it to use the "new" csv module for a long while now, and also I'd like to build a web interface to it

Sigh. One more thing to do...

Web Health Warning Put All Destructive Actions Behind a POST Request

—From Agile Web Development with Rails, Dave Thomas, David Heinemeier Hansson, et. al

Later, we discover that David H and his 37signals cohorts fail to follow this sage imperative but want to blame the world rather than admitting that DHH's main contribution to Agile Web Development was his name.

Google's Accelerator in need of a recall

Google Web Accelerator: Hey, not so fast - an alert for web app designers

Maybe this should be required reading for everyone at 37signals.

Over at about:cmlenz, Chris discusses his thoughts on my previous article wherein I compared Genshi (and other XMLish template engines) to PHP. To be fair, the comparison was a bit tongue-in-cheek since most of my complaints about PHP relate to its use as a programming language, not as a templating engine, but Chris makes a key point that I think is worth addressing:

At least with Genshi and Kid, that “ugly thing to describe web pages with” is pretty much just XHTML. And that, of course, is a feature. For a web developer, reading and writing HTML (as well as CSS and Javascript) is at the core of any front-end work. I don't want a template system that tries to hide the HTML from me; rather, it should enable me to just take some HTML and hook it up to server-side application code.

This seems like a reasonable enough position. However this brings me back to a simple point: a document is independent of the particular syntax used to describe it. Whether XHTML, Latex, PDF, Postscript or whatever, it's the document that matters, not the particular syntax. XHTML is simply one more in a long line of machine formats for describing a document. And this is key. XHTML is not the document. It's a description of a document. Further, XHTML is not special. The only thing that sets XHTML apart from, say a binary Word document is the fact that it's reasonably parsable in its raw form by humans. This hardly qualifies it as an optimal representation for hand manipulation. The most compelling reason for using XHTML today is that everyone else does it (and in some cases, this is in fact a valid argument).

The argument that "I don't want a template engine that tries to hide the XHTML from me" reminds me of back when people argued against high level programming languages in favor of assembly language because HLL's moved them "too far from the machine". For most programmers, the machine doesn't matter. It's the end result that they care about. I feel much the same way about document-generation on the web. The fact that XHTML happens to be the language that browsers currently speak means little to me aside from the fact that I need to somehow feed it to them in order to retrieve my desired document. The plethora of tools for assisting with this suggests that I'm not alone in this desire. Some people actually use tools such as Dreamweaver exclusively for this task and never bother to learn the ins and outs of XHTML. Of course, as a programmer, this option isn't really available to me since I do need to integrate with a backend application (and seeing the output of some of this software discourages using it seriously anyway), but that doesn't mean I have to do it the absolute hardest way possible simply because the easiest way doesn't yet exist.

Also, to be accurate, Breve doesn't "hide" the XHTML, it simply gives you a shorter, cleaner way of writing it. What's the difference between having a vi macro or autocomplete in your editor or having what amounts to the same thing in your template engine? Breve is a shortcut for writing XHTML just as much as having a macro in your editor is.

Talk to anyone who does much XHTML editing and you'll undoubtedly get a laundry list of tools and macros they've written or collected to ease the pain of writing markup. Ask yourself this: if it weren't reasonable to write Python code without extensive help from editing macros and validators, would you still be inclined to use it? Pylons and Rails both include a bunch of "web helpers" to help reduce the need to write markup. My question is, why not just follow this to its logical conclusion and make your entire template engine a "web helper"?

My point in comparing XML-oriented template engines with PHP was not about how PHP is abused, but rather how it mixes two unrelated syntaxes together in a eye-straining mess. XML-based template engines are no different. XHTML is ugly all on its own. There is absolutely no way to make it prettier by stirring in another syntax.

Had some time to hack on Pentropy last night and managed to get RSS 2.0 working (remarkably easy using xsd2breve), one-off pages (i.e. the "about" link) and the beginnings of an admin interface.

To give a quick example of how easy it was to create an RSS feed in Breve, here's the steps I followed:

First I needed to obtain an xsd file for RSS (this took some Googling as apparently there isn't an actual formal standard). I ended up using the one found here.

The next step is to create a test RSS feed. To this end I copied an example off some random site and edited it by hand:

rss (version="2.0") [
    channel [
        title [ 'Google Jobs' ],
        link [ 'http://www.google.com/support/jobs/' ],
        description [ 'Information about job openings at Google Inc.' ],
        item [
            title [ 'HR Analyst - Mountain View' ],
            link [ 'http://www.google.com/support/jobs/bin/topic.py?dep_id=1077&loc_id=1116 ],
            description [ '''
                We have an immediate need for an experienced
                analytical HR professional.
                The ideal candidate has a proven record of developing
                analytical frameworks to make fact-based decisions.
            ''' ]
        ]
    ]
]

I saved this template in templates/feeds/rss20.b. This gives us a bit of test data to verify our custom tags and template are working.

Next we need to create our custom tags and put them in a place where Breve can find them:

$ mkdir custom_tags
$ cd custom_tags
$ xsd2breve http://www.westinkriebel.com/Public/RSS20.xsd > rss20.py

Now we need to add our custom_tags directory to our Python path and tell Breve to actually use them:

# a Pylons controller
import sys; sys.path.append ( 'pentropy/custom_tags' )

class FeedController ( BaseController ):
    def rss20 ( self ):
        return render_response ( 'feeds/rss20?format=rss20' )

Now I can visit '/feeds/rss20' and see the result (a fantastic RSS feed).

The argument format=rss20 tells Breve to import tags from a module with that name.

Next is to put some actual data into the feed. Modify the controller to return the actual data:

def rss20 ( self ):
     c.posts = Post.select (
         order_by = [ desc ( Post.c.post_date ) ],
         limit = 50
     )
     return render_response ( 'feeds/rss20?format=rss20' )

and in our template:

rss ( version="2.0" ) [
    channel [
        title [ 'Pentropy' ],
        link [ 'http://pentropy.twisty-industries.com/' ],
        description [ 'Random Stuff' ],
        [ item [
              title [ _p.title ],
              link [ 'http://pentropy.twisty-industries.com/%s' % _p.slug ],
              description [ cdata ( _p.html_content ) ]
        ] for _p in c.posts ]
    ]
]

Now clearly, at some point I'll want to pull some of that data from the database or a config file, but for now it actually works.

My initial tests with Akismet were not promising. I submitted four comments, all four came back flagged "spam". There was nothing remotely spammish about any of them. I've been told that false positives are a serious issue with Akismet, so I'll move away from that (also the bandwidth required for each post could eventually become an issue). I considered using the Akismet results to decide whether to put comments in a moderation queue, but at that point you may as well simply put them all there and hand-sort them.

Anyway, I found another solution using a Bayesian filter (based on Divmod's Reverend). It requires some training but that's okay.

I utilized Michael Foord's Python library to integrate Akismet into Pentropy. I'm keeping the "are you human" simple addition test in place for now, since that appears to work nicely against bots (although I need to make it less machine-parsable eventually).

The tag cloud on the right side is now a plugin. I utilized the new xinclude feature of Breve to create a component-based page.

The main blog controller:

class PostController ( BaseController ):

    # ...

    @jsonify
    api_tags ( self ):
        tag_index = Tag.select ( order_by = [ asc ( Tag.c._name ) ] )
        tags = [ ( t.name,
                   ( h.url_for ( controller='post', action='by_tag', id=t.id ),
                     len ( t.posts ) ) )
                 for t in tag_index ]
        return dict ( tags )

the plugin controller:

class TagcloudController ( BaseController ):
    def cloud ( self ):
        body = urlopen ( 'http://pentropy.twisty-industries.com/post/api_tags' ).read ( )
        c.tags = simplejson.loads ( body ).items ( )
        c.max = float ( max ( zip ( *zip ( *c.tags )[ 1 ] )[ 1 ] ) )
        return render_response ( 'tagcloud/cloud?fragment=1' )

the plugin's template:

div [
    ol ( class_ = 'tag-list' ) [ [
        li [
            span ( class_ = 'tag-context' ) [ '%d posts are tagged %s' % ( _posts, _tag ) ],

            a ( href = _link, class_ = 'tag',
                style = 'font-size: %0.2fem;' % ( 0.8 + _posts / c.max ) +
                        'color: rgb(%d,100,120);' % ( _posts / c.max * 180 ) +
                        'padding: %0.2fem;' % ( c.max / ( _posts or 1 ) / 10 ) )
            [ _tag ]

        ] for _tag, ( _link, _posts ) in c.tags
    ] ]
]

and finally, the index template looks like this:

# index.b
html [
    body [
        ...
        xinclude ( 'http://pentropy.twisty-industries.com/tagcloud/cloud' )
        ...
    ]
]

The basic sequence of events goes like this: a request is made for index.b (or rather a child template that inherits index, but that's not relevant here), index.b requests the plugin via the xinclude directive. The controller at the url passed to xinclude accesses the published API available in the main controller to get the JSON data describing the available tags (a list of ( tagname, ( link, article_count ) ) ). It then renders its own template (cloud.b) and returns the XHTML output back to the main template which injects it into its own final output.

It seems like a lot of steps for something that could have been (and originally was) simply an included template. However what it provides in return is the ability to define completely encapsulated plugins that have no knowledge of the internal workings of the main application. All they require is a defined JSON (or XMLRPC) interface to retrieve their data from.

The next step is to define a caching mechanism to reduce the number of times this complete sequence must be traversed. In the case of the tag cloud, it's clear we can cache until a new post is made (tags are never created independently of posts in Pentropy). To this end I'll probably add a generic cache directive that will be utilized something like this:

html [
    body [
        cache ( expires = c.tags_changed ) [
            h1 [ 'Tags' ],
            xinclude ( 'http://pentropy.twisty-industries.com/tagcloud/cloud' )
        ]
    ]
]

Having a generic cache directive will allow for static HTML fragments to be stored based upon specific criteria. More on this later.

I've had this idea for a while (it's been on the Breve to-do page as a speculative feature), but I also saw that Genshi supports such a feature and that there's a new template engine "Art" (not released yet) that will also have such a feature.

At first I thought of it as a cool, but not really important feature. I mean, you could do things like embed your last.fm playlist easily but other than that it seemed like fluff, so it was low on my priority list.

However, on the Pylons list, the author of Art mentioned how he planned to leverage this concept to easily support plugins. Duh! It's brilliantly simple (just how I like it). I was thinking in terms of pulling in XML data from other sites, but actually it could be used to pull a fragment from another controller on the same server. This would allow plugins to be nicely encapsulated and easily allow a page to be built from fragments generated by multiple controllers (something not currently possible with Pylons that I'm aware of), all from within the template.

I quickly hacked up an xinclude tag in Breve (mostly a thin wrapper around urllib2.urlopen() and it worked fine for pulling from remote sites. However, when I pointed it to another controller on the same Pylons app, things fell apart. The rendered template was incomplete. Further testing revealed that it only broke if the controller rendered another template (i.e. returning a simple string worked fine). My initial guess (without further testing) is that somehow the subrequest is happening withing the same context as the outer request or the template subsystem has an ugly global somewhere (i.e. the subrequest tries to reuse the existing Template instance). There's also the possibility that it's a bug in Breve, but this seems unlikely at this point (Breve is pretty simple).

My next phase is to narrow down where it's happening with the following tests:

1. Have a Pylons controller directly call another Pylons controller using urlopen and see what we get. Note that it must utilize the template subsystem since simple strings already appear to work.
2. If this works without issue, then work up a simple Breve app (outside of Pylons) and try to replicate the issue there.
3. If that works, then file a bug report on Pylons since I suspect it will be outside my abilities to track down.

Anyway, this feature seems so killer as to be well-worth the effort. I was casting about for ways to support plugins for this blog and this is clearly it.

Follow up:

Lo and behold, the bug was in Breve, and it was a pretty serious one. breve.Template was using register_flattener on a method named __slot which I had expected to not collide with other Template instances but it didn't work. What did work was making it an internal class, but what worked even better was avoiding register_flattener altogether and simply adding a __str__ method to the slot class. 1.0.32 is now the recommended release as it addresses this bug, a memory-leak issue (related to the first bug) and provides an initial (read: testing only) implementation of xinclude (although I think I'll rename it at some point).

Okay, maybe it doesn't. But I do. Fed up with Bitakora, I cracked down and started a blog from scratch (well, building on Splee's SimpleBlog tutorial - mostly for the SQLAlchemy tips).

What you see here is the first basic incarnation of it. It's built using Pylons, Breve and SQLAlchemy. Not that it's required, but it also happens to have PostgreSQL living underneath of it.

This is also my second small Pylons application (the first being the Breve site). This also makes it the second live Breve site in existence (that I'm aware of). This is actually much of the reason I'm doing it: I need to learn Pylons and I need to test Breve on real applications (I'm already seeing features that would make life a bit simpler).

Anyway, I've got a lot to do but my basic goals are as follows:

1. Very basic blog core - no Javascript. No web 2.0. Nothing but posts, comments and tags (and a few themes).
2. Plugins (for adding all the stuff explicitly excluded in #1).
3. One-off pages (e.g. "About", "My Projects")
4. Admin interface.
5. Multiuser

Of these, #1 is around 90% finished. #2 is in exploratory testing. #3 will take all of an hour. #4 and #5 are pipe-dreams for now.

Some of the first plugins I have in mind are:

1. Threaded comment system.
2. Gallery

Oh, and as you can see, the blog is named "Pentropy". I'll have a site/trac/svn up for it soonish.

When I first published TurboStan, I admit I was a bit taken aback by the apparent lack of interest from the majority of TurboGears users. Stan's beauty and clarity seemed so obvious to me that it baffled me that people would choose what I perceived as a lesser PHP over something as clearly elegant as Stan. I tallied the possible reasons for TurboStan's lack of fanfare and came up with the following:

1. Not the default engine for TurboGears. While not a good technical reason, there are reasons for sticking close to the mainstream, especially with an unstable target like TurboGears was in the early days.
2. Dependencies on Twisted, Nevow and zope.interfaces. At least one user mentioned that he was unable to find a version of zope.interfaces that would both install and was compatible with the other two dependencies. I figured that if one person mentioned this problem, then there were probably a dozen more who gave up in silence.
3. Lack of documentation. This plagued me in my early days using Nevow and I didn't improve the situation for others any by extending Stan with even more features that were only documented briefly in my blog (and I changed blogs about as often as my socks in those days).
4. Not really seeing the elegance of Stan.

Since the release of Breve, I've seen a bit more interest than I did with TurboStan (I even received a patch from someone, something that TurboStan never did), but still the warm reception isn't quite what I'd like it to be.

Now I admit that when I first decided to try Nevow (way back before TurboGears was a glimmer in Kevin's eye), I took a look at the two options for templating (Nevow's XML engine and Stan) and went with the XML option. Stan looked... odd. I didn't really see it for what it was. Plus I had some odd sense that XML was a better choice because it was more "standard". I'm not sure when the lights came on and I was able to appreciate the elegance of Stan, but at some point it happened and I'm now loathe to touch the angle brackets on my keyboard. I became so attached to Stan that I refused to try Django, Pylons, CleverHarold or any other framework that I couldn't use Stan with. The main reason I used TurboGears was so I could use TurboStan. Now, I'm sure at least part of this love came from the fact that I was playing with my own creation (not Stan, but the extensions I added) and my ability to direct its future. There's always a lot of fun in that and I can't deny that is part of it.

Still, as I watch TurboGears (and a few other frameworks) adopt Genshi as their default engine, I can't help but wonder at the attraction people find to these sorts of engines. I can't help but notice with a bit of smugness that there's a good chunk of helper functions in Pylons (and Rails) to alleviate the need to spell out HTML, which seems to confirm my belief that their default template engines don't really help as much as they might when it comes to HTML generation (an odd weakness for a web framework to have). When I was first starting on the web (and Python web frameworks were pretty ugly), I was forced to program in PHP/Smarty at my job. Once I got over the novelty of programming amongst the web's various vagaries and oddnesses, I bemoaned these tools and wished for a Python framework that would allow me to escape the ugliness I perceived in them. So what's my point? Well my point is that template engines such as Genshi, Kid, Myghty, et al all smell of PHP to me. Is some of that just cosmetic? You bet. In fact, probably most of it is just cosmetic. There's no doubt in my mind that Genshi is a powerful too and probably full of beautiful Python code under the hood. That doesn't change how it looks to me from the outside. It isn't Python. I find it mildly ironic that Pythonistas are renowned for their loathing of Perl's typographical perversions and then turn around and create equally ugly things to describe web pages with.

Anyway, this isn't so much about how much I dislike those types of engines as it is curiosity as to what reasoning (whether it be founded in technical or taste) other Python programmers have for selecting their preferred template engine. Going along with that, I'm curious why DSL's like Stan, Breve, XIST and others don't generate more enthusiasm amongst Pythonistas.

So if you've got an opinion on this, I'd be interested in hearing it. I'm not looking for an argument, just enlightenment (right now I feel like I'm the enlightened one looking out on the unwashed masses, but there's always the remote possibility I just don't get it, and if that's the case, someone should be kind enough to show me the light) . Also I'm interested to see if there is an actual perceived shortcoming in Breve so that I can perhaps address it.

My work on Breve has revealed more dark corners of the web than I thought I ever wanted to know.

Breve is, at its heart, an XML-generation engine. The fact that it can output HTML and that doing so is its primary purpose is almost incidental, really. What this means is that Breve happily outputs things like <div /> which is logically sound, but technically incorrect.

Now, in a previous, happier life I was blissfully unaware that not all elements can be self-closing in XHTML, especially since such tags will even validate in the W3C's own validator. I ran across the issue when Breve output something similar to the following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
           Strict//EN" "DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
  </head>
  <body>
    <div />
    <div>test</div>
    <div />
  </body>
</html>

Now on the surface, this looks fine. However, this is how Firefox rendered it:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
           Strict//EN"  "DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
  </head>
  <body>
    <div>
      <div>test</div>
      <div></div>
    </div>
  </body>
</html>

I spent about an hour trying to figure out what I was doing wrong, changing doctypes, adding meta tags, etc. As it turns out, I was missing a key element: the HTTP header "text/xml". Now, I didn't think I was missing it, since I'd tried setting that via a meta tag to no avail. Luckily someone on #firefox pointed me to this bug report.

The short story is that there is no practical way to specify that an XML document is, in fact, XML from within the document itself. You must configure the web server to output the proper headers.

The longer story is that I modified Breve to only output self-closing tags for a small subset of XHTML and all is well (except for my faith in people who write RFC's).

Fixed a major issue with how Breve dealt with template paths. I'm not 100% happy with the solution, but it's an issue that the frameworks (Pylons and TurboGears, in this case) tell the template engine nothing about the root path to the templates.

This is an issue for Breve because all Breve paths are relative to a static root path. I've found that this makes it far easier to have fragments in subdirectories that can still inherit from templates in higher-level directories that might then include templates from subdirectories. I tried using things like "inherits ( '../index.stan' )" with TurboStan and it turned into an unmanageable mess real quick. However, TurboStan was only really meant to support TurboGears so I cheated and called out the the TG config file to get a root directory. Supporting both Pylons and TurboGears makes that solution unusable.

Anyway, like it or not, it's been tested with Pylons 0.94 and seems to work. Note that you must specify:

breve_opts = { 'root': 'myproject.templates' }

in your config/middleware.py file.

So it's finally usable. Breve 1.0 is up and out there. Kick the tires (but not too hard) and let me know what you think.

http://breve.twisty-industries.com

So I've settled on a name for my new template engine: Breve. Formerly known as espresso with half-and-half, Breve is now a Stan-like template engine written in pure Python with no external dependencies.

I've also written a Buffet plugin for Breve and it's been tested with Pylons and TurboGears. TurboGears testing was pretty limited (tested that a basic template rendered), but I sat down tonight and reimplemented the Pylons QuickWiki tutorial completely in Breve without issue.

I have to say that Pylons seems pretty straightforward. It's a little more upfront work than TurboGears (mostly due to Routes), but after that it's quite similar in its learning curve. One thing I like better right away is the error messages. Sometimes TurboGears exceptions can be downright obscure (and worse, occasionally being raised far away from the original exception), whereas Pylons gives almost too much information. I know which case I'd rather deal with.

Anyway, I did the Pylons test before the TurboGears test for one reason: the TurboGears site was down all day on the day I wanted to test the Buffet adapter. That meant I couldn't install TurboGears on the laptop I'm currently using. That meant no testing. That meant at least mild aggravation that TurboGears is still failing to recognize its own importance to its community. So far TurboGears has changed hosting services at least twice and still doesn't have any infrastructure to ensure availability (mirrors, anyone?). I've mentioned this to Kevin Dangoor before and I get the feeling he simply doesn't have the time to deal with it. Regardless, I'm starting to feel that simple availability might become the achilles heel of TurboGears. It depends on a couple dozen sub-projects, at least a few of which I've seen become unavailable for download at various times. The sheer number of subprojects that comprise TurboGears makes the likelihood of one of them being unavailable for install far too high. If TurboGears is going to depend on third-party packages, then it needs to do something to ensure not only its own availability, but the availability of all the sub-packages.

Anyway, I'm currently in the middle of a rather largish TurboGears application, so I won't be switching just yet, but I see Pylons in my future.

I've been meaning to tackle some of the problems inherent in TurboStan for a while. One of the biggest issues is the dependencies on Twisted, Nevow and zope.interfaces. That's a pretty heavy dependency tree for a template engine to have.

I started digging in to the code for nevow.stan to see if I could extract the core of Stan from the jungle, but finally changed my mind and decided on a clean rewrite. Stan has lots of features I don't need and lacks features I want to add. Since the basic concept behind Stan is simple to implement it seemed more reasonable to start from scratch.

I had the basic syntax and flatteners working within a couple hours and moved from there to adding features (inheritance, includes, loadable tag definitions, etc). It took a couple days of hacking but I now have the core of a new template engine. I'm quite pleased as the entire code base is not much larger than the original TurboStan plugin (which was mostly just glue code tying Stan and TurboGears together).

I've been using the working name "sexml" (s-expression markup language) but I've also considered "stanly" (i.e. stan-like) and "sexpress" (s-expressions). I guess it's true men think of sex every 10 seconds.

Anyway, I've got to write some documentation and a Buffet adapter (and test it as a TurboStan replacement on a live site), but expect to see a release before the first of the year (i.e. this week).

Oh, and the only external dependency is the os module from the standard Python library.

Today I received the following email:

''' Hello develix.com,

I'm emailing you today to request a link exchange between our website and yours. I found your website http://www.develix.com by searching Google for "Genital Warts". I think our websites have a similar theme to your's, so I am interested in exchanging links.I was wondering if you would like to trade links with my Genital Warts Website.

If interested please send me the Title, URL and Description of your website or the HTML code, so that I can put them on my website of same category and in return you will provide link from Your website . '''

I'm avoiding the temptation to find out if develix.com really does turn up in those search results.

Despite my issues with Lua, I managed to hack together a little Lua script for automatically filtering the multitude of mailing lists I have.

I was inspired by David Reid's script but needed a bit more since his script only deals with the List-Id header, which I found inadequate in dealing with the tons of mailing lists I subscribe to.

By automatic filtering, I mean I don't create a rule for each list, rather the script inspects the email headers and uses bits of information there to create folders and move the mail there without any intervention.

This was more painful than it needed to be since not all mailing lists carry the same headers, and even headers that might be useful (List-Id) often aren't, due to the vagaries in various mailing list software and how people configure that software.

Anyway, the script is below. Enjoy or don't. If you improve it or find bugs, please let me know here.

config.lua:

imap_acct = {
    server = 'mail.yourserver.com',
    username = 'username',
    password = 'password',
}

function parseListId ( header )
    header = header:lower ( )
    header, n = header:gsub ( '[<>]', ' ' )
    baseheader, n = header:gsub('.*x[-]beenthere:%s*([^%s\r\n]*).*', '%1')
    if n == 0 then
        baseheader, n = header:gsub('.*list[-]post:%s*([^%s\r\n;]*).*', '%1')
    end
    if n == 0 then
        baseheader, n = header:gsub('.*list[-]id:%s*([^%s\r\n]*).*', '%1')
    end
    baseheader, _ = baseheader:gsub ( 'mailto:(.*)', '%1' )

    if n > 0 then
        parts = baseheader:find ( '@' )
        if not parts then
            parts = baseheader:find ( '[.]', 1 )
        end
        parent = baseheader:sub ( parts + 1, nil )
        parent, n = parent:gsub ( '(.*[.])(.*)([.].*)', '%2' )
        if n == 0 then
            parent = parent:gsub ( '(.*)([.].*)', '%1' )
        end
        child = baseheader:sub ( 1, parts -1 )

        -- now, if the parent's name exists in the child, delete that from the child
        child, n = child:gsub ( parent .. '[-]', '' )
    end
    return parent, child
end

mailinglist = {
   invert = true,
   'header "X-BeenThere" ""',
   'header "List-Post" ""',
   'header "List-Id" ""',
}

results = match ( imap_acct, 'INBOX', mailinglist )
listids = fetchfields ( imap_acct, 'INBOX',
                        { 'X-BeenThere', 'List-Post', 'List-Id' }, results ) or { }
mailboxes = { }
for message, header in pairs ( listids ) do
    parent, child = parseListId ( header )
    if parent and child then
        mailbox = parent .. '.' .. child
        if not mailboxes [ mailbox ] then
            mailboxes [ mailbox ] = { }
        end
        table.insert ( mailboxes [ mailbox ], message )
    end
end

for mailbox, messages in pairs ( mailboxes ) do
    -- print ( mailbox )
    move ( imap_acct, 'INBOX', imap_acct, 'Lists.' .. mailbox, messages )
end

I've seen a lot of people using Lua these days as an embedded language. I've been meaning to pick up on it as it looks like a nice, simple extension language.

Well that day just came. I discovered imapfilter, a (would-be) wonderful tool for filtering mail on remote IMAP servers, something I need since the mail filtering in Evolution broken-by-design (another story altogether) and I can't stand any other MUA's I've tried.

Anyway, imapfilter uses Lua as the language for writing filters. Seems like a good plan so I dove in. The first thing I discovered is that the Lua documentation leaves something to be desired. If there were just a bit more documentation, I'd call it bad documentation, unfortunately the sample given on the website isn't enough to draw any real conclusions. The second thing I discovered is that regular expressions in Lua aren't very regular. Instead they are sort of like regular expressions, only far less powerful, badly broken and poorly documented (even worse than the main Lua docs).

Doubt me? Okay, explain this:

Lua 5.1.1  Copyright (C) 1994-2006 Lua.org, PUC-Rio
> header = 'X-BeenThere: python-list@python.org'
> print(header:gsub('X-BeenThere:(.*)', '%1'))
X-python-list@python.org       1
>

Eh? Where does that leading "X- " come from? Hard to say. Maybe it's supposed to be there (does "-" mean something? It's hard to know given the ridiculously thin documentation on the matter). Given that pcre is widely adopted amongst all the better scripting languages and most of the bad ones, (not to mention half the non-trivial software on any Linux box) I fail to see why handling regular expressions should be an issue for any language these days. Frankly, in the time I wasted putzing around with the broken string-thing (I refuse to call it a regex parser) in Lua I could have rewritten imapfilter from scratch in Python.

Google, doing no evil.

So I have to wonder how long it is before Google starts using its search capabilities to harvest potential DMCA violations from Gmail and Blogger.

I wonder what other laws Google might decide to help enforce next? Threatening emails sent via gmail? Turning in employees who are slandering their employers?

I suppose it's only a matter of time until we see Google Informant Beta. Of course, in the interests of helping, it'll be free to turn in your neighbors.

I wonder if they get AdSense in prison.

You awaken at 3:50am from a dream where a town of zombies is chasing you, trying to vomit some sort of zombie blood goo on you so that you'll become a zombie too and instead of having a glass of warm milk, hugging your pillow and sobbing, or any other sort of normal activity, you comfort yourself by writing this query:

UPDATE dips
SET revenue = rst.revenue
FROM (
  SELECT d.id, dip_count * r.rate AS revenue
  FROM
    (SELECT d1.*, rate_code
     FROM dips d1 LEFT JOIN accounts
     ON (d1.accounts_id = accounts.id)) AS d
  LEFT JOIN
    rates AS r
  ON (d.carriers_id = r.carriers_id)
  WHERE d.rate_code = r.rate_code
    AND d.revenue IS NULL
    AND d.carriers_id IS NOT NULL
) AS rst
WHERE dips.id = rst.id;

I may indeed be eaten by zombies someday, but thank God I'll never again use MySQL.

If you do web development and Linux is your primary environment, you know how annoying it is to keep a dedicated Windows box around just for testing Internet Explorer. VMware is a good solution but kind of overkill for one app. Here's the easiest way yet:

ies4linux

Enjoy.

I outlined many issues I foresee with Portland's upcoming "free wifi" initiative that will likely cause it to fail, but the picture is even bleaker if it succeeds.

Consider this: the target speeds for the free Wifi are 1Mb/s down, 256Kb/s up. These speeds barely qualify as broadband. Further, these are undoubtedly "optimal" speeds, i.e. the speeds you'll never see in practice. Now consider that the majority of people will probably be willing to settle for these speeds in exchange for the low cost (i.e. hidden in your taxes) and expected ubiquity. What will this mean for existing broadband providers? Well, most likely it will mean drastically reduced customer bases and higher rates for those who remain with them. That means if you don't want the sorry-ass bandwidth the city will provide, you'd better be prepared to shell out.

This is where it gets even more interesting: why aren't we hearing screams of protest from Verizon and Qwest? This service seems to appeal directly to their customer base (DSL users) which would normally have them lobbying and buying off politicians by now. So why are they so silent? Is it because they expect that the free Wifi won't be fast enough to support technologies like VOIP and that VOIP is more of a threat to their revenue stream than the lost revenue from DSL customers? Perhaps. There's really no way of knowing at this point, but the telecom's silence on the matter is certainly curious.

Anyway, it seems pretty obvious that no matter what the outcome, free Wifi is going to have a negative impact on the deployment of true broadband (> 5Mb/s) in our area and that's good for no one.

Portland has just approved a plan to provide free wifi for "95%" of the city. This plan appears to be fundamentally flawed on both a conceptual level and likely to fail on a technical level.

First off, the idea of free internet for everyone seems like a great idea. People who can't afford cable or DSL will suddenly have decent, if not outstanding speeds (the current target is 1Mb/s up, 256Kb/s down). Of course this seems to disregard the fact that people who can't afford cable or DSL probably don't have PC's at all or have outdated PC's that don't have or support Wifi. Think Pentium desktop with Windows 95 (or worse). It ain't gonna happen for those people. What this means is that the most likely bunch of people who will "benefit" from free Wifi are the people who already have connections that are better than what the city plans to offer. I certainly wouldn't sacrifice my $60/mo 30Mbit/768Kb Comcast connection for this. Perhaps a large segment of DSL subscribers would find this a better deal, but I doubt any cable customers will. Anyway, the main point is that basically the entire city is going to subsidize internet for the people who most likely already have it (yes, that means people who can't afford internet won't get it but will help pay for the people who already have it to get it for free - good deal).

There is also the issue of security. Most people already fail miserably to understand how or why hackers target systems. Wifi adds an even greater risk to the casual user's security. How will users know that they are connected to the city's Wifi and not the access point of a malicious person? Probably they will have no way of knowing or even if they did they probably wouldn't notice anyway. This is a potential problem already, but it's at least expected that a user who's looking for an access point has some idea what SSID they are looking for. When Wifi is available on every street corner (undoubtedly with a different, meaningless SSID at each point) users will see no rhyme nor reason to the scheme and it would take a malicious person about 20 minutes to setup camp with their own access point and start phishing people using forged DNS records and sniffing packets. I can sense a good living for the semi-educated criminal element.

Finally, I have serious doubts about whether it's even going to work or be financially feasible. Unless the city is going to use some new segment of bandwidth that doesn't interfere with consumer products then most likely what we'll see is the inability of people who already have superior internet connections and access points at home to use their own equipment due to interference. Should I shroud my house in tinfoil so I can get a good connection on my laptop? If the city does use a different segment of the spectrum, then will everyone be required to buy special hardware to access it, thus raising the entry bar even higher?

Another thing that irks me is where much of the funding is coming from: Portland Public Schools and Metro. That's right: the schools that had to beg for iTax lest they have to close early are helping provide Wifi to the wealthy folks in the Pearl District. I wrote an email to some blockhead on the commission "studying" this proposal some months ago regarding this seeming anomaly and was informed that the "free" (read "paid for by everyone") Wifi would allow schools to get rid of their costly T1's. Uh yeah. Anyone who thinks that Wifi is a good substitute for a T1 (assuming you needed the T1 in the first place) is clearly unqualified to discuss the matter. Further, if reliability truly isn't an issue then there are still far less expensive, faster and more reliable alternatives available. Has anyone approached Verizon to see if FIOS could be provided to the schools? What about Comcast? I'm sure they could do it for far less than the projected $10M (HA!) the Wifi project will cost and provide superior service as well. Further, if the service came over fiber optic rather than Wifi, new and faster technologies will be easily adopted, whereas with the Wifi we'll most likely be stuck with whatever technology is cheapest at the time of deployment for the next decade. That means that by the time the project is complete it will already feel like dialup for most users.

Overall this whole thing smells fishy. The amount of publicity has been pretty minimal given the project's scope and potential impact (good or bad) on residents of Portland, and even when publicity is given, the details seem to get glossed over. I've heard next to zero debate on the topic and I suspect this was decided behind closed doors long ago. It wasn't a matter of if it was going to happen, it was simply who was going to be awarded the contract. Frankly I suspect Portland's strong desire to seem "progressive" and "innovative" has made them even more short-sighted than normal.

Hopefully we don't have to move in order to retain a decent internet connection.

For all those (myself included) who thought that PostgreSQL wasn't as fast as MySQL for most stuff, Jonathan Ellis points to some recent benchmarks that show PostgreSQL to pretty much stomp MySQL all around, especially on SMP/multi-core processors.

If you don't bother reading the articles, simply note this single item:

We see, for instance, that adding a second Woodcrest in MySQL 5.0.20a - costing a good 851 dollars - only yields a 6% performance increase.

One of the comments in Jonathan's blog points to this thread: Single PostgreSQL server faster than MySQL cluster.

Personally I switched from MySQL some years ago for the features, finding performance to be adequate if not stellar. After recent experiences with both however I'd say that PostgreSQL now wins on all the following:

1. ease of installation, configuration, tuning and maintenance
2. features and extensibility
3. performance
4. stability

If you're a programmer, you'll find PostgreSQL to be the most amazing database around, bar none. The ability to program server-side procedures in a plethora of languages (including Python, Ruby, Perl, Tcl and more), terrific client libaries and API's for dozens of languages, and the MVCC approach (vs locking) makes development against PostgreSQL an absolute pleasure.

I chose PostgreSQL for these features in spite of performance doubts. Now that PostgreSQL clearly wins in the performance category as well, choosing it over MySQL is a no-brainer.

Who's a rat?

Javascript lacks string interpolation of any sort.

Python:

s = "%0.2f" % floatN

PHP:

$s = sprintf ( "%0.2f", $floatN );

C:

sprintf ( s, "%0.2f", floatN );

Javascript:

Go find a library.

How the hell did this language make it out of the dumpster?

My friend Josh just got FIOS (4/2Mb) and sent me his download times from a fast Gentoo mirror that's pretty close by (University of Oregon). His times were decent but not astounding:

Length: 20,403,483 (19M), 18,994,870 (18M) remaining [application/x-gzip]
78% [++++++===============================================================>                  ] 16,057,678   628.63K/s    ETA 00:07

Not too bad but not amazing if you're used to Comcast cable. His upstream is really nice, getting around 190KB/s. This is for around $35/month.

Anyway, I decided to test my Comcast cable on the same file from the same mirror:

Length: 20403483 (19M) [application/x-gzip]
Saving to: `emacs-21.4a.tar.gz.2'
92% [====================================>   ] 18,951,374   3.29M/s  eta 0s

Hot damn. I was pretty surprised. I'd seen speeds of around 1.5MB/s coming from my own colocated servers but never this high. Apparently Comcast has increased our speeds again (we noticed a speed jump after we got Comcast digital voice). We pay around $110/month (that's for internet, two digital phone lines and basic cable TV).

Anyway, we've been under a deluge of advertisements from Qwest claiming that Comcast jacks rates (I've had near the same rate from them for several years, even AT&T/@home cost around the same before Comcast bought them). Of course anyone with half a brain knows that Comcast offers an introductory price of $15/month for basic cable and that after six months it goes to the normal price of around $50/month. It really irks me when companies are so blatantly misleading as Qwest is in this case.

My major points of complaint would be:

1. Implying that Qwest DSL is remotely in the same category as Comcast Cable. Judging from my test today I must be getting close to 30Mb/s compared to Qwest's 1.5Mb. This is for around $15 more a month. Even the 5Mbit basic package Comcast offers is far superior to Qwest's offerings.
2. Comcast hasn't changed rates in years. They did cap bandwidth for a while but it was still easily twice as fast as a comparably priced DSL package (and more reliable to boot, I think I've had two outages in 5+ years).
3. Comcast actually wants your business. We run our own business and have struggled with the ups and downs of finances for a while. Comcast has never shut off our service even if we were a couple months late. You can pay in a couple weeks? Not a problem. A friend of mine finished college and was looking for a job and decided to cancel her Comcast account because she was trying to cut expenses. The Comcast rep asked her how long she thought it might be until she might be able to afford service again. My friend told her probably at least three months. They gave her three months of free service to carry her over. The chances of Qwest (or Verizon for that matter) doing that are about nil.

4) Qwest sells your information to telemarkers. When I got Comcast digital voice, two days after getting the new lines (with new numbers) I got a telemarketing call. They knew my name, so I was pretty pissed. I assumed that Comcast had sold my name and number. I pressed the rep until he admitted that Qwest had sold it to them. Apparently even though I buy digital voice from Comcast, that line still has to go through Qwest's network at some point which requires that Comcast notify Qwest of new accounts which Qwest then quickly sells for a couple bucks. This isn't the first time they've done this to me: long ago I had a Qwest land-line and pumping telemarketers who called usually revealed that Qwest had sold them my information. To add insult to injury, Qwest then tries to sell its own customers a service to block telemarketers. Talk about milking both ends of the cow. That's Qwest's "spirit of service". They should just shorten their slogan to "bend over".

I should also mention Verizon at this point. I saw an ad for them last night (on Comcast cable, ironically) taking pretty much the same stance as Qwest: omg, it's cable, run away! What got me with them is they labeled themselves as "a company you can trust". Huh? Is this the Twilight Zone or is this not the same Verizon that just sold all their customer's call histories (and anyone who called someone on their network) to the NSA a few months ago? When they showed all those people following their customers around in their commercials I had no idea how firmly rooted in reality that was. Even if I had I wouldn't have realized those people are all from the FBI and NSA.

Anyway, this whole thing has been rubbing me wrong for some time, so I just had to get it off my chest. Don't buy from Qwest, don't buy from Verizon. If you want to be treated like a person and get your money's worth, buy Comcast.

BTW, here's my full disclosure: I have never worked for any of the aforementioned companies nor do I plan to. So there.

I've been using Bitakora (this blog software) for a while now and I'm fairly happy with it. It doesn't have a lot of features, but I chose it because it seemed to have a promising future.

Well, that future seems to be slipping away. There's been zero updates since I first installed it and the mailing list echoes with the sounds of unanswered emails.

This is pretty sad, because there is a dearth of blogs written in Python. If Bitakora were written in pure Python or perhaps a modern framework like TurboGears, Django, Pylons, etc, then I might be willing to start hacking on it. Unfortunately Bitakora is written in Zope and I have no plans on reading 400 pages of documentation so I can add a captcha to cut down on the spam this blog has started seeing lately.

I've been reading Lee McFadden's SimpleBlog tutorial (mostly to understand SQLAlchemy and how to tie it in to TurboGears) and now I'm thinking about writing my own damn blog based on this tutorial (using TurboStan, of course, rather than Kid or Genshi). Even if it doesn't have every bell and whistle under the sun, at least I'll understand the code and it'll have the bells and whistles I want, which is of course, what's really important.

Anyway, goodbye Bitakora, I barely knew thee.

If you are in the market for a jabber server, the only real advice I can give is to skip Wildfire. After a week you'll find interesting system loads ranging from 40 to over 200. That's with one user on a dual Athlon MP 2800+ box with 3GB of RAM. I can only imagine what would happen if two people registered.

I'll probably give ejabberd a try next. It lacks the pretty web interface Wildfire has, but that's hardly my primary concern at this point.

I decided to take the plunge and upgrade to FC6. I was pretty happy with my FC5 setup that I'd hacked up to include XGL and compiz, but since AIGLX is supported by default in FC6 and there are repositories with Beryl for FC6 but not FC5, I figured there might be some benefit to an upgrade (plus I wanted to reorganize my partitioning scheme, so it seemed like a good time).

Anyway, here's a short list of the good and the bad:

The good:

1. AIGLX support built in. Well, sort of. If you have an nVidia or Intel video card, you may well be set. If you have ATI you have to use XGL, not AIGLX and you probably have a lot of work to do (and chances are it's fruitless work, but you knew that when you bought ATI anyway).
2. Okay. There is no 2.

The bad:

1. XFS retains its place as a second-class citizen on Fedora. Worse, something has broken since FC5 and if you use "linux xfs" to install the end result will be a corrupt FS when you are done. If you want to use XFS you need to partition and format prior to doing the install (I usually just ctrl+alt+F2 after booting the install CD and do it there). I can't help but note the irony that XFS is the sole reason I started using RedHat (back around 7.0) because SGI offered a modified version of RH7 with XFS support. Yet despite this and the fact that it's the second-oldest journaled FS for Linux, the first to offer features such as ACL support and just basically rocks, RedHat/Fedora continues to pretend it's still beta and not worthy of first-class support. Pretty sad. Frankly if I had to choose between Fedora and XFS it'd be goodbye Fedora.
2. It's a bit flaky. Some of this flakiness I can directly attribute to the beta-quality of several components that I'm responsible for installing (beta nVidia driver for AIGLX support, Beryl), but some other things have simply gone downhill a bit since FC5. NetworkManager is flakier. Sometimes it simply refuses to connect to an access point I was just using a moment before, instead choosing to connect to the neighbor's. Other times it simply crashes.
3. Speaking of NetworkManager, why the hell isn't this daemon turned on by default? It's the best thing happening for dynamic networking on Linux, giving an experience similar to what Mac and Windows users are used to. RedHat wrote it, Fedora ships it, but it's off by default. I played with several half-ass solutions before a damn Ubuntu user told me about it. WTF. Note to RedHat/Fedora: when you write really great software, let people know so that your fine work doesn't go to waste.
4. For some reason, the installer chose an i586 kernel. This became an issue when I found I couldn't load the nVidia drivers (my kernel-devel RPM was for i686). Worse, because the kernel version was right, it took me a while to notice what the issue was. Reinstalling the correct architecture fixed it quite easily but it left me scratching my head for a while.
5. Still shipping Firefox 1.5. I've been using 2.0 release candidates for several weeks and FF2.0 is improved in numerous subtle but readily apparent ways. The overall feel is noticeably smoother. Sure, including 2.0 would have meant delaying FC6 for another couple weeks, but frankly I think that would have been a good idea anyway. FC6 feels like it was rushed for some reason which makes little sense given that the most noticeable improvements (from a desktop perspective) are all centered around AIGLX and compiz. Waiting for FF2.0 would have brought a pretty significant feature to FC6.

Anyway, my overall impression is that FC5 with XGL was actually a bit smoother and quite a bit more stable than my current FC6 install. AIGLX may be the future but XGL works a lot better today (it even felt a bit faster than AIGLX which I found surprising). If you've got a nice FC5 install I'd suggest you stick with it for at least another couple months. FC6 should be considered beta software for the near future.

Apparently Adobe has donated Flash to the Mozilla project. Given that I've read that Flash is full of hand-coded micro-optimizations in assembler (hence Macromedia's struggle to keep the various platforms in sync), I can't help but wonder if this may actually create a drag on Mozilla developer resources (do I smell a rewrite?).

My recent discovery of Songbird and rediscovery of AudioScrobbler and last.fm have sent me on a journey to get a music setup I'm happy with.

Since I work mostly on a laptop and my music is on a desktop machine which also happens to control the good speakers, I prefer a client/server type system (yes, I can ssh -Y and run something, but that has serious limitations as you'll see).

My requirements are this: 1) be able to remotely control the player and have the music come out the nice home speakers when I'm at home. 2) be able to stream music over the web so I can still listen to music while I'm away from home. 3) not require that I start 4 different programs depending on location. 4) be able to update last.fm (yes, it's become a requirement).

So, clearly "ssh -Y" won't work as it can't stream music remotely in any elegant way.

I considered all the upcoming client/server music players (XMMS2, BMPx, MPD) and while BMPx would be what I'd choose if UI were the only factor and MPD appears to be the least elegant and least ambitious project, MPD also happens to be the only truly functional player of the group (I'm sure this is largely due to the perhaps overly ambitious scope of the others).

So, server-daemon settled on, I needed a way to both stream and play over the local speakers. Luckily the recent versions of mpd provide icecast as an optional output so that was a non-issue. I'll leave how I solved playing over the local speakers to your imagination.

Anyway, next up was finding a good mpd client. In the past I'd used gmpc, but it's pretty spartan and IMO resembles more of a proof-of-concept than something to be used day-to-day.

First I tried Pympd and almost stayed. It's pretty good. One thing that bothered me was searching. Pympd tries to do a live search as you are typing. Hint to Pympd developers: I've got a little over 17,000 files to search. It's painful when each letter typed causes a 5 second delay while Pympd narrows in on my selection. I'm more than willing to hit enter. Really. Another thing that bothered me is the volume control. I'm not a fan of the drop-down volume slider. Aside from skipping tracks, the volume is the most frequently used feature. It should be controllable with the scrollwheel (like xmms) or at least immediately accessible (yes, that one extra click is annoying, especially on a laptop where the pointing device remains a bit awkward). Also, Pympd fails to jump to the current track automatically. It's nice to be able to see what's playing and be able to select an adjacent song with minimal clicking. Anyway, those complaints aside, Pympd is pretty nice and as I mentioned, I almost stuck with it.

Next up I tried Sonata. It's still not the perfect player but it's so far the best I've discovered in the MPD world. It shares the annoying volume control that Pympd has (although clicking it and scrolling does adjust the volume, it's still suboptimal). Anyway, it's pretty close to what I'm looking for. It has the prettiest song-change popups yet (complete with album art), and provides a nice compact interface (and it collapses down to even more compact). It also has a few other little features that make it nice (keep on all workspaces, keep on top, etc), it's fast, it automatically tracks the playing song in the playlist. It's missing some esoteric features, but IMO those should be the last features implemented, so I think the developers are on the right track. Now if I can just get them to drop that damn iTunes-style volume control...

As it turns out, none of the MPD clients have built-in AudioScrobbler capabilites (well, Pympd can query last.fm, but not update it). Luckily there are several daemon "clients" that can. I tried AudioScribbler but the fact it couldn't handle crossfading was kind of annoying (especially because one of the GUI clients I was testing would inadvertently turn it on all the time). I finally settled on scmpc which seems to be working well.

I think this is about as close to the perfect setup as I can currently hope for (without putting a bunch of hours into writing my own software).

From the Songbird site:

Songbird is a desktop Web player, a digital jukebox and Web browser mash-up. Like Winamp, it supports extensions and skins feathers. Like Firefox, it is built from Mozilla, cross-platform and open source.

I've heard that Songbird was developed by the original developers of WinAmp. So far it looks pretty cool and the fact that it's a XUL app is quite awesome.

I decided to help my friend upgrade the memory in his HP Pavilion zv6000. It's been a while since I'd done anything significant with anything by HP (or Compaq for that matter) because I've always despised their tendency to use proprietary hardware (down to the screws) and bizarre case designs.

Anyway, I always assume people (and corporations) get smarter over time (albeit some more slowly than others), and given that even cheap laptops like GQ make upgrading pretty painless, I didn't expect much difficulty.

Boy, was I wrong.

As it turns out, the zv6000 has two DIMM slots: 1 external and 1 internal. The external slot is about where you'd expect: under a small plastic cover on the back held in place with one screw. The internal slot is under the keyboard. This is already a bummer, but most laptop manufacturers these days make removing the keyboard pretty painless, so I forged ahead.

Anytime I have to get inside a laptop, I always read the fine manual. Laptops are difficult beasts by nature and their disassemblly isn't always obvious. So I went ahead an pulled up the following PDF:

HP Pavilion zv6000 User Guide

In case you don't feel like reading it, here's the steps:

1. Remove four screws that retain the keyboard cover (the plastic strip in front of the screen)
2. Place electrical tape (?!) over a hinge to "protect it". My first "WTF" hit right here.
3. Press down on three or four "pressure points" while simultaneously lifting the cover. Supposedly this "releases" the clips that hold the cover in place (in case the four screws all fell out).

Okay, time to pause here. We are referring to a strip of fairly brittle plastic approximately one inch deep and 10 inches long. You have to be careful with it or it will break. I don't know how many hands HP technicians have, but perhaps they should do a survey of their users before assuming more than two.

Anyway, the "pressure points" turned out to be pointless (ha! a pun!) as they released nothing but whatever the opposite of endorphins might be (in my head, not the laptop).

I finally managed to pry the cover off, fearing a sudden snap every inch of the way, so now onto the next step:

4. Remove four MORE screws that hold the keyboard in place.
5. Remove a metallic sticker that covers the DIMM slot.
6. Insert DIMM
7. Put it all back together (much easier than getting apart).

I'd like to contrast this with how I installed memory in a GQ (Fry's cheap-ass Great Quality brand):

1. Remove one screw from easily accessible panel on back of laptop.
2. Insert DIMM
3. Replace cover and screw and be happy you saved $500 by not buying a piece of crap HP.

Anyway, I feel lucky that nothing broke. And by "lucky", I mean "damn lucky".

Do yourself a favor: don't buy HP.

Oh, and before you tell me "but I have an HP and I freaking love it", ask yourself if you've tried to upgrade the internal DIMM. If you haven't then please feel free to take one for the team and give it a try. If you have and somehow feel I've misrepresented the steps (despite the fact the exact steps are outlined a bit more tersely in the HP manual), then don't bother replying because if you don't hurry you'll miss your appointment with Goddess Ilsa for your weekly degradation session.

I watched the WSGI video (and followed the slides in a second window cause the video is too tiny and compressed) and am sold. WSGI is the future of the Python web framework. While most of the major frameworks (TurboGears, Django, Twisted, et al) support WSGI, only a couple are WSGI from top to bottom: Pylons and Clever Harold. Of these two, Pylons appears to be the most mature and active, so I've decided to give it a shot. Also, the fact that the Pylons crew has ported most of the interesting stuff from Rails doesn't hurt any either.

I have to admit, I'm starting to feel like a framework whore though. Hopefully it starts to pay as well ;-)

Welcome our new dictator.

I think the first people to be prosecuted under this bill should be its authors for attempting to destroy our way of government and life from within. How long has this infiltration been going on?

The Bush administration has systematically destroyed every law ever made to protect American citizens from their own government, and yet I'm still amazed at the temerity of this. Will the next step be not simply not doing things labeled as "terrorist acts", but actual requirements to prove your allegience?

To quote Attorney General Alberto Gonzales: "When courts issue decisions that overturn long-standing traditions or policies without proper support in text or precedent, they cannot - and should not - be shielded from criticism". Ironically, Gonzales doesn't apply this same reasoning to the Executive branch of our government. Apparently the courts should be criticised for overturning long-standing traditions and policies but that anyone who criticises the Executive branch should be imprisoned and tortured until they confess, and then possibly put to death.

You might convince yourself that "well, the wording might allow them to do that, but they never would", but then you'd be a fucking idiot. Bush has already stated publicly that he does whatever the law allows. He further clarified that he meant this in terms of the fullest extent of the law. If the law allows him to kidnap and torture U.S. citizens, he will.

Double-plus ungood.

Lately I've been keeping my eye on alternatives to Python. I've gotten tired of waiting for features in Python that, to put it in GvR's words "ain't gonna happen". Most of these features aren't strictly essential to day-to-day programming, and the lack of them doesn't make Python a bad language, but they are features that give that warm feeling known as "elegance".

What are these features? A very short list (for me) includes:

1. Expression-based syntax
2. Closures
3. Macros (Lisp-style, not C-style)

To be honest, I'd settle for #1, but if I'm looking for an alternative, I may as well look for all three.

You might be wondering why Ruby isn't an option, since it supports at least the first two items and it currently has a thriving community (which is always good to have around a language you plan to use for more than hobbies). Ruby has two major drawbacks to me: first of all the performance is absolutely dismal. Until Ruby has a bytecode compiler or leverages an existing VM such as .NET/Mono, this is unlikely to change (I know work is being done here, but I'm not holding my breath. By the time a Ruby 1.x VM is done, Ruby 2.x may well be released, which would seem be a setback for one or the other). The second reason is I simply don't like the look of Ruby code. Too much syntax. This is more critical than even the performance issues. The performance issues will certainly be fixed at some point, but the language isn't likely to change much cosmetically (lest it become a different language). While Ruby may have the heart of Smalltalk, it has the face of Perl. I may like the ugly girl with the heart of gold, but I'm not going to marry her, thanks.

Anyway, I'm not bashing Ruby here, for all of you who are going to get upset about it. I think Ruby embodies some great ideas that have been languishing in the programming world. The deaths of Smalltalk and Lisp pretty much set the programming world back 20 years, IMO, and the fact that Ruby has brought back to life many of the fundamental concepts embodied in those languages is deeply appreciated. Ruby just isn't for me.

Anyway, I'm old and lazy these days, so whenever I see a new language with a syntax I'm readily familiar with, I perk up. Two languages currently sit on my radar as candidates, Boo and Logix (see bottom of article for references).

Of the two, Logix is the most appealing. Logix isn't so much a language as a metalanguage, with a Python-like syntax definition. It also runs on the Python VM and is, in fact, implemented as a Python module that you import into the Python interpreter. This means you can use the Python standard library which is a big win. It meets every requirement I've outlined above (and then some). My major concern with it is that it appears to be dead or at least stagnating and never appeared to have a community to start with. It's so great, I've considered opening a Trac and trying to jump-start a community, but I doubt my own ability to take the lead on development (language design isn't my area of expertise).

The second language I've recently discovered is Boo. I saw the announcement for Boo on c.l.py a couple years ago where it was met with some bit of scorn by the Python community, but recently I stumbled across it again and see that it's grown into a real project in the interim. Boo isn't as easy a jump as Logix (it's a completely new language, with a Python-like syntax, but some major incompatibilities). Also, it runs on .NET/Mono, not the Python VM. I don't consider this a bad thing, but that VM isn't quite as stable as the Python VM on Linux, at least in my experience. Of course, Boo hasn't hit 1.0 yet, so I expect that by the time it does, the CLR will be production-ready across platforms.

Update: Boo also scores major points for having an MVC web framework called Webbness. Documentation looks pretty thin, but it's a start. I'm guessing (but don't know) that Logix could work with some existing Python framework such as Pylons or TurboGears. That remains to be tested.

UPDATE: The original archive appears to be dead, so I've set up a new Arkivo log here.

Thanks to Splee and his TurboGears-powered Arkivo, the nginx IRC logs are now publicly available.

Visit the #nginx IRC channel at irc.freenode.net and view the archives here.

With the recent announcement that Apple's CalDAV server was built on Twisted and the following announcement that Apple had donated an Xserve to the Twisted project, it's become clear that Apple has some interest in Twisted.

Twisted isn't exactly the most popular framework on the planet, so I found this a little surprising but chalked it up to mysterious forces.

This morning I was showering and I was thinking about how lame it was to get an Xserve. I'd seen benchmarks that demonstrated that OSX simply is not there as a server OS. The blame is apparently to be placed on the overhead associated with spawning threads and processes on OSX under Mach. The performance of server processes like Apache and MySQL is terrible compared to Linux on the same hardware.

Then it dawned on me: this is undoubtedly the root of Apple's interest in Twisted: it's asyncronous, hence, no threads. I've done no benchmarks or even heard of any, but if Twisted is twice as slow as Apache on Linux, and Apache on OSX is five times as slow as Apache on Linux, then it stands to reason that Twisted on OSX is at least twice as fast as Apache on OSX.

Pure speculation of course, since I've neither seen nor done any type of benchmarks, but it's interesting speculation.

It appears the wiki is a great success. I contacted Igor Sysoev to let him know what we were doing and he's put a link on nginx.net so that people can find the wiki.

We've also had more people than ever (up to 8!) in the IRC channel, which isn't bad considering I hung out there by myself for a couple weeks prior to that.

Anyway, to date, most of the work on the wiki has been done by myself and Olle Jonsson, but we've had a few other contributors as well: technoweenie (of Mephisto fame) contributed some info on running Mongrel with Nginx and we've had at least one Russian speaker (Aleksandar Lazic, who wrote the original English draft documents we relied on so heavily) helping us get the translations right.

Overall, I think we can claim that we're around 50-60% done. Most of the documentation is translated, but we need to have reviews to validate that it's correct. We've put links back to the original Russian documentation at the bottom of each page to help people who are able to compare.

Next up, we'll need a fancy logo ;-) Hopefully Igor doesn't mind.

Don't forget: visit the Nginx wiki and help out!

There's also been German and Russian language sections started, but they need work!

I setup a Nginx wiki last night.

Feel free to create an account and add to the knowledge, fix errors, etc.

Also, join us on #nginx on irc.freenode.net!

In A Do-It-Yourself Framework, Ian Bicking demystifies WSGI and shows how to assemble a Python web framework out of components.

Okay, I've managed to move most of the important articles over and spent some time getting things working.

Here's what Bitakora has:

1. Great multiuser support
2. Clean and simple admin interface

Here's what it needs:

1. Split articles (read more...)
2. Spam prevention
3. More configuration options (I had to hack dhtml and py files to cut the number of articles per page down to 5 from 10).
4. RSS feeds by tag

Making of Yuka

After much pain and suffering (as only Zope can inflict), Bitakora is running and it's lookin' grood!

I don't usually discuss politics, but I'm just so fed up with ignorance and the polarization it causes. Today I received an email from actforchange.org, an advocacy group that I subscribe to regarding the upcoming ABC movie "Path to 9/11". Rather than blindly "act for change", I decided to click some of the links in the email and see what the fuss was about.

Apparently this was a mistake.

Not only was the email from Act for Change a complete misrepresentation of their sources, the sources themselves were unsubstantiated (most of the information was without source references). So I'm to understand we should take action based on distortions of other people's distortions and speculation?

Still not having learned my lesson, I continued following links and watched the speculation and arguing about slant in a movie none of these people have actually seen.

That's right. Everyone on both sides is up in arms and they haven't seen the damn movie. Much of the accusations of slant come from rumors that the Clintons pressured Disney and ABC (who are either liberal or conservative depending on whom you ask and in what context you are asking) to make some changes.

I'm supposed to care? How about this instead: I'll care when I or someone else has seen the movie and provided some actual first-hand evidence that the film is slanted in any particular way.

And here you have it: anyone who is arguing about the slant of this film who hasn't actually seen it yet is an utter and absolute buffoon. You heard it here first.

Oh, and Act for Change? You just lost a subscriber. Good work.

As I continue my migration away from Pound and Lighttpd, I'm discovering dark corners of Nginx. Tonight I ran into a problem proxying from Nginx to a Lighttpd backend that's running Mailman and it turns out to be a gem.

Because I've got a lot of sites to migrate, I'm trying to minimize the changes I make at any given point. Many sites rely on custom Lighty configs and with those I'm simply swapping Nginx in for Pound and proxying to Lighty. Later I'll work on eliminating Lighty.

Anyway, I have a site that's using Mailman, and because Nginx doesn't have CGI support, I'm leaving Lighty in place to handle Mailman for the moment. I tried an Nginx config that looked something like this:

location ~ /(mailman|pipermail)/ {
    proxy_pass http://127.0.0.1:8000/;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

This failed to load with the following error:

* Checking nginx' configuration ...
2006/09/04 02:36:33 [emerg] 4067#0: "proxy_pass" may not have URI part in location given by regular expression or inside the "if" statement in /etc/nginx/nginx.conf:7
2006/09/04 02:36:33 [emerg] 4067#0: the configuration file /etc/nginx/nginx.conf test failed
* failed, please correct errors above

This seemed strange. Why would a proxy not be allowed within a regex location?

"Fine", I thought. "I'll just eliminate the regex for now and figure it out later." So I changed the config to look like this:

location /mailman/ {
    proxy_pass http://127.0.0.1:8000/;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

This was accepted but I got a 404 whenever I tried to go to /mailman. Lighty is configured to run on 8000 and I could see from the Lighty log that the request was getting passed through. What was odd is that I could also see that the first part of the request header was getting stripped off. For instance, /mailman/subscribe would become just /subscribe and thus generate a 404.

Finally I realized I'd added a trailing slash to the proxy_pass directive in nginx.conf. Removing this solved the problem. I then decided to try the regex again. This too now worked.

My config now looked like this:

location ~ /(mailman|pipermail)/ {
    proxy_pass http://127.0.0.1:8000;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

So what's the gem? Well, apparently you can strip off the location part of the URL by appending the trailing slash to the proxy_pass directive. This is actually a pretty useful feature if you are proxying to a server such as TurboGears that isn't mounted at the document root. Rather than mucking about trying to get the server root set correctly (which I've never managed to do), you can simply have Nginx strip it off.

[Update]

I wanted to give a more concrete example of using this feature, but didn't have one at the time I originally wrote this entry. I'm moving to a new blog and ended up using this.

The new blogging software I'm using runs on top of Zope, which, unless you are a Zope user, can lead to all sorts of mysterious requirements. Well, one of these requirements is that the blog reside mounted at /users (well, not /users specifically, but not the root). I wanted the blog to be the front page. I could solve this in Zope, I know, but it would be a pain to figure out. Enter Nginx:

Solution 1: rewrite urls.

The obvious and traditional (ala Apache) solution would be to use rewrite rules, and this does in fact, work:

server {
    listen 80;
    server_name blog.twisty-industries.com;
    location = / {
        rewrite (.*) /users last;
    }

    location / {
        proxy_pass http://127.0.0.3:8082;
        include /etc/nginx/proxy.conf;
    }
}

Is there a downside? Shrug. There is a slightly better way however:

Solution 2: proxy path:

server {
    listen 80;
    server_name blog.twisty-industries.com;
    location / {
        proxy_pass http://127.0.0.3:8082/users;
        include /etc/nginx/proxy.conf;
    }
}

I suspect this is a bit more efficient, both in lines of configuration and in request processing time. In short, I like it.

This assumes that SquirrelMail is installed at /var/www/public/webmail.

The Nginx configuration:

location ~ /webmail/.*\.php {
    root /var/www/public;
    fastcgi_pass 127.0.0.1:1025;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/public$fastcgi_script_name;
    fastcgi_param SCRIPT_NAME $fastcgi_script_name;
    fastcgi_param QUERY_STRING $query_string;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param CONTENT_TYPE $content_type;
    fastcgi_param CONTENT_LENGTH $content_length;
    fastcgi_param REQUEST_URI $request_uri;
    fastcgi_param DOCUMENT_URI $document_uri;
    fastcgi_param DOCUMENT_ROOT /var/www/public/webmail;
    fastcgi_param SERVER_PROTOCOL $server_protocol;
    fastcgi_param REMOTE_ADDR $remote_addr;
    fastcgi_param REMOTE_PORT $remote_port;
    fastcgi_param SERVER_ADDR $server_addr;
    fastcgi_param SERVER_PORT $server_port;
    fastcgi_param SERVER_NAME $server_name;
    fastcgi_param REDIRECT_STATUS 200;
}

location ~ /webmail {
    index index.php;
    root /var/www/public;
}

Next, you must run a PHP FastCGI instance. For this purpose I'm using spawn-fcgi from the Lighttpd distribution:

/usr/local/bin/spawn-fcgi -a 127.0.0.1 -p 1025 -u sqmail -g sqmail -f /usr/lib/php4/bin/php-cgi

What's new in CP3

Humberto Diógenes has added Zeroconf support for TurboGears:

View Trac ticket and patch

Pretty interesting idea.

I'd briefly looked at the Yahoo! UI components when they were announced. Somehow I didn't look close enough and missed this:

Yahoo! UI Grids

I guess I'd assumed they were some sort of rich grid widget and didn't look deeper. My mistake.

YouOS

I got a bit frustrated doing real work tonight, and my experiments with clustering TurboGears instances behind Nginx got me thinking about cache coherency (TurboStan has been using an internal dictionary for caching; efficient for a single instance, but not for clustered applications).

I'd never used memcached before, but I quickly located Sean Reifschneider's python-memcached module in the Cheeseshop. It wouldn't easy_install, but that was no problem since it's a single Python file.

Next I set out to rewire the caching in TurboStan to support an additional configuration directive. If you add:

stan.memcached = "127.0.0.1:11211"

to your TurboGears configuration, TurboStan tries to connect to a memcache at that location.

As documented on the Memcached home page, you start memcached with something like:

memcached -d -m 16 -l 127.0.0.1 -p 11211

This starts memcached with a 16Mb cache (hey, I'm not running LiveJournal) using port 11211 on localhost. At first I thought I could get away with 4Mb of cache, but I spent a while trying to figure out why some objects weren't cached before realizing memcached was out of memory.

In order to support both the internal dictionary cache or the memcache, I created a new class that provides the memcache API for a dictionary and allows you to select one mode or the other.

Memcached is remarkably simple to use. One caveat is that because I'm caching code objects and memcache.py uses cPickle, I was forced to marshal code objects prior to passing them to memcache. This means that the code objects are marshalled and pickled. Not too efficient. I think I'll modify memcache.py to always marshall (the special case code it has for pickling or not pickling objects isn't relevant to this purpose since everything will always need to be pickled). The fact that marshal isn't portable across Python versions is irrelevant too since it's a temporary cache.

Anyway, subjective performance was actually marginally slower, but the point of memcached isn't performance but rather scalability (a different thing altogether athough confused by some). And besides, cache coherence is what I was really after (I have methods for destroying cache entries, but in a cluster only one backend would receive the instruction to destroy a cache entry, leaving the other instances with outdated objects in their caches).

References:

• Memcached Home
• Python-Memcached

Over at Hacknot , there's an article criticizing recent trends amongst the Web 2.0 crowd. Admittedly, I've snickered to myself as well when I hear former PHP programmers rave breathlessly about what was previously considered common programming knowledge (well, the acronyms are new anyway), but there's something more afoot than all the hype and in fact it's this very hype that's letting the real deal slip by unnoticed by the author of that article.

I was reading about Amazon's new web services (S3 , EC2 , etc) and it slowly dawned on me what Web 2.0 is really about (yes, I'm quick, some of you are surely thinking). Web 2.0 isn't about MVC frameworks or AJAX (formerly known as "hey my browser actually works") or rounded corners or spinning GIFs or finding a way to make an 800px wide page look like it's not.

Web 2.0 is about putting the web back on the web. By that I mean not on your server . It means the end of shopping carts that aren't run on Amazon. It means the end of Gallery in favor of Flickr. It means the end of blogs not on Blogger. It means your "custom" application will consist of regurgitating data from these services and presenting it in a new way.

There's a key element to all of this: open API's. This is really the meat of "Web 2.0". The API (usually REST, but XML-RPC as well. Even RSS and Atom are part of it) has come to web applications. Much like Microsoft Office brought real vertical applications to the desktop with VBA/COM, Web 2.0 brings vertical applications (aka "mashups") to the web. In a way, the hype is kind of necessary to this process. Since none of this stuff is really new (only new to the web and the programmers who inhabit it), if it were called the same thing it was called 20 years ago, no one would care or even notice.

There's a bit of ego involved in running a website (gasp) and I think many people are reluctant to have, say their shopping cart on Amazon rather than be custom coded so that visitors "never have to leave my site". Well, those days are over. Having your cart on Amazon is not only easier, safer and more reliable, but you'll increase your sales (duh!). It's a bit like the vanity of having your own storefront versus putting your product on the shelf at a large supermarket. The storefront gives you personal satisfaction. The supermarket gives you sales. Since I'm singling out Amazon, I'll point out that even large retailers who can afford to run their own sites are using Amazon (Toys 'R Us for one). It's just smart business. It costs less to develop and moves more product. A real no-brainer. You also don't need to worry about the plethora of problems already solved (redundancy, scalability) by the larger service providers. Of course, with the web API you can have your cake and eat it too, having a regular site that integrates with a larger service, although there are some sacrifices you'll be asked to make by most of the service providers (the external link, OMG!).

I've been slowly building a toolkit that I'm just as slowly realizing is obsolete before it will ever be finished. I've been planning on writing a custom shopping cart for it but am realizing that hey, what's the point. Instead I'll be writing stuff that interfaces with API's and leaving the hard work for others.

Anyway, I suppose I'm not really addressing the article that prompted me to write this, but what's the point. The author apparently thinks Rails is an ORM and Python is only known by a few enthusiasts, so I may as well dredge up a Mesohippus and flog it as argue against him. It appears to me he's caught up in the hype just as surely as the "fanboys" he criticizes, staring at the sun while the polar caps melt around him. It doesn't matter which side of the hype you are caught up in. Pro or con, it interferes in your ability to make unbiased assessments. Frankly, the arrogance of assuming that everyone else is caught up in the hype but you alone are above all that is enough to make DHH seem humble by comparison.

I decided to check once again if there remained a bug with specifying Unix domain sockets in the TurboGears configuration (we're on our third configuration system, so it seemed reasonable to think a bug or two might have gotten shaken out at some point ;-)

And apparently it did (or perhaps the note I make at the end of this article is why I've gotten it to work this time and failed previously).

Nginx configuration:

upstream mycluster {
    server unix:/path/to/socket-1;
    server unix:/path/to/socket-2;
    # ...
}

server {
    # ...
    location / {
        proxy_pass http://mycluster;
    }
}

TurboGears configuration:

server.socket_port=""
server.socket_file="/path/to/socket-1"

Repeat the above configuration in separate config files for each backend you want.

Note that it's necessary to set server.socket_port="" or else someone (TurboGears), somewhere sets it to a default value of 8080. This wouldn't matter except it's how CherryPy determines whether or not we want a domain socket vs a TCP port.

I decided to test out Nginx's load-balancing capabilities. Turns out to be remarkably easy.

CherryPy is not a fast HTTP server (CP3 is supposedly at least twice as fast, but not released yet, and either way, TurboGears is currently tied to CP2). That's okay, but there are some things you can do to make sure that it doesn't become a bottleneck for your TurboGears website.

The first and obvious thing is to serve static files (images, javascript, css, etc) from a regular webserver. Lots of people use Apache or Lighttpd for this purpose. A few others (including myself) prefer to use Nginx.

This solves a lot of the performance issues right off the bat. However there's another technique that can help quite a bit as well: load balancing.

Nginx provides a simple and elegant way to load balance across two or more backend servers. Here's how you do it with TurboGears.

The TurboGears Setup Edit your prod.cfg file and add/change the following lines to look like:

server.socket_host="127.0.0.1"
server.socket_port=8000

Next, copy your prod.cfg file to a file called prod2.cfg and edit it with the following:

server.socket_host="127.0.0.1"
server.socket_port=8001

Next, start two TurboGears instances:

./start-myproject.py prod.cfg&
./start-myproject.py prod2.cfg&

The Nginx Configuration The next step is to edit your Nginx configuration and add the following:

upstream myproject {
    server 127.0.0.1:8000;
    server 127.0.0.1:8001;
}

server {
    listen: 80;
    server_name: www.domain.com;
    location / {
        proxy_pass http://myproject;
    }
}

Now restart Nginx.

Unbelievably, that's it. If you want to see the effects (and convince yourself it works), you can change your dev.cfg files in the same fashion (probably from two separate terminals) and click through your site like mad (or wget it) and watch as each TurboGears backend takes part of the load.

A couple things to note:

• Sessions appear to work fine. I've used this technique with Identity and it doesn't appear to matter which backend the request goes to.
• This probably won't work with SQLite. SQLite is not meant for concurrent access and running two TG servers against the same database is going to be a problem. Besides, if you are doing load balancing, why are you using a toy database anyway? ;-)

In all the years I've used Linux, I've never been inclined to post a screenshot (you've seen one desktop, you've seen them all). But I'm just so damn happy with this background that I had to share.

Alex Does Hoola on my Desk

If you look from left to right rapidly, not only does it give a sort of animated feel, but you'll risk straining an eye muscle.

<voice src="experience">Don't do it.</voice>

I'm an enthusiast. For what? Apparently whatever looks really promising at the moment. That being said, upon hearing that Bob Ippolito has been successfully running Nginx as a high-traffic proxy, I decided to dive right in.

My current standard setup has been Pound proxy (which frankly I've been quite pleased with) in front of various web servers and services, the most common case being in front of Lighttpd and TurboGears. TurboGears doesn't require a separate web server, but CherryPy is suboptimal (to say the least) for serving static content and I host many sites that don't use TurboGears, so there you have it.

Anyway, I'm an enthusiast, but I'm also inclined to be my own crash-test dummy. I don't recommend things until I've at least taken them around the block a few times. So it was time to convert my own sites to Nginx and test the waters.

I started with develix.com as it's pretty lightweight and Nginx's job would consist mainly of serving static files and proxying to the TurboGears app develix.com runs on. Also the fact that develix.com is on it's own IP address simplified the migration. I could leave Pound in place for the other IP addresses and only tinker with develix.com.

I installed Nginx, edited the sample configuration file that came with it, commented out the develix.com entries (quite a few actually, since there are many subdomains), restarted Pound, then fired up Nginx. After a couple minor missteps, develix was happily chugging away on Nginx. Too easy.

Next I decided to migrate a customer's dedicated server. I figured if the experiment were a failure, it would be easy to switch back to his original configuration (Pound proxying to several Lighttpd instances, all running Joomla, Mambo, Sitedepth and Coppermine, i.e. a buttload of crappy, overly complicated PHP code). Further, because this server is mostly about serving large images and videos (yes, it hosts a few of those sites), I figured it would be both a good test of Nginx and a possible performance benefit to my customer.

I notified him of what I wanted to do, and since we'd been having issues with Lighty on that box, he was happy to let me have a go at it.

Well, it wasn't as easy as I'd hoped. Most of the issues were minor, but there were many (remember, Joomla . With lots of modules). Most of these were just a matter of figuring out what the hell was going on, making some rewrite rules, etc. Typical stuff when setting up a website.

One issue however, almost became a showstopper: it seems Nginx doesn't do CGI.

That's right. Of all things. Who would have thought? I mean, after static content, CGI is the simplest thing on the face of the earth. Additionally, it hadn't even occurred to me that he used CGI (only two scripts as it turns out). Anyway, long story short, I ended up writing a small Perl wrapper for the Perl CGI he had that let it be served via FastCGI and then running that from Lighttpd's spawn-fcgi program. This took a surprisingly long time to figure out how to do (think long, painful hours). You'd think there'd be a ready-made Perl script for doing this, but if there is, it's buried under tons of mirrors of the Perl documentation. By the time I finished, the script was quite short, on the order of a dozen lines, but all that meant was I was averaging two lines of code per hour. I won't say that Perl sucks, but I'm sure thinking it.

Anyway, once that was resolved, there was a bit of minor cleanup with file permissions, yada yada, but so far so good.

NOTE: Since I first posted this article, I've downloaded this and I see that it doesn't include the Python source, only .pyc files.

Sorry to say, NAJA IS NOT RECOMMENDED.

Naja has several modules:

1. downloader (HTTP, FTP)
2. websucker (website grabber)
3. newsgrabber, newsreader, newsposter (uue, yEnc, base64, Par2)
4. FTP, FTPS, SFTP, SCP client
5. WebDAV client

Others Features:

1. CSV filter
2. Checksums (CRC32, MD2, MD4, MD5, SHA, SHA1, MDC2, RMD160)
3. Crypt and Decrypt (AES, DES, 3DES ...)
4. CGI & WebDAV Server
5. Web Interface
6. Compress and decompress (zip, tar.gz, tar.bz2)
7. Picture viewer
8. Text Editor
9. HTTP, HTTPS Server
10. Create SSL certificate with Crypto XML Engine

Lately I've gotten fed up with Lighttpd. There's been outstanding bugs that are so familiar they've acquired names. The project's lead, Jan Kneche, seems more interested in schmoozing up to the Rails crowd than providing a decent product (which is ironic given that Mongrel aims to make Lighttpd irrelevant in the Rails world).

Anyway, a discussion today on the TurboGears list brought up an alternative.

Bob Ippolito replied to a discussion between myself and another person about whether to use Pound or Lighttpd as a reverse-proxy in front of TurboGears applications. I held that Pound is the correct solution as it is a proxy, whereas Lighttpd is a web server that can act as one. Further, I expressed my frustration regarding the state of Lighttpd development and the unmaintainability of its config files.

Bob offered up the following information:

One problem with Lighty is that it leaks memory like a sieve [1]. I audited it for a little bit and I gave up, it's a mess. I'd steer clear of it, it will quickly ruin your day if you throw a lot of traffic at it.

The only solution I know of that's extremely high performance that offers all of the features that you want is nginx [2], but its documentation is largely in Russian. I can't read Russian, but I was able to figure it out (the configuration language isn't Russian, neither is C source). I currently have nginx doing reverse proxy of over tens of millions of HTTP requests per day (thats a few hundred per second) on a single server. At peak load it uses about 15MB RAM and 10% CPU on my particular configuration (FreeBSD 6).

Under the same kind of load, apache falls over (after using 1000 or so processes and god knows how much RAM), pound falls over (too many threads, and using 400MB+ of RAM for all the thread stacks), and lighty leaks more than 20MB per hour (and uses more CPU, but not significantly more).

[1] http://trac.lighttpd.net/trac/ticket/758

[2] http://sysoev.ru/en/

I found this interesting, first off because I know that Bob has used Pound in the past and because, well, it's Bob. Also, someone on the #cherokee channel had suggested Nginx as an option, but since the docs were in Russian I was reluctant to commit to it.

Well after Bob's email, I started searching and it turns out that while the official docs are in Russian, there's a bit of English documentation on the web, and apparently some happy users as well:

• http://blog.kovyrin.net/category/web-tech/nginx/
• http://zh.stikipad.com/notes/show/nginx
• http://brainspl.at/articles/2006/08/23/nginx-my-new-favorite-front-end-for-mongrel-cluster

Also of interest is that Nginx happens to live in Gentoo's portage.

Amazon Elastic Compute Cloud

Saw that in an email on the twisted mailing list.

Guido chooses Zope

Guido chooses Django

History seems to indicate GvR's choice of web frameworks spelling doom for both the framework and projects that use them.

Maybe it's because he doesn't know much about the topic. Even if Django does well (and I think it will in spite of his endorsement) his prediction ratio is still only 50% (or less, if you count things like Medusa).

TurboX2 a TurboGears front-end to XMMS2.

So I've had it in mind to use Twisted for doing various ugly networking tasks that CherryPy doesn't seem appropriate for (since it's a threaded server and I don't want to tie up threads doing things that take an unknown amount of time).

So far it's working well. I've implemented two different services (an RSS aggregator and an IRC proxy) that simply do their tasks, then serve up the data as an XML RPC service on localhost.

I did the RSS aggregator a week or so ago, and tonight hacked together the IRC proxy (the idea being a way to offer customer support via IRC without requiring them to install an IRC client).

The single ugliness in the whole thing (from my POV anyway) is having to poll the server from Javascript every n seconds to see if new messages are posted. I'd like to find a way around that. It's not the Javascript itself that's ugly (MochiKit makes it quite simple), rather it's the concept of needlessly polling that bugs me.

I'm wondering about utilizing a timeout (i.e. the server doesn't respond until it actually has data) to this end. If the Javascript gets data within the timeout, it displays it and runs again. If it doesn't, it runs again. This would reduce the latency (no n second delay between updates) and the number of hits on the server. On the downside, it would increase the number of open sockets, but I doubt we're looking at such a huge load of IRC users that this would be an issue.

JSJac

JWChat

In my previous article , I showed that it's pretty easy to have TurboGears generate static HTML files. However, unless they get served somehow, it's pretty useless. In my quest to figure out Nginx, I decided to make this work.

The Nginx Configuration:

location / {
    default_type text/html;
    types { text/html html; }
    root /var/www/public_html/cache;

    if (-f $request_filename.html) {
        rewrite (.*) $request_uri.html last;
    }

    proxy_pass 127.0.0.1:8000;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

The TurboGears Code

You'll need to add a directory under project/static for storing the cached HTML files. I call mine "cache". I think it's a good name so you should too =)

Add the following to your project's controllers.py:

class StaticOutFilter ( BaseFilter ):
    def before_finalize ( self ):
        if cherrypy.response.status is None:
            request = filter ( None, cherrypy.request.path.split ( '/' ) )
            path = os.path.join ( 'var/www/public_html/cache/', *tuple ( request ) ) + '.html'
            try:
                os.makedirs ( os.path.dirname ( path ) )
            except ( OSError, IOError ):
                pass # should log this
            file ( path, 'w' ).writelines ( cherrypy.response.body )

class Root ( controllers.RootController ):
    _cp_filters = [ StaticOutFilter ( ) ]
    # rest of root controller follows ...

At this point, whenever Nginx receives a request, it appends ".html" to the end of the URL and checks that it exists in the cache directory. If it does, it serves the static HTML file, if it doesn't it passes the request on to TurboGears which will generate the file.

Now clearly this simple setup is rife with limitations:

• Changes made in the TurboGears app won't be reflected unless you manually remove the relevant HTML file(s).
• Even if you do remove the files, truly dynamic content (such as displaying a username) won't be reflected.

The first problem is fairly trivial to solve. Assuming you provide an interface for editing content on your TurboGears site, simply add some code that cleans out the relevant cache file.

The second problem is much trickier. There are lots of ways to address it, depending upon your application's needs, but some ideas that come to mind are:

• Add a decorator that indicates whethor or not a page is cacheable and have the filter check this attribute prior to trying to cache it.
• Use AJAX to manage dynamic items.
• For the specific example given of the username (and user-customizable items), it might make sense to have a separate view of the site for authenticated users that bypasses Nginx and goes straight to TurboGears (i.e. myaccount.domain.com rather than www.domain.com). If most of your traffic consists of anonymous users then this might be ideal. Serve static content to anonymous users and dynamic content to authenticated users.

Anyway, this aspect is left as an exercise for the reader (or perhaps myself at a later date). If you have suggestions, feel free to make a comment.

I'm exploring caching options for use with TurboGears, and of course, the penultimate cache is when you are serving static HTML pages from a webserver (like Lighty or Apache) rather than dynamic content from a framework.

The key to generating the static HTML files is to use what is known as a filter. CherryPy lets you attach filters at several points in the request processing chain. In this case, I wanted to intercept the request right before it's returned to the browser and dump it to a file:

from cherrypy.filters.basefilter import BaseFilter
class StaticOutFilter ( BaseFilter ):
    def before_finalize ( self ):
        if cherrypy.response.status is None:
            path = os.path.join ( 'path/to/static/cache',
                                  *tuple ( cherrypy.request.path.split ( '/' ) ) ) + '.html'
            try:
                os.makedirs ( os.path.dirname ( path ) )
            except OSError:
                pass
            file ( path, 'w' ).writelines ( cherrypy.response.body )

class Root ( controllers.RootController ):
    _cp_filters = [ StaticOutFilter ( ) ]

That's it! Now the content is stored as static HTML files in a directory hierarchy that mirrors the URL hierarchy in my application. There are many caveats and limitations (and probable dangers - I'll need to analyze the writer code to make sure no path manipulation can be done), but it's a start.

Read Part 2 of this Article

As some of you already know, we're getting pretty close to making our big transition from Develix to a new name, look and company focus.Our new site is up (for the most part) and is in the refinement stage. We're adding content like mad, refining existing content and overall, going for a whole new image.

It's still under wraps at the moment (the site is publicly accessible, but we've done nothing to market it nor even added a link pointing to it) because we want it to be perfect (or as near as can be) for the big push.

From day one, Develix was about software development. Websites were merely another type of development for us. That's all about to change. We've come to realize that long-term projects wear on us a bit and we're going to focus more on local businesses, web design and marketing. Not to exclude non-local customers (we're happy to work on sites anywhere), but our marketing will be targeting our region and will be quite aggressive.

The move to web design is the culmination of two aspects: our dislike of projects that take six months to complete and the fact that our ongoing development to date has given us a platform that requires minimal programming effort (a CMS type app based on TurboGears and TurboStan). Quite frankly, we no longer need to program very much. We'll still continue improving the product, but it will be more incremental than than in huge rushes.

Frankly, this is a huge relief. We've floundered a bit due to lack of focus and overreaching ourselves a bit (hey, we're a small company!). Already, just having a focus has given us huge doses of inspiration and it's already apparent to us that our upcoming site is absolutely the best work we've ever done. We've literally got more ideas than we can use.

Xara Extreme

I'm often accused of hating all software. I really do hate a lot of it. Some I hate less than others, but overall software fails to deliver the goods for the most part.

There are however, a few bits of software that I don't hate. In fact I love them. So here's my very short list.

1. PostgreSQL. This is undoubtedly the only software of this scale that has never disappointed me.
2. Mochikit. Not really a large impressive project, but interesting in that not only does it not suck, it doesn't suck so much that it makes certain other software (JavaScript) suck less. That's quite an accomplishment.

I'll surely revisit this page to add others as it strikes me.

Oh, and for those who know me and wonder why Linux, Python, TurboGears and Twisted didn't make the list, here's a summary:

1. Linux. It has suckage all over. Luckily they keep moving it so it doesn't get too annoying.
2. Python. It differentates between statements and expressions. 'Nuff said.
3. TurboGears. Built on a threaded web server (CherryPy). Likes Kid too much.
4. Twisted. Awesome software, but how the fuck is a mortal supposed to use this stuff without good examples and documentation? I can't use it so it sucks.

Todo!

XGL on FC5!

I recently added a logic tag to TurboStan with some mixed feelings. One of the things I love about using TurboStan is that I'm constantly discovering new ways to utilize its features.

I'm working on a couple of sites that will have one-off pages, that is, pages that don't fit the standard template for the site. In this case, the only difference was that the home page would not have a sidebar, but the other pages all would. In all cases, index.stan defined the overall site layout, and public.stan filled in slots in this template.

The quick and easy solution seemed to be to use the new "case" tag something like this:

#index.stan
html [
    body [
        div ( id = 'left-sidebar' ) [ slot ( 'left-sidebar' ) ],
        div ( id = 'content' ) [ slot ( 'content' ) ]
    ]
]

#public.stan
inherits ( 'index' ) [
    override ( 'left-sidebar' ) [
        case ( vars.page != '/home' ) [
            xml ( 'left sidebar stuff!' )
        ]
    ],
    override ( 'content' ) [
        div ( render = vars.content, data = vars.page )
    ]
]

This worked fine, except I had a problem: even though the left-sidebar was now empty on the /home page, the CSS attached to it was still in effect. "Easy", I thought, "I'll move the div inside the case tag. So now I ended up with something like:

#index.stan
html [
    body [
        slot ( 'left-sidebar' ),
        slot ( 'content' )
    ]
]

#public.stan
inherits ( 'index' ) [
    override ( 'left-sidebar' ) [
        case ( vars.page != '/home' ) [
            div ( id = 'left-sidebar' ) [
                xml ( 'left sidebar stuff!' )
            ]
        ]
    ],
    override ( 'content' ) [
        div ( id='content', render = vars.content, data = vars.page )
    ]
]

This was looking better. I could see how this small alteration gave me a bit more flexibility. However, I still had a problem: in order to make the left-column and content divs sit side-by-side, I was forced to give them both fixed widths in the CSS. If I removed the widths, the content div ended up underneath the left sidebar. The problem was that the /home page needed to be wider than this.

Anyway, I tossed out several ideas that seemed like kludges until it finally dawned on me: inheritance was the answer. I added a new template home.stan, a new controller that utilized that template, and ended up with the following:

#index.stan
html [
    body [
        slot ( 'left-sidebar' ),
        slot ( 'content' )
    ]
]

#public.stan
inherits ( 'index' ) [
    override ( 'left-sidebar' ) [
        div ( id = 'left-sidebar' ) [
            xml ( 'left sidebar stuff!' )
        ]
    ],
    override ( 'content' ) [
        div ( id='content', render = vars.content, data = vars.page )
    ]
]

#home.stan
inherits ( 'public' ) [
    override ( 'left-sidebar' ),
    override ( 'content' ) [
        div ( id = 'home', render = vars.content, data = vars.page )
    ]
]

Much better! Now not only had I removed the need for actual logic in the template, but I was able to have a separate id for the home page which allowed me to set the width for content div to a different value than the rest of the site.

Like any language, Python has misfeatures. Some of them are huge (like the distinction between statements and expressions), some are minor unexpected behaviours. This one definitely falls into the latter category. A long-standing wart in Python has been the fact that list.sort() doesn't return a reference to the list. It makes sense that it doesn't because sort() is an in-place sort (for efficiency reasons). However, it's different than most of the other list methods in this respect. I think this makes it a wart. The fact that it's an in-place sort is an implementation detail that I shouldn't need to worry about.

Today I got caught out by something similar to this (although more complicated):

>>> import random
>>> l = range ( 10 )
>>> random.shuffle ( l )
>>> l
[4, 6, 3, 2, 0, 8, 1, 7, 5, 9]
>>> list ( l ).sort ( )
>>> l
[4, 6, 3, 2, 0, 8, 1, 7, 5, 9]
>>>

Looks like sort() didn't do anything, doesn't it? Well something got sorted, it just wasn't what you might expect at first glance. Casting to list() created a new, anonymous list that gots sorted, then immediately discarded.

Now if it was expected behaviour in Python were expected to return a new list (or even a reference to the original list), I'd have probably written

l = list ( l ).sort ( )

and all would have been well.

CleverHarold.

DynamicSite

I'm developing a TurboGears app that I want to aggregate some blogs onto the side of. Because CherryPy works from a thread pool, it seemed unwise to tie up a thread (or more) trying to download and parse RSS feeds from potentially unreliable sources.

Enter Twisted.

Twisted seemed a much better solution for fetching and parsing the feeds. Even more so since there's a recipe in the Python Cookbook on ASPN for doing just this. I used the recipe and added an XML-RPC publisher to it so that I could query the results from TurboGears. Overall I added perhaps a half dozen lines to the recipe and was finished.

I'm always amazed by Twisted and I really, really would like to move completely to it for all my server-side development. Someday I will. Someday, after Twisted-Web has a 'twisted-admin' command that builds me a skeleton application like tg-admin does. Someday when Nevow feels more coherent. Someday when I've got the time and deadlines aren't looming. Someday when I can perhaps implement a couple of these things myself.

Someday...

I use Gentoo on a few of my servers and I've had a love/hate relationship with it for quite a while.

The thing I really like about Gentoo is that it makes maintaining custom packages fairly simple. Compared to a binary distro like Fedora, maintaining modified versions of software is a snap.

On Fedora, I download the src.rpm, unpack it, edit the .spec file, repackage it and install it. It's not a huge deal, but more unwieldy than it needs to be. What really concerns me about doing this is that, like most administrators, I like to keep my updates pretty automatic and worry-free. Having to take manual steps means that I'm more likely to forget that say, PHP needs to have mcrypt support compiled in for some application only to see someone's site go down after an upgrade.

In contrast, on Gentoo I just set a flag in a single file and I'm pretty much done.

I also like the fact that there are no major versions of Gentoo. With Fedora, I can expect to have major downtime at least every year or so when a new release comes out. With Gentoo it's a gentle upward slope (well, not really, but I'm getting to that).

So what's to not like then? Well, I think the problem comes down to this: when I'm working on my servers, I'm in administrator mode. That is, I want to deal with things like service configuration and whatnot. Also, in administrator mode, I'm usually dealing with time-constraints and concern over uptime. The problem Gentoo presents to me, as an administrator, is that it makes me concerned about non-adminstrator things, like the build tool chain. I'm not aware of any other distro that can so easily destroy your build tools after a simple update as Gentoo. When has Fedora ever told me that GCC can't create executables? Never. On Gentoo? It's not quite a regular occurrance, but still way too frequent. Each time I end up spending an hour on Google and Gentoo forums tracking down the solution while the server is as useful as a 500 watt brick.

To boil it down to one line: it makes me, as an administrator, worry about things that are the realm of programmers.

Now, I am actually a programmer, so you might think this is a silly complaint. It would be except for my other two concerns as an administrator: time-constraints and uptime. As a programmer, if my tool chain gets hosed, I'll take my sweet time figuring it out. I develop on a standalone PC, so uptime is no concern. As an admin I don't have this luxury. For the same reasons I don't develop on live servers, I find Gentoo to be a suboptimal choice for live servers. Basically, by running Gentoo, you've made the server into a development machine. Worse, it's someone else's development machine.

Now, in all fairness, the approaches Fedora and Gentoo take are very different and it seems quite likely breaking the toolchain after a GCC/libc upgrade is inevitable (I've been told that SourceMage doesn't have this issue, but I don't know from firsthand experience), but understanding and even excusing a problem doesn't make my problem go away.

These sort of issues make the "gentle upward slope" of Gentoo not so gentle. While I may not have two hours of guaranteed downtime because a major release came out, there will be downtime, only it's not planned-for downtime. It usually happens when you least expect it. With Fedora, I know I'm about to do a major upgrade and can schedule it accordingly. With Gentoo, I'll only know if I religiously follow the Gentoo newsletters (and happen to remember all the breaking updates that have happened since the last time I updated). I'm a busy guy with lots of other concerns. It's far too easy for me to miss that Apache changed its configuration layout a couple weeks ago and find that a minor update sends me scrambling for the forums while my customer's sites are down (this happened a few months ago).

Anyway, I still like Gentoo and the ideas behind it, but it doesn't get installed on new servers anymore.

I've been asked, and have previously considered, whether to add logic tags to TurboStan in order to facilitate "presentation logic". I'm still undecided whether this is a good idea or not.

Despite being firmly in the MVC camp, I am also a firm believer that there is such a thing as "presentation logic" that really belongs in the View (template).

For example, in a blog application, a template designer may want to specify that only 10 recent articles are displayed in a sidebar in one situation, and 20 in another. I don't expect a template designer to delve into the controller and add a one-off method to support this. I think the ability to specify things like this is a valid reason to have logic in a template.

On the other hand, once logic flow is added to templates, as it turns out, that is often the easiest place to do logic and it gets abused. Take a look at any moderately sophisticated Smarty template and you'll see what I mean. It quickly becomes a mess and it appears beyond the discipline of programmers and designers alike to avoid it.

So what's the solution? At the moment I've taken to allowing passing extra arguments to renderers in TurboStan, such as:

div ( id = 'sidebar',
      render = render.sequence,
      data = vars.model.Article.data.articles,
      selectby = { 'published': True, 'limit': 10 } )

Of course the controller has to accept these arguments, but that's easily doable.

The question is, is this enough? Is it inherently better than the alternatives? Should I add an "if" tag to TurboStan? You can already (ab)use list comprehensions and any other Python expression within TurboStan, but I wonder if it's wise to open this can of worms.

If you think I should add some sort of logic tag(s) ( I'm talking to you PacoPablo ), maybe a concrete example of something that can't easily be implemented without them would be convincing. An example that either a) cannot be done or b) is just too damn ugly to talk about would be best.

Personally I've found that my desire for such things has dropped pretty much proportionately to how much I've put aside what I'm used to and swallowed the Nevow pill of render and data callbacks, but I'm always willing to at least entertain other's ideas.

I'm seriously entertaining the idea of deprecating the "vars" prefix attached to variables passed to TurboStan templates. There needs to be some sort of prefix to avoid namespace messes, but I'm thinking a bit more separation is in order: I'm considering passing the model as the "model" object and variables passed from the controller as the "view" object, so you'd see stuff like:

h1 [ view.title ],
div ( id = 'content' ) [
    view.content
],
ul ( id = 'articles', render = render.sequence, data = model.Article.data.articles   [
    li ( pattern = 'item', render = model.Article.render.link )
]

I've debated whether to use "view" or "controller" to group variables and methods passed from the controller, but "controller" is awful long and "view" seems a reasonable alternative (especially since TurboGears will be replacing @expose with @view soon).

My main concern is that this would subtly move TurboStan further away from TurboGears in yet another way. Because TurboStan knows nothing about the model, it would be up to the programmer to properly layout the namespaces in the returned dictionary:

return dict = (
    model = model,
    view = dict (
       title = 'A history of the world',
       content = 'Hello, world'
    )
)

Alternatively, I could rely on the fact that TurboGears puts the model in model.py and import that directly in TurboStan. While helping make things a bit more transparent, it also ties TurboStan more closely to TurboGears, which is a downside for people looking to use it outside of TG.

To date, I'm unsure how I feel about this.

Something has been eating at me for a while. There is a fundamental inconsistency in Python, something that has kept me from writing code how I'd like to write it, but I haven't been able to put my finger on it.

I think I finally have.When I first saw anonymous code blocks in Ruby, I thought "aha, that's it", and I was close. But really anonymous blocks aren't it. They just remind me of it. In fact they are a special case of "it". "It" happens to be expressions. Python makes a distinction between statements and expressions and because my background has been with procedural languages (Pascal, C), this distinction seemed natural. But Python's flexibility hints at something more, something that it has not quite provided but I couldn't describe. I liked the presence of functional tools in Python (map, filter, reduce, list comprehensions, etc), but bemoaned lambda's crippled self. I wanted a ternary operator. I've finally realized these things are all just ways of using statements as expressions, and I've also realized that statements really could be expressions, except that they are intentionally crippled in Python.

As anyone who reads this blog knows, I've been doing lots of work with Stan. Stan is an s-expression style DSL for generating XHTML from Python. I decided early on when creating the TurboStan plugin for TurboGears that I would keep Stan templates as pure expressions, even though I realized this would reduce their power to some extent. My feeling was that too much code in the template was wrong. I still feel that way. However, I don't feel that way about my program itself. I feel the language I use should not restrict me in fundamental ways. Many problem domains lend themselves to a functional style of programming and not having language support for that idiom can be painful. I do acknowledge that functional programs can be more difficult to take in at first glance, but they also tend to be more elegant when properly applied. The same argument can be made about many other concepts, such as recursion. Until you are familiar with the concept, recursive code can be like looking into a hall of mirrors. That certainly isn't a reason to not support it in a language. I don't think anyone would take a language seriously that didn't support recursion. Or object-orientation, for that matter. So why does functional-style programming have such a bad rap with the Python devs?

I admire Phillip Eby's work and find his articles thoughtful, so I emailed him about it (he probably thinks I'm a stalker). Anyway he suggested that having statements as expressions leads to nested code and flat is better than nested. This may be true. However so does nested functions and classes. Further, just because something can be abused doesn't mean it's necessary to ban it from the language, especially if having it easily doubles the expressive power of the language itself. Many hacks that have been added to the Python syntax to support special cases of functional programming would no longer be necessary. The syntax itself would be more self-consistent. I certainly don't find Lisp to be an attractive language visually, but I do find the concept it embodies quite elegant and tempting.

In short, I think Python needs to dump the artificial distinction between statements and expressions.

In the meantime, Phillip was kind enough to point me to Logix, which is more or less exactly what I'm asking for. It even generates Python bytecode.

So provided I'm able to actually leverage Logix for use in my day-to-day work, it may be goodbye Python (sort of) and hello, Logix.

Here's the latest release of TurboStan. It includes arbitrarily deep inheritance, code and content caching, and lots of little fixes.Here's a brief overview of what's new:

1. Arbitrarily deep levels of inheritance. Earlier versions only supported inheriting from a single master template. Now templates can inherit from templates which in turn inherit from other templates.
2. Code caching. Templates are precompiled and stored in memory. This gives an enormous performance boost.
3. Fragment caching. Slots can be cached as static HTML via an argument to the override tag. Further, arbitrary cache expiration schemes can be employed via a callback. Also allows for custom cache identifiers if needed.
4. Absolute paths to templates. The relative path scheme employed before was a mess. Now you can set stan.templates = 'path/to/templates' in your TurboGears config file and that becomes the root for all template paths.
5. Passing an arbitrary namespace to included fragments. You can now add locals = dict ( myvar = 'foo' ) and pack the dictionary with variables you want the included fragment to see.

Here's a short example showing how you might implement a time-based caching scheme (i.e. expire the cache at midnight):

controllers.py:

from calendar import weekday, day_name
from time import time, localtime

class Root ( controllers.RootController ):

    @turbogears.expose ( template = 'stan:.public' )
    def index ( self, *args, **kwargs ):

        def render_content ( self ):
            now = tuple ( localtime ( time ( ) ) [ :3 ] )
            today = day_name [ weekday ( *now ) ]

            return ( ''
                This could be pulled from a database or some other source.
                I explain why this is a callback below.
                And to help make the time-based cache expiration make
                a little sense, we'll print what day it is:
                Today is %s.
            '' % today )

        def cache_valid ( self ):
            ''
            Function should return True if the cache content is valid, False if it should
            be flushed and render_content ( ) asked to regenerate the data.
            ''
            return not isMidnight ( )

        return dict (
            render_content = render_content,
            cache_valid = cache_valid
        )

The reason I'm using a callback to generate the content rather than just passing the content back in the return value of the controller is because I don't want to regenerate the content if it's already in the cache. Remember that the controller method is called on every page access prior to rendering the template and prior to checking the cache. This means that if we were to say, pull the content from the database, this database call would be made regardless of whether we later used the cache or not. By implementing this as a callback, we can guarantee that no extraneous functions are called when the cache is used.

Now our index.stan template:

html [
    head [
        title [ 'TurboStan Example' ]
    ],

    html [
        h1 [ 'Welcome to TurboStan!' ],
        div ( id = 'main-content' ) [
            slot ( 'content' )
        ]
    ]
]

And finally, our public.stan template. Note that this is the template referenced in the controller. Index.stan is referenced via inheritance:

inherits ( 'index' ) [
    override ( 'content', cached = vars.cache_valid ) [
        xml ( vars.render_content ( ) )
    ]
]

You can download TurboStan 0.8.7 `here`_.

Feel free to contact me with bug reports, feature requests, etc...

here: http://www.develix.com/downloads/TurboStan/TurboStan-0.8.7.tgz

Timeline is a DHTML-based AJAXy widget for visualizing time-based events. It is like Google Maps for time-based information.

Hm, maybe I should integrate this with my calendar control I've written for TurboGears...

Over at On the Edge, Greg Black has written an article explaining why he's dumping Python.

No, it isn't for Ruby, this time it's for AWK.While I can understand Greg's complaint about changing (even undocumented) behaviour of a standard library function in an incompatible way, Greg's conclusion fails to convince me that this was anything more than a tantrum. To be fair, I have tantrums all the time about software, so I can sympathize. However, Greg also wrote a follow-up article clarifying and solidifying his stance. This takes it beyond a tantrum (which tends to be a temporary thing that you get over rather quickly) into the realm of actual opinion. Therefore I feel compelled to address this opinion with some of my own.

What I found most striking is the deep and fundamental contradiction plainly visible in Greg's stance:

Greg wants to use a dead language, but Python 2.3 isn't going to cut it. Why not? Python 2.3 is certainly as dead as Awk by any measure. If Greg wants an unchanging Python he's found it and yet still complains. This is what I find contradictory. He wants both a living language (i.e. one that is updated) and a dead language (one that doesn't have changes). He's even stumbled upon the perfect solution (locking to 2.3), but is now heart-set upon finding some mythical language that never breaks backwards compatibility (don't try Ruby Greg! They can't even get a book out the door before it's obsolete) but still ships new versions that anyone cares about.

To further the irony, I've taken the title of this article directly from Greg's page where it's presented as some sort of motto above a picture of a BMW taking a corner on two wheels. Apparently life-or-death for a thrill is okay by Greg, but changing some parameters (or filing a bug report) in exchange for a cleaner library or language is a bit too "on the edge."

Go figure.

Having solved the inheritance problem, my next goal is to implement some caching features in TurboStan.There are many issues around caching dynamic content and here's some of my preliminary thoughts with regard to caching TurboStan:

1. Stan is basically a Python expression, so caching .pyc files is an easy option that will give at least minor improvements.
2. Caching and serving static HTML is going to be the fastest.
3. Most pages are a combination of static and dynamic content, so the programmer should be able to specify whether a fragment can be considered static. This way even a dynamic page can benefit from caching the bits of content that are static as HTML.
4. Even dynamic fragments are quite often temporarily static. That is, they change infrequently, so these should be able to be cached as HTML given there is a trigger to indicate when the cache should be rebuilt.
5. Session-based caching might be handy as well. Content may be dynamic across users but static per user. Having a mechanism to support this could be beneficial, but cache storage might explode, so this would need to be carefully considered.

Most of this is pretty standard I think, the question becomes how to specify these concepts within the constraints of TurboGears. Since templates are specified via a decorator, that seems one logical place to pass this information, however, due to the fact that inheritance may bypass this decorator that doesn't seem likely to work. Also I'm not certain that would give the fine-grained cache control I'm looking for.

The solution I'm leaning toward is to specify the cache directives directly within the template. While this will add some clutter to the template, it will also allow the programmer/designer to have cache control tied directly to the item being cached. Also, because this puts cache control directly within TurboStan, it will help insulate the caching system and reduce worries about some future caching system in TurboGears conflicting with whatever scheme I come up with.

# proposed caching directives within a fragment

inherits ( 'index' ) [
    override ( 'title' ) [
        cache [ # static cache, expires only on template change
            xml ( ''
                This never changes
            '' )
        ]
    ],

    override ( 'content' ) [
         cache ( expires = vars.cache_expired ) [
             # expires if the content_changed variable is True
             vars.content
         ],
         cache ( expires = vars.time_expired ) [
             # expires based on static time, i.e. at midnight
             vars.remote_rss_feeds
         ]
    ]
]

What I'm thinking here is that the expires keyword will take a function callback that will return a boolean informing it whether to expire the cache or not. TurboStan can then internally keep a table of these callbacks for each fragment and call them prior to rendering the fragment and if they return False, then instead simply return the static HTML from its cache. If no expires keyword is given, then a failed lookup in that table will return False by default (i.e. the content never expires).

I've never implemented a caching system prior to this so I'm sure there's plenty I'm overlooking. Regardless, I suspect implementation will reveal most of these.

Another idea that occurs to me is to simply provide a communication method with an external cache (i.e. Squid) much like Zope does. By this I mean, don't implement caching of actual static content within TurboStan, but rather simply do cache control, notifying the external cache when the cache is dirty. The downside to this is that it requires an external cache, which isn't always a pleasant option for small sites, so perhaps the ultimate solution would be to provide a simple caching hook within TurboStan that lets the developer choose between internal and external caching.

Sounds like fun.

A longstanding limitation in TurboStan has been that inheritance could only be one level deep. I finally got tired of this and made TurboStan recursively render parent templates. The results were surprising.This limitation seemed a minor nit early in TurboStan's development. After all, Stan itself doesn't even support the concept of inheritance[*]. I added inheritance for two reasons: 1) Kid, TurboGear's primary templating engine supports it and I didn't want TurboStan to lack anything users of Kid might be used to and 2) the CherryPy dispatch system really works best using an inheritance rather than an include model for building pages.

However, I've been trying to refine how I organize my TurboGears applications and that includes keeping the template structure tidy and reducing code duplication in templates. With this in mind, I wanted to use the following structure:

index.stan +-- public.stan +--- page1.stan
           |               |
           |               +--- page2.stan
           |
           +--- admin.stan +--- admin1.stan
                           |
                           +--- admin2.stan

By doing this, I could put the general layout in index.stan, fill in general stuff from public.stan, and provide the content and other custom stuff from each page's template. Further, the admin interface inherits from index.stan, and so acquires the same look as the public face, but can remove or add sections in admin.stan, etc.

So far so good.

Except that this model requires that page[12].stan and admin[12].stan be able to inherit from public.stan and admin.stan which in turn inherit from index.stan. This wasn't doable with TurboStan prior to 0.8.6. So I fixed it by having templates recursively render parent templates. All in all only an hour or so of work, not much.

However, the benefits were immediately noticeable. The target structure has become very elegant, with child templates doing the bare minimum to fill in slots in their parents and index.stan truly controlling the layout of the site. I knew this would be the result, but until I'd actually converted my templates to use the new layout I guess I just failed to realize what a boon this would truly be.

# index.stan

html [
    head [
        title [ 'example ],
        link ( href = "/css/style.css", rel = "stylesheet", type = "text/css" ),
        slot ( 'css' )
    ],

    body [
        div ( id = 'menu' ) [ slot ( 'menu' ) ],
        div ( id = 'content' ) [ slot ( 'content' ) ],
        div ( id = 'sidebar' ) [ slot ( 'sidebar' ) ]
    ]
]

The public view fills in the content, menu and sidebar slots:

# public.stan

inherits ( 'index' ) [
    override ( 'css'), # we'll just use the default stylesheet

    override ( 'menu' ) [
        ul [
            li ( href = '/' ) [ 'Home' ],
            li ( href = '/page1' ) [ 'Page 1' ],
            li ( href = '/page2' ) [ 'Page 2' ],
            li ( href = '/admin' ) [ 'Admin' ]
        ]
    ],

    override ( 'content' ) [
        xml ( ''
            <p>
              Hello, world<br />
              This is some content.  It could have also
              been provided via a variable or several other
              methods, but this keeps it simple.
            </p>
        '' )
    ],

     override ( 'sidebar' ) [
         p [ 'This is some sidebar text' ]
     ]
]

The admin view will remove the sidebar and set a custom menu:

# admin.stan

inherits ( 'index' ) [
    override ( 'css' ) [
        # we'll add some custom CSS for the admin interface here
        link ( href = "/css/admin.css", rel = "stylesheet", type = "text/css" )
    ],

    override ( 'menu' ) [
        ul [
            li ( href = '/admin1' ) [ 'Admin 1' ],
            li ( href = '/admin2' ) [ 'Admin 2' ],
            li ( href = '/' ) [ 'Main Site' ]
        ]
    ],

    override ( 'content' ) [
        div [
            h3 [ 'Hello, administrator' ],
            'Please do your admin duties here'
        ]
    ],

    # sidebar will disappear from the generated output
    override ( 'sidebar' )
]

And finally, page1.stan and admin1.stan will replace the content slot but leave the rest intact:

# page1.stan

inherits ( 'public' ) [
    override ( 'content' ) [
        div [ 'This is the page1 template.' ]
    ]
]

# admin1.stan

inherits ( 'admin' ) [
    override ( 'content' ) [
        div [ 'This is the admin1 template.' ]
    ]
]

Because you can inherit to arbitrary depths, children can "delete" slots in parent templates or add new content with ease. Also, note that the overrides on the css slot cause the <link> tags to get injected between the <head> tags as they should be.

You can download the latest TurboStan here.

The current version (as of this writing) is 0.8.6.

• To be clear, Stan has the slot directive which does the heavy lifting and is what I've leveraged to implement inheritance, but it was meant to be used procedurally from Python code; that is, you would define a slot in Stan, then some Python function would call fillSlots( ). Since a goal for TurboStan was to keep the syntax as a pure s-expression within templates, I added the inherits keyword as a way to express this concept directly within the template without having to resort to doing it procedurally.

My previous article was quite inflammatory: it was a personal rant never meant to see wide dissemination. I make no apologies for its content, however I do apologise to those who came expecting to see a reasoned argument supporting my assertions and came away rightly disappointed. I'll try to remedy that here and perhaps spark a more intelligent discussion than what was had previously.

Before we start, if your comment consists of "you're just a two-bit blogger looking for fame" or some variation of, try to save yourself the pitiful fate of self-fulfilling prophecy. Posting a comment under those pretenses is a bit like trying to put a store out of business by buying all their merchandise. It's probably not going to work. If you don't want me to be famous, go away. It's really for the best.

Part 1: PHP Points the Gun at Your Head

A common adage is that [insert language or tool] will let you shoot yourself in the foot. PHP helps you out even further by just finishing you off for good.

I just thought I'd get that out of the way in case anyone made the mistake of thinking I had softened on my position any.

Let's start out with a list of assumptions, of which I don't think there can be much disagreement

1. PHP is widely assumed to be a good language for beginners.
2. PHP is mainly targeted at web applications
3. PHP has a configuration file with dangerous options in it
4. Many poorly-written PHP applications require that one or more of those options be turned on in order to function.

Whether the designers of PHP intended it to be or not, PHP is widely considered to be a good beginner's language, probably due to its low barrier to entry. Whether they wanted it or not, that's what they've got and this fact must be recognized and accounted for.

Let's go over the various settings in php.ini that can open serious security holes:

1. register_globals - this injects user-supplied data into the global namespace
2. allow_url_fopen - allows remote URLs to be opened as if they are local files
3. magic_quotes_gpc - pretends to protect you from malicious input

Now, before we start, let's just know that these days, PHP ships with these turned off by default. You have to enable them yourself. Exactly how much protection does this provide? Well it would depend on your level of expertise and whether you realized the implications of what you were doing. Many popular PHP apps require that one or more of these be turned on in order to function. If you are a novice PHP programmer (or just a hapless user following installation instructions), then you are quite likely to just do what you are told by people who apparently know more than you (the developers of said PHP application would appear to be experts). And after all, if they were that dangerous, why would the PHP devs provide them in the first place. That's a very good question.

Now, it's been argued that no moderately knowledgable PHP user would fall prey to such well-known exploits. I think the Mambo developers would beg to differ. Mambo, just in the last few months, fell prey to a combination of register_globals and allow_url_fopen which allowed malicious users to overwrite a global variable which contained the name of a Mambo include file and force Mambo to include arbitrary (and I'd expect malicious) PHP code from anywhere in the world and execute it. Now if the PHP experts working on Mambo can't avoid this obvious pitfall, what chance does a beginner have? That's a rhetorical question, btw.

magic_quotes_gpc is a bit more benign, not providing any direct exploits itself, but rather lulling the programmer into a false sense of security. It purports to sanitize user input to help prevent SQL injection attacks and the like. The problem is it only works on a subset of potential malicious input, doesn't deal with two byte characters and also prevents you from doing it right and using prepare() or one of its variants (you get double escaping if you do both, so backslashes appear in your data). Therefore if, magic_quotes_gpc is enabled, the programmer will have to work that much harder to sanitize input. Except of course they probably won't. If you do some quick tests, it appears to work just fine. Most beginners probably wouldn't even do that, but even if they did, there would be little to raise alarms.

Now, if a beginner were just writing their own code, chances are they would not tamper with php.ini. However, I expect that most beginners are going to install one or more existing PHP applications: a blog, a bulletin board... whatever. There's at least a respectable chance that one or more of these applications will want one or more of those settings on in order to function. The stage is set for that user to unwittingly write exploitable code.

If PHP is to truly be a beginner's language (or apparently a language for anyone, judging from the Mambo exploit), then it needs to be a little more difficult to point the gun at your head. Ideally those options would just be removed altogether from PHP. If for some unfathomable reason this can't be done, then perhaps leaving them as compile-time flags would be slightly better (although still not a good idea IMO).

We are talking about a language, targeted at beginners, interfaced via the most hostile environment possible (the internet), with all the cards stacked against the programmer. What could possibly go wrong?

Wifi-Radar

This isn't a particularly original thought (it's been covered many times over), but it's a good summary of the dangers of today's Internet:

Why Web 2.0 will end your privacy

I'm sure everyone is thinking "yes, I'm aware of the problem". If that's the case, then why aren't you doing something about protecting your privacy today, before it's too late?

I've waited for X to get this forever (well, not so much the extensions, but what they provide). Finally dropshadows. I don't consider dropshadows so much eye-candy (well, a little bit), but more like eye-froot-loops. Tasty, yet still nutritious. Mostly.I've managed to get x.org running dual head with the binary NVidia driver including the render/composite/damage extensions. Here's my xorg.conf:

Section "ServerLayout"
        Identifier     "Multihead layout"
        Screen      0  "Screen0" LeftOf "Screen1"
        Screen      1  "Screen1" 0 0
        InputDevice    "Mouse0" "CorePointer"
        InputDevice    "Keyboard0" "CoreKeyboard"
        Option      "Xinerama" "off"
        Option      "Clone" "off"
EndSection

Section "Extensions"
    Option "Composite" "Enable"
    Option "RENDER" "Enable"
    Option "DAMAGE" "Enable"
EndSection

Section "Files"
        RgbPath      "/usr/X11R6/lib/X11/rgb"
        ModulePath   "/usr/X11R6/lib/modules"
        FontPath     "unix/:7100"
EndSection

Section "Module"
        Load  "dbe"
        Load  "extmod"
        Load  "fbdevhw"
        Load  "record"
        Load  "freetype"
        Load  "type1"
        Load  "glx"
        Load  "dri"
        Load  "GLcore"
EndSection

Section "InputDevice"
        Identifier  "Keyboard0"
        Driver      "kbd"
        Option      "XkbModel" "pc105"
        Option      "XkbLayout" "us"
EndSection

Section "InputDevice"
        Identifier  "Mouse0"
        Driver      "mouse"
        Option      "Protocol" "IMPS/2"
        Option      "Device" "/dev/input/mice"
        Option      "ZAxisMapping" "4 5"
        Option      "Emulate3Buttons" "yes"
EndSection

Section "Monitor"
        Identifier   "Monitor0"
        VendorName   "Monitor Vendor"
        ModelName    "ViewSonic G90f-2"
        HorizSync    30.0 - 97.0
        VertRefresh  50.0 - 180.0
        Option      "dpms"
EndSection

Section "Monitor"
        Identifier   "Monitor1"
        VendorName   "Monitor Vendor"
        ModelName    "ViewSonic PF790"
        HorizSync    30.0 - 97.0
        VertRefresh  50.0 - 180.0
        Option      "dpms"
EndSection

Section "Device"
        Identifier  "Card0"
        Driver      "nvidia"
        VendorName  "Videocard vendor"
        BoardName   "NVIDIA GeForce FX (generic)"
        Option "AllowGLXWithComposite" "true"
        Option "RenderAccel" "true"
        Option "BackingStore" "true"
        Option "TwinView" "true"
        Option "ConnectedMonitor" "CRT, CRT"
        Option "MetaModes" "1280x1024,1280x1024"
        Option "AllowGLXWithComposite" "true"
        # Option "NoRenderExtension" "true"
        Option "HorizSync"   "CRT-0: 50-110;  CRT-1: 50-110"
        Option "VertRefresh" "CRT-0: 60-120;  CRT-1: 60-120"
EndSection

Section "Screen"
        Identifier "Screen0"
        Device     "Card0"
        Monitor    "Monitor0"
        DefaultDepth     24
        SubSection "Display"
                Viewport   0 0
                Depth     24
                Modes    "1280x1024" "800x600"
        EndSubSection
EndSection

Section "Screen"
        Identifier "Screen1"
        Device     "Card0"
        Monitor    "Monitor1"
        DefaultDepth     24
        SubSection "Display"
                Viewport   0 0
                Depth     24
                Modes    "1280x1024" "800x600"
        EndSubSection
EndSection

Section "DRI"
        Group        0
        Mode         0666
EndSection

Upon starting X, I run the following command:

xcompmgr -c -f -F -r 10 -l -12 -t -9 -o 0.75 &

This gives me nice drop shadows, fade-in/out effects, etc.

The system isn't without bugs. Occassionally resizing a window will leave artifacts lying around, windows playing video seem to have issues deciding their z-index, etc. And sometimes (albiet quite rarely these days), X will just go to lunch.

I highly recommend using the absolute latest version of the NVidia driver as most of the fixes seem to be coming from that end.

Have fun.

Zabbix

Boxover, a nifty little JavaScript library for doing tooltips.

NOTE: I've just migrated a ton of old articles from Frog (the old blogging software I was using) to this blog and I've yet to convert the comments to the new markup format. Also I deleted a shitload of spam and idiotic comments, so if I accidentally deleted yours, sorry. If you were a spammer or shithead, not so sorry. That should explain any disjointed comments though (and no, I didn't delete all the negative ones, just the ones that added so little as to not be worth the 50 or so bytes of bandwidth they cost).

Please read the latest article before flaming this one. This one will still be here when you get back. I promise.

Open source software has fought a long battle to get where it is today, and at least 3/4 of the so-called LAMP (Linux, Apache, MySQL, PHP) stack are under vigorous attack by many, myself included. Why?

I can't speak for everyone, or for that matter, anyone but myself, but I'm good at doing that so here goes.

I'm quite opposed to using MySQL and PHP, and I'm none too fond of Apache. Anyone who knows me or happens into a conversation with me about development quickly learns of my distaste for these particular projects. To be fair, Apache is the least problematic of the three and if there were no alternatives, I'd use it without a lot of complaint.

MySQL and PHP, on the other hand, really raise my ire. Both of them have two major problems:

1. Bug ridden (by this I am including both misfeatures as well as actual bugs).
2. They encourage bad habits.

I'll start with MySQL. I learned databases on MySQL and used it for several years. Then I discovered PostgreSQL and realized that in fact, I'd learned nothing of databases. For years the MySQL developers were quite vocal that things like referential integrity, transactions, subselects, etc were little more than baggage that could be done better another way. They were after speed, pure and simple. Fair enough, in some respect, since the job MySQL was originally developed for didn't require any of those things. The problem is that a whole generation of database programmers believed them, despite the fact that their applications did require those things. Someone in authority told them they didn't and they bought it (and still buy it). Let me clue you in: you need these things or you need to let someone else handle your database work for you.

Another nit I have with MySQL is the ever-changing ways you must interact with it. Some versions have certain command-line flags, and later versions will ignore those flags (or fail if you pass them),

IMO, MySQL once filled a niche (cheap data store) that today is better filled by SQLite but has moved out of it into a area where it doesn't compete nearly as well (that of an enterprise database) and no longer fits in either. If you need a dumb data store, use SQLite. If you need a FOSS database, use PostgreSQL. They cost the same as MySQL (free) and each does a better job of doing what it was designed for than MySQL.

PHP is another sore spot for me. I've gotten to the point that not only will I not write PHP code, I won't even run applications written in PHP (my long search for decent blogging software was due to the restriction that it not be written in PHP). At some level PHP is a great language because the entry cost is so low. Not so much because the language is so particularly friendly, but because it was designed to work in an extremely simple environment (the web) and because it's quite possible to learn PHP incrementally by intermixing it with HTML. So what's the problem? Well, first of all, as anyone who's done much web programming will tell you, mixing code with markup is not a good thing if you care about maintenance or extensibility. The very thing that makes PHP a great language for beginners is the very thing that makes it a bad language for beginners. At some point they will have to unlearn those habits, except that usually they don't. Also, because it's so easy to whip out a quick PHP webapp, many, if not most, PHP programmers fail to delve very deep into the realm of programming, preferring to sit at the edge and reap the benefits without the work (I'm not talking about coding work, rather the work of understanding your field). PHP programmers practically popularized the most common attack in the world, the SQL-injection attack. Not only is it the most common, it's the most easily avoided. That's how shallow most PHP-programmer's knowledge is. "Professional" programmers are still assembling SQL queries by concatenating strings.

PHP and MySQL are this generation's BASIC, the language that was described thusly by the Free Online Dictionary of Computing

BASIC has become the leading cause of brain-damage in proto-hackers. This is another case (like Pascal) of the cascading lossage that happens when a language deliberately designed as an educational toy gets taken too seriously. A novice can write short BASIC programs (on the order of 10-20 lines) very easily; writing anything longer is (a) very painful, and (b) encourages bad habits that will make it harder to use more powerful languages well. This wouldn't be so bad if historical accidents hadn't made BASIC so common on low-end micros. As it is, it ruins thousands of potential wizards a year.

Replace BASIC with PHP or MySQL (and "low-end micros" with "cheap hosting") and you've got today's most common programmer. Worse, the most common programs in existence today mix the two in a brain-freezing mixture of stupidity.

MySQL and PHP have helped quite a bit in bringing open source into the mainstream. Let's thank them for that and move on, graciously ignoring the millions of minds they've destroyed.

From the sAsync site:

"sAsync offers asynchronous access to the outstanding SQLAlchemy package by Michael Bayer. It also provides SQLAlchemy enhancements including persistent dictionaries, text indexing and searching, and an access broker for conveniently managing database access, table setup, and transactions. Everything can be run in an asynchronous fashion using the Twisted framework and its deferred processing capabilities."

Nufox: XUL from Python

and so much more:

Schevo

Apparently, in an attempt to "embrace and extend" Open Source, Microsoft released some developers they'd previously kept in cages to contribute to certain OSS projects.

At least that's my theory:

(13:27:28) Cliff: apparently in some versions of mysql
(13:27:41) jms@jabber.imr-net.com: nod
(13:27:42) Cliff: theres an option --default-storage-engine=innodb
(13:27:45) Cliff: in others
(13:27:46) Cliff: there isn't
(13:28:04) jms@jabber.imr-net.com: haha
(13:28:05) jms@jabber.imr-net.com: nice
(13:28:11) jms@jabber.imr-net.com: What versions?
(13:28:19) jms@jabber.imr-net.com: You're not using innodb somewhere, or you are?
(13:28:49) Cliff: trying to
(13:28:57) Cliff: i'm not, a hosting customer is
(13:29:04) Cliff: i won't use mysql
(13:29:22) jms@jabber.imr-net.com: Wow
(13:29:23) jms@jabber.imr-net.com: Bitter
(13:29:27) Cliff: it's crap
(13:29:36) jms@jabber.imr-net.com: I wish you would blog this.