Mind Blowing

For the uninitiated, there are a few great articles online that go into great detail about what Python descriptors are, and what they can do. Descriptors are a notoriously mind boggling topic that has left many a curious developer cross-eyed and drooling, and usually more confused than we he started. But they’re simple to understand, really: They’re just object instances posing as dynamic properties which are defined on a class, evaluated at runtime by a method on the descriptor object itself, that are accessible through either an object instance, or statically through a property on the the class itself!

Yeah. "Simple."

To be honest, most of the difficulty in understanding descriptors stems from the fact that (especially in PHP,) classes aren’t really thought of as "objects" much. Classes in general are a little bit "quirky" in PHP, being referenced as a quoted class name string half the time, an unquoted global class name the other half, and as a special keyword the other-other half! Only now are the more advanced class related tools even appearing in the PHP world, and they’re still not complete (which will really screw us later in this article). Plus there’s just not that much information about down and dirty class-level manipulation for PHP out there. Where the official documentation is lacking, it’s only supplemented by helpless user comments flailing wildly trying to understand the murky examples. It’s a real shame because ultimately, we as developers fall victim to the catch 22 of PHP’s evolution as a language: Features will be added after widespread adoption of the feature." Progress happens very slowly because developers lack the understanding (or perhaps the vision) to take advantage of some of these features – much less successfully lobby for improvements in the language itself. Regardless, I’ll admit I never really even thought of classes in such a dynamic way until I switched to Python. So, if you’ve never been exposed to Python and its very object-like class syntax, then it’s going to be an even longer road to enlightenment! No worries though, let’s just break it down slowly and see what descriptors really look like: Descriptors are…

<?php
    // Define the class.
    class TheClass
    {
        // Declare the property.
        public $the_property;
    }

    // Create the object instance.
    $obj = new TheClass();

    // Access the object instance's property.
    echo 'The property: ' . $obj->the_property;

<?php
    // Access a nonexistent property.
    echo $obj->bogus_property;

<?php
    // Define the class.
    class DynamicClass
    {
        /**
         * Returns the value of a dynamic property.
         *
         * This method will issue a 'Notice' level error if a nonexistent
         * property is requested.
         *
         * @param string $name The name of the property to get.
         *
         * @return mixed
         */
        public function __get($name)
        {
            // Check the name of the requested property.
            if ($name == 'the_property') {

                // Return the dynamic value for 'the_property' property.
                return time();
            }

            // Trigger an error if no property was found with the name.
            trigger_error(sprintf(
                'Undefined property: %s::$%s',
                get_class($this),
                $name
            ));
        }
    }

    // Create the object instance.
    $obj = new DynamicClass();

    // Access the object instance's property.
    echo 'The property: ' . $obj->the_property;

<?php
    class StaticPropertyClass
    {
        // Declare the static property.
        public static $the_static_property;

        // Declare the property.
        public $the_property;
    }

    // Create the object instance.
    $obj = new StaticPropertyClass();

    // Access the object instance's property.
    echo 'The property: ' . $obj->the_property;

    // Access the static class property.
    echo 'The class property: ' . StaticPropertyClass::$the_static_property;

    // This will produce an error.
    echo $obj->the_static_property;

    // And so will this.
    echo StaticPropertyClass::$the_property;

<?php
    // Define the descriptor class.
    class DescriptorClass
    {
        /**
         * Calculates and returns the dynamic value for the property
         * associated with the descriptor instance.
         *
         * @return mixed
         */
        public function getDescriptor()
        {
            // Do some special calculations.
            return time();
        }
    }

    // Create a descriptor.
    $the_descriptor = new DescriptorClass();

    // Add the descriptor to the class.
    StaticPropertyClass::$the_static_property = $the_descriptor;

<?php
    // Create the object instance.
    $obj = new StaticPropertyClass();

    // Access the dynamic property, calculated by the descriptor instance.
    echo 'Current time: ' . $obj->the_static_property;

<?php
    // An interface that all descriptors will implement.
    interface DescriptorInterface
    {
        /**
         * Returns the value of the descriptor when accessed.
         *
         * @param object $instance The object instance on which the descriptor was
         *                         accessed. 'null' will be passed if accessed the
         *                         descriptor was accessed statically.
         * @param string $owner    The name of the class which owns the descriptor.
         *
         * @return mixed
         */
        public function getDescriptor($instance, $owner);
    }

<?php
    // Define a descriptor.
    class ChattyDescriptor implements DescriptorInterface
    {
        // Calculate the dynamic value.
        public function getDescriptor($instance, $owner)
        {
            return sprintf(
                "Hey %s, you're a %s.",
                $instance->name,
                $owner
            );
        }
    }

    // Define a simple class.
    class Dude
    {
        public $name;
        public static $descriptor;

        // Assign the name at creation.
        public function __construct($name)
        {
            $this->name = $name;
        }
    }

    // Add the descriptor instance to the class.
    Dude::$descriptor = new ChattyDescriptor();

    // Create an object instance.
    $obj = new Dude('Broseph');

    // Access the descriptor.
    echo $obj->descriptor;

<?php
    // Define a base class for objects that will use descriptors.
    abstract class Descriptable
    {
        // Intercepts requests for nonexistent properties.
        public function __get($name)
        {
            // Get the name of this class.
            $class = get_class($this);

            // Get an introspection reflection for the class.
            $class_ref = new ReflectionClass($class);

            // Check for a static class property with the given name.
            $attr = $class_ref->getStaticPropertyValue($name, null);

            // If the class property is an object, check for a descriptor.
            if (is_object($attr)) {

                // Get an introspection reflection for the property.
                $attr_ref = new ReflectionClass(get_class($attr));

                // Check to see if the static property is a descriptor.
                if ($attr_ref->implementsInterface('DescriptorGet')) {

                    // Call the descriptor method and return the value.
                    return $attr->getDescriptor($this, $class);
                }
            }

            // Trigger an error if no descriptor was found.
            trigger_error(sprintf('Undefined property: %s::$%s', $class, $name));
        }
    }

<?php
    // Define a simple class.
    class WorkingDude extends Descriptable
    {
        public $name;
        public static $descriptor;

        // Assign the name at creation.
        public function __construct($name)
        {
            $this->name = $name;
        }
    }

    // Add the descriptor instance to the class.
    WorkingDude::$descriptor = new ChattyDescriptor();

    // Create an object instance.
    $obj = new WorkingDude('Brosephus');

    // Access the descriptor (works!)
    echo $obj->descriptor;

<?php
    // Define a simple class.
    class UnemployedDude extends Descriptable
    {
        public $name;
        public static $descriptor;

        // Assign the name at creation.
        public function __construct($name)
        {
            $this->name = $name;
        }
    }

    // Add the descriptor instance to the class.
    UnemployedDude::$descriptor = new ChattyDescriptor();

    // Access the descriptor on the class (doesn't work!)
    echo UnemployedDude::$descriptor . " You don't even exist, you lazy bum!";

Meet the Family

Up to this point, we’ve ignored a huge detail. In reality, "getting" isn’t the only access method supported by Python descriptors. There is also support for dynamic value "setting" and "deleting." These actions roughly correspond to the PHP magic methods __set and __unset, and should be fairly self explanatory. Descriptors intercept property "setting" and "unsetting" just as they do for property "getting" currently. To fill out our descriptor implementation, we should define some new interfaces to define how these new methods will be called by the descriptable class.

<?php
    // The descriptor interface for "set" access.
    interface DescriptorSet
    {
        /**
         * Sets the value of the descriptor.
         *
         * @param object $instance The object instance on which the descriptor was
         *                         accessed.
         * @param mixed  $value    The value that was given to the descriptor.
         *
         * @return null
         */
        public function setDescriptor($instance, $value);
    }

    // The descriptor interface for "unset" access.
    interface DescriptorUnset
    {
        /**
         * Unsets the descriptor's value.
         *
         * @param object $instance The object instance on which the descriptor was
         *                         accessed.
         *
         * @return null
         */
        public function unsetDescriptor($instance);
    }

The setDescriptor method will be called when setting the value of a descriptor property like $obj->descriptor = 'new value'; and will receive the new property value in the $value parameter. Note that the unsetDescriptor method only receives the object instance as a parameter. If you’re wondering what happened to the $class parameter that we used in getDescriptor, good catch! It turns out, Python descriptors do not allow static access for "setting" and "deleting (unsetting)" descriptor properties. The reasoning is simple: if these methods were allowed for static property access, you could never remove the descriptor instance from the class. A full restart of the program would be required to empty the static class property. That just isn’t practical, so the Python authors left the feature out completely to prevent confusion. This is an extra bonus for us, since we can’t use static class property access at all with descriptors in PHP. That means we’re really only missing out on the one single use-case.

There is one other detail to descriptors that’s specific to PHP. Many PHP developers use isset() to evaluate whether a property exists and is valid. Unfortunately isset() will return false for any undeclared property, even if we’ve overridden the __get method to return a value. To accurately simulate a real property, we need to override the magic __isset method to return true if a property is evaluated dynamically. With this last piece of the puzzle, we are able to construct a robust descriptable class, completing our PHP descriptor support:

<?php
    // The base descriptor interface.
    interface Descriptor
    {
    }

    // The descriptor interface for "get" access.
    interface DescriptorGet extends Descriptor
    {
        /**
         * Returns the value of the descriptor when accessed.
         *
         * @param object $instance The object instance on which the descriptor was
         *                         accessed. 'null' will be passed if accessed the
         *                         descriptor was accessed statically.
         * @param string $owner    The name of the class which owns the descriptor.
         *
         * @return mixed
         */
        public function getDescriptor($instance, $owner);
    }

    // The descriptor interface for "set" access.
    interface DescriptorSet extends Descriptor
    {
        /**
         * Sets the value of the descriptor.
         *
         * @param object $instance The object instance on which the descriptor was
         *                         accessed.
         * @param mixed  $value    The value that was given to the descriptor.
         *
         * @return null
         */
        public function setDescriptor($instance, $value);
    }

    // The descriptor interface for "unset" access.
    interface DescriptorUnset extends Descriptor
    {
        /**
         * Unsets the descriptor's value.
         *
         * @param object $instance The object instance on which the descriptor was
         *                         accessed.
         *
         * @return null
         */
        public function unsetDescriptor($instance);
    }

    // The base class for all descriptor-bearing subclasses.
    abstract class Descriptable
    {
        /**
         * Finds and returns a descriptor instance for the class.
         *
         * This method will return null if either a descriptor was not found in
         * the class property with the specified name, or if a descriptor was
         * found but does not implement the requested interface. Passing the
         * default interface name 'Descriptor' will return any type of descriptor
         * as long as it is stored in the correct class property.
         *
         * @param string $name  The name of the descriptor property being accessed.
         * @param string $iface The name of the descriptor interface which must
         *                      be supported by the descriptor instance.
         *
         * @return mixed
         */
        protected function _descriptorInstance($name, $iface='Descriptor')
        {
            // Get an introspection reflection for the class.
            $class = get_class($this);
            $class_ref = new ReflectionClass($class);

            // Get the static class property with the given name.
            $attr = $class_ref->getStaticPropertyValue($name, null);

            // Check to see if the property is a descriptor instance.
            if (is_object($attr)) {

                // Get an introspection reflection of the property.
                $attr_ref = new ReflectionClass(get_class($attr));

                // Check to see if the static property has the right interface.
                if ($attr_ref->implementsInterface($iface)) {

                    // Return the found descriptor.
                    return $attr;
                }
            }
            // Return null since we didn't find a matching descriptor.
            return null;
        }

        /**
         * Finds and runs a descriptor method for the class.
         *
         * This method will run the specified descriptor method of the descriptor
         * with the provided name if it exists. The method may be one of 'get',
         * 'set', or 'unset'. Any arguments provided in the '$args' array parameter
         * will be passed to the descriptor method. This method will return 'null'
         * if no matching descriptor instance was found.
         *
         * @param string $method The name of the descriptor method to run.
         * @param string $name   The descriptor property name.
         * @param array  $args   An array of arguments to pass to the descriptor
         *                       method or 'null'.
         *
         * @return mixed
         */
        protected function _descriptorAccess($method, $name, $args=null)
        {
            // Initialize descriptor method arguments.
            if (is_null($args)) {
                $args = array();
            }

            // Get the name of the required descriptor interface.
            $iface = 'Descriptor' . ucfirst($method);

            // Retrieve the descriptor instance matching the name and interface.
            $attr = $this->_descriptorInstance($name, $iface);

            // Check for a valid descriptor instance.
            if (!is_null($attr)) {

                // Call the descriptor instance method with the passed arguments.
                return call_user_func_array(
                    array($attr, $method . 'Descriptor'),
                    $args
                );
            }

            // Trigger an error if the appropriate descriptor wasn't found.
            trigger_error(sprintf('Undefined property: %s::$%s', $class, $name));
            return null;
        }

        /**
         * Gets a dynamic instance property's value.
         *
         * @param string $name The name of the instance property being accessed.
         *
         * @return mixed
         */
        public function __get($name)
        {
            $args = array($this, get_class($this));
            return $this->_descriptorAccess('get', $name, $args);
        }

        /**
         * Sets a dynamic instance property's value.
         *
         * @param string $name  The name of the instance property being set.
         * @param string $value The new value to use for the property.
         *
         * @return null
         */
        public function __set($name, $value)
        {
            $args = array($this, $value);
            return $this->_descriptorAccess('set', $name, $args);
        }

        /**
         * Unsets a dynamic instance property.
         *
         * @param string $name The name of the instance property being unset.
         *
         * @return null
         */
        public function __unset($name)
        {
            $args = array($this);
            return $this->_descriptorAccess('unset', $name, $args);
        }

        /**
         * Returns true if a descriptor instance exists in the named class property.
         *
         * @param string $name The name of the instance property being accessed.
         *
         * @return boolean
         */
        public function __isset($name)
        {
            // Simply test to see if we have any descriptor with that name.
            return is_null($this->_descriptorInstance($name));
        }
    }

The Gotchas

There are some tiny differences between our PHP descriptors and those in Python, aside from the static access restriction discussed previously. Python’s property resolution automatically searches for a class property if an instance does not contain property with the requested name. PHP does not do this, and treats static properties very differently. This is unlikely to become an issue in practice, and is a very specific edge case. Another similar detail is that Python will short circuit its property resolution if a property defines a "setting" access method, always using the descriptor even if the instance defines it’s own property with the same name. Descriptors which only handle "getting" of properties will not be used by default. This is very confusing and really just an intricacy of Python that doesn’t relate to PHP since we don’t have the concept of an object’s "data dict." In all except the wildest of edge cases, these differences may be ignored.

Another small gotcha can pop up when developers erroneously re-include a source file containing a class definition. Since descriptor instances must be assigned to a class property and PHP doesn’t allow evaluated variables as property values in the class definition, we’re forced to add the descriptor instance to the class after it has been defined. This isn’t something that should be feared and it’s certainly not "wrong" according to how PHP works, but it is a little off putting to some developers. Regardless, it’s important to not re-include a file if it adds a descriptor to a class. Otherwise, a new instance of the descriptor may suddenly appear in all of the existing instances of that class. The fix is simple: use require_once like you should be doing in the first place when importing class definitions.

The Payoff

At this point you’re probably very angry (and crosseyed) after having read this exhausting tome, and yet we still haven’t explored why descriptors are useful in the first place. Let’s go over a few benefits descriptors have over other types of dynamic properties.

<?php
    // Define a descriptor to manage retrieving the high score.
    class HighScoreDescriptor
    implements DescriptorGet, DescriptorSet
    {
        protected $_high_score;

        // Returns the current high score.
        public function getDescriptor($instance, $owner)
        {
            // Check to see if we've already retreived the high score.
            if (is_null($this->_high_score)) {

                // Fetch and cache the high score from the db.
                $this->_high_score = get_from_db('high_score');
            }

            // Return the cached high score.
            return $this->_high_score;
        }

        // Sets the current high score.
        public function setDescriptor($instance, $value)
        {
            // Update the cached high score.
            $this->_high_score = $value;

            // Save the new high score to the db.
            update_db('high_score', $this->_high_score);
        }
    }

    // Define a scored game class.
    class ScoredGame extends Descriptable
    {
        // Define a class property to hold the high score descriptor.
        public static $high_score;
    }

    // Add a high score descriptor to the class.
    ScoredGame::$high_score = new HighScoreDescriptor();

    // Create a game instance.
    $game = new ScoredGame();

    // Retrieve the high score for the game (db queried).
    echo 'High Score: ' . $game->high_score;

    // Set the high score, cheater.
    $game->high_score = 31337;

    // Retreive the high score again (db *not* queried).
    echo 'New High Score: ' . $game->high_score;

    // Create another game instance.
    $another_game = new ScoredGame();

    // Retrieve the high score for the new game (db *still* not queried).
    echo 'Same High Score: ' . $another_game->high_score;

Conclusion

Descriptors are fun, they’re cool, and they’re just an all around good tool to have in your toolbox. They’re the ideal solution to some very complex, real world problems that have plagued developers for years. I long for the day when PHP releases support for dynamic class properties as well so we can get our hands on a drop-in replacement for Python’s descriptors. Until then, the descriptor interfaces and descriptable object class we’ve just designed will fit the bill just fine.

Create a virtualenv for TextMate

We’re gonna need to install a few Python packages in order to make this work. That means it’s time to fire up virtualenv and make an isolated Python environment for our fix.

stewart:~ xdissent$ mkvirtualenv TextMate
New python executable in TextMate/bin/python
Installing setuptools............done.

Install pip

Virtualenv’s come pre-loaded with easy_install, but we’re going to be changing a couple of files in some packages so we should use pip to make that easier on ourselves.

(TextMate)stewart:~ xdissent$ easy_install pip
Searching for pip
[...]
Installed /Users/xdissent/.virtualenvs/TextMate/lib/python2.5/site-packages/pip-0.3.1-py2.5.egg
Processing dependencies for pip
Finished processing dependencies for pip

Install Docutils

Docutils is the package that provides the reStructuredText parsers, so we’ll definitely be needing that.

(TextMate)stewart:~ xdissent$ pip install docutils
Downloading/unpacking docutils
  Downloading docutils-0.5.tar.gz (1.3Mb): 1.3Mb downloaded
[...]
Successfully installed docutils

Install reStructuredText TextMate Bundle

TextMate has a bundle repository at http://svn.textmate.org/trunk/Bundles with a reStructuredText bundle already built. We need to download that to a place where TextMate will actually find the bundle and load it. Typically, thats ~/Library/Application Support/TextMate/Bundles.

(TextMate)stewart:~ xdissent$ mkdir -p ~/Library/Application\ Support/TextMate/Bundles
(TextMate)stewart:~ xdissent$ cd ~/Library/Application\ Support/TextMate/Bundles
(TextMate)stewart:Bundles xdissent$ svn co http://svn.textmate.org/trunk/Bundles/reStructuredText.tmbundle
A    reStructuredText.tmbundle/Commands
[...]
A    reStructuredText.tmbundle/Syntaxes/reStructuredText.plist
Checked out revision 10740.

Make reStructuredText Bundle use virtualenv

Now that we have the bundle, we need to let it know that it needs to be called from inside our virtualenv. The Preview command uses rst2html.py to generate the preview, and luckily provides a way to customize the path to rst2html.py. All we need to do is define TM_RST2HTML variable with the correct path in our .bash_profile and TextMate will find the right one.

(TextMate)stewart:~ xdissent$ echo export TM_RST2HTML=`which rst2html.py` >> ~/.bash_profile
(TextMate)stewart:~ xdissent$ source ~/.bash_profile

Reload TextMate Bundles

If you’re running TextMate at this point, you can either use the "Bundles" menu or the following command to reload the bundles and install our new reStructuredText bundle.

(TextMate)stewart:~ xdissent$ osascript -e 'tell app "TextMate" to reload bundles'

Bask in the glory

We’re done. Now we can preview or export reStructuredText easily. Open a reStructuredText file and go wild. It’s incredibly convenient to preview documents as you go, and saving them to HTML makes it simple to blog from TextMate.

(TextMate)stewart:~ xdissent$ mate ~/Documents/TextMate\ Fixes.rst

Colour My World

If you’re like me, you tend to include source code or console sessions in your articles. Pygments is a Python package that can add syntax highlighting to these snippets, which comes in handy for previewing in TextMate. Installation is slightly more advanced, but well worth setting up.

Install Mercurial

We want to hack on our Pygments package, so we’re going to want to install it in editable mode. Plus, the latest stable release doesn’t have the BashSessionLexer that I use quite a bit. Pygments uses Mercurial for source control, but I don’t have a system-wide copy of hg, so I needed to grab one real fast.

(TextMate)stewart:~ xdissent$ pip install mercurial
Downloading/unpacking mercurial
  Downloading mercurial-1.1.2.tar.gz (952Kb): 952Kb downloaded
[...]
Successfully installed mercurial

Enable the fetch command

When pip installs an editable package from a Mercurial repository, it uses the fetch command with hg. We need to enable it in our ~/.hgrc file because it’s disabled by default.

(TextMate)stewart:~ xdissent$ cat >> ~/.hgrc <<EOF
> [extensions]
> fetch =
> EOF

Get Pygments

Now we can install Pygments from Mercurial. Pip will download the source and unpack it into $VIRTUAL_ENV/src before finally creating an .egg-link file to point our interpreter to the source code.

(TextMate)stewart:~ xdissent$ pip install -e hg+http://dev.pocoo.org/hg/pygments-main@799#egg=Pygments
Checking out Pygments from hg+http://dev.pocoo.org/hg/pygments-main@799#egg=Pygments checkout from hg+http://dev.pocoo.org/hg/pygments-main@799#egg=Pygments
[...]
    Installed /Users/xdissent/.virtualenvs/TextMate/src/pygments
Successfully installed Pygments

Install the reStructuredText sourcecode directive

Activating syntax highlighting in a document requires the sourcecode directive which highlights the following block of code. We can switch out different Pygments lexers using an option after the directive:

.. sourcecode:: python

    from hartzog import VERSION
    print 'Running Hartzog %%s' %% VERSION

The rendered version looks something like this:

from hartzog import VERSION
print 'Running Hartzog %%s' %% VERSION

The sourcecode directive isn’t installed automatically, so we’ll need to add it ourselves. Directives are simply python functions that must be registered with docutils before any parsing is done. That means we can’t use rst2html.py with highlighting yet because the directive won’t be loaded. Pygments does come with the directive already in its source tree, but we’re going to have to let virtualenv know where it is, and then modify rst2html.py to register the directive before it begins parsing.

(TextMate)stewart:~ xdissent$ add2virtualenv $VIRTUAL_ENV/src/pygments/external
(TextMate)stewart:~ xdissent$ sed -i '' '/^publish_cmdline/i\
> __import__("rst-directive")
> ' `which rst2html.py`

Colorize the Preview command

What good is all this work if we can’t see the damn colors? As it stands now, pygments is marking up our code to be colorized, but the CSS to actually make the colors show up isn’t being loaded into our preview window. We have to enhance the Preview command in the reStructuredText bundle to include Pygments stylesheets. You can control which Pygments theme will be used by defining a TM_PYGMENTIZE_STYLE environment variable. I’ve created a patch that we can apply to update our bundle.

Index: Commands/Preview.plist
===================================================================
--- Commands/Preview.plist  (revision 10740)
+++ Commands/Preview.plist  (working copy)
@@ -25,6 +25,13 @@
            border: 1px #888 solid;
            padding: 0 1em;
    }' > $css
+    if [[ -f "$TM_PYGMENTIZE" ]]
+    then
+        if [[ "$TM_PYGMENTIZE_STYLE" = "" ]]
+            then TM_PYGMENTIZE_STYLE=default
+        fi
+        $TM_PYGMENTIZE -S $TM_PYGMENTIZE_STYLE -f html >> $css
+    fi
    stylesheet=$css
    tmpCreated="yes"
 fi

We want to apply the patch into the root of the bundle with patch.

(TextMate)stewart:~ xdissent$ patch -d ~/Library/Application\ Support/TextMate/Bundles/reStructuredText.tmbundle -p0 -i rst_bundle.diff
patching file Commands/Preview.plist

Tell TextMate how to use pygmentize

The patch for the Preview command requires an environment variable called TM_PYGMENTIZE that contains the path to the pygmentize script. We need to add that to our .bash_profile along with any custom style we want to use for highlighting.

(TextMate)stewart:~ xdissent$ echo export TM_PYGMENTIZE=$VIRTUAL_ENV/bin/pygmentize >> ~/.bash_profile
(TextMate)stewart:~ xdissent$ echo export TM_PYGMENTIZE_STYLE=fruity >> ~/.bash_profile
(TextMate)stewart:~ xdissent$ source ~/.bash_profile

Inform TextMate of the updated Preview command

Just like before, we want to nudge TextMate so we can check out the new preview command without restarting.

(TextMate)stewart:~ xdissent$ osascript -e 'tell app "TextMate" to reload bundles'

Patch Pygments for virtualenv

Everything should be working perfectly now in full color, but I went one step further in my setup to tweak the BashSessionLexer in Pygments. Because I use virtualenv all the time, my bash prompt usually has a virtual env name at the beginning, surrounded by parentheses. If I were to use the console option for sourcecode with a virtualenv active in the session, the lexer wouldn’t recognize my prompt and it wouldn’t get highlighted. I put together a quick patch for Pygments that correctly highlights virtualenvwrapper style prompts.

diff -r 9fe544c7818b pygments/lexers/other.py
--- a/pygments/lexers/other.py      Sat Feb 14 13:27:41 2009 +0100
+++ b/pygments/lexers/other.py      Sat Feb 28 22:33:39 2009 -0600
@@ -416,7 +416,7 @@

         for match in line_re.finditer(text):
             line = match.group()
-            m = re.match(r'^((?:|sh\S*?|\w+\S+[@:]\S+(?:\s+\S+)?|\[\S+[@:]'
+            m = re.match(r'^((?:\([A-Za-z_0-9-]+\))?(?:|sh\S*?|\w+\S+[@:]\S+(?:\s+\S+)?|\[\S+[@:]'
                          r'[^\n]+\].+)[$#%%])(.*\n?)', line)
             if m:
                 # To support output lexers (say diff output), the output

Mercurial patches are weird, so be sure to use the -p1 option when applying:

(TextMate)stewart:~ xdissent$ patch -d $VIRTUAL_ENV/src/pygments -p1 -i pygments.diff
patching file pygments/lexers/other.py

Chump and a Hoagie

That’s it. Go buck wild on some reStructuredText in TextMate dawg. You’ve earned it.

Beta Blues

I’ve been kind of bummed reading all the negative responses to Safari’s update because I’m so impressed by the damn thing. It seems like most of the gripers are claiming Apple ripped off Chrome’s "Tabs-on-Top" layout and "Top Sites" feature or that Safari 4 isn’t as many times faster than every competitor as Apple claims. So… when Apple notices good design they’re not supposed to act on improving their product? Should Apple have included tabs at all since they certainly were not the first to develop the idea?. Nevermind the fact that Chrome wouldn’t even exist if Apple hadn’t shared WebKit with the world, since Chrome uses it exclusively for rendering. So what’s the problem with a few "good as new" bad-ass features, dudes? And if the absence of a handful of outdated Safari 3 behaviors is driving you nuts, you can easily bring them back.

Silver Bindings

The one cool thing that really put a smile on my face when trying out Safari 4 was discovering the new tab-switching keyboard shortcuts. It seems like every app in the world has tabs these days, and moving around between them is the most common (yet also most application-specific) task you will perform. Since the dawn of tab, developers haven’t been able to agree on a global shortcut to switch tabs and my mouse has suffered great wear as the battle rages on. I would guess this problem is a NIH related artifact from a time when tabs were the killer new feature to have, and no application had yet proven to be the ubiquitous tab-interaction authority. I’ve always found Safari 3’s built-in tab navigation key combo about as awkward as this sentence. Firefox certainly paved the way with some very intuitive shortcuts, but I only use Firefox for UI testing with Firebug, so rarely do I find myself treading water in a sea of tabs. Also, Safari 4’s new developer tools have pretty much guaranteed a serious decline in my Firebug time anyway.

With Safari 4, a great tab usage barrier has been shattered as the Firefox control-tab shortcut replaces Safari’s original loathsome key combo. I was so inspired by this move towards consistency (or additional Apple rip-off attempt as one could claim) that I decided to add it to another tab-defying application that I love so dearly: Coda.

The Leopard Way

Mac OS X provides a System Preferences pane that allows you to add, edit and remove global and application-specific keyboard shortcuts. It’s a handy little tool found under the "Keyboard & Mouse" preferences that somehow I’ve only used once… to change tab behaviors. A while back, I actually had the (impressively moronic) idea to change Coda’s tab shortcuts to command-tab so switching between tabs would be "as easy as switching applications." Now that’s taking Coda’s "one window development" tagline to the eXtR3m#.

Luckily for me, Leopard doesn’t allow you to use the tab key when defining new shortcuts, so I settled for command-~, which is the standard "switch windows within application" key command. "Mostly one window development" turned out to be a real drag though, because I frequently work on two or three Coda Sites at once, and found myself with no way to switch between Sites. Bummer.

It was looking like Apple was going to deny me my tab nirvana.

The Working Way

After poking around on Google, I discovered System Preferences stores application shortcuts in your preferences plist. It uses a dictionary value named NSUserKeyEquivalents to map the commands to the shortcuts and can be changed easily with defaults write. I also found that you can use the ^ character to represent the control key when defining your shortcuts, but what about tab? Brute force experimentation landed me in tab heaven on my first try. Here is how to make control-tab and control-shift-tab map to "Select Next Tab" and "Select Previous Tab" respectively in Coda:

stewart:~ xdissent$ defaults write com.panic.Coda NSUserKeyEquivalents -dict-add "Select Next Tab" '^\t'
stewart:~ xdissent$ defaults write com.panic.Coda NSUserKeyEquivalents -dict-add "Select Previous Tab" '^$\t'

The Fix

Using virtualenv, virtualenvwrapper, and Paver SVN r37, I set up a clean build environment for paver like so:

stewart:~ xdissent$ cd ~/Code
stewart:Code xdissent$ svn co http://paver.googlecode.com/svn/trunk/ paver-trunk
A    paver-trunk/.bzrignore
A    paver-trunk/LICENSE.txt
[...]
A    paver-trunk/paver/doctools.py
A    paver-trunk/paver/virtual.py
U   paver-trunk
Checked out revision 37.
stewart:Code xdissent$ cd paver-trunk
stewart:paver-trunk xdissent$ mkvirtualenv paver-tmp
New python executable in paver-tmp/bin/python
Installing setuptools............done.
(paver-tmp)stewart:paver-trunk xdissent$ add2virtualenv `pwd`
(paver-tmp)stewart:paver-trunk xdissent$ python distutils_scripts/paver minilib generate_setup
Paver 1.0a1
---> paver.misctasks.minilib
Generate paver-minilib.zip
---> paver.misctasks.generate_setup
Write setup.py
(paver-tmp)stewart:paver-trunk xdissent$ mkvirtualenv paver-trunk
New python executable in paver-trunk/bin/python
Installing setuptools............done.
(paver-trunk)stewart:paver-trunk xdissent$ rmvirtualenv paver-tmp
(paver-trunk)stewart:paver-trunk xdissent$ python setup.py develop
python setup.py develop
Paver 1.0a1
warning: no files found matching '*' under directory 'paver/docs'

At this point I have Paver 1.0a1 installed in development mode, so the new paver binary is on my path AND changes I make to ~/Code/paver-trunk are effective immediately.

To demonstrate the problem, I created options_test.py. Here’s a simple task with some options:

from paver.easy import *

options(
   show_opts=Bunch(
       foo_bar='default foobar',
       no_foo=False,
   ),
)

@task
@cmdopts([("foo-bar=", "f", "Foo bar value"),
         ("no-foo", "n", "Foo negation")])
def show_opts(options):
   print options.show_opts

Since we have Paver 1.0a1, we can run the task with the -f option. (Somebody said somethin about shiny new features.)

(paver-trunk)stewart:tmp xdissent$ paver -f options_test.py show_opts
Paver 1.0a1
---> pavement.show_opts
Bunch(foo_bar='default foobar', no_foo=False)
(paver-trunk)stewart:tmp xdissent$ paver -f options_test.py show_opts --foo-bar='new foobar' --no-foo
Paver 1.0a1
---> pavement.show_opts
Bunch(foo-bar='new foobar', foo_bar='default foobar', no-foo=True, no_foo=False)

It becomes clear that cmdopts definitions will never override default options for this task. We can’t use foo-bar as an option name in our Bunch, nor could we access the runtime value using options.show_opts.foo-bar. It turns out distutils and setuptools already noticed this problem and convert hyphens to underscores when determining the storage variable’s name for an option. I’ve attached a small patch for Paver SVN r37 that automatically makes this conversion.

(paver-trunk)stewart:tmp xdissent$ patch -d ~/Code/paver-trunk -p0 -i options_fix.diff
patching file paver/setuputils.py
patching file paver/tasks.py
(paver-trunk)stewart:tmp xdissent$ paver -f options_test.py show_opts
Paver 1.0a1
---> pavement.show_opts
Bunch(foo_bar='default foobar', no_foo=False)
(paver-trunk)stewart:tmp xdissent$ paver -f options_test.py show_opts --foo-bar='new foobar' --no-foo
Paver 1.0a1
---> pavement.show_opts
Bunch(foo_bar='new foobar', no_foo=True)

That’s more like it. I further tested the patch to confirm that distutil commands receive their options correctly and that the easy_install problem has been solved.

(paver-trunk)stewart:paver-trunk xdissent$ paver minilib
Paver 1.0a1
---> paver.misctasks.minilib
Generate paver-minilib.zip
(paver-trunk)stewart:paver-trunk xdissent$ mkvirtualenv paver-tmp
New python executable in paver-tmp/bin/python
Installing setuptools............done.
(paver-tmp)stewart:paver-trunk xdissent$ python setup.py sdist
Paver 1.0a1
warning: no files found matching '*' under directory 'paver/docs'
(paver-tmp)stewart:paver-trunk xdissent$ easy_install dist/Paver-1.0a1.tar.gz
Processing Paver-1.0a1.tar.gz
Running Paver-1.0a1/setup.py -q bdist_egg --dist-dir /var/folders/LU/LUPONriBGYepwd3i9FaBB++++TI/-Tmp-/easy_install-GnYR4C/Paver-1.0a1/egg-dist-tmp-iTfLMZ
Paver 1.0a1
running bdist_egg
running egg_info
[...]
Adding Paver 1.0a1 to easy-install.pth file
Installing paver script to /Users/xdissent/.virtualenvs/paver-tmp/bin

Installed /Users/xdissent/.virtualenvs/paver-tmp/lib/python2.5/site-packages/Paver-1.0a1-py2.5.egg
Processing dependencies for Paver==1.0a1
Finished processing dependencies for Paver==1.0a1
(paver-tmp)stewart:paver-trunk xdissent$ which paver
/Users/xdissent/.virtualenvs/paver-tmp/bin/paver
(paver-tmp)stewart:paver-trunk xdissent$ cd tmp
(paver-tmp)stewart:tmp xdissent$ paver -f options_test.py show_opts
Paver 1.0a1
---> pavement.show_opts
Bunch(foo_bar='default foobar', no_foo=False)
(paver-tmp)stewart:tmp xdissent$ paver -f options_test.py show_opts --foo-bar='it works' --no-foo
Paver 1.0a1
---> pavement.show_opts
Bunch(foo_bar='it works', no_foo=True)

Hopefully 1.0 will come out soon, but until then you can download the patch and apply it from within trunk:

$ patch -p0 -i options_fix.diff