Tools bliki

As I explore parser generator tools for external DomainSpecificLanguages, I've said HelloAntlr and HelloSablecc. If you spend much time looking at parser generators you can't really avoid looking at the old stalwarts lex and yacc (or their gnu counterparts flex and bison). I want to explore the way lex and yacc operate, but my C has got too rusty. As Erich Gamma quipped, I've got too lazy to take out my own garbage. Fortunately there is an implementation of a yaccish system for Java, which is just what I need.

The Java implementation, like the classic lex and yacc, is two independent tools: JFlex and CUP. Although they are developed separately they do provide hooks to work together.

As with my previous posts along these lines, this is a overly-simple example just to get the tools working. I take an input file which says:

item camera
item laser

and turn it them into item objects inside a configuration using the following model:

public class Configuration {
  private Map<String, Item> items = new HashMap<String, Item>();
  public Item getItem(String key) {
    return items.get(key);
  }
  public void addItem(Item arg) {
    items.put(arg.getName(), arg);
  }
public class Item {
  private String name;
  public Item(String name) {
     this.name = name;
   }

to pass the following test

    @Test public void itemsAddedToItemList() {
      Reader input = null;
      try {
        input = new FileReader("rules.txt");
      } catch (FileNotFoundException e) {
        throw new RuntimeException(e);
      }
      Configuration config = CatalogParser.parse(input);
      assertNotNull(config.getItem("camera"));
      assertNotNull(config.getItem("laser"));
    }

The first issue is just to get the build going. As with my previous examples I want to take the grammar input files and generate the lexer and parser into a gen directory. Unlike my previous examples I don't do this directly in ant, instead I'm using ant to call a ruby script.

--- build.xml
 <target name = "gen" >
    <exec executable="ruby" failonerror="true">
      <arg line = "gen.rb"/>
    </exec>
  </target>

--- gen.rb
require 'fileutils'
include FileUtils

system "java -cp lib/JFlex.jar JFlex.Main -d gen/parser src/parser/catalog.l"

system "java -jar lib/java-cup-11a.jar src/parser/catalog.y"
%w[parser.java sym.java].each {|f| mv f, 'gen/parser'} 

Yes, I know it's a long way around, but with a lot of source files I'm using the approach in FlexibleAntlrGeneration to do my generation and I can't be bothered to sort it out in ant as well.

(When I attended CITCON recently, I was surprised to find out that people were much happier with ant than I thought. Grumpy me thinks it's a case of Stockholm Syndrome. Even when less grumpy I'm keeping my eye on things like Raven and BuildR which has now got some documentation. I'm so ready to ditch ant.)

You'll notice that CUP puts its output files in the current directory and I couldn't see how to override that behavior. So I generated them and moved them with a separate command.

Once I generate the code I then compile and test it with ant.

<target name = "compile" depends = "gen">
    <mkdir dir="${dir.build}"/>
    <javac destdir="${dir.build}" classpathref="path.compile">
      <src path = "${dir.src}"/>
      <src path = "${dir.gen}"/>
      <src path = "${dir.test}"/>
    </javac>
  </target>

  <target name = "test" depends="compile">
     <junit haltonfailure = "on" printsummary="on">
      <formatter type="brief"/>
      <classpath refid = "path.compile"/>
      <batchtest todir="${dir.build}" >
        <fileset dir = "test" includes = "**/*Test.java"/>
      </batchtest>
     </junit>
   </target>

Lex and yacc separate the lexer and parser into different files. Each is generated independently and combined during compilation. I'll start with the lexer file (catalog.l). The opening declares the output file's package and imports.

package parser;
import java_cup.runtime.*;

JFlex uses %% markers to break the file into sections. The second section consists of various declarations. The first bit names the output class and tells it to interface with CUP.

%%
%class Lexer
%cup

The next bit is code to be folded into the lexer. Here I define a function to create Symbol objects - again to hook into CUP.

%{
  private Symbol symbol(int type) {
    return new Symbol(type, yytext());
  }
%}

The Symbol class is defined in CUP and is part of its runtime jar. There are various constructors taking various information about the symbol and where it is.

Next up are some macros to define words and whitespace.

Word = [:jletter:]*
WS = [ \t\r\n]

The final section is the actual lexer rules. I define one to return the item keyword and the other to return simple words to the parser.

%%
"item"      {return symbol(sym.K_ITEM);}
{Word}      {return symbol(sym.WORD);}
{WS}        {/* ignore */}

So the lexer will send a stream of K_ITEM and WORD tokens to the parser. I define the parser in catalog.y. Again it starts with package and import declarations.

package parser;
import java_cup.runtime.*;
import model.*;

I'm parsing the data into a configuration object, so I need to declare a place to put that result. Again this code is copied directly into the parser object.

parser code {: Configuration result = new Configuration(); :}

With CUP I need to define all the rule elements that I'll use in the productions.

terminal K_ITEM;
terminal String WORD;
non terminal  catalog, item;

The terminals are the tokens I get from the lexer, the non terminals are the rules I'll build up myself. If I want to get a payload from the token, I need to declare its type, so WORD is a string.

The catalog is a list of items. Unlike with antlr or sablecc I don't have EBNF here, so I can't say item*, instead I need a recursive rule.

catalog ::= item | item catalog;

The item rule itself contains the embedded action to put the item into the configuration.

item ::= K_ITEM WORD:w {: parser.result.addItem(new Item(w)); :}
          ;

A little wrinkle to note here is that the actions are put into a separate class to the parser object, so to get to the result field I defined earlier I have to use the parser field of the actions object. I should also mention that once I do much further with this I start to use an EmbedmentHelper to keep action code simple.

People who have used yacc before might notice that I can label the elements of the rule to refer to them in the actions instead of the $1, $2 convention used in yacc. Similarly if the rule returns a value CUP uses RESULT rather than $$.

My memories of lex and yacc are dim, but these tools do seem to mimic the style of using them pretty well. My biggest beef so far is the error handling, which caused me much more fuss than antlr. My feeling so far is that if you're new to parser generators then antlr is the better choice (particularly due to its book). However if you're familiar with lex and yacc then these two are similar enough to build off that knowledge.

After saying HelloSablecc I also wanted to try out Antlr, which is another compiler-compiler for the Java space. As with that entry, this is just about getting Antlr going with a very simple "hello world" style grammar.

Like SableCC, Antlr is a compiler-compiler tool. It's been around for a while, and I've run into a few projects that use it. Unlike SableCC (and the venerable lex/yacc combo) it generates a recursive descent parser using LL grammars. Compiler heads like to argue about whether LL or LALR are better, I'll not step into that debate here.

My simple case is to parse a file of a list of items like this:

item camera
item laser

Each line has the 'item' keyword followed by a single word for the name of an item. I shall load each item object into a configuration object that keeps them all together.

public class Configuration {
  private Map<String, Item> items = new HashMap<String, Item>();
  public Item getItem(String key) {
    return items.get(key);
  }
  public void addItem(Item arg) {
    items.put(arg.getName(), arg);
  }
public class Item {
  private String name;
  public Item(String name) {
     this.name = name;
   }

Here's a test for that, using the file I showed above.

 @Test public void readTwoItems() {
    Reader input = null;
    try {
      input = new FileReader("catalog.txt");
    } catch (FileNotFoundException e) {
      throw new RuntimeException(e);
    }
    Configuration config = ParserCommand.parse(input);
    assertNotNull(config.getItem("camera"));
    assertNotNull(config.getItem("laser"));
    assertEquals(2, config.getItems().size());
  }

As before - using a compiler-compiler for this problem is silly, but then is printing "hello world" on a console. For the same reason as I always write "hello world" with a new environment, I like to write something dirt simple to just make sure I can get things working at all before I start doing anything real with it.

One hassle with using an compiler-compiler like this is that it makes the build process more complicated. I have to run antlr on the grammar file to create java classes for the parser, then include them in the compilation. So it's time to fight with ant again - here's the ant target:

  <property name = "dir.parser" value = "${dir.gen}/parser"/>
  <path id = "path.antlr">
    <fileset dir = "${dir.lib}">
      <include name = "antlr*.jar"/>
      <include name = "stringtemplate*.jar"/>
    </fileset>
  </path>
  <target name = "gen" >
    <mkdir dir="${dir.parser}"/>
    <java classname="org.antlr.Tool" classpathref="path.antlr" fork = "true" failonerror="true">
      <arg value="-o"/>
      <arg value="${dir.parser}"/>
      <arg value="Catalog.g"/>
     </java>
  </target>

This generates code into the gen directory. This way generated code is separate from source code I write myself. Another target does the compilation

 <property name = "dir.build" value = "classes/production/antlrLair"/> 
 <target name = "compile" depends = "gen">
    <mkdir dir="${dir.build}"/>
    <javac destdir="${dir.build}" classpathref="classpath">
      <src path = "src"/>
      <src path = "${dir.gen}"/>
      <src path = "test"/>
    </javac>
  </target>

I can then run the tests with a final target.

<target name = "test" depends="compile">
    <junit haltonfailure = "on">
      <formatter type="brief"/>
      <classpath refid = "classpath"/>
      <batchtest todir="${dir.build}" >
        <fileset dir = "test" includes = "**/*Test.java"/>
      </batchtest>
     </junit>
   </target>

Antlr works with a grammar file Catalog.g. The grammar file defines the productions in the grammar and also actions that the parser takes when it encounters productions. The grammar file also defines the lexer (you can split them if you want). In this sense Antlr is more traditional (and flexible) than SableCC. SableCC doesn't allow actions, instead you generate a parse tree or AST and walk that with java. Antlr allows arbitrary actions, or it supports building a tree in the same manner as SableCC. (Antlr also uses a grammar file to walk the tree.) Since I'm building up a simple domain model of items and a configuration I'll forgo the tree building and do all the work in my actions.

I'll go through this file in chunks, with descriptions as I go. I start with a grammar heading

grammar Catalog;

Antlr supports a number of points at which you can inject code into the generated parser (instead of the generating a superclass which SableCC does.) I put package declarations and imports into the header.

@header{
package parser;
import model.*;
}
@lexer::header {
package parser;
}

The next code injection is to put code into the body of the generated class. Essentially this adds members to the class, hence the name of the command.

@members {
  public Configuration result = new Configuration();
}

Now I can get into the productions of the grammar. I'll do this top down, since it's a top-down parser. I begin by saying that the catalog consists of multiple item clauses followed by the end of the file.

catalog :  item* EOF;

Next I define the item clause as the literal string 'item' followed by a string.

item 	: 'item'  name=STRING 
   {result.addItem(new Item ($name.text));};

Here I also put in the action, which is to create a new item in the model with the name set to the value of the string. The code inside the curlies is java code which is added to the parser after that term is recognized. I can name elements in the production which I then refer to in the action. Here I've given the string the name 'name', which makes sense in context even though it makes for an awkward write-up.

The last productions define the lexer elements for string and whitespace.

STRING 	: ('a'..'z' | 'A'..'Z')+ ;
WS : (' ' |'\t' | '\r' | '\n')+ {skip();} ;

The action of whitespace is to skip (ignore) it.

There are a few things that make Antlr easier to work with than SableCC. Antlr has a nice IDE called AntlrWorks that can plug into IntelliJ. The tool will give you syntax highlighting and completion on grammar elements, plot syntax diagrams for your grammar, and allow you to enter test fragments to parse - displaying the resulting parse tree. It's a very helpful tool to see what the parser is doing. However there's no highlighting/completion for the code inside actions, which is an understandable pain.

Another good feature of Antlr is the fact that there is a decent book on it in the works. The book gives detailed coverage of how the tool works and useful background on language and compiler principles. It does assume you're working on a full blown language and that you'll be generating code - which isn't necessarily so for DSL work. However the detail it gives looks like it will be invaluable as I probe deeper.

Antlr's actions seem like an easier bet if you want to populate a model - I'm not sure how useful an intermediate parse tree or AST would be here. Again further investigation will give me a better feel. The more complex the language the more useful it is to have an intermediate tree representation. I like Antlr's flexibility in allowing you to do actions or tree building with transformations.

Inevitably I did have problems even with this simple example. My biggest blocker was that I originally defined the catalog term as catalog : item*;, that is without the EOF. I then got confused because the parser didn't indicate an error when it got spurious input (like xitem foo). This wasn't helped by inconsistencies between Antlr and AntlrWorks (the latter did show an error and older versions of AntlrWorks would handle whitespace differently too.)

(Another big cause of trouble was getting ant and JUnit to work. I don't want to have to think about the amount of time I've spent over the years trying to diagnose classpath problems, especially with the infamous "Ant could not find the task or a class this task relies upon." message.)

I've done a small amount of playing around with SableCC recently. It took a bit of effort to get a "Hello World" style parser going, so I thought I'd put some notes here as to what I did to get it working. I'm not saying this is the best way to do it, but it may be useful if you're looking to play with it.

SableCC is a compiler-compiler tool for the Java environment. It handles LALR(1) grammars (for those who remember their grammar categories). In other words it's a bottom up parser (unlike JavaCC and Antlr which are top-down).

Like with most compiler-compiler tools, you define a grammar for your language in a grammar file and run the compiler-compiler to generate a parser for this language. Since this is a hello-world example my language is a bare minimum one just to get the compiler-compiler going. The language just is a list of items like this:

item camera
item laser

Where 'item' is a keyword and the second word is a name. I want the parser to turn this into an instance of a Configuration class which contains a list of items.

public class Configuration {
  private Map<String, Item> items = new HashMap<String, Item>();
  public Item getItem(String key) {
    return items.get(key);
  }
  public void addItem(Item arg) {
    items.put(arg.getName(), arg);
  }
public class Item {
  private String name;
  public Item(String name) {
     this.name = name;
   }

The input I showed earlier should be read in to pass the following test.

    @Test public void itemsAddedToItemList() {
      Reader input = null;
      try {
        input = new FileReader("rules.txt");
      } catch (FileNotFoundException e) {
        throw new RuntimeException(e);
      }
      Configuration config = CatalogParser.parse(input);
      assertNotNull(config.getItem("camera"));
      assertNotNull(config.getItem("laser"));
    }

Of course using a compiler-compiler for this is total overkill for this kind of case, but the point of a hello-world example is to get the environment working on a simple case so you can move onto the interesting cases with the basics working. Whenever I play with a new technology I always like to get something like this going first, before I delve into the interesting aspects.

Using a compiler-compiler makes the build process a bit more involved. You have to first run the compiler-compiler on the grammar file to generate the parser, then compile both your custom code and the generated parser to create the overall program, then run (and test it). So at this point I can't get away with just doing everything inside my IntelliJ and actually have to put together an ant file. It was a while since I'd put together an ant file and so it took me a bit to re-remember how to use ant language. To run SableCC I used the Java task:

   <property name = "gendir" value = "gen"/>
   <target name = "gen" >
      <mkdir dir="${gendir}"/>
      <java jar = "lib/sablecc.jar" fork = "true" failonerror="true">
         <arg value = "-d"/>
         <arg value = "${gendir}"/>
         <arg value = "catalog.sable"/>
      </java>
    </target>

I generate the code into the gen directory, to keep it separate from the code I write myself in the src and test directories. I then compile it all together with a javac task.

 <property name = "builddir" value = "classes/production/sable"/>
  <path id="classpath">
       <fileset dir = "lib">
           <include name = "*.jar"/>
       </fileset>
       <pathelement path = "${builddir}"/>
  </path>
  <target name = "compile" depends = "gen, copyDats">
    <mkdir dir="${builddir}"/>
    <javac destdir="${builddir}" classpathref="classpath">
      <src path = "src"/>
      <src path = "${gendir}"/>
      <src path = "test"/>
    </javac>
  </target>

As well as compiling, I also have to move two data files for the parser into the build directory. The data contains the tables used for the parser and lexer. The build directory is nested in the way I have it so it will work nicely with IntelliJ. (I really ought to separate the tests into a separate output directory too, but I was feeling lazy.)

  <target name = "copyDats">
      <mkdir dir="${builddir}"/>
      <copy todir = "${builddir}">
        <fileset dir = "${gendir}" includes = "**/lexer.dat"/>
        <fileset dir = "${gendir}" includes = "**/parser.dat"/>
      </copy>
    </target>

I then have a test task to run the tests.

  <target name = "test" depends="compile">
     <junit haltonfailure = "on">
      <formatter type="brief"/>
      <classpath refid = "classpath"/>
      <batchtest todir="${builddir}" >
        <fileset dir = "test" includes = "**/*Tester.java"/>
      </batchtest>
     </junit>
   </target>

To get the parser up and going I need to define the grammar for my simple language using SableCC's grammar syntax.

Package catalogParser;

Tokens
  itemdef = 'item';
  string = ['a' .. 'z'] +;
  blank = (' ' | 13 | 10)+;
  
Ignored Tokens
    blank;

Productions
  configuration =
    item *
  ;
  item = 
    itemdef string
  ;

Like most compiler-compilers, SableCC splits the work into a lexer and a parser. The lexer reads in characters and chunks them into tokens as defined by the Tokens section of the grammar file. In this case it's really simple: strings are lower case letters, the keyword item is its own token. I also define what is blank whitespace and tell the lexer to throw it away in the Ignores clause.

The lexer will then feed a stream of itemdef and string tokens to the parser. The parser uses two productions to deal with this. It describes a configuration as multiple items, and each item as an itemdef and a string (for its name).

This defines the grammar for my input, but doesn't say how to get from the input to my configuration and item objects. In order to do this I need to write some code to map between what I've parsed and the objects I want to create. In most compiler-compilers I do this by embedding actions into the grammar. SableCC, however, works another way. It automatically creates a parse tree and then gives me a visitor to walk this parse tree. I can then subclass the visitor to do interesting things. In this case, as I walk the parse tree, I take each item node on the parse tree and turn it into the real items in my model.

public class CatalogParser extends DepthFirstAdapter {
  private Configuration result;
  public void outAItem(AItem itemNode) {
    System.out.println("found item");
    result.addItem(new Item(itemNode.getString().toString().trim()));
  }

The parse tree uses naming conventions to bind the grammar to the objects created in the tree. So the grammar creates nodes called AItem to match the item production in the grammar. The method outAItem is called as the visitor leaves the item node and allows me to access whatever is on the item, in this case the underlying string token. I can then create the item in my model using that string as the name.

The last bit of code is that to run the parser on the file, which I do by making the catalog parser a command object.

  public static Configuration parse(Reader input) {
    Configuration result = new Configuration();
    new CatalogParser(input, result).run();
    return result;
  }
  public CatalogParser(Reader input, Configuration result) {
    this.input = input;
    this.result = result;
  }
  public void run() {
    try {
      createParser(input).parse().apply(this);
    } catch (Exception e) {
      throw new RuntimeException(e);
     }
  }
  private Parser createParser(Reader input) {
    return new Parser(new Lexer(new PushbackReader(input)));
  }

So that's the basics of getting it going. So far I haven't started to delve further, so my thoughts about SableCC are somewhat preliminary, but then the whole point of this blog is to write half-formed thoughts.

SableCC is a bit awkward to use. There's little documentation, other than the author's thesis. Fortunately the thesis is much more understandable than many others I've come across so I was able to figure out how to get things going. During my work I made a mistake in the grammar and found it tricky to get diagnostics. The error messages weren't too informative and I resorted to debuggers and print statements inside the generated parser code. Fortunately the problem was in the tokenizing, so I realized my boo-boo by looking at the output from the lexer. LALR parsers are notoriously hard to follow, so I'm glad I didn't have to delve in there. Antlr scores rather better on this front. Recursive descent parsers are easier to follow and there is an Antlr book in the works which should help me a lot as I explore that.

So far I'm not convinced by the approach of removing parser actions and automatically generating a parse tree. Since it's a parse tree, you have to walk it to do anything useful. Nat Pryce let me know about tree transformation rules in the latest SableCC, which look more useful since it defines an abstract syntax tree rather than a parse tree. You still have to walk it to create objects in the domain model, but it's easier to walk. (The latest version of Antlr has similar features.) One plus in tree walking is that if I'm making changes to the tree walker I don't need to re-generate - which keeps me in IntelliJ. However Antlr, has AntlrWorks which plugs into IntelliJ and looks very nice.

I had a need yesterday to play around with velocity in order to explore some stuff on templating and macros. I like velocity's simple template language, but this was one of those times where I wasn't using it in the context of some Java or .NET work. At that point working with velocity becomes a bit of a pain as you have to setup the context and run the processor in Java.

This kind of situation calls for a scripting language, my preferred scripting language is Ruby, and thus I thought this was a great case for trying JRuby. My conclusion is that it works a treat, but I'll bore you with the details.

I downloaded JRuby and unpacked it in /usr/local/lib with a symlink so I could get to it with /usr/local/lib/jruby.

I then put a simple redirecting shell into /usr/local/bin

JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun
JRUBY_HOME=/usr/local/lib/jruby

/usr/local/lib/jruby/bin/jruby "$@"

I could then run JRuby.

$ jruby -v
ruby 1.8.5 (0) [java]

(Actually it took me ages to get that to work, I eventually found the bug between the chair and the keyboard. I'm too embarrassed to tell you what it was.)

To run velocity you have to add it to JRuby's classpath.

CLASSPATH=path/to/velocity-dep-1.4.jar
export CLASSPATH
jruby "$@"

To help run velocity I wrote a little helper class.

require 'java'

class VelocityLauncher
  def initialize context, template
    @context = context
    @template = template
  end

  include_class 'org.apache.velocity.app.Velocity'
  include_class 'org.apache.velocity.VelocityContext'
  include_class 'java.io.StringWriter'

  def run
    vc = VelocityContext.new(@context)
    writer = StringWriter.new
    Velocity.init
    Velocity.evaluate(vc, writer, "LOG", @template)
    return writer.getBuffer
  end
end

Now I can take write a little template:

This is an announcement from $host

Our chief weapons are:
#foreach ($w in $weapons) 
  - $w 
#end

To populate it, all I need is a ruby hash, which is easy to make.

ct = {'host' => 'Cardinal Fang', 
  'weapons' => ['Fear', 'Surprise', 'Ruthless Efficiency']}
template = File.readlines('template.txt').join

puts VelocityLauncher.new(ct, template).run

I can imagine extending this to a nice command line runner for velocity which would take a context file of the form:

host = 'Cardinal Fang'
weapons = ['Fear', 'Surprise', 'Ruthless Efficiency']

But I don't need that yet, so I'll do it another day (and I'm pretty sure what the binding for 'another day' is).

(Updated with a little more about rake.)

Now that JRuby is getting more and more mature, several people are thinking of finally doing something to improve the world of build scripts by replacing ant with rake.

My former colleague Matt Foemmel has starting doing this for real and is writing up progress on his FoemBlog. Matt's written more build scripts than most and around 2000 we both made the mistake of thinking an XML based build file was the way to go. We also both now believe you need a full scripting language.

The thing with build scripts is that you need both declarative and procedural qualities. The heart of a build file is defining tasks and the dependencies between them. This is the declarative part, and is where tools like ant and make excel. The trouble is that as builds get more complex these structures aren't enough. You begin to need conditional logic; in particular you need the ability to define your own abstractions. (See my rake article for examples.)

Rake's strength is that it gives you both of these. It provides a simple declarative syntax to define tasks and dependencies, but because this syntax is an internal DomainSpecificLanguage, you can seamlessly weave in the full power of Ruby.

A big issue with Rake for Java builds is that it was tricky to avoid lots of starts of the Java VM. JRake runs on top of JRuby which runs inside the Java VM, so this issue goes away.

One argument against using rake for builds that I often hear is that it adds another language that people have to learn. What this argument misses, is the point that ant is really its own language anyway. The fact that it's compliant XML doesn't alter the fact that you still have to understand how all the various ant tasks work and fit together. Of course if you already know ant, then it is extra effort to learn rake; but given a mind free of both I don't think rake + ruby is any harder, and there's lots of other things a scripting language can do for you. (I believe that every programmer should be comfortable with at least one scripting language - there's just so many useful things you can do with them.)

With so much invested in ant, it will still be around for a while, but we think rake is the better solution for the future.

Recently Apple announced the Time Machine, which is the ability to go back in time and see all the alterations to your files, including finding deleted files. For some of us intense geeks, this is not a new feature. Like others, I put my entire working directory under version control, originally CVS now Subversion, and have thus had the ability to easily look at all the changes to everything I work on. It's such a useful feature that I've wondered before about what it would be like to have MoreVersionControl, and perhaps Time Machine is a step in that direction.

Time Machine is seen as an automated backup system, so it doesn't seem to support the notion of thoughtful commits that a versioning system has. I think this is the best way to go, at least initially, so that people get used to the idea of this sort of system. The time-based browser looks interesting, versioning systems need some rethinking of user-interface - and who better than Apple to do this?

I think the more important step is that making this capability more widely available will give a kick to application developers. In MoreVersionControl I said that not enough applications know how to diff and merge. Perhaps Time Machine will start getting people to think about that and start building these capabilities into applications, this would make versioning much more handy.

Versioning is handy on a single desktop, but as anyone who has used it knows, the real benefit is in collaboration. Software projects see an enormous benefit from using a version control system as a collaboration tool. Other efforts can as well - presentations, white papers, excel models can all benefit for versioned collaboration tools. (Again the lack of intelligent diff and merge is a big barrier to this.) Even lonely me benefits hugely with my MultipleDesktops.

So my hope that is that Time Machine will spur development of applications that are aware of versioning and can take advantage of it, which will in turn shift to more effective collaboration. But in any case, I'd strongly urge you to try doing it now. Subversion is free and easy to set up, and even though applications don't diff and merge well there are worthwhile benefits in collaborating with others using a shared versioned repository. It's much better than keeping track of emailed documents or using an unversioned shared drive.