Figure 1:
WORK-IN-PROGRESS: - this material is still under development
Last significant update: 13 Apr 09
The techniques I've written about so far have been around, in some form or another, for a long time. The tools that exisit to support them, such as parser generators for external DSLs, are similarly well-seasoned. In this chapter I'm going to spend some time looking at a set of tools that are rather more new and shiny, tools that I call Language Workbenches.
Language Workbenches are, in essence, tools that support creating DSLs in the style of modern IDEs. The idea is that these tools don't just provide an IDE to help create DSLs, they support building IDEs for editing the DSLs. This way someone writing a DSL script has the same degree of support that a programmer does using a post-intellij IDE.
As I write this, the language workbench field is still very young. There are few tools that are really out enough to make use of. Yet there's immense potential here - these are tools that could change the face of programming as we know it. I don't know whether they will succeed in their endevours, but I am sure that they are worth keeping an eye on.
This immaturity means that there's a strong chance that much of what I write in this chapter will be out of date by the time you read it. As ever in my work, I'm looking for core principles that don't change so much, but they are hard to identify in such a rapidly moving field. So take this with caution and keep an eye on the web to find out about more recent developments.
Although language workbenches differ greatly in what they look like, there are common elements that they share. In particular language workbenches allow you to define three aspects of a DSL environment:
There's much in the above bullets that won't make sense to you on first reading, but don't worry I'll reveal all in a moment.
Throughout this book I've been stressing the usefulness of using a Semantic Model. Every language workbench I've looked at uses a Semantic Model and provides tools to define it.
There is a notable difference between language workbenches' models and the Semantic Models I've used so far in this book. As an OO bigot I naturally build an object oriented Semantic Model that combines both data structure and behavior. However language workbenches don't work that way. They provide an environment for defining the schema of the model, ie its data structure, usually using a particular DSL for the purpose - the schema-definition language. They then leave behavioral semantics as a separate exercise, usually through code generation.
At this point, the word "meta" begins to enter the picture and things start getting like a drawing from Escher. This is because the schema-definition language has a semantic model, which is itself a model. The schema-definition language's Semantic Model is the meta-model for a DSL's Semantic Model. But the schema definition language itself needs a schema, which is defined using a Semantic Model whose meta-model is the schema definition language whose meta-model is the.... (swallows self).
If the above paragraph didn't make perfect sense to you (and it only makes sense to me on Tuesdays) then I'll take things more slowly.
I'll begin with a fragment of the secret panel example, specifically the movement from the active state to the waiting for light state. I can show this fragment with the following state diagram.
This fragment shows two states and a transition connecting them. With the Semantic Model I showed in the introduction I interpret this model as two instances of the state class and one instance of the transition class using the java classes and fields I defined for the Semantic Model. In this case the schema for the Semantic Model is java class definitions. In particular I need four classes: state, event, string, and transition. Here's a simplified form of that schema.
class State {
...
}
class Event {
...
}
class Transition {
State source, target;
Event trigger;
...
}
The java code is one way to represent that schema, here's another way using a diagram.
The schema of a model defines what you can have in the contents of the model. I can't add guards to my transitions on my state diagram unless I add them in the schema. This is just the same as any data structure definition: classes and instances, tables and rows, record types and records. The schema defines what goes into the instances.
The schema in this case is java class definitions, but instead I can have the schema be a bunch of java objects rather than classes. This would allow me to manipulate the schema at runtime. I can do a crude version of this approach with three java classes for classes, fields, and objects.
class MClass... private String name; private Map<String, MField> fields;
class MField... private String name; private MClass target;
class MObject... private String name; private MClass mclass; private Map<String, MObject> fields;
I can use this environment to create a schema of states and transitions.
private MClass state, event, transition;
private void buildTwoStateSchema() {
state = new MClass("State");
event = new MClass("Event");
transition = new MClass("Transition");
transition.addField(new MField("source", state));
transition.addField(new MField("target", state));
transition.addField(new MField("trigger", event));
}
Then I can use this schema to define the simple state model of Figure 1.
private MObject active, waitingForDraw, transitionInstance, lightOn;
private void buildTwoStateModel() {
active = new MObject(state, "active");
waitingForDraw = new MObject(state, "waiting for draw");
lightOn = new MObject(event, "light on");
transitionInstance = new MObject(transition);
transitionInstance.set("source", active);
transitionInstance.set("trigger", lightOn);
transitionInstance.set("target", waitingForDraw);
}
It can be useful to think of this structure as two models. The base model is the fragment of Miss Grant's secret panel, it contains the MObjects. The second model contains the MClasses and MFields and is usually referred to as a meta-model. A meta-model is a model whose instances define the schema for another model.
Since a meta-model is just another Semantic Model, I can easily define a DSL to populate it just like I do for its base model - I call such a DSL a schema-definition language. A schema-definition language is really just a form of data model, with some way of defining entities and relationships between them. There's lots of different schema-definition languages and schemas or meta-models out there.
When rolling a DSL by hand, there usually isn't much point in
creating a meta-model. In most situations using the structural
definition capability of your host language is the best bet. By
using the language you have, it's much easier to follow as you are
using familiar language constructs both for the schema and for the
instances. In my crude example, if I want to find the source state
of my transition I have to say something like
aTransition.get("source") rather than
aTransition.getSource() which makes it much harder to
find what fields are available, forces me to do my own type
checking, and so on. I'm working despite my language rather than
with my language.
Perhaps the biggest argument for not using a meta-model in this situation is that I lose the ability to make my Semantic Model a proper OO domain model. While the meta-model does a tolerable, if kludgy, job of defining the structure of my Semantic Model, it's really hard to define its behavior. If I want proper objects that combine both data and behavior, I'm much better off with using the language's own mechanism for schema definition.
These trade-offs work differently for language workbenches. In order to provide the kind of tooling that a language workbench provides, the workbench needs to examine and manipulate the schema of any model I define. This manipulation is usually much easier when using a meta-model. In addition the tooling of language workbenches overcomes many of the common disadvantages of using a meta-model. As a result most language workbenches use meta-models. The workbench uses the model to drive the definition of editors and to help with adding in the behavior that can't exist in the model.
The meta-model, of course, is just a model. As with any other model, that meta-model has a schema to define its structure. In my crude example that schema is one that describes MClass, MField, and MObject. But there's logically no reason why this schema can't be defined using a meta-model. This then allows you to use the workbench's own tools for working with models to work on the schema-definition system itself, allowing yourself to create meta-models using the same tools that are used to write DSL scripts. In effect the schema-definition language is itself just another DSL in the language workbench.
Many language workbenches take this approach, which I refer to as a bootstrapped workbench. In general a bootstrapped workbench gives you more confidence that the modeling tools will be sufficient for your own work as the tool can define itself.
But this is also the point where start to think of yourself as inside an escher drawing. If models are defined using meta-models, which are just models that are defined using meta-models - where does the it all end? In practice the schema-definition tools are special in some way and there's some stuff that's hard-coded into the workbench to make it work. Essentially the special thing about a schema-definition model is that it's capabable of defining itself. So although you can imagine popping up an infinite ladder of meta-models, at some point you reach a model that can define itself. That's quite weird in its own way, of course. On the whole I find it's easiest not to think about it too hard, even on a Tuesday.
A common question is what is the difference between a schema-definition language and a grammar. The short answer is that a grammar defines the concrete syntax of a (textual) language, while the schema definition language defines the structure of the schema of a Semantic Model. A grammar will thus include lots of things that describe the input language while a schema definition language will be independent of any DSL used to populate the Semantic Model. A grammar also implies the structure of the parse tree, together with tree construction rules it can define the structure of a syntax tree. But a syntax tree is usually different to a semnatic model (as I've discussed elsewhere)[TBD: add link].
[TBD: Talk about structural constraints]One of the most notable features of many language workbenches is that they use a projectional editing system, rather than the source editing system that most programmers are used to. A source based editing system defines the program using a represenation that's editable independently of the tools used to process that representation into a running system. In practice that representation is textual, which means the program can be read and edited with any text editing tool. This text is the source code of the program. We turn it into an executable form by feeding the source code into a compiler or interpreter, but it is the source that is the key representation that we programmers edit and store.
With a projectional editing system, the core representation of the program is held in a format specific to the tool that uses it. This format is a persistant representation of the Semantic Model used by the tool. When we want to edit the program, we start up the tool in its editing environment and the tool can then project editable representations of its Semantic Model for us to read and update. These representations may be text, or they may be diagrams, tables, or forms.
Desktop database tools, such as Microsoft Access, are good examples of projectional editing systems. You don't ever see, let alone edit, textual source code or an entire Access program. Instead you start up access and use various tools to examine the database schema, reports, queries, etc.
Projectional editing gives you a number of advantages over a source based approach. The most obvious of these is that it allows editing through different representations. A state machine is often best thought of in a diagrammatic form, with a projectional editor you can render a state machine as a diagram and edit it directly in that form. With source you can only edit it in text, although you can run that text through a visualizer to see the diagram you can't edit that diagram directly.
A projection like this allows you to control the editing experience to make it easier to enter the correct information and disallow incorrect information. A textual projection can, given a method call on an object, only show you the legal methods for that classs and only allow you to enter a valid method name. This gives you a much tighter feedback cycle between the editor and program and allows the editor to give much more assistance to the programmer.
You can also have multiple projections, either at the same time or as alternatives. A common demonstration of the Intentional Software's Language Workbench shows a conditional expression in a C-like syntax. With the press of a menu you can switch that same expression to a lisp-like syntax, or a tabular form. This allows you use whichever projection best fits the way you want to look at the information for the particular task at hand, or programmer preference. Often you want multiple projections of the same information - such as showing a class's superclass as a field in a form and also in a class hierarchy in another pane of the editing environment. Editing either of these updates the core model which in turn updates all projections.
These representations are projections of an underlying model, and thus encourage semantic transformations of that model. If we want to rename a method, this can be captured in terms of the model rather than in terms of text representations. This allows many changes to be made in semantic terms as operations on a semantic model, rather than in textual terms. This particularly helps with capturing refactorings in a safe and efficient manner.
Projectional editing is hardly new, it's been around for at least as long as I've been programming. It has many advantages, yet most serious programming we do is still source-based. Projectional systems lock all programming into a specific tool which not just makes people nervous about vendor lock-in, but also makes it hard to create an ecosystem where multiple tools collaborate over a common representation. Text, despite its many faults, is a common format; so tools that manipulate text can be used widely.
A particularly good example of where this has made a big difference is in source-code mangagement. There's been a great deal of interesting developments in source code mangagement over the last few years introducing concurrent editing, representation of diffs, automated merging, transactional repository updates, and distributed version control. All of these tools work on a wide range of programming environments because they operate only on text. As a result we see a sad situation where many tools that could really use intelligent repositories, diffs, and merges aren't able to do so. This problem is a big deal for larger software projects, which is one reason why larger software systems still tend to use source-based editing.
Source has other pragmatic advantages. If you want to send someone an email to say how to do something, it's easy to throw in a text snippet, while explaining through projections and screenshots can be much more trouble. Some transformations can be automated very well with text processing tools which is very useful if a projectional system doesn't provide a transformation you need. And while a projectional system's ability to only allow valid input can be helpful, it's often useful to put type in something that doesn't work immediately as a temporary step while thinking through a solution. The difference between helpful restriction and binds on thinking is often a subtle one.
One of the triumphs of modern IDEs is that they provide a way to have your cake and eat it. In this approach you work fundamentally in a source-based way, with all the advantages that implies. However when you load your sources into an IDE it creates a semantic model that allows it to use all the projectional techniques that make editing easier, an approach I call model-assisted source editing. Doing this requires a lot of resources, the tool has to parse all the sources and needs a lot of memory to keep the semantic model, but the result is somewhat the best of both worlds. To be able to do this, and to keep the model updated as the programmer edits, is a difficult task.
One concept I find handy when thinking about the flow of source and projectional editing is the notion of representational roles. Source code plays two roles: it is the editing representation (the representation of the program that we edit) and the storage representation (the representation that we store in persistant form). A compiler changes this representation into one that is executable - that is one that we can run on our machine. With an interpreted language the source is an executable representation as well.
At some point, such as during compilation, an abstract representation is generated. This is a purely computer-oriented construct that makes it easier to process the program. A modern IDE generates an abstract representation in order to assist editing. There may be multiple abstract representations, the one the IDE uses for editing may not be the same as the AST used by the compiler. Modern compilers often create multiple abstract representations for different purposes, a syntax tree for some things, a call graph for others.
With projectional editing these representations are arranged differently. The core representation is the Semantic Model used by the tool. This representation is projected into multiple editing representations. The model is stored using a separate storage representation. The storage representation may be human-readable at some level - eg XML - but it isn't a representation any sane person would use for editing.
[TBD: Sort out sections on editing and code generation]
[TBD: Talk about LW DSLs being closer to Excel]
[TBD: Historical origins of LWs - CASE tools vs language people.]
