Martin Fowler's Bliki

ServiceCustodian

2008-11-14T17:07:00-05:00

Let's imagine a pretty world of SOA-happiness where the computing needs of an enterprise are split into many small applications that provide services to each other to allow effective collaboration. One fine morning a consumer service needs some information from a supplier service. The twist is that although the supplier service has the necessary data and processing logic to get this information, it doesn't yet expose that information through a service interface. The supplier has a potential service, but it isn't actually there yet.

In an ideal world the developers of the consumer service just asks the supplier service to develop the potential service and all is dandy. But life is not ideal - the sticking point here is that the developers of the supplier service have other things to do, usually things that are more important to their customer and management than helping out the consumer service team.

Recently I was chatting with my colleague Erik Dörnenburg and he told me about an approach he saw a client use to deal with this problem. They took a leaf out of the open source play-book and made all their services into internal open source systems. This allows consumer service developers write the service themselves.

I'm sure many readers are rolling their eyes at the visions of chaos this would cause, but just as open source projects don't allow just anyone to edit anything; this client uses open-source-style control mechanisms. In particular each service has a couple of custodians - people whose responsibility it is to keep the service in a healthy state. In the normal course of events the consumer developer wouldn't actually commit changes to the supplier source tree directly, instead they send a patch to the custodian. Just like an open-source maintainer, the custodian receives the patch and reviews it to see if it's good enough to commit. If not there's a dialog with the consumer developer.

As Erik knows well from his own open source work, reviewing a patch is much less effort than making a change yourself. So although the custodian approach doesn't entirely eliminate the problem of consumer developers needing to wait on supplier developers, it does a lot to reduce the difficulty. And again following the open-source model, a consumer developer can be made a committer once the custodians are comfortable. This still means that commits can get reviewed by the custodians, but avoids the custodians becoming a bottleneck.

Related to this was their approach to a service registry. We've seen a lot of fancy products being sold to provide service registry capabilities so that people can lookup services and see how to use them. This client discarded them and used an approach that combined wikis with some interesting data mining (more on that soon).

EstimatedInterest

2008-11-06T10:41:00-05:00

TechnicalDebt is a very useful concept, but it raises the question of how do you measure it? Sadly technical debt isn't like financial debt, so it's not easy to tell how far you are in hock (although we seem to have had some trouble with measuring the financial kind recently).

Here's one idea to consider. When a team completes a feature ask them to tell you how long it took them (the actual effort) and how long they think it would have taken if the system were properly clean. The difference between the two is the interest of the technical debt. (So if it actually took them 5 days but they think it would have taken them 3 days with a clean system, then you paid 2 days of effort as interest on your technical debt.)

There are certainly some serious flaws with this technique. The statement of how long it would have taken on a clean system is an estimate based on an imaginary state - so is difficult to make objective. There's the effort in capturing this information, which is easy to get out of hand. But the result may help project a picture of the state of the code-base in a way that's visible to non-technical staff.

Furthermore it may also help with decisions about whether to pay the principal. Some teams like to add technical debt stories to their product backlog - with estimates on how long it would take to remove them. Such technical debt stories are also estimates, but also provide a picture of how much debt has built up. You could get a bit more clever with the estimated interest payments by apportioning them to these debt stories (I spent an extra day on this feature because of the bad state of the flipper module). Comparing interest payments with the principal may help inform a decision about whether to pay off the principal.

I ran into someone recently who tried something a little like this and found it handy, but it's not something I've run into a lot. Certainly there are flaws with doing it - but it may be worth a try for a few iterations.

EarlyPain

2008-11-04T17:26:00-05:00

A few years ago I was talking with a client who told me something he didn't like about the agile approach we were using: "it's doesn't feel right to have these difficulties this early in the project". Contrary to his reaction, in my mind this early pain is one of the great benefits of an agile or indeed any iterative development process.

I have many complaints about the waterfall process, but probably my greatest problem with it is how it tends to defer discovery of problems till late in the project, at which point there's little time or energy to deal with them effectively. Iterative cycles try to flush out as many problems as possible as early as possible. This gives you more time to cope, or at least raises the problems early enough to cancel before investing too much money and effort in a problematic project.

A useful exercise is to reflect on past projects and think about where problems cropped up late. Now ask yourself how you could make those problems crop up earlier. The more pain you get earlier, the better.

UpcomingTalks

2008-11-04T17:24:00-05:00

Neal Ford and Rebecca Parsons will be working with me again on a DSL tutorial at QCon San Francisco. Rebecca and I will do a keynote.

My last scheduled talking arrangement this year is to return to QCon San Francisco. QCon is a organized by the JAOO team and organized in collaboration with InfoQ. It brings the conference formula that I like so well from JAOO to the US.

I've got two talks planned. One is an all day tutorial on DSLs with Neal Ford and Rebecca Parsons. This tutorial is the one we've done at a number of JAOO/QCon conferences in the last year.

The second talk is a keynote with Rebecca on the relationship between agile thinking and enterprise architecture groups. In our work we often have to bridge the gap between our approaches and somewhat traditional enterprise architecture teams - and Rebecca is usually working right on that boundary. So this talk brings our thoughts on how enterprise architecture fits in with an agile mind-set.

Oslo

2008-10-28T17:15:00-04:00

Oslo is a project at Microsoft, of which various things have been heard but with little details until this week's PDC conference. What we have known is that it has something to do with ModelDrivenSoftwareDevelopment and DomainSpecificLanguages.

A couple of weeks ago I got an early peek behind the curtain as I, and my language-geek colleague Rebecca Parsons, went through a preview of the PDC coming-out talks with Don Box, Gio Della-Libera and Vijaye Raji. It was a very interesting presentation, enough to convince me that Oslo is a technology to watch. It's broadly a Language Workbench. I'm not going to attempt a comprehensive review of the tool here, but just my scattered impressions from the walk-through. It was certainly interesting enough that I thought I'd publish my impressions here. With the public release at the PDC I'm sure you'll be hearing a lot more about it in the coming weeks. As I describe my thoughts I'll use a lot of the language I've been developing for my book, so you may find the terminology a little dense.

Oslo has three main components:

a modeling language (currently code-named M) for textual DSLs
a design surface (named Quadrant) for graphical DSLs
a repository (without a name) that stores semantic models in a relational database.

(All of these names are current code names. The marketing department will still use the same smarts that replaced "Avalon and Indigo" with "WPF and WCF". I'm just hoping they'll rename "Windows" to "Windows Technology Foundation".)

The textual language environment is bootstrapped and provides three base languages:

MGrammar: defines grammars for Syntax Directed Translation.
MSchema: defines schemas for a Semantic Model
MGraph: is a textual language for representing the population of a Semantic Model. So while MSchema represents types, MGraph represents instances. Lispers might think of MGraph as s-expressions with a ugly syntax.

You can represent any model in MGraph, but the syntax is often not too good. With MGrammar you can define a grammar for your own DSL which allows you to write scripts in your own DSL and build a parser to translate them into something more useful.

Using the state machine example from my book introduction, you could define a state machine semantic model with MSchema. You could then populate it (in an ugly way) with MGraph. You can build a decent DSL to populate it using MGrammar to define the syntax and to drive a parser.

There is a grammar compiler (called mg) that will take an input file in MGrammar and compile it into what they call an image file, or .mgx file. This is different to most parser generator tools. Most parser generators tools take the grammar and generate code which has to be compiled into a parser. Instead Oslo's tools compile the grammar into a binary form of the parse rules. There's then a separate tool (mgx) that can take an input script and a compiled grammar and outputs the MGraph representation of the syntax tree of the input script.

More likely you can take the compiled grammar and add it to your own code as a resource. With this you can call a general parser mechanism that Oslo provides as a .NET framework, supply the reference to the compiled grammar file, and generate an in-memory syntax tree. You can then walk this syntax tree and use it to do whatever you will - the parsing strategy I refer to as Tree Construction.

The parser gives you a syntax tree, but that's often not the same as a semantic model. So usually you'll write code to walk the tree and populate a semantic model defined with MSchema. Once you've done this you can easily take that model and store it in the repository so that it can accessed via SQL tools. Their demo showed entering some data via a DSL and accessing corresponding tables in the repository, although we didn't go into complicated structures.

You can also manipulate the semantic model instance with Quadrant. You can define a graphical notation for a schema and then the system can project the model instance creating a diagram using that notation. You can also change the diagram which updates the model. They showed a demo of two graphical projections of a model, updating one updated the other using Observer Synchronization. In that way using Quadrant seems like a similar style of work to a graphical Language Workbench such MetaEdit.

As they've been developing Oslo they have been using it on other Microsoft projects to gain experience in its use. Main ones so far have been with ASP, Workflow, and web services.

More on M

We spent most of the time looking at the textual environment. They have a way of hooking up a compiled grammar to a text editing control to provide a syntax-aware text editor with various completion and highlighting goodness. Unlike tools such as MPS, however, it is still a text editor. As a result you can cut and paste stretches of text and manipulate text freely. The tool will give you squigglies if there's a problem parsing what you've done, but it preserves the editing text experience.

I think I like this. When I first came across it, I rather liked the MPS notion of: "it looks like text, but really it's a structured editor". But recently I've begun to think that we lose a lot that way, so the Oslo way of working is appealing.

Another nice text language tool they have is an editor to help write MGrammars. This is a window divided into three vertical panes. The center pane contains MGrammar code, the left pane contains some input text, and the right pane shows the MGraph representation of parsing the input text with the MGrammar. It's very example driven. (However it is transient, unlike tests.) The tool resembles the capability in Antlr to process sample text right away with a grammar. In the conversation Rebecca referred to this style as "anecdotal testing" which is a phrase I must remember to steal.

The parsing algorithm they use is a GLR parser. The grammar syntax is comparable to EBNF and has notation for Tree Construction expressions. They use their own varient of regex notation in the lexer to be more consistent with their other tools, which will probably throw people like me more used to ISO/Perl regexp notation. It's mostly similar, but different enough to be annoying.

One of the nice features of their grammar notation is that they have provided constructs to easily make parameterized rules - effectively allowing you to write rule subroutines. Rules can also be given attributes (aka annotations), in a similar way to .NET's language attributes. So you can make a whole language case insensitive by marking it with an attribute. (Interestingly they use "@" to mark an attribute, as in the Java syntax.)

The default way a grammar is run is to do tree construction. As it turns out the tree construction is the behavior of the default class that gets called by the grammar while it's processing some input. This class has an interface and you can write your own class that implements this. This would allow you to do embedded translation and embedded interpretation. It's not the same as code actions, as the action code isn't in the grammar, but in this other class. I reckon this could well be better since the code inside actions often swamp grammars.

They talked a bit about the ability to embed one language in another and switch the parsers over to handle this gracefully - heading into territory that's been explored by Converge. We didn't look at this deeply but that would be interesting.

An interesting tidbit they mentioned was that originally they intended to only have the tools for graphical languages. However they found that graphical languages just didn't work well for many problems - including defining schemas. So they developed the textual tools.

(Here's a thought for the marketing department. If you stick with the name "M" you could use this excellent film for marketing inspiration ;-))

Comparisons

Plainly this tool hovers in the same space as tools like Intentional Software and JetBrains MPS that I dubbed as Language Workbenches in 2005. Oslo doesn't exactly fit the definition for a language workbench that I gave back then. In particular the textual component isn't a projectional editor and you don't have to use a storage representation based on the abstract representation (semantic model), instead you can store the textual source in a more conventional style. This lesser reliance on a persistent abstract representation is similar to Xtext. At some point I really need to rethink what I consider the defining elements of a Language Workbench to be. For the moment let's just say that Xtext and Oslo feel like Language Workbenches and until I revisit the definition I'll treat them as such.

One particularly interesting point in this comparison is comparing Oslo with Microsoft's DSL tools. They are different tools with a lot of overlap, which makes you wonder if there's a place for both them. I've heard vague "they fit together" phrases, but am yet to be convinced. It could be one of those situations (common in big companies) where multiple semi-competing projects are developed. Eventually this could lead to one being shelved. But it's hard to speculate about this as much depends on corporate politics and it's thus almost impossible to get a straight answer out of anyone (and even if you do, it's even harder to tell if it is a straight answer).

The key element that Oslo shares with its cousins is that it provides a toolkit to define new languages, integrate them together, and define tooling for those languages. As a result you get the freedom of syntax of external DomainSpecificLanguages with decent tooling - something that deals with one of the main disadvantages of external DSLs.

Oslo supports both textual and graphical DSLs and seems to do so reasonably evenly (although we spent more time on the textual). In this regard it seems to provide more variety than MPS and Intentional (structured textual) and MetaEdit/Microsoft's DSL tools (graphical). It's also different in its textual support in that it provides real free text input not the highly structured text input of Intentional/MPS.

Using a compiled grammar that plugs into a text editor strikes me as a very nice route for supporting entering DSL scripts. Other tools either require you to have the full language workbench machinery or to use code generation to build editors. Passing around a representation of the grammar that I could plug into an editor strikes me as a good way to do it. Of course if that language workbench is Open Source (as I'm told MPS will be), then that may make this issue moot.

One of the big issues with storing stuff like this in a repository is handling version control. The notion that we can all collaborate on a single shared database (the moral equivalent of a team editing one copy of its code on a shared drive) strikes me as close to irresponsible. As a result I tend to look askance at any vendors who suggest this approach. The Oslo team suggests, wisely, that you treat the text files as the authoritative source which allows you to use regular version control tools. Of course the bad news for many Microsoft shops would be that this tool is TFS (or, god-forbid, VSS), but the great advantage of using plain text files as your source is that you can use any of the multitude of version control systems to store it.

A general thing I liked was most of the tools leant towards run-time interpretation rather than code generation and compilation. Traditionally parser generators and many language workbenches assume you are going to generate code from your models rather than interpreting them. Code generation is all very well, but it always has this messy feel to it - and tends to lead to all sorts of ways to trip you up. So I do prefer the run-time emphasis.

It was only a couple of hours, so I can't make any far-reaching judgements about Oslo. I can, however, say it looks like some very interesting technology. What I like about it is that it seems to provide a good pathway to using language workbenches. Having Microsoft behind it would be a big deal although we do need to remember that all sorts of things were promised about Longhorn that never came to pass. But all in all I think this is an interesting addition to the Language Workbench scene and a tool that could make DSLs much more prevalent.

ObservedRequirement

2008-09-16T19:12:00-04:00

Here's one of my favorite software development quotes:

Requirements are the things that you should discover before starting to build your product. Discovering the requirements during construction, or worse, when you client starts using your product, is so expensive and so inefficient, that we will assume that no right-thinking person would do it, and will not mention it again.

--Suzanne and James Robertson

It's the opening paragraph of the first edition of their book "Mastering the Requirements Process". As regular readers might guess, my liking isn't connected to agreement. I like this quote because it sums up the waterfall value system to requirements (indeed the word 'requirement' is inherently waterfallish).

Agile methods violate this underlying assumption by intending to discover the 'requirements' during construction and after delivery. But even this cavalier disregard of the above sage advice is nothing compared to what many leading web sites do these days. These sites explore requirements by observing what the users do on their sites and using that information to generate ideas for new features along the following lines:

Look at what people are trying to do with the site and provide easier ways for them to do it.
Look at where people are abandoning doing something, and look for ways to fix whatever was frustrating them.
Build a new feature and see if people use it.
Build an experimental feature and make it available to a subset of the user base. Not just can you see if they like it, you can also assess how much load it puts on your servers.

To support this kind of analysis, you need to add user logging behavior into the application and build some tools to analyze these logs. Much logging appears for free in web applications, which I suspect was major impetus for people starting to do this. But the logging, and analysis, can go further as it's added to the application.

I haven't found much advice out there on the web on how to do this, and I don't hear much discussion about doing this in practice. Like many things it requires a concentrated effort to spend the time to build monitoring capability and then to use it to probe how to improve the software. Furthermore it's a mighty step away from how the traditional software process, even for agile projects.

But there is a huge potential here. Everyone knows how big the difference is between what people say they want and what people actually need and use. By watching what people actually do with your application, you can find out what actually happens with the software - which can give you much more direct information than other sources. As a result I think more teams should consider adding this approach to their toolkit.

EvolutionarySOA

2008-09-12T09:29:00-04:00

Can SOA be done with an agile approach?

I don't delve too much into the cluttered world of SOA (ServiceOrientedAmbiguity), but I get this question often enough (in some form or other) to be worth a pontification.

When I first came across agile software development (in the form of Extreme Programming) the most troubling aspect for me was its approach to software design. Like many I'd become used to the notion that software should be designed before it was programmed, while XP seemed to encourage an almost wilful embracing of design ignorance. In 2000 I was asked to give the closing keynote at the first Agile/XP conference, and in order to gather my thoughts I ended up writing Is Design Dead? - an essay that examined the fate of design in an agile world.

I'm still quite proud of that essay and I think it's well worth reading today - but for this bliki entry I'll summarize. I talked about two driving approaches to software design: planned and evolutionary. Planned design works out the design of software in one phase and builds (programs) it afterwards. In this case changing the design is hard once you've begun construction. Evolutionary design assumes regular change of the design even once you've done significant programming. I concluded that the practices of XP provided a disciplined approach to evolutionary design, thus making it much more practical than people had realized. This change did not remove software design (it is not dead) but did significantly change how we think about design.

The argument for planned design in an SOA context is that we are building webs of interconnected, loosely coupled systems. In this situation each system is making its services available as a PublishedInterface to the whole enterprise. Published interfaces are hard to change, therefore you need planned design to get them right so you don't have to change them. Planned design is also a reaction to the chaotic system interconnections that people see in most organizations. This chaos is the result of poor design, so the feeling is that better planned design will prevent this happening in future.

Evolutionary design is an essential aspect of agile methods. One of the key foundations of agile methods is the desire to handle, indeed welcome, change. Planned design assumes change is hard, and thus tries to predict where it occurs. If changes occur within the predicted boundaries then it's easy, but if it falls outside those boundaries you're out of luck. Agile thinking assumes unpredictable change is inevitable and tries to enable it in a controlled way.

So as I look at SOA, or any other design context, the fundamental question I ask is "is change predictable?" Only if change is predictable is a planned design approach valid. My sense is that if predictability is hard within the context of a single application, it's doubly hard across an enterprise. If we use planned design in a unpredictable context we find that however good the plans are, they will be undermined by the unpredictable changes, leading to the same mess we are currently in. Usually, however, the mess is worse because a there is significant sunk cost in a planned design, this can easily reduce the business value that an SOA effort is intended to produce, particularly if time-to-market is important.

As a result I think we have to bite the bullet and figure out how to do evolutionary design in this loosely connected context. After all the whole of point of loose coupling is to make change easier. At the center of this is thinking about contracts in terms of change, with such ideas as Consumer Driven Contracts.

This direction leads to things like Jim Webber's notion of Guerilla SOA. This builds up SOA using small steps directed by producing business value. Since producing business value is the whole point, this offers a path to producing a much better return on investment. It's certainly an approach our clients appreciate.

Can evolutionary design scale to SOA sizes? In my view we have an existence proof at a much larger scale than even a big SOA effort - the web itself. The web is built on very loose coupling and lots of unpredictable changes. It is, in many ways, a mess - but it's a mess that works, delivering real value to lots of people every day.

DslQandA

2008-09-09T18:33:00-04:00

I was asked to put together a discussion of DSLs for non-technical types. Maybe I've been reading too much Stephen O'Grady, but I felt an irresistible urge to do it in a Q and A manner. So here it comes.

What is a Domain Specific Language?

A Domain Specific Language (DSL) is a computer programming language of limited expressiveness focused on a particular domain. Most languages you hear of are General Purpose Languages, which can handle most things you run into during a software project. Each DSL can only handle one specific aspect of a system.

So you wouldn't write a whole project in a DSL?

No. Most projects will use one general purpose language and several DSLs

Are they a new idea?

Not at all. DSLs have been used extensively in Unix circles since the early days of that system. The lisp community often talks of creating DSLs in lisp and then using the DSLs to implement the logic. Most IT projects use several DSLs - you might of heard of things like CSS, SQL, regular expressions and the like.

So why are they getting a lot of noise now?

Probably because of Ruby and Rails. Ruby as a language has many features that make it easy to develop DSLs and the people who got involved in the Ruby community have been familiar with this approach from elsewhere, so they took advantage of these features. In particular Rails uses several DSLs which have played a big role in making it so easy to use. This in turn encouraged more people to take up these ideas.

Another reason is that many Java and C# systems need to have some of their behavior defined in a more dynamic way. This led to complex XML files that are difficult to comprehend, which in turn led to people exploring DSLs again.

So DSLs can be used with languages other than Ruby?

Yes, as I indicated DSLs have been around for much longer than Ruby has. Ruby has an unobtrusive syntax and meta-programming features that make it easier to create more elegant internal DSLs than languages like C# and Java. But there are useful internal DSLs in Java and C#.

What's the distinction between internal and external DSLs?

An internal DSL is just a particular idiom of writing code in the host language. So a Ruby internal DSL is Ruby code, just written in particular style which gives a more language-like feel. As such they are often called Fluent Interfaces or Embedded DSLs. An external DSL is a completely separate language that is parsed into data that the host language can understand.

Why are people interested in DSLs?

I see DSLs as having two main benefits. The most common benefit is that they make certain kinds of code easier to comprehend, which makes it much easier to modify, thus improving programmer productivity. This is worthwhile all on its own and is relatively easy to achieve.

The most interesting benefit, however, is that a well designed DSL can be understandable by business people, allowing them to directly comprehend the code that implements their business rules.

So is this the hook - business people write the rules themselves?

In general I don't think so. It's a lot of work to make an environment that allows business people to write their own rules. You have to make a comfortable editing tool, debugging tools, testing tools, and so on. You get most of the benefit of business facing DSLs by doing enough to allow business people to be able to read the rules. They can then review them for accuracy, talk about them with the developers and draft changes for developers to implement properly. Getting DSLs to be business readable is far less effort than business writable, but yields most of the benefits. There are times where it's worth making the effort to make the DSLs business-writable, but it's a more advanced goal.

Do you need special (ie expensive) tools?

In general, no. Internal DSLs just use the regular facilities of the programming language that you are using anyway. External DSLs do require you to use some special tools - but these are open source and are very mature. The biggest problem with these tools is that most developers aren't familiar with them and believe they are harder to use than they really are (a problem exacerbated by poor documentation).

There are exceptions on the horizon, however. These are a class of tools that I call a LanguageWorkbench. These tools allow you to define DSLs more easily, and also provide sophisticated editors for them. Tools like this make it more feasible to make business-writable DSLs.

So is this a repeat of the dream of developing software without programming (or programmers)?

That was the intent of COBOL, and I don't think there's any reason to think that DSLs will succeed where COBOL (and so many others failed). What I think is important is that DSLs allow business people and developers to collaborate more effectively because they can talk about a common set of precise rules that are the executable code.

When should I consider making a DSL?

When you are looking at an aspect of system with rich business rules or work-flow. A well-written DSL should allow customers to understand the rules by which the system works.

Isn't this going to lead to a cacophony of languages that people will find hard to learn?

We already have a cacophony of frameworks that programmers have to learn. That's the inevitable consequence of reusable software, which is the only way we can get a handle on all the things software has to do these days. In essence a DSL is nothing more than a fancy facade over a framework. As a result they contribute little complexity over what is already there. Indeed a good DSL should make things better by making these frameworks easier to use.

But won't people create lots of bad DSLs?

Of course, just like people create bad frameworks. But again I'd argue that bad DSLs don't do much additional harm compared to the cost of bad frameworks.

DslBookRoadmap

2008-08-04T11:34:00-04:00

I've hit a significant, if purely internal, milestone in the DSL book recently. I also find that people regularly ask me what the status is of the book is. So it seems like a good moment to post a note about where I am with the book and where I see things going.

The most common question I get is "when will the book be out?" The way things look at the moment, I'd estimate it appearing some time in 2010. That's an estimate, with all the usual hedges that apply to such things.

To help see how the current state meshes in with that estimate, I'll describe the general progress of how books work, at least for me. The first phase is writing the First Review Draft. This means writing all the material in the book, to the level that I can submit it to peer review. For my larger books (eg Refactoring, P of EAA) this takes about 12-18 months. This book is taking longer, I've already been at it for nearly two years, and I suspect there's another year to go. Once I have the First Review Draft, it goes out to review. This takes time for people to read it, get their comments to me, and for me to modify the text based on the review. I usually do two rounds of review - and it takes around 6 months for that to happen. Once I'm done with formal review the book is a Final Draft. At this point it goes off for production - which includes copy-edit, indexing, layout, printing, etc. That takes another 6 months. So I estimate that the book will appear about a year after I get a First Review Draft.

Not all authors work this way, some write a few chapters and sent them for review while they write a few more chapters. I use the approach I do because it was how I did my first book and it seemed to work very well.

To describe the current state of the book, I need to talk a bit about its structure. The final structure of the book will be a DuplexBook, but while I'm working on it I tend to think of it in terms of a set of subject areas, where each subject area will end up as a set of narratives and topics within the duplex structure.

My recent milestone was completing my first coherent pass at the internal DSL subject area. By this I mean I now have what I think is a coherent draft of the that subject area. It will still need more work before I get to the First Review Draft, but reaching a coherent draft is a big point for me because it means I have reached a point where it's looking in good shape. Here's my current list of subject areas and their status:

Introduction: coherent (this is the first three narratives: An Introductory Example, Using Domain Specific Languages, and Implementing DSLs).
External DSLs: incoherent
Internal DSLs: coherent
Code Generation: coherent, but with some important bits missing
Alternative Computational Models: incoherent
Error Diagnostics: open.
Language Workbenches: open.

I've been working since March on getting the internal DSL section into a coherent state. I suspect it will take me a similar amount of time to get external DSLs into coherence, and a similar time again to get alternative computational models into coherence. This is why I think I'm at least a year off First Review Draft. These are also the next two subject areas that I'm going to work on, in that order.

As well as those two subject areas to make coherent, I also have some holes to fill and a couple of elephants to deal with. The holes are topics within subject areas that aren't really complete yet, although the overall structure is reasonable. Code generation is a good example of that. I'm pretty happy with what I have, and it is coherent. But I need to address at least one topic (the Generation Gap pattern) before I can really say that subject area is done. These holes don't take a huge amount of time to fill and I tend to work on them as the mood takes me - often I like to take a break in the middle of a major topic for something like this.

The two elephants are the open topics of error diagnostics and language workbenches. These are elephants because I don't know how I'm going to tackle them, or indeed if I'm going to put much effort into tackling them. It may be that I'll do little more than a skimpy overview chapter, or I may spend six months each on them. I'm deliberately putting off those decisions while I work on the next two subject areas.

Of course this road-map could all change. One of the reasons I don't like doing incremental review (eg sending off the internal DSL section for a formal review now) is that working on one section can cause me to completely rethink another. This has already happened. I did a substantial chunk of work on the internal DSL section late in 2007 and then worked for a while on what is now the Computational Network and Dependency Network patterns at the end of 2007. Working on those patterns made me realize my internal DSL thinking was all wrong and I thus needed to rewrite that whole section. This kind of thing is common with a topic area like this where there is no pre-existing structure. (This is different to a book on an existing technology, eg UML Distilled, where the thing I'm describing already has an understood structure before I start.)

Having said that, if you're interested in what I'm doing I think I'm at the point where you can take a good look at the introductory and internal DSL sections and it should make sense. I'd certainly be happy for feedback on those sections. Just beware that there are cross links between those and other sections that may still get worked on.

I'll update this page when I hit a major milestone or major re-plan. If you want more granular updates I have a more detailed feed. If you want to send me comments on work so far, please do so. I have a mailing list for the book, I'd prefer all comments to go to that mailing list so that there's an opportunity for discussion. So if you do send me comments, please let me know if you're happy with them going on the list or not and whether you wish to join the list.

MDSDandDSL

2008-07-14T17:10:00-04:00

What is the connection between ModelDrivenSoftwareDevelopment (MDSD) and DomainSpecificLanguages (DSLs)?

It's pretty common to see the term "DSL" crop up in the context of MDSD. Indeed some MDSD people seem to think that DSLs only exist within the MDSD world. I've been writing a lot on DSLs recently for my book, but so far I haven't really touched on the MDSD angle much Instead I've concentrated on DSLs role in more conventional programming. DSLs exist in both the textual language and MDSD worlds and play pretty much the same role for both.

In an MDSD context DSLs are again a language targeted at a specific kind of problem as opposed to general purpose languages such as the UML. As a result they can have the same kind of relationship: build a system in the general purpose modeling language and use DSLs for various specific aspects. Since MDSD hasn't caught on that much, however, you also see a different approach where modeling DSLs are used in the context of a traditional language environment. Here you might use several modeling DSLs that generate Java code to be combined in a Java project. In this case there's no general purpose MDSD model around - you use MDSD for each DSL relatively independently.

In order to use model-oriented DSLs you need a different, RepositoryBasedCode, approach to tooling. This introduces quite a few pragmatic issues as the general support environment for such tools is less established. In order to define your own DSLs you need more specialized tooling - something I call a Language Workbench.

DSLs seem to have a proportionately higher emphasis in the MDSD world than they do in the mainstream programming world. Cynics think this is a result of the MDSD community desperately searching for a way to remain relevant, fans of MDSD regard it as a sign of MDSD's superior sophistication. I think this is mainly due to the fact that the MDSD community is smaller and has far less in the form of established practice.

A particularly visible sub-community of MDSD is centered around ModelDrivenArchitecture (MDA). I'm not much of a fan of MDA in particular, but am particularly skeptical of MDA DSLs.

There is much that model-oriented DSLs share with textual DSLs. I put a lot of emphasis with textual DSLs in basing work around a Semantic Model. MDSD, as its name indicates, is very much about driving a system from that kind of a model. A difference is that most MDSD people assume that you'll want to generate code from that model rather than executing the model directly.

As I write this, I'm not sure how much I'm going to cover language workbenches in my book. Certainly I'll at least discuss the overall concept behind them, but the coverage may not be that deep. This will be partly due to the large amount of material I seem to be generating on textual DSLs and partly due to the fact that language workbenches are much newer and thus more volatile and less mature.