Martin Fowler's Bliki

Hang around my colleagues at ThoughtWorks and you soon get the impression that the only good Enterprise Service Bus (ESB) is a dead ESB. Jim Webber refers to them as Egregious Spaghetti Boxes. So it's not uncommon to hear tales of attempts to get them out of systems that don't need them.

Battle was joined at one client and it brought to mind my younger days playing D&D. Webber swings but misses as the ESB is AC 2, Evan gets a hit and rolls 2d8 for 6 damage. Erik finally kills it by casting "Summon Request Stream Map".

So what was Erik's decisive spell? Essentially the idea was to take a simple request and show how the data for the request and response made their way through the layers of the application. Erik printed out all the code that you needed to read to understand how this would work - which ran to several pages. He also produced this diagram.

It's currently fashionable in agile circles to do Value Stream Mapping as a way to uncover waste in a software development process. I think of this as a request stream map because it similarly takes a request and shows how it moves through the layers allowing us to visualize what's going on and think about the cost and value of the layers.

Layering is an essential tool for building software applications. But like most essential things in life, excess can be almost as much of a problem as too little. A visualization like this (or the multiple pages of code) can help you find where "just enough" is.

One hazard, however. If you do need to transform data from one form to another, it's usually better to a few little transformations than one big transformation. You want to avoid unnecessary transformations not compress the ones you need.

What's the most common programming language in the world?

I'm not sure how you could go about measuring this, but one thing you'd need to do is consider what we mean by programming. My candidate answer considers that the most popular programming language is one used widely by people who do not consider themselves as programmers. This language is Excel, or more generally spreadsheets.

Spreadsheets are easily used for small tasks, but are also used for surprisingly complex and important things. Often I've seen professional programmers gulp when they realize that some vital business function is being run off some spreadsheet that they'd find too complicated to muck with.

In general, we've not had much success with programming languages for these kind of LayProgrammers. Whenever someone talks about some new environment that's going to allow people to specify complex behavior "without programming" I mention COBOL, which was originally designed to get rid of programmers. So it's important to consider what Excel can teach us about programming environments.

One property of spreadsheets, that I think is important, is its ability to fuse the execution of the program together with its definition. When you look at a spreadsheet, the formulae of the spreadsheet are not immediately apparent, instead what you see is the calculated numbers - an illustration of what the program does.

Using examples as a first class element of a programming environment crops up in other places - UI designers also have this. Providing a concrete illustration of the program output helps people understand what the program definition does, so they can more easily reason about behavior.

So why do I feel we need this particular Neologism? Essentially because I think it deserves more thought. We pass by illustrative programming examples without really thinking about them or what makes them special - or even that they are special in some way. We've used illustrative programming for years, but we've not paid enough attention to it. We've not thought enough about what are its essential qualities and what its strengths and weaknesses are.

I've chosen the term "Illustrative Programming" to describe this, partly because "example" is so heavily used (and illustration isn't) but also because the term "illustration" reinforces the explanatory nature of the example execution. Illustrations are meant to help explain a concept by giving you a different way of looking at it - similarly an illustrative execution is there to help you see what your program does as you change it.

When trying to make a concept explicit like this, it's useful to think about the boundary cases. One boundary is the notion of using projections of program information during editing, such as an IDE that shows you the class hierarchy while you are working on the code. In some ways this is similar, as the hierarchy display is continuously updated as you modify the program, but the crucial difference is that the hierarchy can be derived from static information about the program. Illustrative programming requires information from the actual running of the program.

I also see illustrative programming as a concept beyond the classic REPL loop of dynamic languages. REPL loops allow you to explore execution, but they don't make the examples front and center in the way that a spreadsheet does its values. Illustrative programming techniques put the illustration in the foreground of your editing experience. The program retreats to the background, peeping out only when we want to explore a part of the illustration.

I don't think that illustrative programming is all goodness. One problem I've seen with spreadsheets and with GUI designers is that they do a good job of revealing what a program does, but de-emphasizes program structure. As a result complicated spreadsheets and UI panels are often difficult to understand and modify. They are often riven with uncontrolled copy-and-paste programming.

This strikes me as a consequence of the fact that the program is de-emphasized in favor of the illustrations. As a result the programmers don't think to take care of it. We suffer enough from a lack of care of programs even in regular programming, so it's hardly shocking that this occurs with illustrative programs written by lay programmers. But this problem leads us to create programs that quickly become unmaintainable as they grow. The challenge for future illustrative programming environments is to help develop a well structured program behind the illustrations - although the illustrations may also make us rethink what a well structured program is.

The hard part of this may well be the ability to easily create new abstractions. One of my observations of rich client UI software is that they get tangled because the UI builders think only in terms of screens and controls. My experiments here suggest to me that you need to find the right abstractions for you program, which will take a different form. But these abstractions won't be supported by the screen builder as it can only illustrate the abstractions it knows about.

My colleagues Rebecca Parsons and Neal Ford have been spending a lot of time involved in thinking along these lines too. So here's some thoughts that Neal had in an email exchange

One of the few people to take illustrative programming seriously is Jonathan Edwards. He's come up with many very imaginative ideas as to what such an environment should look like. His vision of illustrative programming is also closely bound to the notions of projectional editing and controlled copy-and-paste.

The trigger for me in wanting to coin a term here, is the use of illustrative programming by Language Workbenches by people like IntentionalSoftware. These Language Workbenches encourage you to build illustrative DSLs. Using illustration is important in this case since this should help engage lay-programmers, which is one of the aims of using DSLs. The challenge is to do this without falling into the trap of poor program structure.

One of the most striking things about the Manifesto for Agile Software Development is the format of its values "we favor x over y". I don't remember who came up with that idea, or how it arose. It probably just bubbled up as we tossed around ideas. But it's distinctive format has led a few people to try using that format again.

If you fancy trying this form, there's a couple of things to remember about it, things that people don't always realize. The most important of these is that the unfavored values are valuable too. A phrase like "solving world hunger over slavery" doesn't carry much power because hardly anyone is openly in favor of slavery. The format works when both left and right sides of the "over" are valuable things that most people want. What the comparative values then say is that we want both things, but if push comes to shove we prefer the one on the left. The harder choice there is between right and left, the better the value statement.

So a good mental test is to imagine someone reversing each value statement, indeed reversing all that you have in the set. There should be people that you can imagine proudly and reasonably supporting that opposite position. In our case we saw much of the industry heading towards high-ceremony processes - reversing the values we felt fairly summed up the values of that community. I can easily imagine writing an article extolling why the reverse set of values are a coherent world-view for software development by putting myself into that mind-set.

The right-hand values may be the current state of the world you want to change, or they may be a future state desired by another community. Either way the comparative values are there to highlight the contrast between one and the other.

Another point about the manifesto that I like is its brevity: four comparative values and twelve principles. It's hard to get that kind of brevity, but the briefer you make it - the punchier it is. I'm sure the manifesto would have not had the impact it did if it had forty-six value statements.

I'm one of those who has used this format since. Done well it can really highlight what makes a particular philosophy different to another. Here is another sample, a set of values I wrote to describe how I saw ThoughtWorks as being different from other software organizations.

• Leveraging bright people over Making the most of moderate people
• Flexible career paths over Well-defined roles
• Delivering business value over Leading edge research
• Learning new technologies over Mastering established technologies
• Solving difficult problems over Increasing market share
• Learning from mistakes over Avoidance of taking risks
• Delivery to the client over Quarterly results

We've since replaced this list with another set of principles which try to capture how we want to be. But we did use the comparative values for a while to try to explain both to ourselves and others what made our aspirations different.

As such I think that not just is this format a good way to sum up a world view, when done well it also leads to sharp discussions about what people really want to care about. This usefulness comes directly from the fact that you are making hard choices between things that are all desirable.

Recently some of our developers ran into the accusation that with a dynamic language like ruby you use so many dynamic type checks that you end up effectively writing your own type system. So they thought, since we've written a lot of real ruby code - how often do we make dynamic type checks? Michael Schubert gathered up the data.

The table below contains the data. We define a dynamic type check as the use of the methods is_a?, kind_of?, and instance_of?. The lines of code come from the standard rake stats command in rails.

The moral of this data is that you shouldn't expect to see a lot of type check calls in your ruby code base. This, of course, is true of any dynamic language. It was generally considered bad form in Smalltalk circles I inhabited too.

The methods that were checked for in this data aren't the only ones that can be considered a dynamic type check. Other cases are respond_to? and aClass === anInstance. Our folks felt that these cases were no more common than the ones they checked for.

Most uses are those of dealing with liberal input - eg where a method parameter can be a string, symbol, or array. These crop up in DSLish situations where you want liberal input for the high readability.

A couple of weeks ago there was a Ruby conference in San Francisco called GoGaRuCo (Golden Gate Ruby Conference). This conference has grabbed attention due to a talk at which the presenter illustrated a discussion of CouchDB by using sexually suggestive pictures of women. Unsurprisingly the result has been a fair bit of heated, and occasionally offensive, debate.

The main lines of the debate are familiar. Various people, not all women, lay the charge that the images and general tone was offensive. Such material makes women feel degraded and alienated. This kind of presentation would not be tolerated at most professional events.

Defenders of the presenter point out that the slides were humorous and no offense was intended. The Rails community has always had a edginess to it - in part because much of the Rails community is focused on the rejection of enterprise values - both technologically and socially. David Heinemeier Hansson is happy to proclaim himself as an R rated individual and is happy to consign "professional" to the same pit to which he cast "enterprise".

I'll admit to finding much to like in the general edginess of the Rails world. Innovation often involves seeing a generally accepted line and vaulting over it. There's plenty of precious posturing around the software world that I'm glad to see skewered. Many of us have been delighted at how Rails has cheekily whacked over-complex frameworks, vendor bloatware, and other assorted ills. An important target of this skewer has been the rise of corporate blandness, where a fear of offense has transformed into a fear of any authentic communication and the rise of the anodyne press release. I'm right with the rails people on this - software is too much fun to shriveled up in dry talks and writing.

So the view of the rails leadership seems to be this: that the objections to the presentation are yet another attempt to foist empty corporate values on the thriving Rails ecosystem.

Except on this occasion I don't see the suits as the people doing the complaining. Most of those calling foul are women who have had to struggle with very real sexism in their careers, and men who have seen this and side with those women. They have been fighting the suits since before the Rails leadership were born, and for much higher stakes.

This incident has now grown beyond a conference presentation and a slide-deck on the web. The issue is no longer the presentation, but the reaction of the community to this event. The leaders, particularly David Heinemeier Hansson as the most visible figure, now face an important time in influencing what the future of the community will be.

The reaction of the rails leadership thus far is to deny the offense. I'll say now that I don't believe they are sexist. I believe that they didn't think the talk would give this much offense - and even that they don't think the talk should give offense.

At this point there's an important principle. I can't choose whether someone is offended by my actions. I can choose whether I care. The nub is that whatever the presenter may think, people were offended - both in the talk and those who saw the slides later. It doesn't matter whether or not you think the slides were pornographic. The question is does the presenter, and the wider community, care that women feel disturbed, uncomfortable, marginalized and a little scared.

It's my view that the people in a community have the power to set the tone of that community, to decide what is and is not acceptable behavior within it. If something questionable happens and people remain silent, that is an implicit acceptance of that event. That is why I feel compelled to write this page, because I think that this talk, and more importantly the rails leadership response to this talk, is objectionable.

My observation is that most men in the software business think that there isn't much sexism left in the profession - that this curse is a memory from a previous generation. Yet when I talk to women, I hear a different story. Nearly every one can tell me recent stories where they were clearly expected to feel degraded and belittled because of their gender. So some sexually suggestive pictures aren't a joke to them, they are a pointed reminder of disturbing behavior, and a reminder that such events can happen again at any time. One of the great difficulties for white guys like me is that we haven't been in that position; where prejudice can appear out of any corner, reinforced by the fact that every other face looks different.

This becomes more of an issue because the rails world faces a notable lack of women. The software world struggles with Diversity as it is. It's a problem for our profession, in that we lose access to talent, and it's a problem for many women who don't get the chance to develop a satisfying career in programming. The open-source world in general has even bigger issue, and the rails community perhaps more so. I'm sure it's not the only factor but the encouragement of talks like this creates an unwelcoming atmosphere of NetNastiness which deters many women from starting and staying in the community.

There also seems to be a generational factor in this. My colleagues have noticed that younger women, typically those under 30, are much less conscious of sexism than their older colleagues. This is partly because of the successes their elders have had in opening up the workplace to women. It may also because younger women haven't yet met the glass ceiling (and I hope it will be gone before they get there). Younger women also seem much more tolerant of sexual imagery. Yet I don't think this is cause for complacency. An important element in nurturing women in our profession is to have role-models who can show what's possible. Alienating older women makes it harder to do that.

So where does this go? I won't attempt to predict the future, but there is a scenario where this little presentation may be seen as a defining event in the rails story. This doesn't mean that people will suddenly leave in droves, but it does begin with a few departures, such as Mike Gunderloy's. The community continues with more alienating events, encouraged by the fact that those who are more sensitive are no longer around to object. This encourages more departures as people don't want to be associated with such a community. Thus develops a positive feedback loop making the rails world increasingly brash and unwelcoming for many of us.

I have a different vision - one that sticks it to the suits so hard it will make their eyes water. How about a community where women are valued for their ability to program and not by the thickness of their skin? How about a community that edgily pushes new boundaries without reinforcing long running evils? Perhaps even a community where women reach equal numbers? Such a community would hand the suits the defeat in the long battle women have been fighting for centuries. I'd love to be part of that.

Further Reading

A selection of commentary on this issue on the web. It's not a comprehensive list, just the items I've felt are particularly interesting and relevant to this story.

• The original presentation on slideshare. (Be warned: some people might not be comfortable looking at these slides at work.) This is almost the presentation that was given, although apparently there were some more racy pictures interspersed between the code examples that were removed for the slideshare version.
• Reaction from Sarah Allen, who was there.
• Reaction from Sarah Mei, who was also there. I found her entries in the comments (starting here) most helpful in understanding her feelings.
• Early reflections on this from Audrey Eschright, which try to explain why many women feel offended by this kind of thing.
• Renae Bair offers a different perspective.
• David Heinemeier Hansson indirectly supports this kind of presentation by explaining why he's an R-rated individual and questioning if software is any different to other professions.
• Mike Gunderloy explains why he's resigned from Rails Activists. Notice the shift in focus now from the original presentation to the reaction to it.
• Matt Aimonetti, the presenter, gives his main post reacting to the objections. Note also that he's posted comments on many of the blogs I've referenced earlier.
• Liz Keogh looks at how talks like this lead to cognitive associations that lead to problems.
• Scott Hanselman looks at the region between political correctness and offense.
• Why the Lucky Stiff, posts a mosaic of reactions from women.
• Piers Cawley explains the difference between rails programmers and 80's truck salesmen.
• Josh Susser, who was responsible for the technical program at the conference, apologizes gracefully and explains the background to selecting the talk.
• Tim Bray adds two observations, one link, and some recommendations
• A similar incident strikes the Flash community.

Some thoughts on common statements

These pictures were less revealing and sexual that what you'd find in a mainstream movie

This is where the context matters. Watching a movie is a different social space to being in a software development talk. As result people react differently.

These kinds of sexualized images have long been associated with men's clubs. Condoning a presentation like this can imply that the powerful (the community leadership) wants this atmosphere, to create a context that excludes women. I don't think the rails leadership actually wants to do this, but if someone did want to do create such a group, this would be a good way to go about it.

Women shouldn't get so annoyed, men don't when women make reverse jokes

If you always worry about people being offended, you'll just end up being bland

Yes, that's a real risk. But being aware of causing offense doesn't mean you have to dial all the way down to corporate blandness. It means thinking how what you will cause offense and being comfortable with the result. You may feel that certain people should be deal with being offended, you may think that only a very small amount of people will be offended. That can be a reasonable response, but it has to be a thoughtful response.

As often, I find my black colleague Chad Wathington puts it well:

The presenter made an apology

The people who were offended are being thin-skinned.

That's a comment often made by those who condone NetNastiness, but it doesn't help those who are offended. The crucial point is: do we want to create an environment where "thin-skinned" people aren't welcome? After all the consequence of a society that is tolerant of nastiness and bullying is one where only the thick-skinned need apply. I'd prefer that people are welcomed for their ability in software development, not their ability to withstand offense.

This goes further than just our profession. To be successful as software developers, we need to collaborate with people in other fields. Tolerating this degree of nastiness makes it difficult for people in other walks of life to work with us, which impoverishes us all.

Humor is an important tool to puncture the self-important

I agree, but this only works if the power relationship is in the right direction. Someone of low power poking fun at a powerful person is a different situation to someone with lots of power skewering someone with little power. Women are (still) in a position of low power in our society (particularly in software development) so we have to be more careful with our humor.

Should be organizer be blamed for this?

If an organizer rejected a talk like this, that would be censorship

You're just trying to impose your moral and 'professional' standards on us

This is no worse than what happens in other professions

Can we get off this subject now and on to important technical issues?

Actually I think a social issue like this is worth spending time on. To build software effectively you have to be able to collaborate with other people, both other programmers and people outside of the software community. Discussions like this help us understand how we relate with other people, which makes a huge contribution to both our professional and personal lives.

Acknowledgements

I always find it particularly difficult to write these kinds of pieces. When I do, I find it particularly valuable to bounce thoughts off several of my colleagues and other friends. My thanks to David Heinemeier Hansson, Jez Humble, John Kordyback, Cyndi Mitchell, Mai Skou Nielsen, Rebecca Parsons, Kathy Sierra, Roy Singham, Chris Stevenson, and Chad Wathington for reading and commenting on the drafts. Thanks also to lots of people who have posted their feelings and analysis both on public channels and internal ThoughtWorks mailing lists. I have learned a great deal in the last few days.

Update: video of the Intentional talk at DSL Devon is now available.

Several years ago, my then colleague Matt Foemmel, dissatisfied with the tools with which we were building software, managed to get in touch with Charles Simonyi to find out more about the shadowy Intentional Software. What he saw impressed him, and he persuaded me and other ThoughtWorkers to get involved too. What we saw was a tool with startling potential, but we remained frustrated by the secrecy and lack of urgency to release. That frustration ended last week.

Last week I set off for Chris Sells's DSL Devcon, and Magnus Christerson - Intentional's product manager - suggested I pop in to see how they were going on. After several years of "Real Soon Now", I was unsure, but Rebecca Parsons, my colleague who has been keeping regular contact with Intentional, said that now would be a good time.

I spent a fascinating and exciting day at their office in Bellevue. It's not that I saw anything particularly new - these were all ideas and capabilities that had been around for a while - but there was a realness and maturity that I hadn't seen before. Indeed Intentional had released a version 1.0 of their product a few weeks earlier. The usual approach is to trumpet a version 1.0 release of a ground-breaking product from the mountaintops. Only Intentional would make such a release and not bother to tell anyone. Indeed as I write this there's no mention of their product on their website - if you want more information you have to talk to them.

What's There

This isn't a comprehensive discussion of their tool (called the Intentional Domain Workbench), I haven't had time to put something like that together. But I hope my scattered thoughts and observations will be interesting. The Intentional Domain Workbench is a LanguageWorkbench, indeed it's one of the systems that made me coin that term. A Language Workbench is a tool that allows people to design DomainSpecificLanguages: not simply to parse them, but build a comprehensive environment that includes rich editing. In Intentional's case this is a ProjectionalEditing environment.

One of the examples they have is the state machine example I use for my book introduction. The workbench allows you to define the schema of the semantic model state machine in its schema definition language. In order to manipulate state machines you define projections of the semantic model. One of the striking features of the Intentional Domain Workbench is its ability to support multiple projections of the same semantic model. For the state machine example they've defined projections in several of the DSLs I've used in discussing the example: XML, custom syntax, and Ruby. All three of these projections are reversible, meaning that you can edit through them, updating the semantic model and other projections. Switching between the projections is just a matter of selecting a menu item.

They also had read-only projections in fluent C#, command-query C, and a state machine diagram. Although they hadn't set up the diagram to be editable, the workbench can handle editable diagrammatic representations. In another example they show an electronic circuit which is editable in both a tree structured property sheet projection and in a circuit diagram projection.

The circuit diagram also showed another really powerful feature of the workbench - the ability to fluidly integrate example executions with the program definition. In the electronic circuit case, this means that you can give the various elements of the circuit properties and the model will calculate the impedance of various parts of the circuit and display them as you are editing the circuit. Of course you can build a custom program to do this kind of thing - but the point is that this behavior comes easily as part of a DSL definition in the workbench.

Combining example execution with program definition is one of the features of spreadsheets - and may be a reason why spreadsheets have become so successful as an environment for LayProgrammers. It's also a notion that's been propelling much of Jonathon Edwards's interesting and wild ideas. My sense is that interesting DSLs in language workbenches will have this characteristic, particularly if they are aimed at being used by lay-programmers.

Another way that you can combine execution with specification is with test cases. They have an example of a pension workbench, build with Capgemini, that allows actuaries to enter formulas using full mathematical notation, together with FIT-like tables to show test cases. These test cases are live with the appropriate red/green behavior as you edit the formulas.

The pension workbench also illustrates the combination of multiple languages. When you look at a pension document on the screen, you're looking at three independent languages: word processed text for the prose, mathematical notation for the formulae, and test case tables. These languages are developed independently but integrated in the workbench's core data structure (called the Intentional Tree). This integration extends to the execution too - you can step into a test case and delve into the intermediate values in the mathematical formulae.

In order to make these things run, you have to include behavior with the semantic model. Intentional have developed their own general purpose language, whose working name is CL1, to do this. CL1 can look like superset of C#, but such a view is again a projection of the core semantic model. I found it interesting that this is a similar feature to JetBrains MPS who have their "base language" which projects into a Java-like general purpose language. Increasingly much of these tools are programmed using this in-workbench general purpose language.

The intended way of working is that developers use the Intentional Domain Workbench to build a domain-specific workbench. They provide a runtime (the Intentional Domain Runtime) for them to run without language editing capabilities. So Capgemini used the Intentional Domain Workbench to build the Pension Workbench as their own product. The Intentional Domain Workbench allows you to define new model schemas and projections, while the Pension Workbench allows you to build pension plans using these languages.

The Intentional system is primarily arranged in the .NET ecosystem. Both the workbench and runtime run on the CLR and core parts of them are written in C#. The workbench makes it really easy to generate .NET assemblies that can be automatically loaded into the workbench for testing or run with the runtime. Custom workbenches can generate code for any environment, and Intentional have done some work with another partner that involves generating Java code so that people can specify behavior in the custom workbench and then deploy the resulting system in a Java environment.

An interesting aspect of the implementation is that they handle representational transformations by using lots of little transformations rather than one large one. As an example, code generating C# from a semantic model involves about a dozen small transforms lined up in a pipeline similar to a multi-stage compiler, the last step being a transformation from a C# AST to text. Much of their internal design goes into making this approach efficient so you can happily string together a lot of small transforms without worrying about any efficiency cost. A further consequence is that the pipeline of transforms for code-generation is very similar to that used for editing projections.

A common problem with tools that use projectional editing is how they deal with version control. Often the answer is to just let multiple people edit the same store simultaneously, which makes many serious developers quake. The Intentional Domain Workbench has a built in version control mechanism that records all the changes made to the Intentional Tree and can do commits and merges at the tree level. You then see diffs in languages by doing another projection.

An interesting feature of this version control approach is that you can commit with conflicts and the conflicts are committed into the repository as conflicts. Unlike with text files they don't mess up your text - you have a real data structure present, so you can find the conflicts and fix them. The developers use this feature to commit a conflict they can't sort out to a branch so that developers more familiar with the conflicted area can update to the branch and fix it.

The fact that editing is done on an intentional tree rather than text also changes some other things. For example unordered collections are tagged so that a change in the ordering of the elements in an editor doesn't trigger a conflict. You can also include domain-specific conflict detection and resolution behavior.

Going Public

Historically the lack of releasing of Intentional has been one problem, their secrecy is another. To see anything real about the Intentional Domain Workbench has required what Neal Ford refers to as an UnforgivenContract. Intentional have given some public talks, but they've really boiled down to saying "trust us, we have some really cool technology". We'd known that indeed they had, but couldn't explain to people why.

So I awaited the talk at DSL DevCon, given by Magnus and Shane Clifford (their development manager), with quite some expectation. They said they were going to finally open the curtain. Would they - and how would people react?

They started worryingly, with the usual unrevealing Powerpoints, but then they switched to showing the workbench and the curtain finally opened. To gauge the reaction, take a look at Twitter.

• @pandemonial Quite impressed! This is sweet! Multiple domains, multiple langs, no question is going unanswered
• @csells OK, watching a live electrical circuit rendered and working in a C# file is pretty damn cool.
• @jolson Two words to say about the Electronics demo for Intentional Software: HOLY CRAPOLA. That's it, my brain has finally exploded.
• @gblock This is not about snazzy demos, this is about completely changing the world we know it.
• @twleung ok, the intellisense for the actuarial formulas is just awesome
• @lobrien This is like seeing a 100-mpg carburetor : OMG someone is going to buy this and put it in a vault!

Afterwards a couple of people said it was the most important demo they'd ever seen, comparing it even to the Mother of all Demos. For many there was a sense that the whole world of software development had just changed.

(Many thanks to Chris Sells and co for organizing this conference and inviting me to speak. They also made a video of the talk available.)

So now what? There's more to all this than a demo can reveal. Right now we want to get several of our hands on the workbench and kick its tires - hard. Assuming it passes that test, we want to use it on commercial projects and see how works for real. No system designed using the Intentional Domain Workbench has yet gone live, and as any agilist knows you never really understand something till you deploy it into production every week.

Shortly the other major similar workbench to this - JetBrains's Meta Programming System - will have version 1.0 released as open-source. So this year could well be the year when these Language Workbenches will finally step out into the light and see their first external pilot projects. (I should also mention that the MetaEdit workbench has been out for a while, although it hasn't had much visibility.) I don't know whether these workbenches will change the face of programming as we know it, after all I once thought Smalltalk was going to be our future; but these workbenches do have the potential to be such a profound change. Certainly I'm excited that we're now on the next, more public, stage of this journey.

Many computer systems are built to house data and turn it into useful information for humans. When we do this there is a natural desire to make that information consistent. After all what use is there of a computer system that's in two minds about things?

But sometimes computer systems should record contradictory data and help humans deal with that. This issue came foremost to my mind many years ago when working in health care for the UK National Health Service. We were building a conceptual model for health care delivery - essentially a conceptual schema for an electronic health care record.

Looking back on it, there were certainly plenty of things I'd do differently now. But one thing in particular was something very precious and important - the model was very much a collaborative effort between myself, another software developer, two doctors and a nurse. The clinicians understood the model and played a full part in developing it - they were not merely passive reviewers. As a result I think the ideas we developed were particularly valuable in thinking about what a clinical practitioner wants to see in an electronic health care record.

One thing the clinicians were very strong about was this need to capture contradictory information. I might have a note from the Royal Hope Hospital saying my blood type is A and another note from the Sisters of Plenitude saying my blood type is B. This would clearly be nonsense, blood types don't change. But that doesn't mean we cannot record these two bits of data. Without further investigation we don't know which one is correct. Even if we test again and confirm one of them, we can't just throw away the bad one as it may have been the basis for further clinical action. And of course there are lots of cases where the contradiction isn't as clear cut. We may never be able to find out which of two contradictory bits of data was wrong or may find a change over time that is extremely unlikely but not impossible.

The key to handling this issue is to represent my blood type not as an attribute of a person class, but as a fully fledged class in its own right - which we called observation. Each observation applies to a particular patient, but also records such information as when it was made, who made it, and how it was made.

We also saw that observations can be about the absence of things as much as about their presence. So in some circumstances it may not be possible to figure out my blood group, but it is possible to say that it isn't blood group O. This we could represent as an observation of an absence of blood group O. (I have no idea if this example is possible or reasonable, but it can get tricky to think up realistic examples quickly.) Often observing the absence of things is crucial in a diagnostic process.

Using observations changes the way we determine information about a patient. Rather than simply asking for a patient's blood group, we look at all the patient's blood group observations. If they are all the same, then we just use that value. If they differ, we need to delve deeper. In many cases observations do sensibly change over time, so we might look at all the observations of my weight over time to plot how my weight changes.

Although we need to keep contradictory observations, we also need to capture if we think one of them was wrong. Some observations, such as a broken leg, will become untrue over time, but the blood group example above is more likely to be a error. In the erroneous case we have the notion of rejecting one observation with another. So we might have a further test in the Albion Hospital that finds I'm Blood Group A, this observation would then reject the Sisters of Plenitude's observation. Rejecting an observation says that we believe it was never true. We never delete the old observation, instead we mark it as rejected and link it to Albion Hospital's observation.

An important property of information is that it's used to guide behavior. A rejected observation may have been used as evidence for further observations or to justify interventions. Keeping these links in the record is essential since once an observation is rejected we can then follow those links to investigate the consequences. If the observation we've just rejected is a crucial part of evidence for another observation, that should be questioned and maybe rejected as well. Observations thus form a web of evidence that we can examine as we learn more about the patient.

Most of the time, of course, we don't use complicated schemes like this. We mostly program in a world that we assume is consistent. But there are times where we have to step away from that comfortable assumption. When that happens then explicit observations are a useful tool

(If you are interested in more of this, see Chapter 3 of Analysis Patterns. I'm sure I'd write it better if I were to do it again now, but the core concepts still seem to hold up pretty well. I'd also like to call out my colleagues on this work: Tom Cairns, Anne Casey, Mark Thursz, and Hazim Timimi)