Martin Fowler's Bliki

RequestStreamMap

2009-07-01T14:58:00-04:00

Hang around my colleagues at ThoughtWorks and you soon get the impression that the only good Enterprise Service Bus (ESB) is a dead ESB. Jim Webber refers to them as Egregious Spaghetti Boxes. So it's not uncommon to hear tales of attempts to get them out of systems that don't need them.

Battle was joined at one client and it brought to mind my younger days playing D&D. Webber swings but misses as the ESB is AC 2, Evan gets a hit and rolls 2d8 for 6 damage. Erik finally kills it by casting "Summon Request Stream Map".

So what was Erik's decisive spell? Essentially the idea was to take a simple request and show how the data for the request and response made their way through the layers of the application. Erik printed out all the code that you needed to read to understand how this would work - which ran to several pages. He also produced this diagram.

It's currently fashionable in agile circles to do Value Stream Mapping as a way to uncover waste in a software development process. I think of this as a request stream map because it similarly takes a request and shows how it moves through the layers allowing us to visualize what's going on and think about the cost and value of the layers.

Layering is an essential tool for building software applications. But like most essential things in life, excess can be almost as much of a problem as too little. A visualization like this (or the multiple pages of code) can help you find where "just enough" is.

One hazard, however. If you do need to transform data from one form to another, it's usually better to a few little transformations than one big transformation. You want to avoid unnecessary transformations not compress the ones you need.

IllustrativeProgramming

2009-06-30T15:23:00-04:00

What's the most common programming language in the world?

I'm not sure how you could go about measuring this, but one thing you'd need to do is consider what we mean by programming. My candidate answer considers that the most popular programming language is one used widely by people who do not consider themselves as programmers. This language is Excel, or more generally spreadsheets.

Spreadsheets are easily used for small tasks, but are also used for surprisingly complex and important things. Often I've seen professional programmers gulp when they realize that some vital business function is being run off some spreadsheet that they'd find too complicated to muck with.

In general, we've not had much success with programming languages for these kind of LayProgrammers. Whenever someone talks about some new environment that's going to allow people to specify complex behavior "without programming" I mention COBOL, which was originally designed to get rid of programmers. So it's important to consider what Excel can teach us about programming environments.

One property of spreadsheets, that I think is important, is its ability to fuse the execution of the program together with its definition. When you look at a spreadsheet, the formulae of the spreadsheet are not immediately apparent, instead what you see is the calculated numbers - an illustration of what the program does.

Using examples as a first class element of a programming environment crops up in other places - UI designers also have this. Providing a concrete illustration of the program output helps people understand what the program definition does, so they can more easily reason about behavior.

So why do I feel we need this particular Neologism? Essentially because I think it deserves more thought. We pass by illustrative programming examples without really thinking about them or what makes them special - or even that they are special in some way. We've used illustrative programming for years, but we've not paid enough attention to it. We've not thought enough about what are its essential qualities and what its strengths and weaknesses are.

I've chosen the term "Illustrative Programming" to describe this, partly because "example" is so heavily used (and illustration isn't) but also because the term "illustration" reinforces the explanatory nature of the example execution. Illustrations are meant to help explain a concept by giving you a different way of looking at it - similarly an illustrative execution is there to help you see what your program does as you change it.

When trying to make a concept explicit like this, it's useful to think about the boundary cases. One boundary is the notion of using projections of program information during editing, such as an IDE that shows you the class hierarchy while you are working on the code. In some ways this is similar, as the hierarchy display is continuously updated as you modify the program, but the crucial difference is that the hierarchy can be derived from static information about the program. Illustrative programming requires information from the actual running of the program.

I also see illustrative programming as a concept beyond the classic REPL loop of dynamic languages. REPL loops allow you to explore execution, but they don't make the examples front and center in the way that a spreadsheet does its values. Illustrative programming techniques put the illustration in the foreground of your editing experience. The program retreats to the background, peeping out only when we want to explore a part of the illustration.

I don't think that illustrative programming is all goodness. One problem I've seen with spreadsheets and with GUI designers is that they do a good job of revealing what a program does, but de-emphasizes program structure. As a result complicated spreadsheets and UI panels are often difficult to understand and modify. They are often riven with uncontrolled copy-and-paste programming.

This strikes me as a consequence of the fact that the program is de-emphasized in favor of the illustrations. As a result the programmers don't think to take care of it. We suffer enough from a lack of care of programs even in regular programming, so it's hardly shocking that this occurs with illustrative programs written by lay programmers. But this problem leads us to create programs that quickly become unmaintainable as they grow. The challenge for future illustrative programming environments is to help develop a well structured program behind the illustrations - although the illustrations may also make us rethink what a well structured program is.

The hard part of this may well be the ability to easily create new abstractions. One of my observations of rich client UI software is that they get tangled because the UI builders think only in terms of screens and controls. My experiments here suggest to me that you need to find the right abstractions for you program, which will take a different form. But these abstractions won't be supported by the screen builder as it can only illustrate the abstractions it knows about.

My colleagues Rebecca Parsons and Neal Ford have been spending a lot of time involved in thinking along these lines too. So here's some thoughts that Neal had in an email exchange

I think these tools work best for lay people (thus, your link to LayProgrammers). However, in general, tools like this slow down experienced/power users. When you mention UI panels, the Mac is rife with these types of controls. I spend a great deal of time in Keynote, fiddling with the inspector. At least all those controls are in one place (not like the new ribbon stuff). I would much prefer a markup language I could use to directly define stuff, with macros, snippets, and all the other things I'm accustomed to as a developer.
as these tools grow, they get unwieldy (perhaps because they are ceasing to be domain specific enough?) Look at Word, Excel, and PowerPoint. They had to invent new UI metaphors to expose all the functionality of those tools. APIs in programming languages scale much better, with several orders of magnitude more density before they become hard to navigate.
All the best-practices and tools don't exist there: refactoring, levels of testing, etc. Also, you loose the connection to text, meaning that macro facilities either don't exist or complex one-offs. I think a good comparison that highlights the limitations of Illustrative Programming is the comparison between bash (large, arcane, powerful, quirky) to Automator. I almost never use Automator because it suffers from Dietzler's Law: it's always lacking 10% of what I need. I gladly deal with the crufty surface area of bash because of the more power afforded.
I share your bullishness around these types of tools, but they are a long time from being useful for full-bore Agile development. I hope they mature fast.

--Neal Ford

One of the few people to take illustrative programming seriously is Jonathan Edwards. He's come up with many very imaginative ideas as to what such an environment should look like. His vision of illustrative programming is also closely bound to the notions of projectional editing and controlled copy-and-paste.

The trigger for me in wanting to coin a term here, is the use of illustrative programming by Language Workbenches by people like IntentionalSoftware. These Language Workbenches encourage you to build illustrative DSLs. Using illustration is important in this case since this should help engage lay-programmers, which is one of the aims of using DSLs. The challenge is to do this without falling into the trap of poor program structure.

Revitalizing Enterprise Software

2009-06-29T19:14:00-04:00

web site news: AMP, an Australian financial services company, ran an internal conference called Amplify. They asked me to talk about agile software development. I thought about how to make this best fit into the overall flow of the conference, particularly since I expected a significant part of the audience to not be part of IT. I settled on talking about how IT projects can be infrastructural or strategic. This classification alters how you approach the projects, in particular on the way IT and business people should collaborate.

Agilists and Architects: Allies not Adversaries

2009-06-26T11:24:00-04:00

web site news: At QCon San Francisco 2008 Rebecca Parsons and I gave a talk about how agile approaches work with enterprise architecture groups. At the moment there's a lot of distrust and conflict between agile project teams and architecture groups. We dig into why this is so, and explore ways that these groups can work together.

Ruby at ThoughtWorks

2009-06-11T16:57:00-04:00

web site news: ThoughtWorks started using Ruby for production projects in 2006, from then till the end of 2008 we had done 41 ruby projects. In preparation for a talk at QCon I surveyed these projects to examine what lessons we can draw from the experience. I describe our thoughts so far on common questions about Ruby's productivity, speed and maintainability. So far our conclusions are that Ruby is a viable platform that should be seriously considered for many forms of applications - in particular web applications using Ruby on Rails. I also go through some technical lessons, including some thoughts on testing with Active Record.

Google I/O Talk on Cloud

2009-06-10T10:04:00-04:00

web site news: Rebecca Parsons and I talk about Google App Engine and the general world of clouds. In the first bit I talk about things various ThoughtWorkers learned from experiementing with App Engine, highlighting issues with testing, persistance, and concurrency. In the second part Rebecca talks about the broader issues enterprises will face with moving to the cloud.

ComparativeValues

2009-06-05T16:44:00-04:00

One of the most striking things about the Manifesto for Agile Software Development is the format of its values "we favor x over y". I don't remember who came up with that idea, or how it arose. It probably just bubbled up as we tossed around ideas. But it's distinctive format has led a few people to try using that format again.

If you fancy trying this form, there's a couple of things to remember about it, things that people don't always realize. The most important of these is that the unfavored values are valuable too. A phrase like "solving world hunger over slavery" doesn't carry much power because hardly anyone is openly in favor of slavery. The format works when both left and right sides of the "over" are valuable things that most people want. What the comparative values then say is that we want both things, but if push comes to shove we prefer the one on the left. The harder choice there is between right and left, the better the value statement.

So a good mental test is to imagine someone reversing each value statement, indeed reversing all that you have in the set. There should be people that you can imagine proudly and reasonably supporting that opposite position. In our case we saw much of the industry heading towards high-ceremony processes - reversing the values we felt fairly summed up the values of that community. I can easily imagine writing an article extolling why the reverse set of values are a coherent world-view for software development by putting myself into that mind-set.

The right-hand values may be the current state of the world you want to change, or they may be a future state desired by another community. Either way the comparative values are there to highlight the contrast between one and the other.

Another point about the manifesto that I like is its brevity: four comparative values and twelve principles. It's hard to get that kind of brevity, but the briefer you make it - the punchier it is. I'm sure the manifesto would have not had the impact it did if it had forty-six value statements.

I'm one of those who has used this format since. Done well it can really highlight what makes a particular philosophy different to another. Here is another sample, a set of values I wrote to describe how I saw ThoughtWorks as being different from other software organizations.

Leveraging bright people over Making the most of moderate people
Flexible career paths over Well-defined roles
Delivering business value over Leading edge research
Learning new technologies over Mastering established technologies
Solving difficult problems over Increasing market share
Learning from mistakes over Avoidance of taking risks
Delivery to the client over Quarterly results

We've since replaced this list with another set of principles which try to capture how we want to be. But we did use the comparative values for a while to try to explain both to ourselves and others what made our aspirations different.

As such I think that not just is this format a good way to sum up a world view, when done well it also leads to sharp discussions about what people really want to care about. This usefulness comes directly from the fact that you are making hard choices between things that are all desirable.

Hot topics panel Jun 10, Chicago.

2009-06-04T14:31:00-04:00

web site news: I'm in Chicago next week for a major get-together of many of the leading technologists in ThoughtWorks world-wide. The Chicago Business Development people are determined to take advantage of us, so they've set up a panel to discuss hot topics in technology at lunchtime. It's quite the amazing group of people: Rebecca Parsons, Ian Cartwright, Ola Bini, Erik Döernenburg, Neal Ford, Pramod Sadalage, and Josh Graham.

Catching up with videos

2009-06-04T14:21:00-04:00

web site news: One of the main things I've not been getting around to announcing are various videos that have appeared of me jabbering on. Newish ones include a meta-introduction to DSLs, and interview on DSLs with Chris Sells (where I win a gold star for insensitivity by calling his newest pride-and-joy "19th century"), a JAOO 2008 panel interview on DSLs, and the QCon London keynote on software used by the Obama campaign.

Using Twitter

2009-06-04T14:17:00-04:00

web site news: I've been remiss about keeping the news column here up to date, partly because it needs some serious work on the publishing code, partly because I'm trying to focus on the DSL book, but partly because I've started to announce things on twitter.

DynamicTypeCheck

2009-06-02T19:47:00-04:00

Recently some of our developers ran into the accusation that with a dynamic language like ruby you use so many dynamic type checks that you end up effectively writing your own type system. So they thought, since we've written a lot of real ruby code - how often do we make dynamic type checks? Michael Schubert gathered up the data.

The table below contains the data. We define a dynamic type check as the use of the methods is_a?, kind_of?, and instance_of?. The lines of code come from the standard rake stats command in rails.

Project ID	Code		Test		LOC / type check	test LOC / code LOC
Project ID	type checks	Lines of Code	type checks	Lines of Code	LOC / type check	test LOC / code LOC
A	16	13318	0	9856	1448	0.7
B	14	19138	0	17123	2590	0.9
C	0	2607	0	2981	∞	1.1
D	7	4265	3	4069	833	1.0
E	32	29619	60	97688	1384	3.3
F	18	~9500	N/A	N/A	528	N/A
G	0	2455	0	3290	∞	1.3
H	9	2220	6	6404	575	2.9
I	23	10633	2	12331	919	1.2
J	196	40461	24	88511	586	2.2
K	17	5769	6	9848	679	1.7

The moral of this data is that you shouldn't expect to see a lot of type check calls in your ruby code base. This, of course, is true of any dynamic language. It was generally considered bad form in Smalltalk circles I inhabited too.

The methods that were checked for in this data aren't the only ones that can be considered a dynamic type check. Other cases are respond_to? and aClass === anInstance. Our folks felt that these cases were no more common than the ones they checked for.

Most uses are those of dealing with liberal input - eg where a method parameter can be a string, symbol, or array. These crop up in DSLish situations where you want liberal input for the high readability.

SmutOnRails

2009-04-30T13:59:00-04:00

A couple of weeks ago there was a Ruby conference in San Francisco called GoGaRuCo (Golden Gate Ruby Conference). This conference has grabbed attention due to a talk at which the presenter illustrated a discussion of CouchDB by using sexually suggestive pictures of women. Unsurprisingly the result has been a fair bit of heated, and occasionally offensive, debate.

The main lines of the debate are familiar. Various people, not all women, lay the charge that the images and general tone was offensive. Such material makes women feel degraded and alienated. This kind of presentation would not be tolerated at most professional events.

Defenders of the presenter point out that the slides were humorous and no offense was intended. The Rails community has always had a edginess to it - in part because much of the Rails community is focused on the rejection of enterprise values - both technologically and socially. David Heinemeier Hansson is happy to proclaim himself as an R rated individual and is happy to consign "professional" to the same pit to which he cast "enterprise".

I'll admit to finding much to like in the general edginess of the Rails world. Innovation often involves seeing a generally accepted line and vaulting over it. There's plenty of precious posturing around the software world that I'm glad to see skewered. Many of us have been delighted at how Rails has cheekily whacked over-complex frameworks, vendor bloatware, and other assorted ills. An important target of this skewer has been the rise of corporate blandness, where a fear of offense has transformed into a fear of any authentic communication and the rise of the anodyne press release. I'm right with the rails people on this - software is too much fun to shriveled up in dry talks and writing.

So the view of the rails leadership seems to be this: that the objections to the presentation are yet another attempt to foist empty corporate values on the thriving Rails ecosystem.

Except on this occasion I don't see the suits as the people doing the complaining. Most of those calling foul are women who have had to struggle with very real sexism in their careers, and men who have seen this and side with those women. They have been fighting the suits since before the Rails leadership were born, and for much higher stakes.

This incident has now grown beyond a conference presentation and a slide-deck on the web. The issue is no longer the presentation, but the reaction of the community to this event. The leaders, particularly David Heinemeier Hansson as the most visible figure, now face an important time in influencing what the future of the community will be.

The reaction of the rails leadership thus far is to deny the offense. I'll say now that I don't believe they are sexist. I believe that they didn't think the talk would give this much offense - and even that they don't think the talk should give offense.

At this point there's an important principle. I can't choose whether someone is offended by my actions. I can choose whether I care. The nub is that whatever the presenter may think, people were offended - both in the talk and those who saw the slides later. It doesn't matter whether or not you think the slides were pornographic. The question is does the presenter, and the wider community, care that women feel disturbed, uncomfortable, marginalized and a little scared.

It's my view that the people in a community have the power to set the tone of that community, to decide what is and is not acceptable behavior within it. If something questionable happens and people remain silent, that is an implicit acceptance of that event. That is why I feel compelled to write this page, because I think that this talk, and more importantly the rails leadership response to this talk, is objectionable.

My observation is that most men in the software business think that there isn't much sexism left in the profession - that this curse is a memory from a previous generation. Yet when I talk to women, I hear a different story. Nearly every one can tell me recent stories where they were clearly expected to feel degraded and belittled because of their gender. So some sexually suggestive pictures aren't a joke to them, they are a pointed reminder of disturbing behavior, and a reminder that such events can happen again at any time. One of the great difficulties for white guys like me is that we haven't been in that position; where prejudice can appear out of any corner, reinforced by the fact that every other face looks different.

This becomes more of an issue because the rails world faces a notable lack of women. The software world struggles with Diversity as it is. It's a problem for our profession, in that we lose access to talent, and it's a problem for many women who don't get the chance to develop a satisfying career in programming. The open-source world in general has even bigger issue, and the rails community perhaps more so. I'm sure it's not the only factor but the encouragement of talks like this creates an unwelcoming atmosphere of NetNastiness which deters many women from starting and staying in the community.

There also seems to be a generational factor in this. My colleagues have noticed that younger women, typically those under 30, are much less conscious of sexism than their older colleagues. This is partly because of the successes their elders have had in opening up the workplace to women. It may also because younger women haven't yet met the glass ceiling (and I hope it will be gone before they get there). Younger women also seem much more tolerant of sexual imagery. Yet I don't think this is cause for complacency. An important element in nurturing women in our profession is to have role-models who can show what's possible. Alienating older women makes it harder to do that.

So where does this go? I won't attempt to predict the future, but there is a scenario where this little presentation may be seen as a defining event in the rails story. This doesn't mean that people will suddenly leave in droves, but it does begin with a few departures, such as Mike Gunderloy's. The community continues with more alienating events, encouraged by the fact that those who are more sensitive are no longer around to object. This encourages more departures as people don't want to be associated with such a community. Thus develops a positive feedback loop making the rails world increasingly brash and unwelcoming for many of us.

I have a different vision - one that sticks it to the suits so hard it will make their eyes water. How about a community where women are valued for their ability to program and not by the thickness of their skin? How about a community that edgily pushes new boundaries without reinforcing long running evils? Perhaps even a community where women reach equal numbers? Such a community would hand the suits the defeat in the long battle women have been fighting for centuries. I'd love to be part of that.

Some thoughts on common statements

These pictures were less revealing and sexual that what you'd find in a mainstream movie

This is where the context matters. Watching a movie is a different social space to being in a software development talk. As result people react differently.

It is important to realize that the same behavior can be appropriate or inappropriate for different people in different positions. The risqué banter between partners and often in teams (mixed and unmixed) can be normal and healthy. But a newcomer who is not part of that group may perceive that same banter as demeaning or threatening. The corollary is that when strangers are around you need to be more careful about what you say.

--Chris Stevenson

These kinds of sexualized images have long been associated with men's clubs. Condoning a presentation like this can imply that the powerful (the community leadership) wants this atmosphere, to create a context that excludes women. I don't think the rails leadership actually wants to do this, but if someone did want to do create such a group, this would be a good way to go about it.

Women shouldn't get so annoyed, men don't when women make reverse jokes

You can't ignore history. Women have been comprehensively discriminated against for generations, indeed in most societies in the world they still are. It's the same reason why it's insensitive to make jokes about blacks and slavery or jews and the holocaust. The joke makes it look like you think the actual wrong was no big deal.

If you always worry about people being offended, you'll just end up being bland

Yes, that's a real risk. But being aware of causing offense doesn't mean you have to dial all the way down to corporate blandness. It means thinking how what you will cause offense and being comfortable with the result. You may feel that certain people should be deal with being offended, you may think that only a very small amount of people will be offended. That can be a reasonable response, but it has to be a thoughtful response.

As often, I find my black colleague Chad Wathington puts it well:

I think we don't have to get caught up in managing to every insult. I do think that people who have privilege need to do their best to not offend marginalized groups, realizing that no one is going to be perfect. Best effort is good enough as long as we respond gracefully and truthfully when we fail. As someone on the receiving end, I've always maintained that my job is be compassionate during those failures

--Chad Wathington

The presenter made an apology

The presenter effectively said "I'm sorry you were offended" - that translates to "don't be so thin-skinned". The presenter claims that he wasn't intending to be offensive, and I can believe that. But his failing is not realizing that what he considers to be offensive isn't the same as that of some of the audience. His pseudo-apology suggests to me that either he doesn't care that those people were offended, or doesn't understand how they could be offended - probably the latter.

The people who were offended are being thin-skinned.

That's a comment often made by those who condone NetNastiness, but it doesn't help those who are offended. The crucial point is: do we want to create an environment where "thin-skinned" people aren't welcome? After all the consequence of a society that is tolerant of nastiness and bullying is one where only the thick-skinned need apply. I'd prefer that people are welcomed for their ability in software development, not their ability to withstand offense.

This goes further than just our profession. To be successful as software developers, we need to collaborate with people in other fields. Tolerating this degree of nastiness makes it difficult for people in other walks of life to work with us, which impoverishes us all.

Humor is an important tool to puncture the self-important

I agree, but this only works if the power relationship is in the right direction. Someone of low power poking fun at a powerful person is a different situation to someone with lots of power skewering someone with little power. Women are (still) in a position of low power in our society (particularly in software development) so we have to be more careful with our humor.

Should be organizer be blamed for this?

No. It's not up to the organizers to vet talks. Certainly it's up to the organizers to choose talks, but there's no way they can be responsible for what happens on the day. It's sad that all of this has landed on GoGaRuCo and I think Josh Susser's apology was very gracious.

If an organizer rejected a talk like this, that would be censorship

It's not censorship. Censorship is when the powerful stop people organizing their own conference, or prevent people publishing their own web site or pamphlets. A conference organizer or a web site host has the responsibility to set the tone for that space. An important part of this is selecting content. Not every talk that's offered gets accepted, and the choices the organizer makes determines what the conference is like. Organizers will reject poor quality talks all the time, and it's reasonable to say that offensive talks are poor quality.

You're just trying to impose your moral and 'professional' standards on us

You might be surprised by my personal attitude to sexuality. But the point is not about judging various standards, it's about whether we want to make a group of people feel alienated. When looking at this I ask: "who is being offended" and "do I care about that group".

This is no worse than what happens in other professions

I don't know, although I've certainly encountered more overt sexism in worlds other than software. But I don't think that's relevant - we should do what we can to make our environment so that it doesn't exclude worthwhile people.

Can we get off this subject now and on to important technical issues?

Actually I think a social issue like this is worth spending time on. To build software effectively you have to be able to collaborate with other people, both other programmers and people outside of the software community. Discussions like this help us understand how we relate with other people, which makes a huge contribution to both our professional and personal lives.

Being a professional isn't just about being good at your job, it's about being proud of the impact you have on the world in general. We have a duty to make the world a better place.

--Jez Humble

Acknowledgements

I always find it particularly difficult to write these kinds of pieces. When I do, I find it particularly valuable to bounce thoughts off several of my colleagues and other friends. My thanks to David Heinemeier Hansson, Jez Humble, John Kordyback, Cyndi Mitchell, Mai Skou Nielsen, Rebecca Parsons, Kathy Sierra, Roy Singham, Chris Stevenson, and Chad Wathington for reading and commenting on the drafts. Thanks also to lots of people who have posted their feelings and analysis both on public channels and internal ThoughtWorks mailing lists. I have learned a great deal in the last few days.

IntentionalSoftware

2009-04-20T17:51:00-04:00

Update: video of the Intentional talk at DSL Devon is now available.

Several years ago, my then colleague Matt Foemmel, dissatisfied with the tools with which we were building software, managed to get in touch with Charles Simonyi to find out more about the shadowy Intentional Software. What he saw impressed him, and he persuaded me and other ThoughtWorkers to get involved too. What we saw was a tool with startling potential, but we remained frustrated by the secrecy and lack of urgency to release. That frustration ended last week.

Last week I set off for Chris Sells's DSL Devcon, and Magnus Christerson - Intentional's product manager - suggested I pop in to see how they were going on. After several years of "Real Soon Now", I was unsure, but Rebecca Parsons, my colleague who has been keeping regular contact with Intentional, said that now would be a good time.

I spent a fascinating and exciting day at their office in Bellevue. It's not that I saw anything particularly new - these were all ideas and capabilities that had been around for a while - but there was a realness and maturity that I hadn't seen before. Indeed Intentional had released a version 1.0 of their product a few weeks earlier. The usual approach is to trumpet a version 1.0 release of a ground-breaking product from the mountaintops. Only Intentional would make such a release and not bother to tell anyone. Indeed as I write this there's no mention of their product on their website - if you want more information you have to talk to them.

What's There

This isn't a comprehensive discussion of their tool (called the Intentional Domain Workbench), I haven't had time to put something like that together. But I hope my scattered thoughts and observations will be interesting. The Intentional Domain Workbench is a LanguageWorkbench, indeed it's one of the systems that made me coin that term. A Language Workbench is a tool that allows people to design DomainSpecificLanguages: not simply to parse them, but build a comprehensive environment that includes rich editing. In Intentional's case this is a ProjectionalEditing environment.

One of the examples they have is the state machine example I use for my book introduction. The workbench allows you to define the schema of the semantic model state machine in its schema definition language. In order to manipulate state machines you define projections of the semantic model. One of the striking features of the Intentional Domain Workbench is its ability to support multiple projections of the same semantic model. For the state machine example they've defined projections in several of the DSLs I've used in discussing the example: XML, custom syntax, and Ruby. All three of these projections are reversible, meaning that you can edit through them, updating the semantic model and other projections. Switching between the projections is just a matter of selecting a menu item.

They also had read-only projections in fluent C#, command-query C, and a state machine diagram. Although they hadn't set up the diagram to be editable, the workbench can handle editable diagrammatic representations. In another example they show an electronic circuit which is editable in both a tree structured property sheet projection and in a circuit diagram projection.

The circuit diagram also showed another really powerful feature of the workbench - the ability to fluidly integrate example executions with the program definition. In the electronic circuit case, this means that you can give the various elements of the circuit properties and the model will calculate the impedance of various parts of the circuit and display them as you are editing the circuit. Of course you can build a custom program to do this kind of thing - but the point is that this behavior comes easily as part of a DSL definition in the workbench.

Combining example execution with program definition is one of the features of spreadsheets - and may be a reason why spreadsheets have become so successful as an environment for LayProgrammers. It's also a notion that's been propelling much of Jonathon Edwards's interesting and wild ideas. My sense is that interesting DSLs in language workbenches will have this characteristic, particularly if they are aimed at being used by lay-programmers.

Another way that you can combine execution with specification is with test cases. They have an example of a pension workbench, build with Capgemini, that allows actuaries to enter formulas using full mathematical notation, together with FIT-like tables to show test cases. These test cases are live with the appropriate red/green behavior as you edit the formulas.

The pension workbench also illustrates the combination of multiple languages. When you look at a pension document on the screen, you're looking at three independent languages: word processed text for the prose, mathematical notation for the formulae, and test case tables. These languages are developed independently but integrated in the workbench's core data structure (called the Intentional Tree). This integration extends to the execution too - you can step into a test case and delve into the intermediate values in the mathematical formulae.

In order to make these things run, you have to include behavior with the semantic model. Intentional have developed their own general purpose language, whose working name is CL1, to do this. CL1 can look like superset of C#, but such a view is again a projection of the core semantic model. I found it interesting that this is a similar feature to JetBrains MPS who have their "base language" which projects into a Java-like general purpose language. Increasingly much of these tools are programmed using this in-workbench general purpose language.

The intended way of working is that developers use the Intentional Domain Workbench to build a domain-specific workbench. They provide a runtime (the Intentional Domain Runtime) for them to run without language editing capabilities. So Capgemini used the Intentional Domain Workbench to build the Pension Workbench as their own product. The Intentional Domain Workbench allows you to define new model schemas and projections, while the Pension Workbench allows you to build pension plans using these languages.

The Intentional system is primarily arranged in the .NET ecosystem. Both the workbench and runtime run on the CLR and core parts of them are written in C#. The workbench makes it really easy to generate .NET assemblies that can be automatically loaded into the workbench for testing or run with the runtime. Custom workbenches can generate code for any environment, and Intentional have done some work with another partner that involves generating Java code so that people can specify behavior in the custom workbench and then deploy the resulting system in a Java environment.

An interesting aspect of the implementation is that they handle representational transformations by using lots of little transformations rather than one large one. As an example, code generating C# from a semantic model involves about a dozen small transforms lined up in a pipeline similar to a multi-stage compiler, the last step being a transformation from a C# AST to text. Much of their internal design goes into making this approach efficient so you can happily string together a lot of small transforms without worrying about any efficiency cost. A further consequence is that the pipeline of transforms for code-generation is very similar to that used for editing projections.

A common problem with tools that use projectional editing is how they deal with version control. Often the answer is to just let multiple people edit the same store simultaneously, which makes many serious developers quake. The Intentional Domain Workbench has a built in version control mechanism that records all the changes made to the Intentional Tree and can do commits and merges at the tree level. You then see diffs in languages by doing another projection.

An interesting feature of this version control approach is that you can commit with conflicts and the conflicts are committed into the repository as conflicts. Unlike with text files they don't mess up your text - you have a real data structure present, so you can find the conflicts and fix them. The developers use this feature to commit a conflict they can't sort out to a branch so that developers more familiar with the conflicted area can update to the branch and fix it.

The fact that editing is done on an intentional tree rather than text also changes some other things. For example unordered collections are tagged so that a change in the ordering of the elements in an editor doesn't trigger a conflict. You can also include domain-specific conflict detection and resolution behavior.

Going Public

Historically the lack of releasing of Intentional has been one problem, their secrecy is another. To see anything real about the Intentional Domain Workbench has required what Neal Ford refers to as an UnforgivenContract. Intentional have given some public talks, but they've really boiled down to saying "trust us, we have some really cool technology". We'd known that indeed they had, but couldn't explain to people why.

So I awaited the talk at DSL DevCon, given by Magnus and Shane Clifford (their development manager), with quite some expectation. They said they were going to finally open the curtain. Would they - and how would people react?

They started worryingly, with the usual unrevealing Powerpoints, but then they switched to showing the workbench and the curtain finally opened. To gauge the reaction, take a look at Twitter.

@pandemonial Quite impressed! This is sweet! Multiple domains, multiple langs, no question is going unanswered
@csells OK, watching a live electrical circuit rendered and working in a C# file is pretty damn cool.
@jolson Two words to say about the Electronics demo for Intentional Software: HOLY CRAPOLA. That's it, my brain has finally exploded.
@gblock This is not about snazzy demos, this is about completely changing the world we know it.
@twleung ok, the intellisense for the actuarial formulas is just awesome
@lobrien This is like seeing a 100-mpg carburetor : OMG someone is going to buy this and put it in a vault!

Afterwards a couple of people said it was the most important demo they'd ever seen, comparing it even to the Mother of all Demos. For many there was a sense that the whole world of software development had just changed.

(Many thanks to Chris Sells and co for organizing this conference and inviting me to speak. They also made a video of the talk available.)

So now what? There's more to all this than a demo can reveal. Right now we want to get several of our hands on the workbench and kick its tires - hard. Assuming it passes that test, we want to use it on commercial projects and see how works for real. No system designed using the Intentional Domain Workbench has yet gone live, and as any agilist knows you never really understand something till you deploy it into production every week.

Shortly the other major similar workbench to this - JetBrains's Meta Programming System - will have version 1.0 released as open-source. So this year could well be the year when these Language Workbenches will finally step out into the light and see their first external pilot projects. (I should also mention that the MetaEdit workbench has been out for a while, although it hasn't had much visibility.) I don't know whether these workbenches will change the face of programming as we know it, after all I once thought Smalltalk was going to be our future; but these workbenches do have the potential to be such a profound change. Certainly I'm excited that we're now on the next, more public, stage of this journey.

ContradictoryObservations

2009-03-03T11:02:00-05:00

Many computer systems are built to house data and turn it into useful information for humans. When we do this there is a natural desire to make that information consistent. After all what use is there of a computer system that's in two minds about things?

But sometimes computer systems should record contradictory data and help humans deal with that. This issue came foremost to my mind many years ago when working in health care for the UK National Health Service. We were building a conceptual model for health care delivery - essentially a conceptual schema for an electronic health care record.

Looking back on it, there were certainly plenty of things I'd do differently now. But one thing in particular was something very precious and important - the model was very much a collaborative effort between myself, another software developer, two doctors and a nurse. The clinicians understood the model and played a full part in developing it - they were not merely passive reviewers. As a result I think the ideas we developed were particularly valuable in thinking about what a clinical practitioner wants to see in an electronic health care record.

One thing the clinicians were very strong about was this need to capture contradictory information. I might have a note from the Royal Hope Hospital saying my blood type is A and another note from the Sisters of Plenitude saying my blood type is B. This would clearly be nonsense, blood types don't change. But that doesn't mean we cannot record these two bits of data. Without further investigation we don't know which one is correct. Even if we test again and confirm one of them, we can't just throw away the bad one as it may have been the basis for further clinical action. And of course there are lots of cases where the contradiction isn't as clear cut. We may never be able to find out which of two contradictory bits of data was wrong or may find a change over time that is extremely unlikely but not impossible.

The key to handling this issue is to represent my blood type not as an attribute of a person class, but as a fully fledged class in its own right - which we called observation. Each observation applies to a particular patient, but also records such information as when it was made, who made it, and how it was made.

We also saw that observations can be about the absence of things as much as about their presence. So in some circumstances it may not be possible to figure out my blood group, but it is possible to say that it isn't blood group O. This we could represent as an observation of an absence of blood group O. (I have no idea if this example is possible or reasonable, but it can get tricky to think up realistic examples quickly.) Often observing the absence of things is crucial in a diagnostic process.

Using observations changes the way we determine information about a patient. Rather than simply asking for a patient's blood group, we look at all the patient's blood group observations. If they are all the same, then we just use that value. If they differ, we need to delve deeper. In many cases observations do sensibly change over time, so we might look at all the observations of my weight over time to plot how my weight changes.

Although we need to keep contradictory observations, we also need to capture if we think one of them was wrong. Some observations, such as a broken leg, will become untrue over time, but the blood group example above is more likely to be a error. In the erroneous case we have the notion of rejecting one observation with another. So we might have a further test in the Albion Hospital that finds I'm Blood Group A, this observation would then reject the Sisters of Plenitude's observation. Rejecting an observation says that we believe it was never true. We never delete the old observation, instead we mark it as rejected and link it to Albion Hospital's observation.

An important property of information is that it's used to guide behavior. A rejected observation may have been used as evidence for further observations or to justify interventions. Keeping these links in the record is essential since once an observation is rejected we can then follow those links to investigate the consequences. If the observation we've just rejected is a crucial part of evidence for another observation, that should be questioned and maybe rejected as well. Observations thus form a web of evidence that we can examine as we learn more about the patient.

Most of the time, of course, we don't use complicated schemes like this. We mostly program in a world that we assume is consistent. But there are times where we have to step away from that comfortable assumption. When that happens then explicit observations are a useful tool

(If you are interested in more of this, see Chapter 3 of Analysis Patterns. I'm sure I'd write it better if I were to do it again now, but the core concepts still seem to hold up pretty well. I'd also like to call out my colleagues on this work: Tom Cairns, Anne Casey, Mark Thursz, and Hazim Timimi)

TechnicalDebt

2009-02-26T13:18:00-05:00

Update: Added a link to Ward Cunningham's video opinion

You have a piece of functionality that you need to add to your system. You see two ways to do it, one is quick to do but is messy - you are sure that it will make further changes harder in the future. The other results in a cleaner design, but will take longer to put in place.

Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.

The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline. The all too common problem is that development organizations let their debt get out of control and spend most of their future development effort paying crippling interest payments.

The tricky thing about technical debt, of course, is that unlike money it's impossible to measure effectively. The interest payments hurt a team's productivity, but since we CannotMeasureProductivity, we can't really see the true effect of our technical debt.

One thing that is easily missed is that you only make money on your loan by delivering. Following the DesignStaminaHypothesis, you need to deliver before you reach the design payoff line to give you any chance of making a gain on your debt. Even below the line you have to trade-off the value you get from early delivery against the interest payments and principal pay-down that you'll incur.

(As far as I can tell, Ward first introduced this concept in an experience report for OOPSLA 1992. It has also been discussed on the wiki.)

Additional Comments

Ward Cunningham has a video talk where he discusses this metaphor he created.

A couple of readers sent in some similarly good names. David Panariti refers to ugly programming as deficit programming. Apparantly he originally started using a few years ago when it fitted in with government policy; I suppose it's natural again now.

Scott Wood suggested "Technical Inflation could be viewed as the ground lost when the current level of technology surpasses that of the foundation of your product to the extent that it begins losing compatibility with the industry. Examples of this would be falling behind in versions of a language to the point where your code is no longer compatible with main stream compilers."

Steve McConnell brings out several good points in the metaphor, particularly how keeping your unintended debt down gives you more room to intentionally take on debt when it's useful to do so. I also like his notion of minimum payments (which are very high to fix issues with embedded systems as opposed to web sites).

(Original posting 3 Aug 2004.)

NashvilleProject

2009-02-25T14:45:00-05:00

I spent some time recently with one of my favorite ever ThoughtWorks projects. It's a project that started in 1998, using then new J2EE technology. Over the years it's had a fascinating history: starting with EJBs, ripping them out, going offshore to Bangalore, coming back to Chicago. Many people have moved in and out of the project and the project has varied in head-count between 6 and 60. Overall the project has had over 300 staff-years of effort on it and weighs in at around 100 KLOC.

It's a favorite of mine because it exhibits an important property of my preferred view of software development: a long term support of a business function enabled by a well-designed code-base. The fact that they are still adding useful business value after ten years is an big dollop of kudos. They are able to rapidly add new features when needed so haven't fallen into the typical morass of a legacy app.

On this visit a couple of thoughts grabbed me.

Firstly they've had an interesting evolution in their approach to acceptance tests and how they update them as they add new features. In their original (and common) world view, each time you implement a new story you add one or more tests. This leads you to a simple tracing structure where each story is verified by one or more acceptance tests. But the problem with this approach is that over time the tests grow in complexity with much duplication.

In their new world view there is a suite of acceptance tests that describe the application behavior in SpecificationByExample style. Each time they play a new story, they decide how to update this suite to reflect the new behavior. This breaks the simple story-to-test relationship, but results in a much simpler and coherent suite of tests.

The second interesting aspect of the project is how it continues to work at improving the code base. They came up with a good, if informal, metric for describing this. A few years ago, if they wanted to take on someone new they wanted that person committed for at least a year, so they could get contributions that would be worthwhile after coming up to speed on the code base. Now that time is down to three months. For a ten year old app with that many hands on it, that's quite an achievement.

For me the key purpose of good design is that it allows you to continue working rapidly with the code (the DesignStaminaHypothesis). Assessing how long it takes a developer to be productive with a code base is a good way to sense this design quality. The minimum-commitment length metric is another spin on this same idea. It's not something we can measure objectively, but it is something that a team can consider looking at.

I'm hoping we'll get more people from the project talking about their experiences. They did do a podcast last year (go to thoughtworks podcasts and look for "Keeping Grey Code Fit").

DslBookRoadmap

2009-02-23T15:59:00-05:00

It's been a while since I posted an update on the status on my DSL book. The most common question I get is "when will the book be out?" The way things look at the moment, I'd estimate it appearing some time in 2010. That's an estimate, with all the usual hedges that apply to such things.

To help see how the current state meshes in with that estimate, I'll describe the general progress of how books work, at least for me. The first phase is writing the First Review Draft. This means writing all the material in the book, to the level that I can submit it to peer review. For my larger books (eg Refactoring, P of EAA) this takes about 12-18 months. This book is taking longer, I've already been at it for over two years, and I suspect there's another six months or so to go. Once I have the First Review Draft, it goes out to review. This takes time for people to read it, get their comments to me, and for me to modify the text based on the review. I usually do two rounds of review - and it takes around 6 months for that to happen. Once I'm done with formal review the book is a Final Draft. At this point it goes off for production - which includes copy-edit, indexing, layout, printing, etc. That takes another 6 months. So I estimate that the book will appear about a year after I get a First Review Draft.

Not all authors work this way, some write a few chapters and sent them for review while they write a few more chapters. I use the approach I do because it was how I did my first book and it seemed to work very well.

The current state of the book is that I now have the broad structure and pass through the material done and in a coherent state. However it's not yet at a First Review Draft state as there are some significant holes to filled, and I think I'll need a few months to fill them. At that point I think it will be worth reviewers spending their time on it. However since it is reasonable coherent now, I think adventurous people may get something out of it.

My next task is to figure out what the holes are and how to fill them in. My mental writing mode is now shifting however. No longer do I want to spend a lot of time investigating new material in any depth, my aim is to push hard to get the book into a useful state for publication. Books can never be complete, the real test of them is whether they are useful. I think I have enough now to be useful, so I need to work on getting it shipped. There's enough there for later work to build on it. I'm currently looking at over 400 pages and this will only grow as the holes get filled. (It has a duplex book structure with the narratives currently clocking in at 100 pages, so that's reasonable but I'd rather it not grow too much more.)

In particular this means that two big areas will get a much skimpier treatment than they probably deserve. One of the biggest issues for me is how to talk about error handling in DSL parsing. In the end I've decided to mostly let it go - not because I don't think it's important but because I really need to cut scope in order to get the book out. I suspect that there'll just be a few pages on this at most.

I'm also not going to go too deeply into the world of Language Workbenches. I am going to write a section about them, but I won't go into too many details about the different ones, how they work, and how to use them. My reason for this is two-fold - partly the need to ship and partly that they are still new and evolving technologies. I like to write books that will last a long time, and anything detailed I write about Language Workbenches in the book will not be current for long. I think there will still be a substantial chapter on this subject, as most of it I can write without doing much more investigation, but it's less than I could.