Design bliki

One of the commonly accepted beliefs in the software world is that talented programmers are more productive. Since we CannotMeasureProductivity this is a belief that cannot be proven, but it seems reasonable. After all just about every human endeavor shows some people better than others, often markedly so. It's also commonly observed by programmers themselves, although it always seems to be remarked on by those who consider themselves to be in the better talented category.

Naturally better programmers cost more, either as full-time hires or in contracting. But the interesting question is, despite this, are more expensive programmers actually cheaper?

On the face of it, this seems a silly question. How can a more expensive resource end up being cheaper? The trick, as it is so often, is to think about the broader picture of cost and value.

Although the technorati generally agree that talented programmers are more productive than the average, the impossibility of measurement means they cannot come up with an actual figure. So let's invent one for argument sake: 2. If you can find a factor-2 talented programmer for less than twice of the salary of an average programmer - then that programmer ends up being cheaper. To state this more generally: If the cost premium for a more productive developer is less than the higher productivity of that developer, then it's cheaper to hire the more expensive developer. The cheaper talent hypothesis is that the cost premium is indeed less, and thus it's cheaper to hire more productive developers even if they are more expensive.

In case anyone hasn't noticed this hypothesis is a key part of our philosophy at ThoughtWorks and is one of the main reasons why I ended up switching from an independent consultant to join. We believe we actually end up cheaper for our clients, even though our rates were higher. Of course, we do have difficulty persuading many clients that this is true - that lack of objective productivity measures strikes again. I still remember a meeting with one prospective client complaining about how our rates were higher than a company who had made a previous, failed, attempt at the system we were bidding on. We had to politely point out that paying less rates for a project that delivered no value was hardly a financially prudent strategy.

There are some notable consequences to the the cheaper talent hypothesis. Most notably is one that it actually follows a positive scaling effect - the bigger the team the bigger the benefits of cheaper talent. Let's assume we actually have put together a team of ten talented developers to run a project in some alternative universe where we have actually measures that they are twice as productive as the average - and thus do cost exactly twice as much to hire. In this case you might naturally assume that a rival team of average programmers would be a team of twenty.

The trouble is that that assumption assumes productivity scales linearly with team size, which again observation indicates isn't the case. Software development depends very much on communication between team members. The biggest issue on software teams is making sure everyone understands what everyone else is doing. As a result productivity scales a good bit less than linearly with team size. As usual we have no clear measure, but I'm inclined to guess at it being closer to the square root. If we use my evidence-free guess as the basis then to get double the productivity we need to quadruple the team size. So our average talent team needs to have forty people to match our ten talented people - at which point it costs twice as much.

Another factor that plays a role here is time-to-market. Let's assume two teams of four people, one talented and one average. To stack the deck of our argument against our talented team, discount the previous paragraphs, and assume the talented team is only twice as productive as the average team. If the talented team charges twice as much then can we assume that it doesn't matter financially which team we pick?

I'm afraid the talented team wins again. They'll complete the project in half of the time of the average team, which means that the customer will start yielding value from the delivered software earlier. This earlier value, compounded by the time value of money, represents a financial gain for picking the talented team, even thought their cost per output is the same.

Agile development further accelerates this effect. A talented team has a faster cycle time than an average team. This allows the full team to explore options faster: building, evaluating, optimizing. This accelerates producing better software, thus generating higher value. This compounds the time-to-market effect. (And it's natural to assume that a talented team is more likely to produce better software in any case.)

Faster cycle time leads to a better external product, but perhaps the greatest contribution a talented team can make is to produce software with greater internal quality. It strikes to me that the productivity difference between a talented programmer and an average programmer is probably less than the productivity difference between a good code-base and an average code-base. Since talented programmer tend to produce good code-bases, this implies that the productivity advantages compound over time due to internal quality too.

All this sounds, at least to me, like a highly compelling argument. It's also one that's widely accepted (at least by programmers who consider themselves talented). But it's far off being accepted by the software industry as a whole. We can tell this because the premium for talented developers (in terms of salary/contracting fees) is less than the productivity difference. Probably the major reason for this the inability to objectively measure productivity. A hirer cannot have objective proof that a more expensive programmer is actually more productive. Only the higher cost is objective. As a result a hirer has to match a subjective judgment of higher value against an objective higher cost. Many hirers, even if they believe the talented programmer is worthwhile personally, isn't prepared to justify the full higher cost to managers, HR, and purchasing.

This effect is compounded by the difficulty in making even a subjective assessment. At ThoughtWorks we rely on peer assessment - developers abilities are assessed by fellow team members. The result is hardly pinpoint precision, but it's the best anyone can do.

Which all points out that hiring and retaining talented programmers is hard work. Hiring and assessment is hard work. You have to deal with people with very individual desires, which are even more important to track as they are effectively underpaid. So a hirer is faced with certain extra work and higher costs versus only a judgment call for higher productivity.

So I understand the situation but don't accept it. I believe that if the software industry is to fulfill its potential it needs to recognize the cheaper talent hypothesis and close the gap between high productivity and higher compensation.

An alternative to SourceBasedCode is the idea that the core definition of a system should be held in a model and edited through projections.

To talk about this style of environment I find it handy to think in terms of multiple representations of the system:

• editable representation: what you edit in order to change the system.
• storage representation: the persistent record of the system definition.
• executable representation: what is executed to make the system run - the executable.
• abstract representation: used to manipulate and reason about system definition.
• visualization representation: a non-editable view of the system definition.

A source based system combines the editable and storage representations in the source file. It executes the source by transforming the source into an executable representation either in one observable step (interpretation) or multiple steps via a compiler. In order to do this it usually transforms the source into an abstract representation as an intermediate step, but this abstract representation is transitory and only around during compilation. The source is seen as the core definition of the system.

With a repository based system the abstract representation is the is core definition of the system. A tool manipulates the abstract representation and projects multiple editable representations for the programmer to change the definition of the system. The tool persists the abstract representation in a storage representation, but this is entirely separated from any of the editable representations that it projects. The relationship to the executable representation is pretty much the same - the executable is produced through a series of transformations from the abstract representation.

An important difference between repository and source based environments is the split between persistent storage and editing. Repositories can choose any persistence mechanism that they choose, while source systems need to have some universal storage mechanism - which is why they are almost always text files.

The abstract representation may be edited through multiple projections, each projection can show a limited amount of the total information which isn't tied to the actual structure of the abstract representation. Repository systems thus usually show a wider range of editing environments - including graphical and tabular structures - rather than just a textual form.

Sophisticated source based IDEs also show multiple projections - for instance a side pane showing a list of methods for a class with graphical annotations to indicate their AccessModifiers. However these projections are usually very much secondary to a source editor, and often the projections can't be edited directly - you have to change the source and see the projection update.

Such PostIntelliJ IDEs do this by creating an abstract representation when they load the source files (which is why they can take a while to start up). They also use the abstract representation to do perform lots of other code-assistance features such as contextual code completion and refactoring.

A significant pragmatic problem with repository based systems is the fact that there is no generally accepted way format for the storage representation. The fact that programmer-readable text is the universal choice for source files means that a whole slew of tools can be built to process them: editors, source-code control, difference visualizers etc. Repositories have to do all this themselves, which is often why these things are often lacking. In particular many repository based environments suffer greatly because they don't have a decent configuration control system, which makes it much harder for multiple people to collaborate on the same system definition. This is a big contrast to source based environments that have a plethora of source code control systems to do this task.

Repository based systems are closely connected with Model-Driven Development (MDD), although I don't think the two are entirely synonyms. In an MDD context the abstract representation is usually referred to as the model. Certainly almost all MDD tools are repository based, but many all repository based tools, eg Microsoft Access, would not consider themselves to be MDD.

(I first explored this way of looking at environments in my essay on Language Workbenches. I've described it here because I think the notion of repository based environments is broader than just in Language Workbenches.)

As my career has turned into full-time authorship, I often worry about distancing myself from the realities of day-to-day software development. I've seen other well-known figures lose contact with reality, and I fear the same fate. My greatest source of resistance to this is ThoughtWorks, which acts as a regular dose of reality to keep my feet on the ground.

ThoughtWorks also acts as a source of ideas from the field, and I enjoy writing about useful things that my colleagues have discovered and developed. Usually these are helpful ideas, that I hope that some of my readers will be able to use. My topic today isn't such a pleasant topic. It's a problem and one that we don't have an answer for.

The scenario runs like this. We carry out a project for a client and hand over a shiny new piece of software. As is our habit these days, we also hand over a bevy of automated tests for this software (typically there are as many lines of code of tests as there are of functional code). These tests are usually a mix of unit tests and broader ranging functional and acceptance tests. Either way the tests act as an active description of what the software does and a bug detector to quickly find problems as we evolve the software. We treasure these tests, they are a key to our success in building software systems.

Some months later the happy customer calls us back to do some further work on the software, adding new features and capabilities. We come in, keen to work on a code base that may have faults - but at least are our faults. Then we make an unpleasant discovery.

The tests no longer run.

Sometimes the tests are excluded from the build scripts, and haven't been run in months. Sometimes the "tests" are run, but a good proportion of them are commented out. Either way our precious tests are afflicted with a nasty cancer that is time-consuming and frustrating to eradicate.

We ask what happened and are told things like "we made a change and some tests broke, so we removed the tests". You can look at this as our failing - we haven't managed to fully teach the client teams about the value of the tests. We need to do more to pass on that failing tests need to be investigated, not simply ignored. But whatever anyone says, we've discovered that cancer of the tests is a common disease.

We don't think that the fact that Test Cancer appears is a reason against writing tests. Even if a particularly virulant strain wipes them all out the day after we leave, we still got value from them while we were building the system. And tests don't always get cancer. We recently spoke to a developer who had become a convert to TDD after maintaining a system we'd handed over a few years ago. The tests made our code much easier to work with than code that other firms had added later.

Last weekend I attended the Alt.NET conference. It was the first named gathering of a group of people I've been watching on the blogosphere for quite a long time. A group of long-time users of Microsoft technologies who feel that their development philosophy has been getting out of sync with the perceived orthodoxy from Redmond. While some have considered moving away this group is keen to stay and try to influence the Microsoft world.

The term alt.net was coined by David Laribee in his blog. The conference was organized using OpenSpace, a style that I thought was particularly apropos to the nature of this community. I'm not claiming to speak for this community, or define it; these words are my interpretations from what I saw and heard.

Alternative

One question that came up was the name alternative. This made a few people uncomfortable as it suggested to them that this was a group in opposition to Microsoft. A different view of "alternative" is that it's something that embraces choice. Many communities believe that having many alternatives makes them stronger (the Unix community leaps to mind). In software development it's rare that one solution is the right one for all possible circumstances. Having alternatives does mean you have to think about which solution is the right one for your situation, but I prefer that to trying to use a hammer to turn a nut. And yes, this does mean that personal experience and preference leads to different choices. We may be programmers but we are still human, not embodied algorithms.

The alt.net mind-set is one that is very familiar to me. It has that mix of agile + object-orientation + patterns + TDD + DDD which is very much the school of software development that I favor. (Lacking a proper name for it, I'm inclined to call it the OOPSLA school of software development.) There is certainly a belief that there is a mainstream Microsoft orthodoxy at the moment, one that doesn't fit the OOSPLA school. And there's some frustration with that. But the point here is that it's not that the alt.net community thinks that the perceived mainstream Microsoft route should be erased, but that the Microsoft world is big enough for different approaches.

As well as discussion about whether "alternative" is a good name, there's also discussion about whether you need a name at all. As a compulsive maker of Neologisms you shouldn't be surprised to find out that I think a name is useful. There's clearly a common style of software development that's formed here and giving it a name makes it easier to talk about. Inevitability some people will be annoyed by coining terms, but I think the usefulness outweighs that objection.

Participative Community

An important feature of alt.net is that it's a participative community. Traditionally when user conferences get together, the vendor is in charge of the agenda. Most sessions are about the vendor showing the community how to use the tools it provides. Good vendors listen to their customer community, reacting by developing new products that respond to their community's wishes.

A participative community is different, they don't just want the vendor to listen and provide suitable products - they want to participate in the development of new products. It's just such a participative community that's taken the initiative in the Java world. JUnit, IBatis, Spring, Hibernate et al didn't come out of the vendors, but were developed by "customers". One of the things about the nature of the software industry is that many customers are every bit as capable of producing vital products as vendor companies, especially when combined with the community and ethos of open source.

The great question ahead for Microsoft is how to engage with a participative and opinionated community like this. Treating such a group as an opponent will result in the loss of valuable products, and more importantly the capable people connected with them. Engaging with a community like this brings great opportunity. I would argue that the participative community around enterprise Java has saved the enterprise Java platform. A big challenge for Microsoft in all this is that this means finding a way to accommodate with open source development. Recent signs, particularly around Iron Ruby, suggest that at least some bits of Microsoft are heading in the right direction.

More signs of the right direction was Scott Guthrie's demonstration of the ASP.NET MVC framework (also see Scott Hanselmans's video). This was very interesting to me, not so much because of the product itself, which didn't seem to have anything particularly innovative (and that's not a complaint), but the philosophical signs around it.

First off there's the fact that Scott Guthrie made the commitment to come to a small, conference like this to reveal this product. Next was the clear design goal around testability. Add a rich sauce of clear understanding and learning from other work in this space. Add a garnish of plugability that allows the framework to work with non-Microsoft tools and encourages extension. Many people present said they hadn't been so excited by a product announcement since .NET.

It's also a fine example of "alternative". The MVC framework isn't intended to replace webforms, programmers can choose whether to use webforms or MVC.

One other issue in a community like this is that it's a community that doesn't equate criticism with animosity. Many vendors suffer from the belief that anyone who criticizes them is their enemy. In truth often your friends are at their most valuable when they are critical. Like any large corporation, Microsoft can exhibit contradictory reactions. There are certainly parts of the organization that do think that friends should never criticize. Part of working with a participative community is to learn to value friendly criticism. Equally people in the community need to learn how to criticize without being nasty - a particularly rare quality in our profession.

Exclusive Community

There's been some debate (particularly in the blogosphere) about whether the alt.net community is an exclusive community (in a bad way). I find a useful way to think about it is a question that did come up a couple of times during the conference. Should people form separate alt.net user groups or should they try to influence and change the current user groups? My answer to this is "both". A focused alt.net user group sets an expectation about the material being discussed and the kind of values and principles that ground the group. People who are doing this style of development need to talk to others doing it so they can learn from them. I've been engaged in this style for over a decade but I still have tons to learn about it. The advantage of a focused gathering is that I get to concentrate on this type of content.

A specialized alt.net user group is only exclusive if it tries to exclude others. The alt.net conference was not exclusive because, at least until the conference filled up, anyone could turn up. As long as an alt.net user group is open to accepting anyone, it's a good thing.

At the same time it's important for alt.net people to engage the wider .NET community. I encourage this style of developing software because I genuinely think it's effective. Therefore I think it's important for me to try and communicate how and why to do it to as big an audience as I can. This way other people can be exposed to the ideas and get the opportunity to understand the techniques so they can choose whether they want to try them. I expect to see many alt.neters looking to talk about these techniques and their experiences with them at a wide range of Microsoft-oriented conferences.

I have high hopes for the alt.net community. I believe this kind of community is important to the viable future of the Microsoft ecosystem, and I want a healthy Microsoft world. My hope is that Microsoft participates in this community, so that many leading Microsofties can happily state they are alt.neters. I hope that alt.neters can strike the tricky balance between sustaining themselves and being open to people coming into the community. I hope I can play a role in making this happen. There was an excellent spirit at this first conference, one that provides lots of good fuel for the future.

I was in Boston, about to fly out to our office in Calgary. I look at my calendar to see if I have a meeting. First one is at 10.30am - cool no need to rush out of bed in the morning.

I turn up and am two hours late. What happened was that I was invited to the meeting that begun at 8.30am Calgary time. Lotus Notes saw my computer was set to Boston time, and helpfully converted the time zones for the two hour shift.

You could argue that this was my fault for not paying attention. After all I know this is how Notes works and was being careless when I read my diary. I don't buy this, Donald Norman noted a while ago that we tend to blame ourselves for errors that are due to bad usability - like a door with a handle that you should push.

Time zones are particularly vulnerable to this kind of problem It's most notably a problem in calendaring applications, but you see this issue in enterprise software as well. There is a temptation to try to be clever with handling time zones, but this temptation leads to trouble if the software isn't quite clever enough, which is what happened in this case.

I'd prefer Notes ignore time zones completely. You set the time for the place the meeting occurs in and that's what it should show you. Who cares about the time zones? It's only the time on the ground that usually matters. When I look at my calendar for a day in Calgary, I want to see the Calgary times for that day - wherever I happen to be when I'm looking at it.

The exception, of course, is a phone meeting that spans time zones. But here you should do something exceptional for a phone meeting rather than complicate handling a physical one. It could be something that allows you to set a flag for a phone meeting, and then you get a different time display. Or it could be as simple as just allowing you to put the timezone on the time display, and leaving the conversion to the reader. With phone meetings you think more about time-zones than you do for physical meetings (or at least I do).

The important lesson here is to make the most common case (physical meetings) simple and only do complicated things for less common cases as exceptions (quite possibly manual exceptions). Calendaring time zones get into trouble because they make the common and simple case be more complex. The problem occurs because the designers wanted to use the same data for both the simple and phone cases - but that just gets the simple case in trouble.

Usually when I hand out a prize for worst-user experience Lotus Notes is at the top of the queue. (Indeed I find it embarrassing to admit that ThoughtWorks uses the damn thing.) But the worst time-zone experience award goes to Microsoft Office, although to be fair this was many years ago. I had recently bought a PDA (running Windows CE version 2). I had put some all day meetings into my calendar, flown to Chicago, and to get the PDA to alert me to meetings I changed the timezone of the PDA to one hour earlier.

Suddenly every one of my all day meetings shifted to a day earlier. This was due to a catalog of errors. First off they had stored an all day meeting as a meeting from midnight to midnight - that's the kind of representation error on a TimePoint that often gets people into trouble. Then it was compounded by shifting the times of meeting when I changed time zones - so all day meetings were now 11pm to 11pm. This, of course, is due to putting the time zone into the meeting so it looks like it shifted when I changed time zone on the PDA. Then to cap it off, the software was clever enough to know that an all day meeting should only show the day of the meeting, but the day it chose was the day of the start of the meeting - that was now one day earlier. That's the poor timepoint representation biting back.

I was in the Calgary office last week and had a good chat with John Kordyback, one of our most trusted technical leads. He's worked on, and dug into, a number of travel loyalty software systems (frequent flyer/sleeper etc) and we talked about the nature of these kinds of things and how to think about them in a more fruitful manner.

The core of of a loyalty system is a system to keep track of points (or miles). This should allow customers to see their points and also for the company to manage the unredeemed points. Although it seems that most people don't see it this way, this is essentially an accounting system, just switching points for dollars. John's observation was that repeatedly he runs into what people see as difficult problems that are much easier to deal with once you put accounting spectacles on.

An example of this is dealing with ad-hoc changes. However good your automated rules processing is, there always cases when something odd happens and you have to intervene manually. The result for many systems is and ad hoc change to totals that is error prone and unaudited. With an accounting frame of mind, however, you look at these changes as accounting adjustments and the patterns for this are well understood.

A notable difference between a loyalty program and most accounting systems is that a loyalty program is more about managing liabilities rather than managing assets. Hence there's more focus on things like risk management, as well as common themes like taxes and revenue reporting.

Many loyalty systems have multiple kinds of points, such as such as regular miles and elite qualifying miles. This is a common point of complexity. If you use an accounting viewpoint, however, you can track these easily as multiple currencies.

An interesting twist on this is potential points. If I book a flight for next month, the airline needs to know that there are miles I will earn when I fly next month (potential miles). These potential miles affect their liabilities. However it's only when I fly that they turn into real miles. Again accounting thinking can help here, we can use multiple currencies again, or use an accounts payable notion. The mechanisms are there and well understood, we just have to apply the model to the situation.

We fleshed this out in practice where we also found it really helpful to use TestDrivenDevelopment. A group of people spent a couple of weeks trying to sort out potential miles with planned design, but the core issue was cracked in a couple of days with TDD. The crucial part of this was focusing on examples to make the problem concrete.

The accounting analogy also applies, although partly less directly, to deciding how to award miles for activity. Any program has activity rules that need to be very flexible and need to cope with constant changes to the loyalty program. We can look at this as following the model of domain events triggering accounting entries through using Agreement Dispatchers. This is a pattern John and I have used lots of times and works well to these kinds of changing rules. Essentially we have agreements that represent the overall program rules for a class of participant. Each agreement consists of a set of posting rules keyed by the type of event and a date range. When an domain event occurs (a hotel stay) we look up the agreement dispatcher for the customer, and use the event to look up the right posting rule. We then run the posting rule to create the appropriate accounting entries to represent the miles for the event. The time dating of the events allow us to change posting rules over time but still be able to handle old events and correctly do automated processing of adjustments. (Some day I'll finish writing up these patterns, but what I have on the web is hopefully enough to give you some ideas.)

The second aspect of a loyalty system is tracking the customer experience. Since the accounting requires the system to record the customer's activity, the loyalty system acts as a natural base to learn from the customer's interactions with the company. Much of this is data mining - looking for patterns in customer behavior which can lead to new products and promotions. You can also use this activity history to assess the success of promotions - if you offer a mileage bonus for flying a route what is the response like?

Like me, John is a strong proponent of using ReportingDatabases, and this is a good fit for this kind of problem. The accounting side needs a very different set of data structures and uses regular updates as activities occur. The customer experience analysis is all read only, so you can use less normalized structures with regular, but not necessarily real time, feeds from the accounting side.

Taking it further, it seems reasonable to completely decouple the accounting and customer experience systems. They are both usually lodged together as a single customer loyalty system because they track the same events. Yet since they differ so much on the inside it may make more sense to treat them as two separate systems that feed off the same event stream (the accounting side would probably generate some events for the customer experience side too).

One of the habits of customer experience tracking is frequent changes to the system to support new kinds of analysis. We speculated that we could try an approach that had a single stored event log of customer activity, and plug in relatively independent 'miners' that would transform selected information from the log into more particular data structures to do different kinds of analysis. The miners could be relatively independent of each other and thus easier to build.

As you can see, our discussion did shift from looking at John's experiences to some of our joint speculations about how a system like this could be built in the future. What's clear to us is that there is a lot of room for exploring new ideas in this space that could introduce a new set of abstractions that would lead to systems that can provide better support to this business activity. More and more attention is being paid to this these days, so this seems like a fruitful territory for us to work in.