Agile bliki

For nth, and I'm sure not last time, I'm sliding into a conversation about defining practices, labeling some of them as "best", and probably the C-word (certification). It's a familiar discussion, and although we've barely started it, I can predict much of where it will go. It's driven by a perfectly reasonable desire to identify who are the better software developers, and how existing developers can improve their abilities.

When people get into these conversations, they usually end up in trouble. Either the group gets into heated discussions and cracks up, or the group doesn't have heated discussions and produces something that others deride. The heart of why this happens, and why I don't see any single, widely-recognized certification program for software development coming soon is that there is no single, well-agreed way to develop software effectively.

Instead what we see is a situation where there are several schools of software development, each with its own definitions and statements of good practice. As a profession we need to recognize that multiple schools exist, and their approach to software development development is quite different. Different to the point that what one school considers to exemplary is considered by other schools to be incompetent. Furthermore, we don't know which schools are right (in part because we CannotMeasureProductivity) although each school thinks of itself as right, with varying degrees of tolerance for the others.

I'm using "school" here in the style of this definition:

4 a: a group of persons who hold a common doctrine or follow the same teacher (as in philosophy, theology, or medicine) <the Aristotelian school>; also : the doctrine or practice of such a group b: a group of artists under a common influence c: a group of persons of similar opinions or behavior; also : the shared opinions or behavior of such a group <other schools of thought>

--Merriam-Webster

I came across this notion explicitly from the Context-Driven School of Software Testing (see James Bach and Brett Pettichord). I like their way of looking at this because it's a model that explains why intelligent software developers have such different approaches.

The Context-Driven folks have done some looking at different schools within the testing world, but I don't know of any good attempt to classify the schools within the broader world of software development. I feel a sense of belonging to a school, one that for me is rooted in the people I met through OOPSLA in the 90's. Object-orientation is a key practice of this school, as is agile methods. You could reasonably argue that this is the agile school, except I think that agile methods are a core component of this school's thinking but not the whole picture. The leaders of this school include people like Ward Cunningham, Ralph Johnson, Kent Beck, and Robert Martin. ThoughtWorks is, on the whole, an organization that follows this school (which is why I'm comfortable here).

But despite this sense of a somewhat coherent school, there's still many open questions. Is it best to think of the agile world as one school or many (are Scrum and XP different schools or part of the same)? What are the major schools out there? What exactly defines a school of thought?

I don't have much of an answer to these questions, but the key point to remember is that there are multiple schools of thought about how to develop software effectively. We may not think much of the other schools to our particular one, but we are foolish not to recognize that other schools exist.

Imagine a hiring situation. There's two candidates both with a few years of experience. In the blue corner we have someone with good broad design skills in the style of design that you favor (in my case that would be things like DRY, judicious use of patterns, TDD, communicative code etc, but the actual list isn't important - just that it's what you favor). However she knows nothing of the particular platform technology that you're using. In the red corner we have someone who has little knowledge (or interest) in those issues, but knows your platform really well - edge cases in the language, what libraries are available, fingers move naturally over the tools. Assume all else about them is equal (which it never is except for thought experiments like this) and that your team doesn't have any gaping holes that this candidate might fill. Which one would you prefer?

My answer is simple, I'd take the one in the with broad design skills. I've always held the view that a good programmer should be able to pick up a new platform relatively quickly. Learning basic design aesthetics is both harder and carries over better to new platforms. Good design practices that matter in Java are equally valuable in .NET. Not being familiar with the platform does slow you down (how do I get a literal class name in C# again?), but producing well designed code is what really makes a difference.

Broad design skills aren't completely portable. Java and .NET are mostly equivalent as languages - moving to Ruby, however, changes more. Moving to a significantly different beast, like functional languages, is a bigger shift. In any case, you can't just blindly replicate all design habits in a new environment. But if you're aware of the new world, an awful lot does carry over.

We've seen this principle prove itself at ThoughtWorks. In our early days with Java, we found the skills experienced developers had learned in Forte gave us excellent instincts for working with Java. We moved away early from the EJB-dominant thinking, and I think it was experience with other platforms that guided us. We saw it even more strongly with .NET. Time and again we saw that good developers with a Java background were rapidly more effective than those with a longer .NET or Microsoft background who lacked those skills. The difference was visible in weeks, not months (and sometimes days).

At the moment we see this shift most notably in Ruby. We've had quite the run of Ruby projects this year, and often we turn to people with long experience in curly-brace languages to fill the need. Again we've seen the value that broad design skills gives us.

It's not always a sure thing. I have seen cases where someone experienced in another platform just doesn't desire to get in and learn the new one. Desire to learn is a necessary component here - I'd take the single platform specialist if he wanted to learn broad design and the broad designer didn't want to learn the new platform. It's also essential to have someone on the team who knows the platform well.

I'd say most people at ThoughtWorks prefer design skills over platform knowledge. Many clients don't share that point of view - which can lead to some difficult pragmatic and ethical choices.

What happens if you have someone you want to bring onto the team with strong design skills and no platform background - yet the client insists on at least two years experience on the platform. In your professional judgment, the broad candidate is going to be more productive than anyone else available. You need to be honest with your client, but on the other hand he is paying you for your professional judgment. Does this change if the client has given you responsibility for delivery of the project?

For us these questions are more charged because there is a financial interest involved. If we add a ThoughtWorker to a team, then we bill for that person. If a client hires a platform specialist contractor, we don't get that income. For many people this is a crucial fact in the situation, yet I expect our project managers are wise enough to know that the risk of adding the wrong person is much more important than one billable income.

Consider another case where you're open with the client and the client demands a reduced rate for the broad design person due to her lack of platform knowledge as she'll be learning on the job. You're sure that she, despite that lack, will be more productive than the competing platform expert due to those design skills. Should you accept a reduced rate?

It's the nature of our, and most other, professions that you learn on the job. A platform specialist also has to learn broad design skills if he's going to produce maintainable code. Here it's important to remember that not just is it usually harder to learn design than platforms, it's also less certain. Given a motivated broad-designer, I can be pretty sure she'll pick up a platform in time. But there's no guarantee the other way around. Some people are good at learning details of a platform, but never figure out how to write clear code.

I've talked here about broad design skills - and I do believe this makes a difference on the technical axis. But there are other dimensions of broadness too. Most risk in software projects lies in the communication between businesspeople and programmers, so a candidate who can communicate well with users brings a great deal to a team.

A similar issue is knowledge of the problem domain. Often clients want people who already know their business, yet are surprised when someone rapidly gains enough understanding to be useful. I've long held that it's the ability to collaborate with others which is central here. By collaborating with a domain expert, or a platform expert, someone with broad skills can be become effective very quickly. Knowledge of other domains often introduces surprising insights into a project and similarities often crop up in sup-rising places. It's remarkable how often things like core accounting patterns crop up in places that don't look like accounting on the surface. In the end it's the ability to work with others, coupled with being a fast learner, that is the critical skill.

I'm not dismissing deep platform knowledge here. In an ideal world every team member would be excellent broad programmers with several years platform experience, good familiarity with the problem domain, and written similar systems at least twice before. But we all know how far our world is from that ideal. You need some platform knowledge on a team, and if it were a gap I would reach for the platform specialist to fill it. But that doesn't alter my default position to prefer broad design skills most of the time.

A key property of agile development is figuring out how to make a system go live with a small subset of features. We build software for the business value it offers, the quicker we go live, the faster we get at least some of that business value.

My colleague Dave Leigh-Fellows told me one of my favorite examples of this kind of thinking. It came when we has working for a brokerage firm. They had a new kind of product that they wanted to get into the market. The full software support for this was a web page that the customer filled in that generated the necessary transactions against the back-end system. But Dave came up with a way to get the product into the market faster than that.

• Version 1 was a static web page that described the product and provided a telephone number to call. Some temporary staff then spoke to the customer and entered the information into the back end system.
• Version 2 was a web form that captured the data the customer filled in. However this version didn't load that data into to the back end system. Instead the web form generated a fax. They hired some more temps to get the orders from the fax machine to the people that keyed the information into the back end system. Since the fax machines were a bit of a distance away, this is where the roller-skates came in.
• Version 3 hooked the web form into the back-end system directly.

The first two versions may not have been the most elegant solutions ever conceived, but they did get the product into the market much more quickly. I've not come across any other examples of iterative development that use roller-skates, but that may be more due to lack of imagination rather than lack of need.

I'm a big fan of Continuous Integration, it's an relatively simple practice that can make a huge difference to most development teams. However like most practices it has its flaws^H^H^H^H^H opportunities for improvement. Paul Duvall, author of the soon-to-be-standard book on the subject, pointed out one of these recently. If the commit build breaks, the whole team is affected and potentially slowed until it's fixed.

When we first started doing Continuous Integration at ThoughtWorks, this one of the of the things that worried me about the way we were doing it. It worried me because there was an important difference between between the ThoughtWorks 2000 style and the style we'd used at C3.

The ThoughtWorks 2000 style is pretty much the canonical style of CI used today. Once you are happy with your work you commit it to the repository, and then build it on the build machine (either manually for with a CI server like CruiseControl). The problem lies if your commit is bad, anyone who updates will get failing code until you fix it.

In the C3 way of doing it we didn't commit to the head of repository directly. C3 was a Smalltalk project and used Envy, a Smalltalk-oriented repository system. Envy had some different concepts to mainstream repositories. Since it's ages since I used it my memory on exactly how it worked has gone all fuzzy, but the basic idea was that when you were working on your feature you committed to editions. An edition was like a private branch, visible to everyone but not blessed. Only when you had a successful build on the build machine would you upgrade your edition into a release, which was the equivalent to the mainline. This way you never got broken code into the mainline of the project.

Envy made it easy to work this way, the version control systems we mostly use now make it more tricky. Ideally you want to create a working copy that updates from the true head (to keep you in sync) but commits to a different pending-head branch. Only a successful integration build can actually commit to the true project head. A continuous integration server would check out from the pending head and, if successful, commit to the true head.

How difficult is it to set something like this up yourself? I'm not sure, I haven't chatted with a team that's done it2. However a number of team oriented tools are providing this kind of capability. For example JetBrains's TeamCity does it under the name "delayed commit". Paul also mentions Borland's Gauntlet.

The other question is how much it matters. Despite my worries we didn't get enough pain from broken builds to want to install a pending head in 2000. If you get a lot of broken integration builds there are other ways to fix it. Often the main problem is that people aren't doing a private build before they commit. As usual the people-issue is often a more important issue to deal with before introducing more complicated technology.

How do you improve the productivity of software developers?

An answer I've given regularly for many years now, and one that applies to almost anyone who uses a computer, is give them a bigger screen.

I used to raise eyebrows fifteen years ago by recommending that every developer should work on a 21 inch screen. These days I say that everyone should have at least two 20 inch screens.

Why is this important? If I have a small screen I can't see as much at once, so to see different things I have to keep popping windows to the front. With my two screens I can put a whole bunch of stuff on the screen at once and all I have to do is move my head. My eyes can flick between the text I'm typing now in Emacs and the rendered result in firefox. I can keep open an IDE with lots of subwindows and have documentation in a browser right there. I don't have look around on the task bar to see where I put that terminal window, I just mouse and type. It's often hard to imagine the improvement before you try, but I can really feel the difference since I doubled my screen.

And it's not as expensive as most people think. Chatting with a friend I looked up the price for my twin Samsung 204Bs - $700 for the pair at my local circuit city. Developers are expensive, it doesn't take much to recover that kind of cost (Hmmm, there's room on my desk for a third.)

Too often these days I see pair programming done on laptops with low and small screens. This is silly, big screens make a wise investment.

A common, perhaps dominant, practice of agile methods is to develop a list of features (often called stories) for the software that's being built. These features are tracked with index cards, work queues, burndown charts, backlogs, or whatever your tool of choice is.

On the whole I like this kind of approach. By breaking down everything you need to do into small tasks that you can complete in a week or few, you can visualize progress and get a sense of how much you can get done. I've often said that the key benefit of iterative development is to reduce risk by forcing completion of software in chunks instead of the waterfall habit of leaving long and hard to manage activities (testing, integration) till late in the project.

The problem comes when this list suddenly grows horns and fangs and becomes a Fixed-Price Fixed-Scope Big Up-Front Project Plan. Craig Larman once joked that the waterfall process has strong antibodies that reject iterative processes by warping them into some form of waterfall. RUP has been a common victim of these antibodies, seeing its phases turn into some variant of the analysis-design-build-test conveyor.

The key to beating off the waterfall is to realize that, as Dan puts it, agilists value Outcomes over Features. The feature list is a valuable tool, but it's a means not an end. What really matters is the overall outcome, which I think of as value to the customers.

An important part of this thinking is that you expect the feature list to change as the project goes on. This happens you discover new things that you can do, and re-prioritize old things. This is the essence of adaptive planning, which has always been a key indicator of agile thinking. This results a big shift in how people think about a plan. In plan-driven projects, success and failure is often worded in terms of "did things go according to the plan?" In agile projects this is a meaningless question, because plans change so often. The plan is a tool, primarily one that you use to gauge the effect of changes: "how will adding this feature affect what we do". The plan is a tool to figure out what should fit in the FivePoundBag. If your plan's not constantly changing, you are very unlikely to be doing adaptive planning, and hence aren't agile.

Feature lists have another problem - you easily lose sight of the context that makes the feature valuable. This is a reason why Alistair Cockburn is a proponent of use cases, because they concentrate on a narrative of how someone uses a system. Marc NcNeil also talks about this in terms of Customer Journeys. The weakness of use cases in planning is that they don't give you clear units to tick off so you can assess progress and project consequences of choices into the future. That makes them less useful as a planning tool, but that doesn't negate their value as tool for imagining what a good outcome would be.