Agile bliki

Here's a common misconception about agile methods. It centers on the way user stories are created and flow through the development activity. The misconception is that the product owner (or business analysts) creates user stories and then put them in front of developers to implement. The notion is that this is a flow from product owner to development, with the product owner responsible for determining what needs to be done and the developers how to do it.

A justification for this approach is that this separates the responsibilities along the lines of competence. The product owner knows the business, what the software is for, and thus what needs to be done. The developers know technology and know how to do things, so they can figure out how to realize the demands of the product owner.

This notion of product owners coming up with DecreedStories is a profound misunderstanding of the way agile development should work. When we were brainstorming names at Snowbird, I remember Kent suggesting "conversational". This emphasized the fact that the heart of our thinking was of an on-going conversation between customers and developers about how a development project should proceed.

In terms of coming up with stories, what this means is that they are always something to be refined through conversation - and that developers should play an active role in helping that definition.

• spotting inconsistencies and gaps between the stories
• using technical knowledge to come up with new stories that seem to fit the product owner's vision
• seeing alternative stories that would be cheaper to build given the technological landscape
• split stories to make them easier to plan or implement

This is the Negotiable principle in Bill Wake's INVEST test for stories. Any member of an agile team can create stories and suggest modifications. It may be that just a few members of a team gravitate to writing most of the stories. That's up to the team's self-organization as to how they want that to happen. But everyone should be engaged in coming up and refining stories. (This involvement is in addition to the develpers' responsibility to estimate stories.)

The product owner does have a special responsibility. In the end the product owner is the final decider on stories, particularly their prioritization. This reflects the fact that the product owner should be the best person to judge that slippery attribute of business value. But having a final decision maker should never stop others from participating, and should not lead people astray into a decreed model of stories.

There's a mess I've heard about with quite a few projects recently. It works out like this:

• They want to use an agile process, and pick Scrum
• They adopt the Scrum practices, and maybe even the principles
• After a while progress is slow because the code base is a mess

What's happened is that they haven't paid enough attention to the internal quality of their software. If you make that mistake you'll soon find your productivity dragged down because it's much harder to add new features than you'd like. You've taken on a crippling TechnicalDebt and your scrum has gone weak at the knees. (And if you've been in a real scrum, you'll know that's a Bad Thing.)

I've mentioned Scrum because when we see this problem, Scrum seems to be particularly common as the nominative process the team is following. For many people, this situation is exacerbated by Scrum because Scrum is process that's centered on project management techniques and deliberately omits any technical practices, in contrast to (for example) Extreme Programming.

In defense of Scrum, it's important to point out that just because it doesn't include technical activities within its scope should not lead anyone to conclude that it doesn't think they are important. Whenever I've listened to prominent Scrummers they've always emphasized that you must have good technical practices to succeed with a Scrum project. They don't mandate what those technical practices should be, but you do need them. After all projects get into trouble for poor internal quality all the time, the fact that a lot crop up under Scrum's flag may be more due to the fact that Scrum is so popular at the moment then anything particular to Scrum. Popularity and SemanticDiffusion tend to go together.

So what to do about it?

The scrum community needs to redouble its efforts to ensure that people understand the importance of strong technical practices. Certainly any kind of project review should include examining what kinds of technical practices are present. If you're involved or connected to such a project, make a fuss if the technical side is being neglected.

If you're looking to introduce scrum, make sure you pay good attention to technical practices. We tend to apply many of those from Extreme Programming and they fit just fine. XPers often joke, with some justification, that Scrum is just XP without the technical practices that make it work. Sniping aside, the XP practices make a good starting point - and are certainly going to be much better than doing nothing at all.

I always like to point out that it isn't methodologies that succeed or fail, it's teams that succeed or fail. Taking on a process can help a team raise it's game, but in the end it's the team that matters and carries the responsibility to do what works for them. I'm sure that the many Flaccid Scrum projects being run will harm Scrum's reputation, and probably the broader agile reputation as well. But since I see SemanticDiffusion as an inevitability I'm not unduly alarmed. Teams that fail will probably fail whatever methodology they mis-apply, teams that succeed will build their practices on good ideas and the scrum community's role is to spread these good ideas around widely.

Many people are looking to Lean as the Next Big Agile Thing. But the more popular lean becomes the more it will run into the same kind of issues as Scrum is facing now. That doesn't make Lean (or Scrum) worthless, it just reminds us Individuals and Interactions are more valuable than Processes and Tools.

Update at End

TechnicalDebt is a very useful concept, but it raises the question of how do you measure it? Sadly technical debt isn't like financial debt, so it's not easy to tell how far you are in hock (although we seem to have had some trouble with measuring the financial kind recently).

Here's one idea to consider. When a team completes a feature ask them to tell you how long it took them (the actual effort) and how long they think it would have taken if the system were properly clean. The difference between the two is the interest of the technical debt. (So if it actually took them 5 days but they think it would have taken them 3 days with a clean system, then you paid 2 days of effort as interest on your technical debt.)

There are certainly some serious flaws with this technique. The statement of how long it would have taken on a clean system is an estimate based on an imaginary state - so is difficult to make objective. There's the effort in capturing this information, which is easy to get out of hand. But the result may help project a picture of the state of the code-base in a way that's visible to non-technical staff.

Furthermore it may also help with decisions about whether to pay the principal. Some teams like to add technical debt stories to their product backlog - with estimates on how long it would take to remove them. Such technical debt stories are also estimates, but also provide a picture of how much debt has built up. You could get a bit more clever with the estimated interest payments by apportioning them to these debt stories (I spent an extra day on this feature because of the bad state of the flipper module). Comparing interest payments with the principal may help inform a decision about whether to pay off the principal.

I ran into someone recently who tried something a little like this and found it handy, but it's not something I've run into a lot. Certainly there are flaws with doing it - but it may be worth a try for a few iterations.

Update: A recent discussion surfaced another way to capture the estimated interest. During a retrospective (which wise teams do at the end of each iteration) capture an estimate of interest paid against each of the problem areas of the system. Doing this estimate against recent completed work may be easier than forward estimates against future stories.

A few years ago I was talking with a client who told me something he didn't like about the agile approach we were using: "it's doesn't feel right to have these difficulties this early in the project". Contrary to his reaction, in my mind this early pain is one of the great benefits of an agile or indeed any iterative development process.

I have many complaints about the waterfall process, but probably my greatest problem with it is how it tends to defer discovery of problems till late in the project, at which point there's little time or energy to deal with them effectively. Iterative cycles try to flush out as many problems as possible as early as possible. This gives you more time to cope, or at least raises the problems early enough to cancel before investing too much money and effort in a problematic project.

A useful exercise is to reflect on past projects and think about where problems cropped up late. Now ask yourself how you could make those problems crop up earlier. The more pain you get earlier, the better.

Here's one of my favorite software development quotes:

It's the opening paragraph of the first edition of their book "Mastering the Requirements Process". As regular readers might guess, my liking isn't connected to agreement. I like this quote because it sums up the waterfall value system to requirements (indeed the word 'requirement' is inherently waterfallish).

Agile methods violate this underlying assumption by intending to discover the 'requirements' during construction and after delivery. But even this cavalier disregard of the above sage advice is nothing compared to what many leading web sites do these days. These sites explore requirements by observing what the users do on their sites and using that information to generate ideas for new features along the following lines:

• Look at what people are trying to do with the site and provide easier ways for them to do it.
• Look at where people are abandoning doing something, and look for ways to fix whatever was frustrating them.
• Build a new feature and see if people use it.
• Build an experimental feature and make it available to a subset of the user base. Not just can you see if they like it, you can also assess how much load it puts on your servers.

To support this kind of analysis, you need to add user logging behavior into the application and build some tools to analyze these logs. Much logging appears for free in web applications, which I suspect was major impetus for people starting to do this. But the logging, and analysis, can go further as it's added to the application.

I haven't found much advice out there on the web on how to do this, and I don't hear much discussion about doing this in practice. Like many things it requires a concentrated effort to spend the time to build monitoring capability and then to use it to probe how to improve the software. Furthermore it's a mighty step away from how the traditional software process, even for agile projects.

But there is a huge potential here. Everyone knows how big the difference is between what people say they want and what people actually need and use. By watching what people actually do with your application, you can find out what actually happens with the software - which can give you much more direct information than other sources. As a result I think more teams should consider adding this approach to their toolkit.

Can SOA be done with an agile approach?

I don't delve too much into the cluttered world of SOA (ServiceOrientedAmbiguity), but I get this question often enough (in some form or other) to be worth a pontification.

When I first came across agile software development (in the form of Extreme Programming) the most troubling aspect for me was its approach to software design. Like many I'd become used to the notion that software should be designed before it was programmed, while XP seemed to encourage an almost wilful embracing of design ignorance. In 2000 I was asked to give the closing keynote at the first Agile/XP conference, and in order to gather my thoughts I ended up writing Is Design Dead? - an essay that examined the fate of design in an agile world.

I'm still quite proud of that essay and I think it's well worth reading today - but for this bliki entry I'll summarize. I talked about two driving approaches to software design: planned and evolutionary. Planned design works out the design of software in one phase and builds (programs) it afterwards. In this case changing the design is hard once you've begun construction. Evolutionary design assumes regular change of the design even once you've done significant programming. I concluded that the practices of XP provided a disciplined approach to evolutionary design, thus making it much more practical than people had realized. This change did not remove software design (it is not dead) but did significantly change how we think about design.

The argument for planned design in an SOA context is that we are building webs of interconnected, loosely coupled systems. In this situation each system is making its services available as a PublishedInterface to the whole enterprise. Published interfaces are hard to change, therefore you need planned design to get them right so you don't have to change them. Planned design is also a reaction to the chaotic system interconnections that people see in most organizations. This chaos is the result of poor design, so the feeling is that better planned design will prevent this happening in future.

Evolutionary design is an essential aspect of agile methods. One of the key foundations of agile methods is the desire to handle, indeed welcome, change. Planned design assumes change is hard, and thus tries to predict where it occurs. If changes occur within the predicted boundaries then it's easy, but if it falls outside those boundaries you're out of luck. Agile thinking assumes unpredictable change is inevitable and tries to enable it in a controlled way.

So as I look at SOA, or any other design context, the fundamental question I ask is "is change predictable?" Only if change is predictable is a planned design approach valid. My sense is that if predictability is hard within the context of a single application, it's doubly hard across an enterprise. If we use planned design in a unpredictable context we find that however good the plans are, they will be undermined by the unpredictable changes, leading to the same mess we are currently in. Usually, however, the mess is worse because a there is significant sunk cost in a planned design, this can easily reduce the business value that an SOA effort is intended to produce, particularly if time-to-market is important.

As a result I think we have to bite the bullet and figure out how to do evolutionary design in this loosely connected context. After all the whole of point of loose coupling is to make change easier. At the center of this is thinking about contracts in terms of change, with such ideas as Consumer Driven Contracts.

This direction leads to things like Jim Webber's notion of Guerilla SOA. This builds up SOA using small steps directed by producing business value. Since producing business value is the whole point, this offers a path to producing a much better return on investment. It's certainly an approach our clients appreciate.

Can evolutionary design scale to SOA sizes? In my view we have an existence proof at a much larger scale than even a big SOA effort - the web itself. The web is built on very loose coupling and lots of unpredictable changes. It is, in many ways, a mess - but it's a mess that works, delivering real value to lots of people every day.