Agile bliki

One of the steady themes I've seen throughout my career is that of the nature and importance of software development. Recently a prospect told one of our salespeople that "software is like sewage pipes, I want it to work reliably and I don't want to know about the details". This is the kind of approach that Nicholas Carr talked about in IT Doesn't Matter. On a contrasting note we've done work for many businesses where IT has been a clearer strategic enabler to their business, allowing them to enter new markets or significantly increase their market share. So is IT a utility, like sewage pipes, or a strategic asset?

I take that the view is that it can be either, depending on the system. A classic example of a utility IT project is payroll, everyone needs it, but it's something that most people want to 'just work'.

So what is the distinguishing factor between utility and strategic projects? To my mind it's all about whether the underlying business function is a differentiator or not. If how you do this function is a crucial part of what makes you better than the competition, then the software that supports this function needs to be as good as you can make it. Note that this distinction isn't about the software, but the business function. As Ross Pettit puts it "This is not a separation of IT by the nature of the technology, but into what technology does for the host business".

The most important point about this dichotomy is to realize that there are two kinds of software projects and they need to be treated entirely differently. They way you staff, run, and budget a strategic project is entirely different to how you do a utility project. Too often people assume that what is good for one is good for the other - and trouble inevitably follows.

Another consequence is that only a few projects are strategic. The 80/20 rule applies, except it may be more like 95/5. While certainly it's most common for people to not recognize the dichotomy at all, it's also common for people to think that too many projects are strategic.

One of the most important ways in which these efforts differ is where the risks lie. For utility projects the biggest risk is some kind of catastrophic error - you don't want the sewage pipe to break, or to miss payroll. So you need enough attention to make sure that doesn't happen, but other than that you want costs to be as low as possible. However with strategic projects, the biggest risk is not doing something before your competitors do. So you need to be able to react quickly. Cost is much less of an issue because the opportunity cost of not doing something is far greater than costs of software development itself.

This is not a static dichotomy. Business activities that are strategic can become a utility as time passes. Less often, a utility can become strategic if a company figures out how to make that activity a differentiator. (Apple did something like this with the design of personal computers.)

One way this dichotomy helps is in deciding between building custom software and installing a package. Since the definition of utility is that there's no differentiator, the obvious thing is to go with the package. For a strategic function you don't want the same software as your competitors because that would cripple your ability to differentiate.

Often people realize this and buy a package for a utility project, but then spend huge amounts of money customizing this - which is just as wasteful. My view is that for a utility function you buy the package and adjust your business process to match the software. Usually this is politically infeasible, so the workaround is to put a low grade software team to work on it. Provide enough care to avoid catastrophe, but otherwise you don't need a high-grade team.

Another way the dichotomy makes its influence felt is the role of agile methods. Most agilists tend to come from a strategic mindset, and the flexibility and rapid time-to-market that characterizes agile is crucial for strategic projects. For utility projects, however, the advantages of agile don't matter that much. I'm not sure whether using an agile approach for a utility project would be the wrong choice, but I am sure that it doesn't matter that much.

Like many classifications, there's a lot of grey in between. Yet this is one of those rare cases where I think there's a strong argument to turn up the contrast and force more binary thinking. As Ross commented in a discussion of a draft of this post: "'shades of grey' give license to pile things into the wrong category; things that are really utility will be given an inflated importance, rather than dispositioned as the utilities they really are." Forcing a binary decision, tilted to minimize what's in the strategic bucket, would help provide the focus that's often lacking in IT initiatives.

Ross goes so far as to argue that there shouldn't be a single IT department that's responsible for both utility and strategic work. The mindset and management attitudes that are needed for the two are just too different. It's like expecting the same people who design warehouses to design an arts museum.

To explore more...

• Ross's article calls for a Glass-Stegall of IT departments.
• Marc McNeil talks of projects as Tractors, nuclear power plants and the bleeding edge
• Neal Ford points out that integration efforts, such as SOA, can't be strategic.
• There is video of the talk I gave on this theme for AMP last year.

A common thing you find in agile projects is that the development team sits in a single open team room. It was advocated early on in Extreme Programming and called out as one of primary practices in the second edition. Agilists favor a open team room as it promotes lots of informal and deep communication between people on the team.

Why

Software development is an intense exercise in collaboration. An open space encourages regular conversations and interactions between people. You can see what everyone's doing and easily ask for help when you need to. Often you get serendipitous communication, where you overhear something that's really useful.

Hearing this, some people are concerned about noise, and would prefer private offices. In practice I find that teams don't find noise to be a serious issue. There's usually a hum of conversation going on, after all pair programming often accompanies this style of development. But the conversation isn't usually that distracting, particularly as you're focused on the conversation with your pair. I suspect the reason it's not that distracting is because the team has a common purpose around a collaborative activity. It isn't comparable to an open-plan office where everyone is doing something different.

Tips for a good Team Room

First make sure it is the right size for the team. While a team room should be open within itself, it should be closed to everyone else. In an ideal world you'd like flexible walls that can insulate one team from another, so an office consists of pods of teams. This is hard to do in practice. Our offices tend to be completely open, with little barriers between teams. This seems to work well-enough, although there are some complaints about the inter-team noise.

Pay attention to natural light. Humans are used to seeing the outside world, and all sorts of natural rhythms work off of light. So it's no surprise that people get very cranky if there isn't enough light about. I've spent plenty of days in enclosed conference rooms, and it certainly wears down my energy.

Provide enough space: about 50 square feet per person (4.5 square meters for those on metric).

With the right kind of space the next key thing is to ensure that the team has control over that space. An important part of agile thinking is that the team is responsible for how it works, and how it organizes its space is part of that. Ideally you want a team to have complete control over their space, with freedom to configure it how they like and reconfigure it at will. Things should be done so it's easy to move things around, because during the project the team will need to change things as the project changes.

The purpose-built tables in our Beijing office have handy baskets for power and other cables.

An immediate consequence of this is to ditch any kind of modular furniture that requires a facilities group to move more than an inch. Most teams I see use simple tables, and you can certainly go cheap here.

The biggest hassle is wires - primarily for power and network access. Ideally you want these underfloor or overhead so that people can easily route wires to the tables, wherever the tables happen to be.

The place to spend money on furniture is for good quality chairs. Programmers spend a lot of time sitting, and any physical damage due to poor posture will have a direct effect on the team's productivity - so don't stint here. It may be that some people will want strange chairs, such as balls or kneeling chairs. Do your best to accommodate them.

Some people are big fans of tables that can be adjusted between a sitting and standing height, as they find that standing for a while helps with back pain. These are harder to find, but worth looking into if your team members need it. Back pain is a common issue, but everyone's pain (and treatment) is different.

You'll need lots of wall space, as agilists love their information radiators. You want plenty of room for story walls, architectural diagrams, whatever people want to stick on the wall. A good bit of this wall space should be whiteboards so people can draw something whenever the mood takes them. Include some wheeled whiteboards. Make sure there's a digital camera around so people can easily record what's on the board. Now that displays are so cheap, consider getting some just for wall displays - this is particularly handy for dynamic displays such as build status. I've seen one team have projectors trained on a wall with various kinds of information displays.

The traditional layout is to have people working around clumps of desks. This gives you regular eye contact with the rest of your team. However I've heard many people sing the praises of the UPod.

People will occasionally need some private space, so ensure there are one or two small conference rooms available, with telephones. These can be used for privacy or when there's a concern about distraction. A large meeting room that the team can gather in for meetings away from the working space is also handy.

I've always been a big proponent of lots of monitor space. Clever software with multiple virtual workspaces is a great feature, but nothing is faster than just moving your eyes. Every workstation should have a couple of 20 inch monitors as a bare minimum. My desk has a pair of 20 inch monitors for my Ubuntu machine and a 25 inch for my mac laptop. I don't think that's at all excessive.

Software development is inherently creative, so expect to see lots of baubly distractions. Toys are commonly found around our teams (as Neal Ford says: "every team needs a plastic kangaroo"). There are good cognitive reasons why this is valuable, it's all about keeping the brain stimulated and creative.

Similarly do provide easy access to snacks and drinks. This helps support informal, conversational breaks in the team area. It's hard to be creative when you have to pay for awful coffee.

If you're working with remote workers, make it easy to set up a video link. Indeed many teams like having permanent video links to any remote workers so you can always maintain that casual video contact.

Useful Links

• InfoQ summary of things for a team room.
• William Pietri describes a team room he's used.
• Bill Wake's gallery of team rooms and charts.
• Rachel Davies's advice on building an agile environment.
• Joel Spolsky is perhaps the best illustration for using individual offices rather than a single team room.
• Rajeev Singh's thoughts on moving into this kind of team room at ThoughtWorks India.

My thanks to my ThoughtWorks colleagues for helping compile this information.

Translations: | Portuguese |

SEMAT (Software Engineering Method and Theory) is an effort initiated by Ivar Jacobson, Bertrand Meyer, and Richard Soley. Its stated aim is to "refound software engineering based on a solid theory, proven principles and best practices". Like many notorious people in the software world I was invited to participate. Thus far I've declined and feel the need to explain why.

The call for action seemed somewhat vague to me. Although it derided fads and fashions, it seemed very much like a fashion statement itself. To better understand what was going on I took a further look at some of the initial publications in Dr Dobb's Journal. From here I got the distinct impression that the central thrust of the initiative is to create a software meta-method-kernel - essentially a set of common process elements for software developments that you can rigorously compose into a method for your own project.

At this point I lost interest. I spent a fair bit of time in the 80's and 90's mucking around with this idea. In the end I decided that it was too hard and of limited value. Why this is so was primarily crystallized for me by Alistair Cockburn who explained that since people are the central element in software development, and people are inherently non-linear and unpredictable - such an effort is fundamentally doomed. Or at least it is until people become predictable agents that can be described with tractable mathematics - for which event I'm not holding my breath.

My views have since gone along the lines that software process is a much more multi-faceted thing than what a meta-method-kernel would usefully describe. Alistair's work on describing methods strikes me as a much more realistic approach.

Alistair actually did get involved in SEMAT and attended their inaugural meeting. His decision to withdraw further reinforces my lack of interest to participate myself.

One of the arguments used to support the adoption of lean techniques in software is the success of Toyota. So do Toyota's recent quality failings undermine the case for lean software development?

One answer for this is to take a sense of proportion. Lean manufacturing techniques were the underpinning of Toyota's rise from an insignificant company in the 1950's to a global giant in the 2000's. By the 1990's other car companies, and many other manufacturers, were busily copying Toyota's techniques. The general sense is that copying these techniques did much to raise the overall quality of cars in the last decade or so. I would be very surprised if the recent problems at Toyota are enough negate that half-century of success.

But a better answer is to remember that Lean manufacturing is about manufacturing not software. The application of lean ideas to software development is a consequence of MetaphoricQuestioning. Lean ideas can help us come up with better ideas for software development, and as such are valuable. But in the end their usefulness lies with how they are used in software and they should be judged on their record here. Their history in manufacturing, both good and bad, is another industry.

Here's a common misconception about agile methods. It centers on the way user stories are created and flow through the development activity. The misconception is that the product owner (or business analysts) creates user stories and then put them in front of developers to implement. The notion is that this is a flow from product owner to development, with the product owner responsible for determining what needs to be done and the developers how to do it.

A justification for this approach is that this separates the responsibilities along the lines of competence. The product owner knows the business, what the software is for, and thus what needs to be done. The developers know technology and know how to do things, so they can figure out how to realize the demands of the product owner.

This notion of product owners coming up with DecreedStories is a profound misunderstanding of the way agile development should work. When we were brainstorming names at Snowbird, I remember Kent suggesting "conversational". This emphasized the fact that the heart of our thinking was of an on-going conversation between customers and developers about how a development project should proceed.

In terms of coming up with stories, what this means is that they are always something to be refined through conversation - and that developers should play an active role in helping that definition.

• spotting inconsistencies and gaps between the stories
• using technical knowledge to come up with new stories that seem to fit the product owner's vision
• seeing alternative stories that would be cheaper to build given the technological landscape
• split stories to make them easier to plan or implement

This is the Negotiable principle in Bill Wake's INVEST test for stories. Any member of an agile team can create stories and suggest modifications. It may be that just a few members of a team gravitate to writing most of the stories. That's up to the team's self-organization as to how they want that to happen. But everyone should be engaged in coming up and refining stories. (This involvement is in addition to the develpers' responsibility to estimate stories.)

The product owner does have a special responsibility. In the end the product owner is the final decider on stories, particularly their prioritization. This reflects the fact that the product owner should be the best person to judge that slippery attribute of business value. But having a final decision maker should never stop others from participating, and should not lead people astray into a decreed model of stories.

There's a mess I've heard about with quite a few projects recently. It works out like this:

• They want to use an agile process, and pick Scrum
• They adopt the Scrum practices, and maybe even the principles
• After a while progress is slow because the code base is a mess

What's happened is that they haven't paid enough attention to the internal quality of their software. If you make that mistake you'll soon find your productivity dragged down because it's much harder to add new features than you'd like. You've taken on a crippling TechnicalDebt and your scrum has gone weak at the knees. (And if you've been in a real scrum, you'll know that's a Bad Thing.)

I've mentioned Scrum because when we see this problem, Scrum seems to be particularly common as the nominative process the team is following. For many people, this situation is exacerbated by Scrum because Scrum is process that's centered on project management techniques and deliberately omits any technical practices, in contrast to (for example) Extreme Programming.

In defense of Scrum, it's important to point out that just because it doesn't include technical activities within its scope should not lead anyone to conclude that it doesn't think they are important. Whenever I've listened to prominent Scrummers they've always emphasized that you must have good technical practices to succeed with a Scrum project. They don't mandate what those technical practices should be, but you do need them. After all projects get into trouble for poor internal quality all the time, the fact that a lot crop up under Scrum's flag may be more due to the fact that Scrum is so popular at the moment then anything particular to Scrum. Popularity and SemanticDiffusion tend to go together.

So what to do about it?

The scrum community needs to redouble its efforts to ensure that people understand the importance of strong technical practices. Certainly any kind of project review should include examining what kinds of technical practices are present. If you're involved or connected to such a project, make a fuss if the technical side is being neglected.

If you're looking to introduce scrum, make sure you pay good attention to technical practices. We tend to apply many of those from Extreme Programming and they fit just fine. XPers often joke, with some justification, that Scrum is just XP without the technical practices that make it work. Sniping aside, the XP practices make a good starting point - and are certainly going to be much better than doing nothing at all.

I always like to point out that it isn't methodologies that succeed or fail, it's teams that succeed or fail. Taking on a process can help a team raise it's game, but in the end it's the team that matters and carries the responsibility to do what works for them. I'm sure that the many Flaccid Scrum projects being run will harm Scrum's reputation, and probably the broader agile reputation as well. But since I see SemanticDiffusion as an inevitability I'm not unduly alarmed. Teams that fail will probably fail whatever methodology they mis-apply, teams that succeed will build their practices on good ideas and the scrum community's role is to spread these good ideas around widely.

Many people are looking to Lean as the Next Big Agile Thing. But the more popular lean becomes the more it will run into the same kind of issues as Scrum is facing now. That doesn't make Lean (or Scrum) worthless, it just reminds us Individuals and Interactions are more valuable than Processes and Tools.