Martin Fowler
Chief Scientist,
ThoughtWorks
Since the begining of the new millenium, we've been running an interesting XP project. It's interesting not just because it was one of the first XP projects at ThoughtWorks, but also due to its size: around 50 people are involved. Here we talk about how we run this project by focusing around the activities that need to be done for a single iteration, and how the various subteams work around that iteration.
Note: This article was written in late 2000 and was based on the experiences we had at that time. There is much that both ThoughtWorks and the agile community has learned since then, so bear that in mind as you read this. Sadly we don't have time to comprehensively re-assess the content of the article, so just beware that we don't necessarily agree now with what we believed then. -- MF
The original version of this article was published in the Cutter IT Journal.
ThoughtWorks, Inc. is a Chicago based system integration and consulting company of around 300. We specialize in building demanding business applications using technology that is both new and strategic for the industry. We also work on complex business applications that usually require stitching together various components and augmenting these components with complicated and changing business logic. We've done a lot of work in the leasing industry, which is not known for simple and logical contractual agreements. If you've ever read a leasing agreement, you'll get the picture.
For quite a while now, we've been working on a particularly demanding leasing application, built with Enterprise Java. It's a big application, the kind of thing where existing vendors have spent over a decade developing their systems. Before completely revamping our project approach, we spent 18 months gathering requirements, creating prototypes, and beginning development. We ran into many of the problems that are typical for these kinds of projects. The requirements were poorly understood and changing. Simply controlling a project that size is demanding both for the development and for the customer relationship. During this time we came into contact with the ideas of Extreme Programming (XP) and felt that it was a decent fit for the nature of our project. XP was also a good fit for the culture of our company. So eight months ago we began working in an XP style.
Although the fit is decent, it's by no means perfect. XP is designed for projects involving a maximum of a dozen developers. We have a team of fifty people, half of whom are developers. XP also encourages frequent production releases. Since we are in the game of replacing complicated existing systems, we knew that we faced a long time before we could release something the customer would use for real deals. We also realized from the beginning that these factors meant we couldn't do XP by the book. (But then we never do things by the book even if we wrote it!) So an important part of our process involves changing it a little bit every month with the participation of the entire team to re-design our XP approach to suit our needs.
When we started with XP, we mapped out a release plan. This took 18 months of story cards (roughly equivalent to a feature) and allocated them to iterations. XP runs a project by dividing into short iterations, each of which has to deliver production quality software that contains newly implemented story cards. We use one month iterations (slightly longer than vanilla XP). Every month the customer can see the system growing with new features.
This essay discusses how we are currently doing this project. We intend to change it periodically as we change our process, and get round to updating the article. We'll also discuss some of the changes we've made and why we made them.
We've got a 50-person team. The project management office consists of two account managers who are primarily responsible for client and sponsor face time. The day-to-day planning work is done by a release manager and iteration manager. The release manager concentrates on the long range release plan, which is very volatile and has a horizon of six months to a year. The iteration manager focuses on the iteration plan, which is the details of what happens in a single iteration.
The split between release and iteration managers reflects the style and preferences of the two individuals that do it. Originally they alternated as "odd" and "even" iteration managers, but we found that Matt was better at the track the ongoing iterations and Cara was better at the broader release plans. So they adjusted their roles to suit their strengths and interests. We don't think our solution is one that necessarily everyone should follow. An important part of XP is that people sign up for what they are interested in doing, this applies to managers as well.
In XP terms, our "customer" is a 12-person team of in-house domain experts and clients who will receive the beta release of the application.
There are roughly 25 developers on the project, divided into two teams. The domain team is solely responsible for coding the new functionality in a given iteration. The SWAT team is responsible for bug fixes, the build process, automation of testing, and all other development tasks not related to the new functionality. We split the team like this to maintain a coherent design and to prevent people from stepping over each other. We also didn't want more than 13 people adding new functionality to the code base at a given time. At each iteration, a few developers rotate between the teams; every developer has been on both teams.
Our quality assurance staff is an 8 person team responsible for daily regression testing, determining what story cards will be delivered at the end of an iteration, and the preparation of the release notes for each delivery.
From the domain developer's point of view, the iterative process is very simple. Each iteration begins with a new batch of stories which they implement, hand off to QA, then start the next iteration. However there's a lot of work surrounding this simple story. Each part of the team has it's own cycle that it works on. Most teams spend a typical month working on some material from 3 given iterations: the current iteration, the previous iteration and next iteration.
To illustrate our XP approach, we're going to focus on what needs to happen for a particualr iteration. We've picked number 6 to make it easier to talk about, but what we describe is what we do now rather than what we did on 6. (We didn't want to update all the numbers every time we update the article.). Iteration 6 was scheduled for June, but part of the work also occurs in May and July. (It's a helpful coincidence that we started at the beginning of the year, but that's not mandatory!) As we talk about the projects we always focus on the iteration that the domain team is working on, so we say things like "in the first week of iteration 5, the planners get the list of story cards for iteration 6." There's cycle tables here for each team showing the activities they do during iterations 5, 6, and 7. To see the flow of activities better in the table, we've shaded all the activities for iteration 6 so you can easily see which activities occur and whether they occur before or after the actual development of the iteration. We've included a seperate cycle table for each team, and you can look at the consolidated cycle table to see the activities for all the teams in one place.
The first hazy plans for an iteration occur somewhere in the mists of time as the release plan. The release plan is never static and shifts and changes as we learn about what the customer wants and what the developers can do. In the release plan there is a bundle of stories, chunks of functionality, for iteration 6. The release plan may be shifting, but that doesn't stop it from being an essential tool. It gives us a way of seeing what may be acheivable, evaluating options, and assessing the effects of events that hit us.
Early in May, shortly after iteration 5 begins, the release manager starts looking in earnest at the content of iteration 6. The customer team works with the release manager to decide which stories will actually make it for presentation to the developers in June. With iteration 5 started, we know what will and what won't get worked on during iteration 5. With that information in hand, you know what story cards will not be worked on in the current iteration, and you are ready to evaluate what should be worked on next. The release manager now starts coordinating getting everything together for iteration 6.
This primarily involves coordinating the prioritization effort with the customer to develop the next list of story cards to be worked on. The customer can select any story cards they wish as long as the estimated developer weeks on the cards add up to the number of available developer weeks for an iteration. This is called the velocity. In this case, we scheduled 28 weeks of ideal developer time, which is based on the amount we got done in iteration four.
| Dispose Asset | |||
| Create A/R record for sales price with or without tax calculation. Create journal entries for A/R record. Create expenses for A/R records to ahndle disposal costs and appropriate journal entries. Change asset status to "disposed". Create journal entries for releiving (fixed asset and accumulated) inventory and calculated gain or loss. Set stop dates on book and tax depreciation. Create transaction history record. Asset must be active - in inventory. | 2 weeks | Analyst: Terri Developer: Biren QA: Shelly |
|
The customer has the opportunity to outline new functionality not currently scheduled in the release plan, pull in functionality slated for later, push out functionality scheduled for the iteration, or reprioritize the functionality within a card. In terms of story cards, this is a customer's opportunity to, based on their current business needs and strategies, create new story cards, move them around in the plan, move them out of the plan entirely or split them into several cards which will get prioritized differently. We also involve developers by asking them to re-estimate the stories that we considering for the iteration. With a better sense of the project and more focus on the stories these estimates have become very accurate - to the point that we now mostly are dead on with our estimates of what will fit in the next iteration.
This planning results in the release plan being laid out again. The process of selecting the next month's functionality usually takes about a week. As soon as this is finished, the analysis cycle begins.
In preparation for the iteration 6 planning meeting, the release manager must determine the exact number of developer days available during the month. This means, taking into account people's vacation and training days, the number of domain developer days available during the month and converting that number based on the previous iteration's velocity into the number of ideal developer days available for the upcoming iteration.
At the begining of iteration 6, the iteration manager hands over the planning work of the iteration. In the iteration planning meeting the developers come up with an iteration plan. The iteration manager then tracks the iteration plan and coordinates the closing of the iteration and the delivery of the iteration to the customer.
| Planning Cycle | |||||||||||
| Iteration 5 | Iteration 6 | Iteration 7 | |||||||||
| Get list of story cards for Iteration 6 | Plan resources for Iteration 6 | Deliver Iteration 4 | Close Iteration 5 | Get list of story cards for Iteration 7 | Plan resources for Iteration 7 | Deliver Iteration 5 | Close Iteration 6 | Get list of story cards for Iteration 8 | Plan resources for Iteration 8 | Deliver Iteration 6 | Close Iteration 7 |
| Manage Iteration 5 | Manage Iteration 6 | Manage Iteration 7 | |||||||||
| Prepare for Delivery of Iteration 4 | Work with Analysts to articulate Iteration 6 funcionality. | Prepare Iteration 6 planning meeting | Prepare for Delivery of Iteration 5 | Work with Analysts to articulate Iteration 7 funcionality. | Prepare Iteration 7 planning meeting | Prepare for Delivery of Iteration 6 | Work with Analysts to articulate Iteration 8 funcionality. | Prepare Iteration 8 planning meeting | |||
| Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 |
Two weeks before the beginning of iteration 6, the domain experts meet to discuss the upcoming functionality. After reaching consensus on the details of how the new functionality should work, they divide up the responsibility of articulating that functionality in a work product we call a narrative. A narrative is written for each story card slated for the iteration. A narrative is similar to a use case, but is less formal. We've chosen to articulate the new functionality in a more prose description than usually seen in a use case.
At the end of the narrative, a sketch of the relevant test scenarios is laid out. At a minimum, the test scenarios must be articulated sufficiently to allow a developer to estimate the detailed programming tasks related to the story card. These test scenarios are expanded to the final tests to necessarily and sufficiently test the functionality of the story card. Time permitting, these test scenarios are expanded as much as possible prior to the iteration 6 planning meeting. In practice, test scripts are generally finalized mid-way through iteration 6.
The domain experts meet again prior to the iteration 6 planning meeting to review the narratives and sign up for the story card. Signing up for a card means agreeing to presenting the narrative during the iteration planning meeting, being the point of contact for the development staff during the iteration, ensuring the quality and completeness of that cards test scripts, and handing off the functionality to the QA team at iteration close.
| Analysis Cycle | |||||||||||
| Iteration 5 | Iteration 6 | Iteration 7 | |||||||||
| Working with Developers on Iteration 5 | Working with Developers on Iteration 6 | Working with Developers on Iteration 7 | |||||||||
| Prepare for Delivery of Iteration 4 | Articulate Iteration 6 funcionality. Write Narratives | Close Iteration 5 | Prepare for Delivery of Iteration 5 | Articulate Iteration 7 funcionality. Write Narratives | Close Iteration 6 | Prepare for Delivery of Iteration 6 | Articulate Iteration 8 funcionality. Write Narratives | Close Iteration 7 | |||
| Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 |
The iteration planning meeting is the whopper meeting that gets the entire team on board with exactly what functionality will be worked on during the iteration. All the developers and analysts attend (QA find it comes at a very bad time in their cycle, so just a QA representative comes along). We sit in one big room with poster sized story cards stuck to the wall. There are a few ground rules: use the mike, no cell phones, no laptops, and no whining!
We provide everyone with copies of the narratives and a scope document. The scope document is divided into the broad areas of functionality for the system. For each area it shows which story cards have been done, which are slated for this iteration, and which are slated for later iterations. We've found this very helpful in clarifying what exactly is in or out of this iteration.
After bagels and coffee, the point of contact analyst leads a discussion about the functionality outlined in the narrative of the first card on the agenda. We discuss the scenarios. Analysts clarify the functionality and developers articulate what it will take to code the functionality. The developers outline a complete list of development tasks for each card. As the developers call out the tasks, they are written on big sticky notes and stuck to the card.
After lunch, we finish the functionality discussion, breaking the story card into tasks. The development staff meets as a group to estimate each task. For most of the project, the total number of developer days on the targeted cards exceeds the estimated capability of the team. So the customer picks entire cards or several tasks, under advisement from developers, to pull from the iteration. During the negotiation of what will be pulled out, the individual tasks or whole cards physically get pulled off the wall.
Although this pulling has been a regular feature of the project, it hasn't happened in the last few iterations. This is because our release planning has improved and we now estimate pretty accurately how many stories to bring to the iteration planning meeting. This both helps planning and the smooth running of the meeting. A key part of this has been involving developers in the pre-iteration planning process to give more accurate estimates. Also practice has made a big difference: you get better at estimating as you do more of it. So don't let early innacuracies put you off.
| Write dispose asset transactor | 1 day | Biren |
| Make journal entries | 2 days | Biren |
| GUI - create asset disposal screen | 3 days | Caleb |
| Calculate gain/loss | 2 days | Biren |
| Create pessimistic session bean | 1 day | Biren |
| Generate disposal charge | 1 day | Biren |
| Update statuses | 1 day | Biren |
| Coded Functional Tests | 2 days | Caleb |
Once the functionality has been selected, developers, sign up for tasks. Developer sign up has some basic rules; each developer should stick to tasks on just a few cards and no card should have too many different people signed up for its tasks. Each story has an analyst and QA person sign up before the meeting. A developer also signs up for a card, that involves coordinating those developers who've signed up for tasks on the card.
The result is a well articulated and reasonable list of functionality that will be developed during the iteration, a list of developer estimated development tasks sufficient and necessary to code that functionality, and resources allocated to each story card and task. Everyone on the team has participated in this process and is familiar with the plan for the next month's development of new functionality outlined on posters that remain on the wall throughout the entire iteration.
Historically our iteration plans have been stunningly accurate. We rarely fail to complete stories that were signed up in the iteration plan, and this confidence carries through to everyone involved. The iteration plan is seen as a very stable prediciton of what will appear in a month's time.
We wrap up the day-long suffering (these meetings can be extremely painful) with a keg.
How to present the functionality to the developers is something that we still feel we have much to learn about. A key principle is that the presentation and narrative are by no means the final word. During the iteration developers and analysts have frequent contacts on the details of the function. At the iteration planning the developers don't need all the information, but they do need enough to break the stories down into tasks and thus form the iteration plan. This step requires some initial design work.
The big problem is that the analysts need to be sure that they have told the developers everything of importance, but the developers want a minimum of information as there's only so much you can absorb in a day. So we face the difficulty that the developers want more brevity, but the analysts are concerned that they could leave out something important that will affect the expectations of the client.
We've tried doing things without narratives and having the release manger present the narratives instead of the analysts doing it. Neither worked well. Our latest attempt is to have preliminary meetings where a subset of the narratives are looked over by an interested subset of the developers. This helps work out many of the issues in advance.
The domain developers are responsible for adding the iteration's new functionality into the code base. Throughout the iteration they work closely with the domain experts and with each other to design and complete the development tasks they signed up for during the planning meeting. The domain experts are on-site and available to work face to face with developers. Collaboration among the developers occurs in several ways. Every other day the developers meet for a quick stand up meeting to discuss what they are currently working on and resolve development issues as a group.
Although one developer bears the responsibility of completing a task, as a rule the developers pair program throughout the iteration. Factored into the estimate and the work of completing any development task is writing unit tests. Code check in practice is that code doesn't get checked in unless there are unit tests (that pass) and that the new code doesn't break the unit tests. The iteration's time frame is centered around the domain developers activities. What we call the current iteration is whatever iteration's work the domain developers are working on. The only duty of a domain developer during the iteration is coding that iteration's functionality.
| Domain Developers Cycle | |||||||||||
| Iteration 5 | Iteration 6 | Iteration 7 | |||||||||
| Develop Iteration 5 functionality | Develop Iteration 6 functionality | Develop Iteration 7 functionality | |||||||||
| Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 |
The SWAT team is responsible for all development duties that take place during an iteration that are not related to the new functionality. It is SWAT's job to take care of as many other development tasks as they can to ensure that the domain development staff can stay fully focussed on coding new functionality. This includes bug fixes from previous iterations' development, automating testing, streamlining the build process, performance tuning, upgrades, etc.
Note: Using a SWAT team in this way to fix outbound problems isn't something I've seen much since this project and is not something I would now recommend. These days I would urge people to ensure these kinds of problems are removed during the development iteration by the development team itself. -- MF
At the close of an iteration, the SWAT team branches the iteration's build and fixes only the most critical bugs to get our delivery build. Once the iteration closes, very little functionality is changed. It is risky having too much new functionality developed on the code branch. It's better having the domain development team work on any unfinished functionality in the next iteration and deliver it one iteration late.
| SWAT Developers Cycle | |||||||||||
| Iteration 5 | Iteration 6 | Iteration 7 | |||||||||
| Prepare build for Iteration 4 | Bug fixes, automate functional tests for Iteration 4. On-going non-functional development tasks | Prepare build for Iteration 5 | Bug fixes, automate functional tests for Iteration 5. On-going non-functional development tasks | Prepare build for Iteration 6 | Bug fixes, automate functional tests for Iteration 6. On-going non-functional development tasks | ||||||
| Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 |
| Quality Assurance Cycle | |||||||||||
| Iteration 5 | Iteration 6 | Iteration 7 | |||||||||
| Add Iteration 4 tests to regression test suite. Run regression tests | Add Iteration 5 tests to regression test suite. Run regression tests | Add Iteration 6 tests to regression test suite. Run regression tests | |||||||||
| Test Iteration 4 delivery build | Prepare Iteration 4 release notes | Deliver Iteration 4 | Close Iteration 5 | Test Iteration 5 delivery build | Prepare Iteration 5 release notes | Deliver Iteration 5 | Close Iteration 6 | Test Iteration 6 delivery build | Prepare Iteration 6 release notes | Deliver Iteration 6 | Close Iteration 7 |
| Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 | Week 1 | Week 2 | Week 3 | Week 4 |
Overall, we've found this XP approach to be fruitful.
One of the biggest benefits we've seen from this project is how well we've grown senior developers. It's been noticeable that many people who were junior developers a year ago are now strong technical leads. We attribute this to the fact that every developer is able to take responsibility from the start and that we encourage rotation between different parts of the app. The rotation has some frustrations, particularly for analysts who have to explain how things should work again to a new developer, but the result is that we now have several technical leaders who can lead development, which makes the technical leadership of the project much more stable, even as the project has lost several of the early senior deveopers.
We now face a new problem. In the next six months we have to deliver to our first live users while at the same time coninuing to evolve a generic product. We have to figure out how to divide our work between site specific delivery and the ongoing generic development. We've put together a process, and we'll see how it works out. We know our initial process won't be right, but we also know that we'll learn from experience - and we trust ourselves to succeed.
Cara Taber has been with ThoughtWorks for 3.5 years. During that time she has been a developer, designer, analyst and project manager. Before ThoughtWorks, she studied esoteric mathematics at a prestigious institution from which she received the title of Ph.D. drop-out. Hoodwinked into project management, she hopes that someday she will be allowed to code again. Her latest strategy is to work her way back into the good graces of the development staff by supplying the monthly iteration planning meeting with a margarita machine. She claims to live in Austin, TX, but most weeks can be found in ThoughtWorks' Chicago office.
Martin Fowler is the Chief Scientist for ThoughtWorks, an internet application development and consulting company. For a decade he was an independent consultant pioneering the use of objects in developing business information systems. He's worked with technologies including Smalltalk, C++, object and relational databases, and EJB with domains including leasing, payroll, derivatives trading and healthcare. He is particularly known for his work in patterns, the UML, lightweight methodologies, and refactoring. He is the author of Analysis Patterns, Refactoring, Planning Extreme Programming, and the award winning UML Distilled'
Here's a list of the major updates to this paper
© Martin Fowler | Privacy Policy | Disclosures