Agile methods are solidly in the mainstream, but that popularity hasn't been without its problems. Organizational leaders are complaining that they’re not getting the benefits they expected. This article presents a fluency model that will help you get the most out of agile ideas. Fluency evolves through four distinct zones, each with its own benefits, required proficiencies, and key metrics.

After my positive experiences with Extreme Programming in the 90s, I became curious about similar sounding approaches such as Scrum, Crystal, and DSDM. Digging into them, I distilled the common characteristics of these new methodologies: preferring adaptive planning to predictive planning, and treating people as more important to success than what process was used. In time these approaches gathered under the banner of agile software development, (and I revised the article) but I still find the perspective in this article a good way to understand the essence of agility.

It may not have started it all, but the manifesto gave the movement a name together with a dollop of initial energy. A decade later it still captures the essence of what agile methods are about.

A group of seventeen people got together in Snowbird, UT in February 2001 to talk about new styles of lightweight methods. One result of this is coining the word agile to represent a new breed of agile processes for software development. We also put together a Manifesto for Agile Software Development which describes the values and principles of these agile methods. Jim Highsmith and I wrote this article for Software Development magazine to further explain the manifesto.

For many that come briefly into contact with Extreme Programming, it seems that XP calls for the death of software design. Not just is much design activity ridiculed as “Big Up Front Design”, but such design techniques as the UML, flexible frameworks, and even patterns are de-emphasized or downright ignored. In fact XP involves a lot of design, but does it in a different way than established software processes. XP has rejuvenated the notion of evolutionary design with practices that allow evolution to become a viable design strategy. It also provides new challenges and skills as designers need to learn how to do a simple design, how to use refactoring to keep a design clean, and how to use patterns in an evolutionary style.

Over the last decade we've developed and refined a number of techniques that allow a database design to evolve as an application develops. This is a very important capability for agile methodologies. The techniques rely on applying continuous integration and automated refactoring to database development, together with a close collaboration between DBAs and application developers. The techniques work in both pre-production and released systems, in green field projects as well as legacy systems.

On the surface, the world of agile software development is bright, since it is now mainstream. But the reality is troubling, because much of what is done is faux-agile, disregarding agile's values and principles. The three main challenges we should focus on are: fighting the Agile Industrial Complex and its habit of imposing process upon teams, raising the importance of technical excellence, and organizing our teams around products (rather than projects). Despite the problems, the community's great strength is its ability to learn and adapt, tackling problems that we original manifesto authors didn't imagine.

Ryan Lockard (Agile Uprising) invited me to join Rebecca Wirfs-Brock for a podcast conversation on architecture on agile projects. Rebecca developed Responsibility-Driven Design, which was a big influence for me when I started my career. We talked about how we define architecture, the impact of tests on architecture, the role of domain models, what kind of documentation to prepare, and how much architecture needs to be done up-front.

When an organization takes on an agile mindset, enterprise architecture doesn't go away, but the role of enterprise architects changes. Enterprise Architects no longer make choices, but help others make the right choice and then radiate that information. Enterprise Architects still need to form a vision, but then need to build bridges between teams to build communities of learning. These will allow teams to explore new approaches and learn from each other, with Enterprise Architects as partners in that growth.

10 years after we wrote the agile manifesto, we authors were invited to a special event during the Agile 2011 conference to celebrate the anniversary. Fifteen of the seventeen authors came along and we ran a park bench panel answering questions and comments from the audience. I think we were all surprised by how good it was to meet again and how easily we fell back into a comfortable collaboration and discussion. Our discussion included some background to the writing of the manifesto, looking at things over the last decade that we were pleased and unhappy with, future developments in agile, and the relationship between agile and lean.

There isn't a simple dichotomy of remote versus co-located work, instead there are several patterns of distribution for teams each of which has different trade-offs and effective techniques suitable for them. While it's impossible to determine conclusive evidence, my sense is that most groups are more productive working in a co-located manner. But you can build a more productive team by using a distributed working model, because it gives you access to a wider talent pool.

This is the second part of my keynote at OOP 2014 in Munich and is a tricky talk to describe. Usually I like a title and abstract to describe what the talk is about - but this talk is a journey, and I don't want to tell you where I'm going, but instead to explore the ground with me. I will say that it starts with my biggest problem with most adoption of agile software development - the nature of the interaction between users, analysts, and programmers. It goes on to explore these roles, raising questions about the relationship of programmers to users, our responsibilities to them, and finally the Two Great Challenges that I think programmers need to face up to.

Neal Ford and I gave a talk at USI in Paris (2010) on some aspects of why agile works (as opposed to how). This probes at some of the core forces that makes agile effective, rather than looking at techniques. In particular we look at role of communication and feedback and how they interplay in agile environments.

Why we need methods that assume software should be soft and open to change.

Interview with Paulo Caroli and me at Agile Brazil

Continuous Integration is a software development practice where each member of a team merges their changes into a codebase together with their colleagues changes at least daily. Each of these integrations is verified by an automated build (including test) to detect integration errors as quickly as possible. Teams find that this approach reduces the risk of delivery delays, reduces the effort of integration, and enables practices that foster a healthy codebase for rapid enhancement with new features.

Daily stand-up meetings have become a common ritual of many teams, especially in Agile software development. However, there are many subtle details that distinguish effective stand-ups and a waste of time.

SD Times interview on 10 years since the agile manifesto

A keynote for QCon 2007 that I did with my colleague Dan North. We both see the gap between developers and their customers/users as the biggest problem in software development. (We'd call it a chasm, but that word is so overused.) Here we talk about this gap, why it's important, and what we need to do to cross it. In particular we argue that the traditional role of an intermediary Business Analyst acts as a ferry, while what we really need is a bridge that enables directly contact between developers and their customers (and analysts can build and maintain that bridge). This is one of my favorite joint keynotes, both because I think the topic is so important, and because Dan is such stimulating co-speaker.

In February 2001 a group of seventeen software pundits got together in Snowbird UT to discuss the growing field of what used to be called lightweight methods. We decide to use the term agile to describe this new breed of agile methods. We also wrote the Manifesto for Agile Software Development , setting out the values and principles of these agile processes. I was one of these self-elected visionaries and have since come across many questions about this group's origins and the subsequent founding of the agile alliance. This is my recollection of those events.

For the last four years Thoughtworks has operated a lab in Bangalore India to support our software development projects in North America and Europe. Traditional approaches to offshore development are based on plan-driven methodologies, but we are very firmly in the agile camp. Here I discuss our experiences and lessons learned in doing offshore agile development. So far we've discovered that we can make it work, although the benefits are still open to debate. (Although this article was last updated in 2006, I our visited our offshore work in 2011 and found the lessons to still be relevant and thus the article did not need a further significant revision.)

When I went to Snowbird in 2001 for the meeting that led to the Manifesto, Jim interviewed me for a book he was working on. It provides a snapshot on my thinking on extreme programming and this thing that a few days later we called agile software development.

As XP and other Agile methods gain popularity, questions are beginning to surface regarding how to scale XP beyond teams of 10-12 people. In mid February 2003 a workshop dedicated to the subject was held in Banff Alberta Canada. In this article we report on the keynote speeches from Ken Schwaber, and Martin Fowler, as well as other leading practitioners.

Refactoring has grown into a well-known technique, and most software development teams at least claim to be doing refactoring regularly. Many teams, however, don't appreciate the different workflows that refactoring can be used in, and thus miss opportunities to effectively incorporate refactoring into their development activities. In this deck I explore various different workflows. I hope it will encourage teams to integrate refactoring more deeply into their work, resulting in a better designed code-bases that will make it quicker and easier to add new features.

Over the last decade or so, Refactoring has become a widely used technique to keep a high internal quality for a codebase. However most teams don't make enough use of refactoring because they aren't aware of the various workflows in which you can use it. In this keynote talk from OOP 2014 in Munich, I explore some of these workflows: such as Litter-Pickup Refactoring, Comprehension Refactoring, and Preparatory Refactoring. I also remind people why common justifications for refactoring will sabotage your best efforts. (This talk also has a treatment as an infodeck.)

The Agile Uprising podcast has been doing a series of interviews with the authors of the Agile Manifesto. This is my turn in the interview seat. I don't remember much about the Snowbird workshop itself, but I was able to describe a bit about the context leading up to the manifesto.

Scattered impressions from my recent trip to Australia for the Agile Australia conference.

Should there be a certification program for agile methods?

One of the most common questions I see about agile projects is how they deal with handover to another team. If you have a development team that leaves and hands over support to a support team, how do they cope when agile projects tend to produce much less documentation than plan-driven processes?

According to the current board of the Agile Alliance, agile methods have “crossed the chasm” , which I think means they are becoming more widespread. While this has its advantages, it also brings problems. As a methodology or design approach becomes fashionable, then we see a lot people using it, or teaching it, who are focusing on the fashion rather than the real details. This can lead to reports of things done in agile's name which are a polar opposite to the principles of movement's founders.

The meeting at Snowbird Utah in 2001 that decided to use the word 'Agile' and began the 'Manifesto for Agile Software Development'.

I'm thinking of using agile software development - but should I use Lean software development instead?

Last week I attended the Agile 2010 conference in Orlando. Agile 20xx is the major US agile-oriented conference whose roots go back to XP Universe and the Agile Development Conference. I've not been a regular attender of the main agile conferences, but I did go last year as well. Rather than make an attempt at a consolidated description, here are a few scattered impressions.

C3 was the short name of the Chrysler Comprehensive Compensation project, a payroll project at Chrysler which has since become famous as the 'birth project' of Extreme Programming.

One of the common elements of agile methods is that they raise programming to a central role in software development - one much greater than the software engineering community usually does. Part of this is classifying the code as a major, if not the primary documentation of a software system.

Here's a common misconception about agile methods. It centers on the way user stories are created and flow through the development activity. The misconception is that the product owner (or business analysts) creates user stories and then put them in front of developers to implement. The notion is that this is a flow from product owner to development, with the product owner responsible for determining what needs to be done and the developers how to do it.

Daniel Terhorst-North's recent blog post on software craftsmanship has unleashed a lot of blog discussions (which I summarize below, if you're interested). There's a lot in there, but one of his themes particularly resonated with me, hence this post.

When someone is looking at what makes up a top-class enterprise software developer, often the conversation may turn to knowledge of frameworks and languages, or perhaps the ability to understand complicated algorithms and data structures. For me, one of the most important traits in a programmer, or indeed in a development team, is something that I'll call Customer Affinity. This is the interest and closeness that the developers have in the business problem that the software is addressing, and in the people who live in that business world.

A few years ago I was talking with a client who told me something he didn't like about the agile approach we were using: “it doesn't feel right to have these difficulties this early in the project”. Contrary to his reaction, in my mind this early pain is one of the great benefits of an agile or indeed any iterative development process.

Extreme Programming (XP) is a software development methodology developed primarily by Kent Beck. XP was one of the first agile methods, indeed XP was the dominant agile method in the late 90s and early 00s before Scrum became dominant as the noughties passed. Many people (including myself) consider XP to be the primary catalyst that got attention to agile methods, and superior to Scrum as a base for starting out in agile development.

A common, perhaps dominant, practice of agile methods is to develop a list of features (often called stories) for the software that's being built. These features are tracked with index cards, work queues, burndown charts, backlogs, or whatever your tool of choice is.

Many people belive that you can't do a fixed price contract in an agile project. Since the whole point of an agile process is that you cannot predict the future, this isn't an unreasonable supposition. However this doesn't mean you can't come up with a fixed price agile contract, what it really means is that you can't come up with a fixed scope contract.

Many companies like the idea of writing a contract that fixes scope and price because they think it lowers their risk. The mirage says that their financial obligation is fixed at the price of the deal. If they don't get satisfactory software, then it won't cost them.

There's a mess I've heard about with quite a few projects recently. It works out like this:

• They want to use an agile process, and pick Scrum
• They adopt the Scrum practices, and maybe even the principles
• After a while progress is slow because the code base is a mess

One of my favorite soundbites is: if it hurts, do it more often. It has the happy property of seeming nonsensical on the surface, but yielding some valuable meaning when you dig deeper

Can average developers use agile methods?

A common question is whether large projects can be done with agile techniques. After all many agile approaches are designed for smaller projects and the heavyweight ideas that they resist are more needed on bigger projects.

A bunch of common misconceptions about Pair Programming.

One of the most difficult things for many people to understand about agile methods is the people orientation of agile. Those who are interested in agile processes all agree that process is a second-order effect on project success. The first value of the agile manifesto is that Individuals and Interactions are more valuable than Process and Tools.

All agile methods stress the importance of direct interaction between the developers of a system and customers who are its eventual beneficiaries. The agile manifesto said “Business people and developers must work together daily throughout the project”, which is there to stress the high frequency of interaction. Extreme Programming stresses this through its practice of OnsiteCustomer.

The Retrospective Prime Directive is an important part of retrospective practice, and has been in integral part of retrospective thinking since Norm Kerth first launched the practice. Recently I read Pat Kua's new Retrospective Handbook, which is based on the extensive experience with retros that Pat has had as a tech lead at Thoughtworks. I find Pat's advice for the Prime Directive revolting, yet have to say he's almost certainly correct.

I often run into a complaint that agile methods don't have a rigorous definition. The complainer may talk about how this means that you can't tell if a particular team is using an agile method or not. They may also say that this makes it hard to teach people how to do agile methods - what's the curriculum?

To some degree I do feel the pain of this complaint - but I accept there is no cure. This lack of rigorousness is part of the defining nature of agile methods, part of its core philosophy.

For nth, and I'm sure not last time, I'm sliding into a conversation about defining practices, labeling some of them as “best”, and probably the C-word (certification). It's a familiar discussion, and although we've barely started it, I can predict much of where it will go. It's driven by a perfectly reasonable desire to identify who are the better software developers, and how existing developers can improve their abilities.

Self-Testing Code is the name I used in Refactoring to refer to the practice of writing comprehensive automated tests in conjunction with the functional software. When done well this allows you to invoke a single command that executes the tests - and you are confident that these tests will illuminate any bugs hiding in your code.

From time to time people question whether a particular specialty can be used incremental way: “You can't do (security | user interface design | databases | internationalization | * ) with an agile project because this aspect has to be done up front.”

A common thing you find in agile projects is that the development team sits in a single open team room. It was advocated early on in Extreme Programming and called out as one of primary practices in the second edition. Agilists favor a open team room as it promotes lots of informal and deep communication between people on the team.

Timeboxed Iterations are a way to divide and schedule up the work on a project, particularly associated with agile software projects. The team breaks down the visible features of the software into User Stories, and breaks down time into fixed periods (e.g. one week), called iterations. They then schedule the stories by allocating them to the iterations. Stories are roughly estimated so that the team can figure out how many stories fit in an iteration.

User Stories are chunks of desired behavior of a software system. They are widely used in agile software approaches to divide up a large amount of functionality into smaller pieces for planning purposes. You also hear the same concept referred to as a feature, but the term “story” or “user story” has become prevalent in agile circles these days.