The State of Agile Software in 2018

This is a transcript of my talk at Agile Australia, Melbourne 2018. I did the talk off-the-cuff, with just the outline to guide me. I have edited the transcript to make the text read less incoherently, while following the track of the talk. You can find a video of the talk at InfoQ.

How many people have seen me talk here before at Agile Australia?. And you came back? Wow, I'm impressed. If you've seen me talk before at Agile Australia, or indeed at any conference, you know that pretty much every time I give a talk, I call it “Software Design in the 21st Century”, or something like that: because it's a vague title and I can talk about whatever I like. And I was going to do that again here, but I decided that I'm going to do something specific: to talk about where we are: the agile community in 2018. I decided I'd do not something that's a particularly deeply planned talk with lots of slides and clever diagrams and beautiful transitions - but just me yakking. Which I've done before, but it's been a while, so we'll see how we go.

As we look at the state of agile: in many ways, on a kind of surface level, things are looking very good. I mean, look at the size of this crowd for instance. We're huge, we're fitting in this big conference place - actually not fitting in very well because it was pretty crowded out there. You go to all sorts of places and you see agile scattered around. Somebody sent me a Harvard Business Review cover with “agile” on it. I mean, it's all over the place. That's a big shift from 10 years ago here, or even longer when we were in that ski place at Snowbird talking about what the hell we should call ourselves. And it sounds like success, but you talk to a lot of old-time agilists, people who were doing it before it was called “agile” back in the late 90s, and there's actually a lot of disquiet, a lot of disappointment, a lot of unhappiness in the air.

That's actually not unusual because it's been like that pretty much the whole time as far as I can tell1. And that's actually a good thing, because that dissatisfaction is a sign of wanting to improve. But it does lead to that sense of: “why are we struggling”?

What's the current challenge that we're having to face? Back 10 years ago, the challenge was people taking agile at all seriously. I remember going into one of the big Thoughtworks clients in Australia. They wanted me to give them the usual “Martin Fowler come and give us a talk” kind of routine. Somebody from the clients said, “Yeah, we want you to talk about whatever you like, but don't say anything about the agile stuff.” Which was a bit of an alarming thing in the mid-2000s when I was talking a lot about agile, but that was the sense then. A sense that this was something kind-of bad and you don't want to talk about it.

Now agile is everywhere, it's popular, but there's been an important shift. It was summed up quite nicely by a colleague of mine who said, “In the old days when we talked about doing agile, there was always this pushback right from the beginning from a client, and that would bring out some important conversations that we would have. Now, they say, 'Oh, yeah, we're doing agile already', but you go in there and you suddenly find there's some very big differences to what we expect to be doing. As Thoughtworks, we like to think we're very deeply steeped in agile notions, and yet we're going to a company that says, “Yeah, we're doing agile, it's no problem,” and we find a very different world to what we expect.

Our challenge at the moment isn't making agile a thing that people want to do, it's dealing with what I call faux-agile: agile that's just the name, but none of the practices and values in place. Ron Jeffries often refers to it as “Dark Agile”, or specifically “Dark Scrum“. This is actually even worse than just pretending to do agile, it's actively using the name “agile” against the basic principles of what we were trying to do, when we talked about doing this kind of work in the late 90s and at Snowbird.

So that's our current battle. It's not about getting agile respectable enough to have a crowd like this come to a conference like this, it's realizing that a lot of what people are doing and calling agile, just isn't. We have to recognize that and fight against it because some people have said, “Oh, we're going to 'post-agile', we've got to come up with some new word,” - but that doesn't help the fundamental problem. It's the values and principles that count and we have to address and keep pushing those forwards and we might as well use the same label, but we've got to let people know what it really stands for. 2

If that's the broad level of the problem, how do we focus on particular things? I want to focus on three main problem areas that I think are the ones that I would like to highlight. The ones that I think are most worthy of our attention.

The first one of these is what I would call the Agile Industrial Complex. To be fair, I'm part of it, right? I'm standing on the stage here talking about agile, indeed we're all part of it to some degree, many of us are parts of some kind of agile consulting firms, probably with agile in the title. But a lot of what is being pushed is being pushed in a way that, as I said, goes really against a lot of our precepts.

In particular, one of the really central things that impressed me very much about the early agile advocates was this realization that people are operating at the best when they choose how they want to work.

When we talked about the Agile Manifesto and laid out the four value statements, with most of those value statements, we didn't care very much about what order they came in. But we did have an opinion about the first one: which is “Individuals and Interactions over Processes and Tools”. To me that crystallized a very important part of what agile thinking is about. If you want to succeed in doing software development, you need to find good people. You need to find good people that work together at a human level, so they can collaborate effectively. The choice of what tools they use or what process they should follow is second order. I thought that's a very important statement to come from what was basically a gathering of process weenies. I mean we were all process guys to some extent or other, and yet we were acknowledging that what we were talking about was actually of secondary importance. What matters is that the team chooses its own path.

It goes further than that for the team should not just choose the process that they follow, but they should be actively encouraged to continue to evolve it and change it as they go. One of the things about any kind of agile processes or agile method is that it is inherently very slippery. It changes week to week, month to month. One of the quotes that I used to flash around people was “if you're doing Extreme Programming the same way as you were doing it a year ago, you're no longer doing Extreme Programming”. Because if you don't take charge and you don't alter things to fit your circumstance, then you are missing the key part of it. There are various rituals and things that we can set up to make this work, and retrospectives are clearly a technique that lots of people find to be really very central. In fact, I think Ron Jeffries joked that Allistair Cockburn's approach to agile was “come together in peace and harmony, deliver software every week, and do a retro every time to figure out how you can improve”. The retrospective is really such a central part of the practice.

Now, it actually doesn't matter whether you actually have a formal retrospective. It doesn't matter whether you have four or five labels of things on your retro board, or exactly how you do the retro. What does matter is the notion of thinking about what we're doing and how we can do better, and it is the team that's doing the work that does this, that is the central thing.

This is a reaction against the whole Frederick Taylor, separate process people. How many people here know the story of Frederick Taylor and his approach? [a few hands go up] How many people have come across the name Frederick Taylor or even heard of it? A few more. A lot more of you should raise your hands. He's probably one of the most important figures in the history of the 20th century in terms of how he's actually affected people's day to day lives. He was from the late 19th century, in America, and he was very interested in trying to make people more efficient in the industrial workplaces that were developing at that time. His view of the average worker was that they were lazy, venal, and stupid. Therefore you didn't want them to decide how they should make a particular piece of machinery, but somebody else, somebody who was a more intelligent and educated, they should figure out exactly the best way to do it. Even going down to: do I do this and then that or do I do that and then this. This is a very scripted sense of motion and movement. The whole time and motion industry came out of that. At the heart of this notion was that the people who are doing the work should not decide how to do it. It should be a separate group of planners who does this, and that strongly affected manufacturing and factory work through much of the early 20th century.

And it affected the software industry as well - people said, “we need software process experts to figure out how to do things, and then programmers just fit into the slots”. But interestingly, just as software people were talking about how we need to kind of follow this very Taylorist notion as the future of software development, (I heard people saying that back in the 80s and 90s), the manufacturing world was moving away from it. The whole notion of what was going on in a lot in manufacturing places was the people doing the work need to have much more of a say in this because they actually see what's happening.

The agile movement was part of trying to push that, to try to say, “The teams involved in doing the work should decide how it gets done,” because let's face it, we're talking about software developers here. People who are well paid, well educated, hopefully, well-motivated people, and so they should figure out what in their particular case is necessary.

“Particular case” becomes important because different kinds of software are different. I've lived most of my life in enterprise applications: database-backed, GUI/web frontend kind of world. That's what most of the people I know do but that's very different to designing telephone switches or designing embedded software. Even within the world that I'm relatively familiar with, different teams have different kinds of situations, we have different legacy environments to coordinate with, and we have different dynamics between individuals on a team. With so many differences, how can we say there's one way that's going to work for everybody? We can't. And yet what I'm hearing so much is the Agile Industrial Complex imposing methods upon people, and that to me is an absolute travesty.

I was gonna say “tragedy”, but I think “travesty” is the better word because in the end there is no one-size-fits-all in software development. Even the agile advocates wouldn't say that agile is necessarily the best thing to use everywhere. The point is, the team doing work decides how to do it. That is a fundamental agile principle. That even means if the team doesn't want to work in an agile way, then agile probably isn't appropriate in that context, and [not using agile] is the most agile way they can do things in some kind of strangely twisted world of logic.

So that's the first problem: the Agile Industrial Complex and this imposition of one-best-way of doing things. That's something we must fight against.

The second problem is the lack of recognition of the importance of technical excellence to what we do. A lot of agile conferences I go to don't tend to talk very much about the actual techniques of writing software. How many people here are software developers by the way? [a few people raise hands] Smattering, but very much a minority. The same problem was true at the Agile Alliance's main conference for quite a while, but they realized that they were getting a lot of people who were involved in the project management side and things of that kind, but not very many people who are the technical people who actually did the work. And that's actually quite tragic. It led to an even more tragic thing of some software developers saying, “Oh, we need to create a whole new world for ourselves. The software craftsmanship movement where we can go away, get away from all of these business experts and project managers and business analysts, and just talk about our technical stuff.” But that's a terrible thing because the whole point of agile is to combine across these different areas. The lowliest, juniorest programmer tapping away on JavaScript should be connected to people who are out there thinking about the business issues and business strategies of the group that they're working with.

And I'll say a bit more about that in my third point, but that means we've got to pay attention to these technical skills, and we have to think about how do we nurture those, how do they grow, how do we make them important? I've spent the last couple of years as my primary writing exercise coming up with a new edition of the book about refactoring. How many people here have heard about refactoring? [a lot of hands] Good. How many people could accurately describe it to people? [less hands] Rather less. It's a very core technique to the whole agile way of thinking because it fits in with the whole way in which we can build software in a way that it can change easily. When I summarize agile to people, I usually say there's two main pieces to it. One, I've already talked about, the primacy of the team, and the team's choices of how they do things, but the other is our ability to change rapidly, to be able to deal with change easily.

I always loved Mary Poppendieck's phrase, “A late change in requirements is a competitive advantage.” But to do that you need software that's designed in a way that's able to react to that kind of change. Refactoring is central to this because refactoring is a disciplined way of making changes. Now it's a horrible thing when somebody tweets something along the lines of: “I'm refactoring our software at the moment, so it's going to be broken for the next two weeks.” Bzzz - that's not refactoring. Refactoring is small changes, each of which keeps everything working. It doesn't change the observable behavior of the software, that's its definition. And I should know because I was the one who got to define it.

Refactoring is lots of small changes, none of which change the observable part of the software, but all of which change its internal structure. Usually (you refactor) because you want to make some new feature, and the current internal structure doesn't fit very well for that new feature. So you change the structure to make the new feature easy to fit in, all the time refactoring and not breaking anything, and then you can put it in. Kent Beck summarizes this. “When you want to make a change, first, make the change easy. (Warning, this may be hard.) Then make the easy change”. That's a very crucial way to use refactoring but it goes beyond that because refactoring is a constant thing. You're looking at some code and you're trying to say, “What does this do? I don't really understand what this does. Let me think about this. Let's burrow in. Ah, now I understand what this code does.” Then, before you move on, in the words of Ward Cunningham: “You take the understanding out of your head and you put it into the code,”: by restructuring it, by renaming it. Naming is a really important part of all of this. So that the next time you or somebody else comes through that same piece of code, you don't have to go through that puzzle exercise.

for each desired change, make the change easy (warning: this may be hard), then make the easy change

-- Kent Beck

Why is this important? Because if you want to make changes, you want to add things quickly, you've got to quickly understand which parts of the program matter, what do they do, and how do I work so that I can quickly make that change. This also burrows up into modularity. If I've got a well-modularized piece of software, instead of having to understand the whole thing, I can just understand part of it. Technical excellence is about being able to build that kind of adaptive software.

Then a self-reinforcing principle comes in. Once you realize I can change software quickly to add new things, I don't try to make the software do everything I want it to do right at the beginning. Because I don't need to. Because I'll be able to change things later on. This is the principle of Yagni - “You aren't gonna need it”. Don't add features to the software until you need them because if you do, it bloats the software and makes it harder to understand. But that strategy is totally hopeless without good refactoring techniques. And then refactoring relies on testing, and refactoring also relies on continuous integration, and together with continuous integration, you have the practice of continuous delivery and the notion that we're gonna be able to release the software very, very frequently. And in some organizations, we actually do release the software very, very frequently.

Now, there's a notion out there that says that rapid release of software can only be done if you're prepared to put up with lots and lots of errors. And if you want software to be reliable, you do things in much more slow, deliberate way. Wrong. And we're seeing more and more evidence pile up, that this is very, very wrong. My favorite book of the year so far, I think it's going to be the best book of the year is “Accelerate” by Nicole Forsgren, Jez Humble and Gene Kim. (And I say that with great sadness because my book's going to come out this year, but I'm hoping for number two) In this book they, through looking at lots of surveys of organizations, show how different practices affect things. One of the things that they demonstrate is releasing many times a day and low defect rates go together. Because you can't release many times a day unless you figure out how to get your defect rates down. That requires the kinds of practices we talk about: automated testing, refactoring, yagni - all of this stuff flows together. The most important thing about Extreme Programming is that the individual practices of Extreme Programming reinforce each other.

The third thing that I want to stress is the importance of getting rid of software projects as a notion. Instead we want to switch to a product-oriented view of the world where instead of projects that you spin up, run for a while and then stop; you instead say, “Let's focus on things that are much more long-lasting and organize a product team around that.” Another way of thinking about it is: what are the business capabilities that your organization has, and then organize the teams around those. These business capabilities will be long-lasting and will necessarily mean combining together technical people and the people who are on the business side of things into the same team.

A popular notion at the moment is the Amazon Two Pizza Team - people talk about this all the time: “Organize yourself into two-pizza teams.” Maybe with a little side joke about how American pizzas are really, really big, so they can be bigger teams than you might think. But when they give that description, they often miss off something that's clear whenever I hear Amazon people talk about this; which is that each of those teams needs to connect - right the way through to the customer. Each team is focused on some aspect of the customer's experience, some aspect of what Kathy Sierra calls making the customer kick ass at what they do. This alters our notion of what that small team does because if that small team is focused on some piece of customer experience, some way of making the customer do what they do better, then that tells us how we should draw lines between our small teams. Now, it's not always easy to do this, but it should, I think be the driving notion.

But again, we often see it violated. I went to one supposedly agile organization. I was chatting with a development team. There was about half a dozen programmers. There were only four users of the software that they were writing, senior planners in retail planning. Six developers, four users. They had never spoken to each other. They weren't allowed to. Instead, there was a Scrum product owner who was managing the conversation between them. Well, actually, there had been four product owners over the course of a year. I mean, what nightmare world of agile is this that they can dare use the word “agile” to describe it? When [at Snowbird] we were talking about names for agile software development, Kent Beck suggested “conversational”, because there should be an ongoing conversation between software developers, business people, and anybody involved in making that customer experience better.

If I was on that team as a developer, I'd want to be on first-name terms with all of the users. I would want to be talking. I would want to watch what they do, I'd want to understand how they think about making their customers happier and see that whole flow. So we need to think about that. That's my third challenge.

So my three things, that we should face as challenges:

• Get rid of the Agile Industrial Complex and the idea of imposing stuff on teams. Let teams work out the way they should work themselves.
• Raise the importance of technical excellence, and never forget that when writing software, the technology side is really vital, and
• organize around products.

So that all sounds a little bit negative because I've talked about the problems of this current state of agile. I have two minutes and 45 seconds left to offer one reason why I'm not too depressed about the situation. It comes from probably my favorite thing about the whole Agile Manifesto. This didn't occur in Snowbird when we wrote it, but occurred about six months later in Tampa, Florida at OOPSLA where most of the people who wrote the manifesto were together with a bunch of other interested people. At this point, the Agile Manifesto had taken off in a way that we could never have imagined. Suddenly it was clear there was this huge opportunity for doing interesting things, and many people said, “We need to set up a real organization here.”

One of the questions was: “Should the original 17 people that wrote the manifesto take a special place in this ongoing effort?” The thing I'm proud of, that we the 17 authors did, is that we said “no”. We wrote the manifesto. We did a nice piece of work, we'll be part of what goes on in the future, but we have no special role in that future. That role has to grow forwards. We said “new people will come in and do great things”, which is indeed what happened.

One of the ways where that was important was about a big problem with the people who wrote the manifesto, particularly looking at it now with 2018 glasses. 17 people: 17 white, middle-aged guys. 3 Not a thread of diversity there, but because we let go, the agile world could incorporate many people from all sorts of backgrounds who took part. Mary Poppendieck was one of the big leaders of the early agile alliance efforts. Rebecca Parsons, my boss at Thoughtworks, was the chair of agile alliance for a long time. I'd go to an agile conference, and I tend to see much more women than I do in other kinds of software conferences. That's a good thing. Certainly, it wasn't the work of us at the beginning. We weren't even thinking about that. But the point was: because we let go, we ended up with a community that could develop things themselves. That could tackle challenges that we hadn't even imagined and work on them. And it's that constant learning and growing and changing as a community, which is our greatest strength.

I spoke not that long ago to someone who is a latecomer to the agile world, who said that they found a degree of welcoming and openness that wasn't true of many other groups. That made me really, really happy, because as long as we can be that flexible, as long as we can always change, then whatever challenges we face, I think we have a future. Thank you.

---