I hated carrots when I was growing up, hating the smell and texture of the things. But after I left home and started to cook for myself I started to like them. Nothing changed about the carrots, nor did my taste buds get a radical overhaul, the difference was in the cooking. My mother, like so many English people of her generation, wasn't a great cook - particularly of vegetables. Her approach was to boil carrots for twenty minutes or more. I since learned that if you cook them properly, carrots are a totally different experience.
This isn't a site about cooking, but about software development. But I find that often a technique or tool is like the poor carrot - blamed for being awful when the real problem is that the technique is being done incorrectly.
Let's take a couple of examples. Several friends of mine commented how stored procedures were a disaster because they weren't kept in version control (instead they had names like GetCust01, GetCust02, GetCust02B etc). That's not a problem with stored procedures, that's a problem with people using bad practices with them. Similarly a criticism that TDD led to a brittle design on further questioning led to the discovery that the team in question hadn't done any refactoring - and refactoring is a critical step in TDD.
Both of these are boiled carrots - useful tools that have been misused. I've seen teams gain value out of both stored procedures and TDD. If we discard them without taking their usage into account we lose useful tools from our toolbox.
Not every failure of technique is a boiled carrot. I'm reliably informed that there's no way to cook squirrels so they are appetizing to anyone who isn't desperate (which is a shame considering what they've been doing to our garden this spring). If I come across a team working on code in a shared folder, without any version control, there's no way to cook that technique which isn't similarly appalling.
So when we hear of techniques failing, we need to ask a lot more questions.
- Was it the technique itself that had problems, or was some other thing being missed out. Does the technique have an influence on this? (Version control is a separate thing to stored procedures, but it can be harder to use version control with stored procedures due to nature of tools involved.)
- Was the technique used in a context that wasn't suitable for it? (Don't use wide-scale manual refactoring when you don't have tests.) Remember that software development is a very human activity, often techniques aren't suitable for a context because of culture and personality.
- Were important pieces missed out of the technique?
- Were people focused on outward signs that didn't correspond to the reality? This kind of thing is what Steve McConnell called Cargo Cult Software Engineering.
- Is the technique something that works at some scale, but is being used outside its zone of applicability? It's worth remembering Paracelsus's principle that the difference between a medicine and a poison is the dosage. Testing a system through the UI is useful with a few scenarios, but if you use it as your main testing approach you'll end up with slow and brittle tests which will either slow you down or get ignored.
An interesting aspect of this is whether certain techniques are fragile; i.e are they hard to apply correctly and thus more prone to a faulty application? If it's hard to use a technique properly, that's a reasonable limitation on the technique, reducing the context when it can be used. Some delicate foods have to left to a master chef. That doesn't make them a bad technique, but it does reduce their applicability to more skillful teams. I'd argue this is the fundamental problem with late integration of components. While some teams can develop components to careful specifications that can integrate together late in the day, in practice few teams are able to pull that off, and late integration ends up being like Fugu.
While we need to be wary of boiled carrots, we also need to bear in mind we also get the situation that I've observed as "no methodology has ever failed". With any failure (assuming you can know WhatIsFailure) you can find some variation from the methodology - which leads its defenders to say it wasn't followed and thus didn't fail. There's a genuine tension here, one that can't be resolved without a deep understanding of the deeper principles underlying a technique. The real point is that such techniques aren't rigorously describable, just as Spaghetti Carbonara does not have one precise recipe that can be followed without thinking. In the end what really counts is the dish, the technique to prepare it can inspire and guide a good chef but cannot guarantee success to someone without a certain degree of skill.
Like any profession, we can advance faster if we learn from each others' experiences. Reports of people using techniques and tools are important so that we can judge what to try in our own work. However a simple label usually isn't enough to go on. We are as unable to measure compliance with the proper use of a technique as we are unable to measure their successfulness. The important thing to do is whenever you hear of a technique failing - always dig deeper to see if the carrot's been in the pot too long. Otherwise we risk missing out on something worthwhile.
I originally wrote about this topic under the heading "Faulty Technique Dichotomy", but now feel the boiled carrot metaphor is more memorable.
I liked Ron Jeffries parable for a similar phenomenon "We tried baseball and it didn't work".
AcknowledgementsHenrique Souza, Jeantine Mankelow, Karl Brown, Kief Morris, Kyle Hodgson, Matteo Vaccari, Patrick Kua, Rebecca Parsons, Ricardo Cavalcanti, Roni Greenwood, Sriram Narayan, and Steven Lowe discussed drafts of this post on our internal mailing list.
Bimodal IT is the flawed notion that software systems should be divided into these two distinct categories for management and control.
- Front Office systems should be optimized for rapid feature development. These systems of engagement need to react rapidly to changing customer needs and business opportunities. Defects should be tolerated as the necessary cost for this rapid development cycle.
- Back Office systems should be optimized for reliability. As systems of record, it's important that you don't get defects that damage the enterprise. Consequently you slow down the rate of change.
When I first heard about this approach, I was pleased - thinking that these august organizations had come to same conclusion that I had with my UtilityVsStrategicDichotomy, but as I read further I realized that Bimodal IT was a different animal. And worse I think that Bimodal IT is really a path down the wrong direction.
My first problem is that the separation is based on software systems rather than business activity. If you want to rapidly cycle new ideas, you are going to need to modify the back office systems of record just as frequently as the front office systems of engagement. You can't come up with clever pricing plans without modifying the systems of record that support them.
My second issue is that the bimodal idea is founded on the TradableQualityHypothesis, the idea that quality is something you trade-off for speed. It's a common notion, but a false one. One of the striking things that we learned at ThoughtWorks when we started using agile approaches for rapid feature delivery is that we also saw a dramatic decline in production defects. It's not uncommon to see us go live with an order of magnitude fewer defects than is usual for our clients, even in their systems of record. The key point is that high quality (and low defects) are a crucial enabler for rapid cycle-time. By not paying attention to quality, people following a bimodal approach will actually end up slowing down their pace of innovation in their "systems of engagement".
So my advice here that it is wise to use different management approaches to different kinds of software projects, but don't make this distinction based on the bimodal approach. Instead take a BusinessCapabilityCentric approach, and look at whether your business capabilities are utility or strategic.
Sriram Narayan's book - Agile IT Organization Design - looks at this kind of problem in much more depth.
Jez Humble provides a worthwhile critique of Bimodal IT
Simon Wardley prefers a three-level model of Pioneers, Settlers, and Town Planners.
1: Sadly all their substantial material is available to subscribers only.
AcknowledgementsBrian Oxley, Dave Elliman, Jonny LeRoy, Ken McCormack, Mark Taylor, Patrick Kua, Paulo Caroli, and Praful J Todkar discussed drafts of this post on our internal mailing list
Serverless architectures are internet based systems where the application development does not use the usual server process. Instead they rely solely on a combination of third-party services, client-side logic, and service hosted remote procedure calls (FaaS).
Serverless applications often make extensive use of third party services to accomplish tasks that are traditionally taken care of by servers. These services could be rich ecosystems of services that interoperate, such as Amazon AWS and Azure, or they could be a single service that attempt to provide turnkey set of capabilities such a Parse or Firebase. The abstraction provided by these services could be infrastructural (such as message queues, databases, edge caching…) or higher-level (federated identity, role and capability management, search…).
The most substantive part of a server based web application is the work that happens between the controller and the infrastructure; the business logic. A long lived server hosts the code that implements this logic and performs the required processing for as long as the application stays alive. In serverless applications, custom code components have a lifecycle that is much shorter, closer to the timeline of a single HTTP request/response cycle. The code activates when a request arrives, processes the request and becomes dormant as soon as the activity dies down. This code often lives in a managed environment such as Amazon Lambda, Azure Function or Google Cloud Functions, which takes care of the lifecycle management and scaling of the code. (This style of organizing software is sometimes called “Function as a Service” - FaaS.) The short per-request lifecycle offers itself to a per-request pricing model too, which results in significant cost savings for some teams. 
A new style, a new set of tradeoffs
All design is about tradeoffs. There are some distinct advantages to applications built in this style and certainly some problems too.
The most commonly asserted benefit is cost. In systems with bursty traffic patterns, the cost of having a beefy server run cold the majority of the time in order to accomodate the bursts is both wasteful and expensive. The demand based pricing model of cloud based infrastructure services can offer significant reduction in costs for teams that have to deal with this type of traffic. In addition, in a traditional server based application the scalability of the application and all associated infrastructural components are the responsibility of the development team. This is often harder than using services that scale transparently behind the simple abstraction of an API available over a URL. Thus teams often find that serverless applications can be made to scale more easily.
On the other hand, there are some new costs. The conceptual overhead of splitting a single application into something that woven from a fabric of services is significant and increases with the number and variety of services used. Local development and unit testing is also harder when applications have significant parts implemented and running in external services. Teams often use Broad Stack Tests and semantic monitoring to offset this to some extent.
Lastly, there is a perceived benefit of serverless systems being easier to operate and maintain. Third-party services spend significant resources on security, availability, scaling and performance. These things often require specialized skills and may not be in the wheelhouse of smaller development teams. But this doesn't mean teams can forget about operations. It still falls on development teams to deal with the problems caused by service outage, downtime, decomissioning and slowdowns and to prevent these from having a cacading impact on their own applications.
I would like to thank Martin Fowler for his help with the illustration, editorial advice and guidance with this post. In addition, many thanks to Mike Roberts, Paul Hammant and Ken McCormack for their input and to Chris Turner, Ian Carvell, Patrick Debois and Jay Sandhaus for taking time to discuss their experiences building serverless applications. Jean-Noël Rouvignac alerted me to some typos.
Mike Roberts is writing a more detailed article on serverless architectures, which includes examples, further details on trade-offs and contrast with similar styles.
Patrick Debois talks more about the reality of operations for serverless architectures in his talk from serverlessconf 2016
YAaaS: Yet Another as a Service
These days everything seems to need to be "as a service", so we need a meta-term for this linguistic trend. My thanks to my colleague Birgitta Böckeler for coming up with one. So now we can say things like "FaaS is a YAaaS for 'Function'".
A business-capability centric team is one whose work is aligned long-term to a certain area of the business. The team lives as long as the said business-capability is relevant to the business. This is in contrast to project teams that only last as long as it takes to deliver project scope.
For example, an e-commerce business has capabilities such as buying and merchandising, catalog, marketing, order management, fulfilment and customer service. An insurance business has capabilities such as policy administration, claims administration, and new business. A telecom business has capabilities such as network management, service provisioning and assurance, billing, and revenue management. They may be further divided into fine-grained capabilities so that they can be owned by teams of manageable size.
Business-capability centric teams are “think-it, build-it and run-it” teams. They do not hand over to other teams for testing, deploying or supporting what they build. They own problems in their area end-to-end. They also own the IT systems (applications, APIs and data) that primarily support the business-capability. The underlying technology platforms (e.g. Java, .NET) and application platforms (e.g. Salesforce, SAP, Peoplesoft) may be shared across teams.
The team size for each capability is determined periodically, such as annually, based on guidance from IT strategy. This is a different kind of portfolio management where budget in the form of team capacity is allocated across a portfolio of long-lived business-capabilities whereas in traditional portfolio management, budget in the form of funds is allocated across a portfolio of comparatively short-lived projects/programs. Business-capability centric teams need to be OutcomeOriented to maximize their potential. Unless they are empowered to work towards a business outcome, they may devolve to being scope-delivery oriented.
Consider the example of a typical application landscape with a mix of up-to-date and legacy systems, some homegrown applications and some commercial off-the-shelf (COTS) applications, some SaaS applications, a heterogeneous API layer served by some new microservices, some mega-services and everything tied together with a combination of ad-hoc integration, an enterprise service bus and other boutique middleware. Each business-capability centric team would own a cohesive subset of the above that primarily relates to its business area. However, some applications are cross-capability by nature e.g. the end-to-end lookup-to-checkout customer journey in an e-commerce application. They might need a team of their own (or two teams, one for mobile and one for the laptop). It is a non-trivial exercise to draw boundaries within the application landscape and parcel it out to teams. Outcome-orientation is a good guiding principle. Consider if each parcel can be held responsible for a business outcome or sub-outcome (expressed as a business metric).
Some people are concerned that having a single team manage several systems within a business capability will act against Conway’s Law. But Conway's law isn't against a single team being responsible for multiple related components. It allows for high cohesion of component ownership and low coupling between teams and thus makes for better responsiveness.
Implications for Headcount
A business-capability centric configuration may require a slightly greater headcount than a project-centric model of execution. This is because a project’s remit is typically only to “build-it, handover to support/ops and disband” while a business-capability centric team’s remit is to “think-it, build-it and run-it” for as long as the business-capability is relevant. This requires us to maintain at least a minimal team at all times for each business-capability. As it turns out, this is desirable for a number of reasons. The project-centric model usually ends up compromising the architectural integrity of the application landscape because each project team only cares about delivering its scope by the promised date. In the process, it may take shortcuts such as:
- Ad-hoc integrations with the systems it depends on
- Integrating with or adding functionality to systems that are meant to be sunsetted because it would take more effort to do so with the replacement systems.
- Tacking on quick-and-dirty code on top of a previous team’s efforts and making it a maintenance nightmare in the process.
Some of this may be avoided with capable oversight from enterprise architecture but it remains a challenge nevertheless because the project-centric model often results in a different team for every new release and the new teams have to learn business rules and do’s and don’ts of the surrounding application landscape all over again. Outsourcing complicates this further.
On the other hand, the project-centric model is no stranger to huge headcounts when funding is plentiful and projects are initiated recklessly without regard to the carrying capacity of the existing codebase and application landscape. The lack of work-in-process limits at a project portfolio level leads to many projects getting started and few finishing or delivering desired outcomes.
Strategic and Utility Capabilities
Business capabilities may be categorized as strategic or utility over a given time horizon. Sometimes, it is more useful to label a few sub-capabilities within a capability as strategic. On the other hand, corporate IT capabilities such as payroll, accounting, legal, HR and workplace collaboration are usually classified as utility. Although still organized as business-capability centric teams, utility capabilities are often delivered with packaged software (buy over build). So they are "think-it, buy-it, customize/configure/integrate-it, run-it" teams rather than "think-it, build-it, run-it" teams. Utility capabilities are also commonly outsourced to external business-capability centric teams supplied by managed service providers. Even when delivered in-house, it is practical to staff these teams for keep-the-lights-on skills rather than top-notch development skills. In the same vein, although outcome-orientation is important for utility capabilities, they could be led by lower grade product owners.
AcknowledgementsSpecial thanks to Martin Fowler for his guidance with the content, help with publishing and for the nice illustrations.
Any significant software development effort requires several different activities to occur: analysis, user experience design, development, testing, etc. Activity-oriented teams organize around these activities, so that you have dedicated teams for user-experience design, development, testing etc. Activity-orientation promises many benefits, but software development is usually better done with OutcomeOriented teams.
Traditionally, big businesses with large IT departments (Enterprise IT) have tended to execute IT development projects with a bunch of activity-oriented teams drawn from a matrix IT organization (functional organization). The solid-lined arms of the matrix (headed by a VP of development, testing and so on) are usually along activity boundaries and they loan out “resources” to dotted-lined project or program organizations. Common justifications for doing so include:
- It helps standardization of conventions and techniques in development if all developers report into a single organization (arm of the matrix). Same for testing etc.
- It helps the cause of mentoring, training and nurturing the competency in general if all developers have long-lived development/engineering managers. Same for testing etc.
- It helps maximize utilization of talent (and thereby improve cost-efficiency) by staffing projects from pools of supposedly fungible developers, testers etc.
However, activity-oriented teams are prone to optimize for their own activity and not for the bigger picture of delivering useful software. This is a consequence of what they are held responsible for and how they are measured. It is common for a team of only developers to only be measured by their velocity. If they are only tasked with delivering scope, they will not think about whether it is going to solve the problems it was meant to. Even if they do so, they may be discouraged by the product management team - another activity-oriented team that is only responsible determining the spec.
Organizing by activity gets in the way of lowering batch size of work that is handed-off between teams. A separate team of testers won’t accept one story at a time from a team of developers. They’d rather test a release worth of stories or at least all stories in a feature at a time. This lengthens the feedback loop, increases end-to-end cycle time and hurts overall responsiveness.
High speed IT calls for motivated teams. Autonomy is a key motivator for teams. However, activity oriented organization can only be granted so much autonomy because they tend to use it for optimizing their sphere of activity.
A variation of activity-oriented organization are super-specialized teams that may result in the following ways:
- Tool or skill centric teams: e.g. a WebSphere Portal Server team or a BizTalk team
- Architectural Layer teams: e.g. a presentation layer team, middleware team, data layer team.
They are problematic because they have a narrow focus and they tend to optimize for team performance rather than the big picture. For sure, some tools may need specialists but that is no reason to isolate them in a separate team. Specialization isn’t the problem; organizing along lines of specialization is.
What works better
Software development is an iterative design process. In order to achieve true iteration and realize the value of fast feedback, its activities need to be performed with a single team (having a common reporting line) as far as possible. Many internet business and independent software vendors (ISVs) already operate in this way.
Activity-oriented organization may be attractive from the perspective of maximizing utilization but it comes in the way of maximizing value-addition and end-to-end responsiveness. In today’s business climate, market responsiveness (time-efficiency) is more important than cost-efficiency especially when it comes to capital expenditure items such as software development. Besides, solid-line reporting should be aligned to the most important goals which are responsiveness and value-addition.
In order to effectively nurture competencies in the absence of an aligned reporting structure, set up communities of practice along with expert community leads who have the ability to mentor and the budget to organize community events, buy books and arrange for training. These communities also help with standardizing conventions and techniques without going overboard with standardization. As for having a stable reporting manager, it is only an issue in project-centric IT. A BusinessCapabilityCentric setup allows for stable reporting managers.
AcknowledgementsThanks to Bill Codding, David Wang (Wang Wei), James Lewis, Kief Morris, Patrick Kua, Patrick Sarnacke, Steven Lowe and Venkatesh Ponniah for their comments. Special thanks to Martin Fowler for his guidance with the content, help with publishing and for the nice illustrations.
People who sponsor development of software usually aren’t very interested in development metrics such as velocity or frequency of deployment to production. They care more about business benefits that the software will deliver such as lower manual effort, better sales conversion, greater customer satisfaction, i.e business outcomes. Outcome-oriented teams are those that are mandated and equipped to deliver business outcomes, such teams have people with the capability to carry out all necessary activities to realize the outcome.. By contrast, ActivityOriented teams are neither equipped nor mandated to do so. They can only perform one of several activities required to realize an outcome.
A mandate to deliver a business outcome is very different from a mandate to deliver a certain amount of scope. Scope delivery is easy, relatively speaking. Outcome realization requires real collaboration between those who understand the problem and those who can fashion various levels of solution for it. Initial attempts at solution lead to a better understanding of the problem which leads to further attempts at better solutions. This doesn’t work where the product management organization is separate from the development (scope-delivery) organization.
Outcome-oriented teams are necessarily cross-functional (multidisciplinary) whereas ActivityOriented teams are typically mono-functional (single specialty). In the most traditional scenario, an outcome might simply be defined in terms of a project. The project is funded on the basis of a business case and therefore the desired outcome is to realize what is promised in the business case. However, depending on the size of the project it may be organized as one or more teams. When these teams are set up along activity boundaries it becomes an activity-oriented project (or program) organization. On the other hand, we achieve an outcome-oriented organization by dividing the overall outcome into sub-outcomes and assigning sub-outcomes to cross-functional teams that are self-sufficient in terms of people required to deliver the sub-outcome.
A potential problem of outcome-orientation is that people working in the same activity don’t get to mentor and learn from each other. To mitigate this, set up communities of practice along with expert community leads who have the ability to mentor and the budget to organize community events, buy books and arrange for training.
Executing big projects with an outcome-oriented setup is definitely better than doing so with an activity-oriented setup. That said, an outcome-oriented approach really shines when we move away from a project-centric funding model. In projects executed with outcome-oriented teams, the teams don’t live as long as the outcome is relevant to the business. They are disbanded at the end of the project which means we lose sight of the outcome until another release in the same area is funded with another team. This also leads to a problem of instability of management and reporting structure. These are limitations of the project-centric mode of IT execution, which can be overcome with a BusinessCapabilityCentric organization.