language feature


It's now common in many programming environments to represent data structures as a composite of lists and hashmaps. Most major languages now provide standard versions of these data structures, together with a rich range of operations, in particular Collection Pipelines, to manipulate them. These data structures are very flexible, allowing us to represent most forms of hierarchy in a manner that's easy to process and manipulate. [1]

The essence of this data structure is that there are (usually) two composite data types:

The leaves of the tree can be any other element, commonly the basic primitives in the language (such as integers and strings), but also any other structure that isn't treatable as a list or hash.

In most cases there are separate data types for the list and hash, since their access operations differ. However, as any lisper can tell you, it's easy to represent a hash as a list of key-value pairs. Similarly you can treat a hash with numeric indexes as a list (which is what Lua's tables do).

A list 'n' hash structure is by default schemaless, the lists can contain disparate elements and the hashes any combination of keys. This allows the data structure to be very flexible, but we must remember that we nearly always have an implicit schema when we manipulate a schemaless data structure, in that we expect certain data to be represented with certain keys.

A strength of the list and hash structure is that you can manipulate it with generic operations which know nothing of the actual keys present. These operations can then be parameterized with the keys that you wish to manipulate. The generic operations, usually arranged into a collection pipeline, provide a lot of navigation features to allow you to pluck what you need from the data structure without having to manipulate the individual pieces.

Although the usual way is to use flexible hashes for records, you can take a structure that uses defined record structures (or objects) and manipulate it in the same way as a hash if those record structures provide reflective operations. While such a structure will restrict what you can put in it (which is often a Good Thing), using generic operations to manipulate it can be very useful. But this does require the language environment to provide the mechanism to query records as if they are hashes.

List and hash structures can easily be serialized, commonly into a textual form. JSON is a particularly effective form of serialization for such a data structure, and is my default choice for this. Often XML is used to serialize list 'n' hash structures, it does a serviceable job, but is verbose and the distinction between attributes and elements makes no sense for these structures (although it makes plenty of sense for marking up text).

Despite the fact that list 'n' hashes are very common, there are times I wish I was using a thoughtful tree representation. Such a model can provide richer navigation operations. When working with the serialized XML structures in Nokogiri, I find it handy to be able to use XPath or CSS selectors to navigate the data structure. Some kind of common path specification such as these is handy for larger documents. Another issue is that it can be more awkward than it should to find the parent or ancestors of a given node in the tree. The presence of rich lists and hashes as standard equipment in modern languages has been one of the definite improvements in my programming life since I started programming in Fortran IV, but there's no need to stop there.


David Johnston, Marzieh Morovatpasand, Peter Gillard-Moss, Philip Duldig, Rebecca Parsons, Ryan Murray, and Steven Lowe discussed this post on our internal mailing list.


1: I find it awkward that there's no generally accepted, cross-language term for this kind of data structure. I could do with such a term, hence my desire to make a Neologism

if you found this article useful, please share it. I appreciate the feedback and encouragement




When I was starting out on my writing career, I began with writing articles for technical magazines. Now, when I write article length pieces, they are all written for the web. Paper magazines still exist, but they are a shrinking minority, probably doomed to extinction. Yet despite the withering of paper magazines, many of the assumptions of paper magazines still exact a hold on writers and publishers. This has particularly risen up in some recent conversations with people working on articles I want to publish on my site.

Most web sites still follow the model of the Paper Age. These sites consist of articles that are grouped primarily due to when they were published. Such articles are usually written in one episode and published as a whole. Occasionally longer articles are split into parts, so they can be published in stages over time (if so they also may be written in parts).

Yet these are constraints of a paper medium, where updating something already published is mostly impossible. [1] There's no reason to have an article split over distinct parts on the web, instead you can publish the first part and revise it by adding material later on. You can also substantially revise an existing article by changing the sections you've already published.

I do this whenever I feel the need on my site. Most of the longer-form articles that I've published on my site in the last couple of years were published in installments. For example, the popular article on Microservices was originally published over nine installments in March 2014. Yet it was written and conceived as a single article, and since that final installment, it's existed on the web as a single article.

Our first rationale for publishing in installments is the notion that people tend to prefer reading shorter snippets these days, so by releasing a 6000 word article in nine parts, we could keep each new slug to a size that people would prefer to read. A second reason is that multiple publications allows for more opportunities to grab people's attention, so makes it more likely that an article will find interested readers.

When I publish in installments, I add an item to my news feed and tweet for each installment. Since I'm describing an update, I link with a fragment URL to take readers to the new section (in future I may link to a temporary explanatory box to highlight what's in the new installment).

But whatever the way the article is released to the world, it is still a single conceptual item, so its best permanent form is a single article. Many people have read the microservices article since that March, and I suspect hardly any of them knew or cared that it was originally published in installments.

In that case we wrote the entire article before we started the installment publishing, but there's no reason against writing it in stages too. For my collection pipelines article, I wrote and published the original article over five installments in July 2014. As I was writing it, I was conscious that there were additional sections I could add. I decided to wait to see how the article was received before I put the effort in to write those sections. Since it was pretty popular, I made a number of revisions, for each one I announced it with a tweet and an item on my feed.

Letting an article evolve like this is the kind of thing that's difficult in a print medium, but exactly the right thing to do on the web. A reader doesn't care that I revised the article to improve it, she just wants to read the best explanation of the topic at hand.

I do like to provide some traces of such revisions. At the end of each article, I include a revision history which briefly summarizes the changes. For a couple of revisions, such as the 2006 revision of my article on Continuous Integration, I made the original article available on a different URL with a link from the revised article. I don't think the original article is useful to most readers, only really to those tracing the intellectual history of the idea, so shifting the original to a new URL makes sense.

The role of the feed is important in this. The traditional blog reinforces the Paper Age model by encouraging people to match an article with its feed entry. For longer articles, I prefer to consider them as different things, the feed is a notice of a new article or revision, which links to the article concerned. That way I generate feed entries each installment where the feed summarizes what's been added.

The point of all this is that we should consider web articles as information resources, resources that can and should be extended and revised as our understanding increases and as time and energy allow. We shouldn't let the Print Age notions of how articles should be constructed dictate the patterns of the Internet Age.


1: There is a sort of an update mechanism, in that a series of articles might be republished as a single work. But that is relatively rare.

if you found this article useful, please share it. I appreciate the feedback and encouragement


API design · object collaboration design


A required interface is an interface that is defined by the client of an interaction that specifies what a supplier component needs to do so that it can be used in that interaction.

A good example of required interface is an interface commonly referred to as “comparable”. Such an interface is usually required by a sort function. Imagine I have a set of albums, and I want to sort them by title, but ignoring articles such as "The", "A", and "An". I can arrange them to be sorted in this way by implementing the required interface for any sort functions.

In Java it would look something like this.

class Album...

  public class Album implements Comparable<Album> {
    private String title;
    public Album(String title) {
      this.title = title;
    public String getTitle() {
      return title;
    public int compareTo(Album o) {
      return this.sortKey().compareTo(o.sortKey());
    private String sortKey() {
      return ignoreSortPrefixes(title).toLowerCase();
    private static String ignoreSortPrefixes(String arg) {
      final String[] prefixes = {"an", "a", "the"};
              .map(s -> s + " ")
              .filter(s -> arg.toLowerCase().startsWith(s))
              .map(s -> arg.substring(s.length(), arg.length()))

In this case Comparable is the required interface of the various Java sort functions. More complicated examples can have a richer interface with several methods defined on it.

Often people think about interfaces as a decision by the supplier about what to expose to clients. But required interfaces are specified (and often defined) by the client. You often get more useful interfaces by thinking about what clients require - leading towards thinking about RoleInterfaces.

Using an Adapter

A common problem comes up if I want to plug together two modules that have been defined independently. Here we can run into difficulties even if we get names that match.

Consider a task list with a required interface of tasks.

class TaskList...

  private List<Task> tasks;
  private LocalDate deadline;
  public LocalDate latestStart() {
    return deadline.minusDays( -> t.shortestLength()).sum());

interface Task…

  int shortestLength();

Let's imagine I want to integrate it with an Activity class I got from a different supplier.

class Activity…

  public int shortestLength() {

Even though the activity has a method whose signature happens to match the required interface's, I (rightly) can't create a task list of activities because the type definitions don't match. If I can't modify the activity class I need to use an adapter.

public class ActivityAdapter implements Task {
  private Activity activity;

  public ActivityAdapter(Activity activity) {
    this.activity = activity;
  public int shortestLength() {
    return activity.shortestLength();

In the software world we use the term adapter pretty freely, but here I'm using strictly in the sense of the Gang of Four book. In this usage an adapter is an object that maps one object to the required interface of another.

In this case, I don't need an adapter if I'm using a dynamic language, but I do if the activity class used a method with a different signature.


Alexander Zagniotov and Bruno Trecenti commented on drafts of this post.
if you found this article useful, please share it. I appreciate the feedback and encouragement


team organization · database · encapsulation · application architecture · web development


One of the most common ways to modularize an information-rich program is to separate it into three broad layers: presentation (UI), domain logic (aka business logic), and data access. So you often see web applications divided into a web layer that knows about handling http requests and rendering HTML, a business logic layer that contains validations and calculations, and a data access layer that sorts out how to manage persistant data in a database or remote services.

On the whole I've found this to be an effective form of modularization for many applications and one that I regularly use and encourage. It's biggest advantage (for me) is that it allows me to reduce the scope of my attention by allowing me to think about the three topics relatively independently. When I'm working on domain logic code I can mostly ignore the UI and treat any interaction with data sources as an abstract set of functions that give me the data I need and update it as I wish. When I'm working on the data access layer I focus on the details of wrangling the data into the form required by my interface. When I'm working on the presentation I can focus on the UI behavior, treating any data to display or update as magically appearing by function calls. By separating these elements I narrow the scope of my thinking in each piece, which makes it easier for me to follow what I need to do.

This narrowing of scope doesn't imply any sequence to programming them - I usually find I need to iterate between the layers. I might build the data and domain layers off my initial understanding of the UX, but when refining the UX I need to change the domain which necessitates a change to the data layer. But even with that kind of cross-layer iteration, I find it easier to focus on one layer at a time as I make changes. It's similar to the switching of thinking modes you get with refactoring's two hats.

Another reason to modularize is to allow me to substitute different implementations of modules. This separation allows me to build multiple presentations on top of the same domain logic without duplicating it. Multiple presentations could be separate pages in a web app, having a web app plus mobile native apps, an API for scripting purposes, or even an old fashioned command line interface. Modularizing the data source allows me to cope gracefully with a change in database technology, or to support services for persistance that may change with little notice. However I have to mention that while I often hear about data access substitution being a driver for separating the data source layer, I rarely hear of someone actually doing it.

Modularity also supports testability, which naturally appeals to me as a big fan of SelfTestingCode. Module boundaries expose seams that are good affordance for testing. UI code is often tricky to test, so it's good to get as much logic as you can into a domain layer which is easily tested without having to do gymnastics to access the program through a UI [1]. Data access is often slow and awkward, so using TestDoubles around the data layer often makes domain logic testing much easier and responsive.

While substitutability and testability are certainly benefits of this layering, I must stress that even without either of these reasons I would still divide into layers like this. The reduced scope of attention reason is sufficient on its own.

When talking about this we can either look at it as one pattern (presentation-domain-data) or split it into two patterns (presentation-domain, and domain-data). Both points of view are useful - I think of presentation-domain-data as a composite of presentation-domain and domain-data.

I consider these layers to be a form of module, which is a generic word I use for how we clump our software into relatively independent pieces. Exactly how this corresponds to code depends on the programming environment we're in. Usually the lowest level is some form of subroutine or function. An object-oriented language will have a notion of class that collects functions and data structure. Most languages have some form of higher level called packages or namespaces, which often can be formed into a hierarchy. Modules may correspond to separately deployable units: libraries, or services, but they don't have to.

Layering can occur at any of these levels. A small program may just put separate functions for the layers into different files. A larger system may have layers corresponding to namespaces with many classes in each.

I've mentioned three layers here, but it's common to see architectures with more than three layers. A common variation is to put a service layer between the domain and presentation, or to split the presentation layer into separate layers with something like Presentation Model. I don't find that more layers breaks the essential pattern, since the core separations still remain.

The dependencies generally run from top to bottom through the layer stack: presentation depends on the domain, which then depends on the data source. A common variation is to arrange things so that the domain does not depend on its data sources by introducing a mapper between the domain and data source layers. This approach is often referred to as a Hexagonal Architecture.

Although presentation-domain-data separation is a common approach, it should only be applied at a relatively small granularity. As an application grows, each layer can get sufficiently complex on its own that you need to modularize further. When this happens it's usually not best to use presentation-domain-data as the higher level of modules. Often frameworks encourage you to have something like view-model-data as the top level namespaces; that's ok for smaller systems, but once any of these layers gets too big you should split your top level into domain oriented modules which are internally layered.

Developers don't have to be full-stack but teams should be.

One common way I've seen this layering lead organizations astray is the AntiPattern of separating development teams by these layers. This looks appealing because front-end and back-end development require different frameworks (or even languages) making it easy for developers to specialize in one or the other. Putting those people with common skills together supports skill sharing and allows the organization to treat the team as a provider of a single, well-delineated type of work. In the same way, putting all the database specialists together fits in with the common centralization of databases and schemas. But the rich interplay between these layers necessitates frequent swapping between them. This isn't too hard when you have specialists in the same team who can casually collaborate, but team boundaries add considerable friction, as well as reducing an individual's motivation to develop the important cross-layer understanding of a system. Worse, separating the layers into teams adds distance between developers and users. Developers don't have to be full-stack (although that is laudable) but teams should be.

Further Reading

I've written about this separation from a number of different angles elsewhere. This layering drives the structure of P of EAA and chapter 1 of that book talks more about this layering. I didn't make this layering a pattern in its own right in that book but have toyed with that territory with Separated Presentation and PresentationDomainSeparation.

For more on why presentation-domain-data shouldn't be the highest level modules in a larger system, take a look at the writing and speaking of Simon Brown. I also agree with him that software architecture should be embedded in code.

I had a fascinating discussion with my colleague Badri Janakiraman about the nature of hexagonal architectures. The context was mostly around applications using Ruby on Rails, but much of the thinking applies to other cases when you may be considering this approach.


James Lewis, Jeroen Soeters, Marcos Brizeno, Rouan Wilsenach, and Sean Newham discussed drafts of this post with me.


1: A PageObject is also an important tool to help testing around UIs.

if you found this article useful, please share it. I appreciate the feedback and encouragement


bad things · writing


Andrew Koenig first coined the term "antipattern" in an article in JOOP[1], which is sadly not available on the internet. The essential idea (as I remember it [2]) was that an antipattern was something that seems like a good idea when you begin, but leads you into trouble. Since then the term has often been used just to indicate any bad idea, but I think the original focus is more useful.

In the paper Koenig said

An antipattern is just like a pattern, except that instead of a solution it gives something that looks superficially like a solution but isn't one.

-- Andrew Koenig

This is what makes a good antipattern something separate to just a bad thing to point and laugh at. The fact that it looks like a good solution is its essential danger. Since it looks good, sensible people will take the path - only once you've put a lot of effort into it will you know it's bad result.

When writing a description of an antipattern it's valuable to describe how to get out of trouble if you've taken the bad path. I see that as useful but not necessary. If there's no good way to get out of it, that doesn't reduce the value of the warning.

It's useful to remember that the same solution can be a good pattern in some contexts and an antipattern in others. The value of a solution depends on the context that you use it.


1: Journal of Object-Oriented Programming, Vol 8, no. 1. March/April 1995. It was then reprinted in "The Patterns Handbook", edited by Linda Rising (Cambridge University Press)

2: I don't have a copy of the paper, so I'm going primarily off memory and some old notes.

if you found this article useful, please share it. I appreciate the feedback and encouragement


team organization · project planning · collaboration


Alignment maps are organizational information radiators that help visualize the alignment of ongoing work with business outcomes. The work may be regular functionality addition or technical work such as re-architecting or repaying technical debt or improving the build and deployment pipeline. Team members use alignment maps to understand what business outcomes their day-to-day work is meant to improve. Business and IT sponsors use them to understand how ongoing work relates to the business outcomes they care about.

Here’s an example scenario (inspired by real life) that illustrates how these maps may be useful. A team of developers had inefficiently implemented a catalog search function as N+1 calls. The first call to the catalog index returned a set of SKU IDs. For each ID returned, a query was then made to retrieve product detail. The implementation came to the attention of an architect when it failed performance tests. He advised the team to get rid of the N+1 implementation.

“Search-in-one” was the mantra he offered the team as a way to remember their objective. Given the organizational boundary between architects and developers and the low frequency of communication between them, the mantra was taken literally. The team moved heaven and earth to implement a combined index query and detail query in a single call. They lost sight of the real objective of improving search performance and slogged away in an attempt to achieve acceptable performance in exactly one call. Funding ran out in a few months and after some heated discussions, the project was cancelled and the team disbanded.

The above example may seem absurd but sadly, enterprise IT is no stranger to architecture and business projects that are cancelled after a while because they lost sight of why they were funded in the first place. In the terminology of organizational design, these are problems of alignment.

Visualizing Alignment

Broadly, IT strategy has to align with business strategy and IT outcomes with desired business outcomes. A business outcome may be supported (in part) by one or more IT outcomes. Each IT outcome may be realized by one or more initiatives (program of work—architectural or business). At this point, it may also be useful to identify an owner for each initiative who then sponsors work (action items) across multiple teams as part of executing the initiative. Depending on the initiative the owner may be a product owner, architect, tech lead or manager. Here's an alignment map for the “search-in-one” case. Had it been in public display in the team’s work area, it might have prompted someone to take a step back and ask what their work was really meant to achieve.

Global Map

A global alignment map for the IT (appdev+ops) organization may look more like this (although real maps tend to be much larger).

As with all information radiators, such a map is a snapshot in time and needs to be updated regularly (say once a month). Each team displays a big printout of the global map in its work area.

Big organizations are likely to realize value early in this exercise by collaborating to come up with a version 1.0 of such a map that everyone agrees to. The discussions around who owns what initiatives and what outcomes an initiative contributes to leads to a fair bit of organizational clarity of what everyone is up to. Usually, the absence of well-articulated and commonly understood business and IT strategies come in the way of converging on a set of business and IT outcomes. Well-facilitated workshops with deep and wide participation across the relevant parts of the organization can help address this.

Tracing alignment paths

Once a global alignment map is in place, it allows us to trace alignment from either end. IT and business sponsors can trace what action items are in play under a given initiative. Development team members can trace through the map to understand the real purpose of items they are working on. In addition to in-progress items, we could also include action items that are planned, done or blocked.

As illustrated in the map above, each team highlights their section of the map on their copy of the global map.

Qualitative benefits validation

Once a month (or quarter), IT and business people get together to validate if all the IT activity has made any difference to business outcomes. Business people come to the meeting with red-amber-green (RAG) statues for business outcomes and IT people may come with RAG statues for their side of the map. Both parties need to be able to back up their RAG assessments with data and/or real stories from the trenches (narrative evidence).

These maps can be combined

Sriram's recent book explores how best to design an IT organization to be agile enough to survive in today's competitive jungle.

With this the group may realize that:

  • Some outcomes have turned green as compared to the previous meeting. Perhaps customer retention turned green after the last release of the responsive rewrite initiative.
  • Not all IT activity is making the expected difference to business outcomes. This provides an opportunity to discuss why this may be the case. Perhaps because:
    • It is a little early in the day. Other planned items need to complete before we can expect a difference. This is probably why, in the map above, customer acquisition is red even though site UX is green. Platform unbundling is still incomplete.
    • The initiatives and action items are sensible but a different execution approach is needed (this is the reason in case of “search-in-one”).
    • A different initiative or set of actions are required and existing ones are better cancelled. Something outside of IT has to fall into place before the business can realize value.
  • A few business outcomes are green even though the related IT initiatives aren’t. This probably means IT matters less to this outcome than other non-IT factors. In the map above, this is probably why customer retention is green even though site performance isn’t. Perhaps IT means to say that performance isn’t where it should be although it hasn’t affected retention just yet.

To summarize, alignment maps provide a common organization-wide tool to discuss the extent to which different IT initiatives are paying off. They could also improve the ability to make sense of ongoing work and bring it greater alignment with business objectives. I haven't used this technique enough yet to claim general effectiveness, although I do think it shows enough promise. If you try this out I'd be glad to hear about experiences with it.


Thanks to Jim Gumbley, Kief Morris and Vinod Sankaranarayanan for their inputs. Special thanks to Martin Fowler for his guidance with the content and help with publishing.

Further Reading

I describe other information radiators that help the cause of organizational agility in my book Agile IT Organization Design. My companion web site at contains links to further writing and my talks.

Lars Barkman has posted details on how to construct an alignment map with graphviz.

if you found this article useful, please share it. I appreciate the feedback and encouragement


delivery · agile adoption · team organization · collaboration


Agile software development has broken down some of the silos between requirements analysis, testing and development. Deployment, operations and maintenance are other activities which have suffered a similar separation from the rest of the software development process. The DevOps movement is aimed at removing these silos and encouraging collaboration between development and operations.

DevOps has become possible largely due to a combination of new operations tools and established agile engineering practices [1], but these are not enough to realize the benefits of DevOps. Even with the best tools, DevOps is just another buzzword if you don't have the right culture.

The primary characteristic of DevOps culture is increased collaboration between the roles of development and operations. There are some important cultural shifts, within teams and at an organizational level, that support this collaboration.

An attitude of shared responsibility is an aspect of DevOps culture that encourages closer collaboration. It’s easy for a development team to become disinterested in the operation and maintenance of a system if it is handed over to another team to look after. If a development team shares the responsibility of looking after a system over the course of its lifetime, they are able to share the operations staff’s pain and so identify ways to simplify deployment and maintenance (e.g. by automating deployments and improving logging). They may also gain additional ObservedRequirements from monitoring the system in production. When operations staff share responsibility of a system’s business goals, they are able to work more closely with developers to better understand the operational needs of a system and help meet these. In practice, collaboration often begins with an increased awareness from developers of operational concerns (such as deployment and monitoring) and the adoption of new automation tools and practices by operations staff.

Some organizational shifts are required to support a culture of shared responsibilities. There should be no silos between development and operations. Handover periods and documentation are a poor substitute for working together on a solution from the start. It is helpful to adjust resourcing structures to allow operations staff to get involved with teams early. Having the developers and operations staff co-located will help them to work together. Handovers and sign-offs discourage people from sharing responsibility and contributes to a culture of blame. Instead, developers and operations staff should both be responsible for the successes and failures of a system. DevOps culture blurs the line between the roles of developer and operations staff and may eventually eliminate the distinction. One common anti-pattern when introducing DevOps to an organization is to assign someone the role of 'DevOps' or to call a team a 'DevOps team'. Doing so perpetuates the kinds of silos that DevOps aims to break down and prevents DevOps culture and practices from spreading and being adopted by the wider organization.

Another valuable organizational shift is to support autonomous teams. In order to collaborate effectively, developers and operations staff need to be able to make decisions and apply changes without convoluted decision making processes. This involves trusting teams, changing the way risk is managed and creating an environment that is free of a fear of failure. For example, a team that has to produce a list of changes for sign-off in order to deploy to a testing environment is likely to be delayed frequently. Instead of requiring such a manual check, it is possible to rely on version control, which is fully auditable. Changes in version control can even be linked to tickets in the team's project management tool. Without the manual sign-off, the team can automate their deployments and speed up their testing cycle.

One effect of a shift towards DevOps culture is that it becomes easier to put new code in production. This necessitates some further cultural changes. In order to ensure that changes in production are sound, the team needs to value building quality into the development process. This includes cross-functional concerns such as performance and security. The techniques of ContinuousDelivery, including SelfTestingCode, form a basis which allows regular, low-risk deployments.

It is also important for the team to value feedback, in order to continuously improve the way in which developers and operations staff work together as well as the system itself. Production monitoring is a helpful feedback loop for diagnosing issues and spotting potential improvements.

Automation is a cornerstone of the DevOps movement and facilitates collaboration. Automating tasks such as testing, configuration and deployment frees people up to focus on other valuable activities and reduces the chance of human error. A helpful side effect of automation is that automated scripts and tests serve as useful, always up-to-date documentation of the system. Automating server configuration, for example, removes the guesswork associated with a SnowflakeServer and means that developers and operations staff are equally able to know and change how a server is configured.


1: Operations tools include virtualization, cloud computing and automated configuration management. These are often supported by engineering practices such as Continuous Integration, evolutionary design and clean code.

if you found this article useful, please share it. I appreciate the feedback and encouragement