ImmutableServer

delivery · build scripting

tags:

Automated configuration tools (such as CFEngine, Puppet, or Chef) allow you to specify how servers should be configured, and bring new and existing machines into compliance. This helps to avoid the problem of fragile SnowflakeServers. Such tools can create PhoenixServers that can be torn down and rebuilt at will. An Immutable Server is the logical conclusion of this approach, a server that once deployed, is never modified, merely replaced with a new updated instance.

Automated configuration tools are usually used with ConfigurationSynchronization where you leave a server running for a potentially long period of time, repeatedly applying configuration to bring it into line with the latest specification. In theory servers can be allowed to run indefinitely, and they'll be kept completely consistent and up to date. In practice it's not possible to manage a server's configuration completely, so there is considerable scope for configuration drift, and unexpected changes to running servers.

This is where a PhoenixServer is helpful.

By frequently destroying and rebuilding servers from the base image, 100% of the server's elements are reset to a known state, without spending a ridiculous amount of time specifying and maintaining detailed configuration specifications.

Once you're using phoenixes, the focus of configuration management shifts to the management of base images. Fixes, changes, and updates are applied to the base image rather than to running systems. Each time you want a new update you modify the base instance and run it through an automated test harness. You only create new servers when they pass these steps.

So a phoenix server's complete state is built from a combination of base image + automated configuration management + data [1], which reduces the pressure to have automated configuration manage 100% of the server.

But while we can continue to run configuration management updates on a server during its brief lifetime, there's less value in doing so. In fact, there is considerable value in not doing so, since any change to a running system introduces risk.

This leads naturally to immutable servers [2].

Once you've spun up a server instance from a well-tested base image, you shouldn't run configuration management tools, since they create opportunities for untested changes to the instance. Any changes that are needed can be made to the base image, tested, and then rolled out. Servers without the change are torn down and replaced.

If this sounds familiar, it's because it follows the practices of ContinuousIntegration and ContinuousDelivery. With Continuous Delivery of software, it's safer to compile a given version of an application into a deployable artifact only once, and know that you are deploying and running a consistently built application in all environments. With an immutable server, you make each change to a base image, and then you know that all instances created from that image are consistent.

The main differences between instances of a server role come from configuration settings, which should come from outside the server. For example, most virtualization and cloud platforms offer a way to set metadata values when provisioning a new instance, which can then be read by the running server. New servers may also pull configuration values from a central registry like Zookeeper.

It's advisable to minimize the number and scope of per-instance configuration items for immutable servers, and to run changes to these through automated testing where feasible.

---

Handling data

If servers are disposable, the data that lives on them often is not. When implementing phoenix or immutable servers you should consider what data needs to be persisted as servers are destroyed and created, and what data must be replicated in order to scale by adding additional servers.

You can ship data off of the instance when it has value but isn't needed at runtime, for example sending logfiles to a central syslog server. A shared file system like NFS can make files available to servers, perhaps living on a SAN. Cloud platforms generally offer mountable storage devices like AWS EBS volumes which can be attached to new servers instances when the old ones are destroyed, or quickly duplicated and attached to replicas when scaling a cluster. Often you can pass the buck to a service which someone else maintains, like Amazon's RDS database service.

---

ConfigurationSynchronization

delivery

tags:

Automated configuration tools (such as CFEngine, Puppet, or Chef) allow you to avoid SnowflakeServers by providing recipes to describe the configuration of elements of a server. Configuration synchronization continually applies these specifications, either on a regular schedule or when it changes, to server instances throughout their lifetime. If someone makes a change to a server outside the tool, it will be reverted to the centrally specified configuration the next time the server is synchronized. If some configuration change is needed, it's made in the configuration specification (recipes, manifests, or whatever the particular configuration tool calls it), and is then applied to all relevant servers across the infrastructure.

In theory, once a server has been created, configuration synchronization will keep it up to date, applying upgrades and patches and preventing configuration drift for a potentially long lifetime.

In practice, however, it isn't feasible to keep servers completely consistent using current automated configuration management tools, so over time configuration synchronization leads to inconsistency. [1]

Each element of server configuration managed by an automated tool requires work to write, test, and maintain the recipe or manifest. It simply isn't reasonable to attempt to manage every single element of a typical server using these tools. There are far too many of them. And for each additional element you do specify, the work involved grows non-linearly thanks to the potential interactions, integrations, and dependencies between them.

Software packages are a particular challenge, given the dependencies they may have on yet more packages. Tools such as yum, apt-get, gems, etc. automatically work out dependencies and install them from repositories. So a large number of the packages on a system are only implicitly managed, and may change without notice. Although you can micro-manage the versions and dependencies of software packages installed, it's an intractable job considering the number of packages involved.

If managing the stuff you want to have on your server is difficult, managing the things you don't want is worse, given there are an effectively inifinite number of them. Current configuration management tools require you to explicitly list each file, package, or other element you want removed if found. This means additional things can be manually added onto some servers, creating inconsistent and unexpected behaviour.

In reality, people using automated configuration tools get by well enough without specifying 100% of the configurable surface area of their servers. Teams apply the 80/20 rule to automated configuration, focusing 80% (or more like 95%) of their attention on defining that 20% (or 5%) of the system which is most relevant to their particular needs, leaving the rest to the defaults installed by the base OS. So the majority of the system is not under automated configuration, and generally, this works. Unfortuantely, when it doesn't - when some part of the system not managed by automation tools does cause an issue, the effects are unexpected and can be difficult to track down.

This problem grows worse over the lifespan of a server, and with the frequency of change. The longer a server stays up, the more it may deviate from other servers, especially newer ones.

This issue leads some people to use PhoenixServers. By ensuring that a server's lifespan is kept short, frequently rebuilding fresh ones from a base image, the potential for configuration drift is kept small, without the overhead of specifying more of the server's configuration in a management tool than is really necessary. Taking the phoenix server to its logical conclusion leads to ImmutableServers, which avoid any changes made to a server during it's lifespan.

---

EmbeddedDocument

noSQL · encapsulation · object collaboration design

tags:

Flowing JSON data structures through a server is something I'm seeing more these days. JSON documents can be persisted directly, either by using an AggregateOrientedDatabase or a serialized LOB in a relational database. JSON documents can also be served directly to web browsers or used to transfer data to server-side page renderers. When JSON is being used in this way, I hear people saying that using an object-oriented language gets in the way because the JSON needs to be translated into objects only to be rendered out again - a waste of programming effort [1]. I agree with the point about waste, but I argue that it's not a problem with objects but a failure to understand encapsulation.

Let's imagine we're storing an order as a JSON document and serving it up with minor server-side processing, again as JSON. An example document might be like this.

{ "id": 1234,
  "customer": "martin",
  "items": [
    {"product": "talisker", "quantity": 500},
    {"product": "macallan", "quantity": 800},
    {"product": "ledaig",   "quantity": 1100}
  ],
  "deliveries": [
    { "id": 7722,
      "shipDate": "2013-04-19",
      "items": [
        {"product": "talisker", "quantity": 300},
        {"product": "ledaig",   "quantity": 500}
      ]
    },
    { "id": 6533,
      "shipDate": "2013-04-18",
      "items": [
        {"product": "talisker", "quantity": 200},
        {"product": "ledaig",   "quantity": 300},
        {"product": "macallan", "quantity": 300}
      ]
    }
  ]
}

We'll assume we haven't got much server-side processing to do, but we do have some. Let's also assume we're using an OO language. A naive approach might be to read in the JSON document, convert the data to the appropriate object graph (with orders, line-items, and deliveries), apply any processing, and then serialize the object graph to JSON for the client.

In many of these situtiations a better way to proceed is to keep the data in a JSONish form, but still wrap it with objects to coordinate manipulation. Most programming environments provide generic libraries that take a document and deserialize it to generic data structures. So a JSON document would deserialize to a structure of lists and dictionaries, an xml document to a tree of xml nodes. We can then take this generic data structure and put it into a field of an order object - here's an example with Ruby and JSON.

class Order...
  def initialize jsonDocument
    @data = JSON.parse(jsonDocument)
  end

When we want to manipulate the data, we can define methods on the object as usual, and implement them by accessing this data structre.

class Order...
  def customer
    @data['customer']
  end
  def quantity_for aProduct
    item = @data['items'].detect{|i| aProduct == i['product']}
    return item ? item['quantity'] : 0
  end

This includes cases with more complex logic. [2]

class Order...
  def outstanding_delivery_for aProduct
    delivered_amount = @data['deliveries'].
      map{|d| d['items']}.
      flatten.
      select{|d| aProduct == d['product']}.
      inject(0){|res, d| res += d['quantity']}
    return quantity_for(aProduct) - delivered_amount
  end

The embedded document can be enriched before sending to the client.

class Order...
  def enrich
    @data['earliestShipDate'] = 
      @data['deliveries'].
      map{|d| Date.parse(d['shipDate'])}.
      min.
      to_s
  end

If needed, you can form similar objects on sub-trees of the embedded document.

class Order...
  def deliveries
    @data['deliveries'].map{|d| Delivery.new(d)}
  end

class Delivery...
  def initialize hash
    @data = hash
  end
  def ship_date
    Date.parse(@data['shipDate'])
  end

One thing be wary of here is that such object wrappers aren't quite the same as normal objects. The delivery objects returned in the above code fragment don't have the same equality semantics that you'd expect from objects arranged in the more usual structure.

Despite its compartive rareity, an embedded document fits well with object-orientation. The point of encapsulated data is the hiding of the data structure, so that users of the object don't know or care about the internal structure of the order.

Those familiar with functional programming will recognize the style of flowing a generic data structure through a series of functions - you can think of the object as providing a namespace for manipulating the generic data stuctures.

The sweet spot for an embedded document is when you're providing the document in the same form that you get it from the data store, but still want to do some manipulation of that data. If you don't have any need to access the contents of the JSON document, then there's no need to even deserialize it into a generic data structure. The order object needs only a constructor and a method to return its JSON representaiton. On the other hand as you do more work on the data - more server side logic, transforming into different representations - then it's worth considering whether it's easier to turn the data into an object graph.

---

API_Copyright

API design · legal

tags:

Yesterday the Electronic Frontier Foundation released an amicus brief for a lawsuit on the topic of copyrighting APIs. The brief is a statement on behalf programmers who oppose the copyrighting of APIs. I'm one of the signatories to the amicus brief.

The software industry has long had a complicated relationship with intellectual property law. It's long been held that source code can be protected by copyright, preventing someone from just copying someone else's program without their consent. On the whole I'm in favor of using copyright in this way - as one who writes a lot of prose describing software development, the mechanism seems to fit well.

But that fit breaks down when it comes to interfaces of software components. The whole point of components, as I see it, is that they represent software elements that are independently replaceable and upgradeable [1]. Breaking down software systems into components allows competition to develop, as different groups compete with better or cheaper components. For this to work, components need to have common interfaces, which raises the question of how such interfaces can develop.

So far common interfaces mostly appear by interfaces becoming widely used and then widely copied. This can be intentionally, with component authors documenting their interfaces, or by reverse engineering of closed interfaces. Reverse engineering of closed interfaces has played a vital role in the development of software systems we use today - the amicus brief describes examples such as various elements of Unix, the BIOS of the IBM PC, and the SMB protocol for network sharing of hard drives.

This background was challenged recently by a case brought by Oracle against Google's use of Java for Android. There are a number of parts to this case, but a key question was whether the API to core java libraries was copyrightable. Last year a federal district court ruled that APIs were not copyrightable. Oracle is appealing against this, the amicus brief that we signed was organized by the EFF to uphold the district court's ruling.

I am not opposed to intellectual property in general, or the use of copyright for software. I do think it's reasonable for software developers to be able to protect their hard work through the course of an admittedly imperfect legal system. Nor do I deny that designing a good API is significant intellectual work. But like everything else in property law, it's about balancing interests. Copyrightable APIs would erect considerable barriers to innovation and competition, further limiting the ability of new entrants to disrupt markets and improve the software ecosystem. This threat is not a small one, and I'm happy to play a small part in opposing it.

---

ContinuousDelivery

delivery · version control · continuous integration

tags:

Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

You’re doing continuous delivery when: [1]

• Your software is deployable throughout its lifecycle
• Your team prioritizes keeping the software deployable over working on new features
• Anybody can get fast, automated feedback on the production readiness of their systems any time somebody makes a change to them
• You can perform push-button deployments of any version of the software to any environment on demand

You achieve continuous delivery by continuously integrating the software done by the development team, building executables, and running automated tests on those executables to detect problems. Furthermore you push the executables into increasingly production-like environments to ensure the software will work in production. To do this you use a DeploymentPipeline.

The key test is that a business sponsor could request that the current development version of the software can be deployed into production at a moment's notice - and nobody would bat an eyelid, let alone panic.

To achieve continuous delivery you need:

• a close, collaborative working relationship between everyone involved in delivery (often referred to as a "DevOps culture" [2]).
• extensive automation of all possible parts of the delivery process, usually using a DeploymentPipeline

Continuous Delivery is sometimes confused with Continuous Deployment. Continuous Deployment means that every change goes through the pipeline and automatically gets put into production, resulting in many production deployments every day. Continuous Delivery just means that you are able to do frequent deployments but may choose not to do it, usually due to businesses preferring a slower rate of deployment. In order to do Continuous Deployment you must be doing Continuous Delivery.

Continuous Integration usually refers to integrating, building, and testing code within the development environment. Continuous Delivery builds on this, dealing the the final stages required for production deployment.

For more information see the resources on my guide page, in particular the book.

---

DeploymentPipeline

delivery · build scripting

tags:

One of the challenges of an automated build and test environment is you want your build to be fast, so that you can get fast feedback, but comprehensive tests take a long time to run. A deployment pipeline is a way to deal with this by breaking up your build into stages. Each stage provides increasing confidence, usually at the cost of extra time. Early stages can find most problems yielding faster feedback, while later stages provide slower and more through probing. Deployment pipelines are a central part of ContinuousDelivery.

Usually the first stage of a deployment pipeline will do any compilation and provide binaries for later stages. Later stages may include manual checks, such as any tests that can't be automated. Stages can be automatic, or require human authorization to proceed, they may be parallelized over many machines to speed up the build. Deploying into production is usually the final stage in a pipeline.

More broadly the deployment pipeline's job is to detect any changes that will lead to problems in production. These can include performance, security, or usability issues. A deployment pipeline should enable collaboration between the various groups involved in delivering software and provide everyone visibility about the flow of changes in the system, together with a thorough audit trail.

A good way to introduce continuous delivery is to model your current delivery process as a deployment pipeline, then examine this for bottlenecks, opportunities for automation, and collaboration points.

For more information see chapter 5 of the Continuous Delivery book, available as a free download.

---

StoryTest

testing · bad things

tags:

Story tests are BusinessFacingTests used to describe and verify the software delivered as part of a UserStory. When a story is elaborated the team creates several story tests that act as acceptance criteria for the story. The story tests can be combined into a regression suite for the software and provide traceability from the requirements (user stories) to tests and (through execution) to the behavior of the system. Story tests are usually BroadStackTests.

User stories are popular because they offer a simple workflow, each story adds new tests to the story test suite. However story tests often lead to problems. Regularly adding story tests leads to a large body of tests, often with significant duplication between them. When behavior needs to change in later iterations of the project, duplication in tests can take a painful amount of time to update. Furthermore broad-stack story tests take a long time to execute, which is why having a lot of them violates the TestPyramid. As a result many people recommend using just a few UserJourneyTests together with business-facing ComponentTests.

tags

API design · academia · agile · agile adoption · agile history · analysis patterns · application architecture · application integration · bad things · big data · build scripting · certification · clean code · collaboration · conferences · continuous integration · data analytics · database · delivery · dictionary · diversions · diversity · documentation · domain driven design · domain specific language · domestic · encapsulation · estimation · evolutionary design · expositional architectures · extreme programming · gadgets · infodecks · internet culture · language feature · lean · legacy rehab · legal · metrics · microsoft · mobile · noSQL · object collaboration design · parser generators · photography · presentations · process theory · productivity · programming platforms · project planning · projects · recruiting · refactoring · refactoring boundary · requirements analysis · retrospective · ruby · scrum · software craftsmanship · team environment · team organization · technical debt · technical leadership · testing · thoughtworks · tools · travel · uml · version control · web · website · writing

All Content

translations

Japanese · Spanish · Korean · Chinese · Thai

blogroll

ThoughtBlogs

TW Alumni · Tim Anderson · Kent Beck · Tim Bray · Paul Duvall · Elizabeth Hendrickson · Jeremy Miller · Ralph Johnson · Dan Pritchett · xkcd