Two Stack CMS

We build many websites with rich content, often using popular Content Management Systems (CMS). A recent project involved a marketing website for a global manufacturer which demanded complex interactive content with high availability and traffic needs.

Our response was to apply the editing-publishing separation pattern and build two distinct stacks of software for content creation and delivery. In this deck you can see an overview of this architecture and our response to the issues of integration between the stacks, providing a secure preview of the live site, and handling the evolution and scaling of the system.

6 October 2014

This page is a fallback page for the proper infodeck.

There are couple of reasons why you are seeing this page

You are using an older browser that can’t display the infodeck. This probably due to a lack of support for SVG, most commonly due to an older version of Internet Explorer (pre 9.0). If this is the case please try this link again with a modern browser.
You may have navigated directly to this page - using a URL that ends with “fallback.html”. If so to read the full infodeck please go to https://martinfowler.com/articles/two-stack-cms

The following is dump of the text in the deck to help search engines perform indexing

We applied the editing-publishing separation pattern in building a two-stack CMS to support a global readership while supporting complex coordination of legacy editing tools

by Sunit Parekh and Martin FowlerBuilding a Two-Stack CMS for a global product catalog

Many thanks to all our friends and colleagues who were involved in building this system and helped with the review of the Infodeck.

2014-10-06

Sunit Parekh is an application developer from Thoughtworks who likes to work on complex business problems with modern technologies. He specializes in distributed agile development working from Pune, India.

Martin Fowler is an author and speaker who enjoys helping his colleagues write about their experiences in building important software systems.

Hints for using this deck

A call to build a marketing website for a global manufacturer had two big challengesComplex Content

The site needs to be fresh and engaging, requiring design-heavy page structure, lots of graphics and video, interactive pages and links to internal systems for product specifications.

Writers for the website use a wide range of tools, some purchased, some built in-house. We must support these tools and provide a way to coordinate the workflow between tools and the various groups involved.

An important part of the coordination problem is translation - the global site must support over twenty countries with nearly a hundred locales. Imagery, product availability and pricing all vary between countries.

Global Audience

A global marketing site serves heavy traffic: daily visitors in six figures and over two million daily page views.

The site needs high availability, with no downtime for the frequent updates it will receive.

Traditional CMS designs conflate two diverse needs

When editing, you have a few expert users who make frequent updates to the content.

When published, you have very many readers, but updates are rare.

We call this the Editing-Publishing Separation pattern.

So we used different stacks for content creation versus delivery

Editors use the Content Creation Stack (CCS) to develop content iteratively using specialized systems,

The Content Delivery Stack (CDS) handles delivery worldwide to readers of the web site and syndication to external platforms such as dealers.

Editors are specialists in particular kinds of content and use a range of custom and purchased systems that are designed specifically for that kind of content. For example photographers work with the Digital Asset Management system to acquire imagery required for website.

Publishing is controlled by a single transfer operation that copies content from the CCS to the CDS. This transfer operation is the only writer to the CDS system.

Availability and scalability requirements are very different for the two stacks. The CCS can be managed as a more traditional back-office set of applications, while the CDS needs to be highly scalable and available as there's no good time to be down with a global system.

We need a secure preview for the content we are going to publish

Business users and content authors need to review the content before it is published to the general public. However, with specialised CCS systems it is difficult to view the content in published-like state.

Putting all the content in CDS for preview means the CDS needs to be aware of varying contents for different user bases and be responsible for protecting it leaking out before it should.

Content needs to be reviewed and approved before publishing, with ability to go live at specified date and time.

We split the CDS into separate slices for preview and live

Both slices run the same codebase with different content in the database. Preview allows business users and content authors to review the site in a published-like state and approve it. Live serves the content to end-users.

Each system in CCS publishes content independently. Once published, content is available for preview immediately.

The preview slice is secured to only allow access for selected users.

Preview is also used as staging for new features and content showcase.

When ready, we publish the content to CDS-Live

The slices have different availability and scalability needs. Live is highly available and scalable for a large, global readership. Preview only caters to a few internal users and can tolerate downtime

Content metadata controls flow to preview and live

While working on the content, the authors publish from the CCS to the CDS-Preview. The CDS preview only has the current version of any content, any past history is kept by the CCS. When the authors are ready for review they mark the content's status as ready for review and send the link to the reviewers. When the reviewers approve of the content it's marked as ready for live.

All content is marked with a sunrise and sunset date. We only copy content from CDS-Preview to CDS-Live once we are past the sunrise date and the content is ready for live. Once the sunset date is passed that content is deleted from both CDS stacks.

We need to coordinate workflow for content creation and delivery

The tools used for content creation are independent tools, yet content creation has workflows that are particular to the host organization. An example of this is that some products need immediate publication of new text in one language while sending this text out for translation and publishing it when it's done.

The CCS tools often can't be modified to take into account our workflow, and even with in-house tools, it's better to keep the CCS tools decoupled from workflow needs.

We built an adapter layer to integrate CCS together and with CDS

The adapter layer is responsible for all communications between CCS systems and in transferring content to the CDS. We used the Apache Camel framework to set up an asynchronous communication flow.

To illustrate, we'll use the example workflow of product publishing.

STEP 1: Writers write a product description in the master locale (language) inside the Product Catalog system,. As needed they export using the product catalog's built-in export mechanism into the adapter layer for publishing to CDS Preview.

After publishing, the product description is immediately available for preview in master locale.

STEP 2: Once the master locale content is approved, the Product Catalog system exports the master content to the adapter layer with a list of locales for translation.

The adapter breaks down the content into items (a block of text) and send them to the Content Translation System.

STEP 3: The Content Translation System coordinates the work of translators to translate each item and return it to the adapter.

The adapter keeps track of all the split content and received translations. Once it's got all the translations it weaves them together and sends them back to the Product Catalog system

STEP 4: the Product Catalog system re-exports the product description (now with all its translations) to the adaptor layer for preview.

The adapter layer succeeded in that it allowed us to use existing tools for content creation, and allowed us to upgrade and replace these tools independently. However many of these components had severe limitations around localization, cross-system validation, and content composition. These led to lots of complexity in the adapter layer to handle all the workarounds.

Content delivery needs to evolve and scale

The Content Delivery Stack consists of independently evolvable components

The preview and live stack use the same components to ensure that the preview is an accurate reflection of the live system. The component architecture is designed around the more demanding live case. We want the components to be independently upgradable so we can easily add new features.

We provide two APIs for accessing the content. The underlying software is the same, but the two APIs offer different services and security characteristics.

Content Publishing Service API is a RESTful read/write API used for content uploading. It is accessible via HTTPS only and protected with authentication and authorization controls.

Digital Assets are uploaded to the Digital Asset Server. All other textual content is stored in the Aggregated Content Store.

The Aggregated Content Store is MongoDB NoSQL document database. A document store is an ideal choice to store schemaless editorial marketing content as well as multi-attribute product information.

The Content Delivery Service API is a RESTful, read-only API that extracts content from the Aggregated Content Store to serve it to the Presentation Front End over HTTP. It provides the content in JSON.

It is locale aware, providing only the content that's available in a specified locale.

It also syndicates presentation neutral (raw) content in JSON form to other systems such as product feed syndication to eCommerce and dealers like Amazon.

The presentation front end translates the JSON from the content delivery service into HTML pages. It composes the page using modules, which are capable of representing JSON data in different ways as needed by the UI designers. This transformation from JSON to HTML enables Separated Presentation.

It also is responsible for syndicating rich content to external systems.

The Digital Asset Server supplies images. It has the ability to perform on-the-fly image transformation which is needed to support responsive design and serving different sizes based on device resolution.

We use replication and caching on the live infrastructure to maximize availability and performance

Overall we need the Live slice on CDS to be both highly available and scalable. However the Publishing API does not need the same levels of availability and scalability - instead it has security needs such as authentication and authorizations with HTTPS.

Aggregated Content Store is three large servers running MongoDB replicas set to allow reads from any server.

Content Delivery Services and Presentation FrontEnd are setup to use Blue-Green Deployment for high availability and zero-downtime deployments for the global readership.

Zero-downtime deployments enable us to do continuous delivery. We release new features every 3 weeks to production. Features that take longer than a single release cycle use Feature Flags.

All requests from the Presentation FrontEnd to the Content Delivery Service are cached for a short duration. We use a load balancer in front of the APIs to cache the JSON responses with 15 min TTL (max-age).

We use CDN edge servers to cache HTTP responses at the page level to provide the best end-user performance. This keeps 92% of page responses within 1 second.

We use Scene7 SaaS from Adobe as Digital Asset Server for Images and Videos. Scene7 provides on-the-fly image transformations.

Some things we learned...Async communication is good for system characteristics but problematic for user-experience

We chose asynchronous communication for content publishing from CCS to CDS. It was a good design from system perspective, reducing coupling and allowing the CCS to be more responsive. However this led to difficulties with feedback to the content creators.

Sometimes it could take a couple of minutes for the content to be available in preview. Editors could not tell how long it would take. Furthermore there was no direct feedback to the editors in the CCS system about failures in the CDS.

In response to these problems we built a reporting API in the CDS that we used in the CCS to show editors what was happening. We marked content red when it was changed in CCS and not published due to a failure, yellow for content published but waiting to be visible in preview, and green for content now available in preview. The lesson for the future is to pay careful attention to the user-experience consequences of asynchronous communications.

Debugging and reporting in distributed systems is difficult

Since many systems were involved in content creation, it was difficult to report on progress for a big launch. Also it became difficult to debug any issues across systems, such as content waiting for translation.

We resolved this with structured logging using Splunk both to generate alerts for specific errors and to build progress reports for progress of big launches.

Two Stack CMS is a good choice when...We want to leverage existing systems

In almost every organisation specialized systems (Product Catalog, Media Store etc) already exist providing structured content to other systems such as an eCommerce system.

Using a Two Stack CMS, we can continue to use these specialized systems without duplicating the content. Keeping these systems allows us to use the capability of specialized content creation users to efficiently handle large volume of content such as thousands of images, hundreds of product specifications etc.

We have different cross-functional needs

The CCS supports a few editors making frequent updates, the Preview CDS supports a few readers with secure access, while the Live CDS supports high volumes of readers with relative little churn in content.

So when should I not use the Two Stack CMS architecture?

When content volume is low and only few trained people are responsible for content creation and maintenance, any added complexity of a two stack CMS isn't worthwhile. Such complexity makes it harder for content creators to learn how to use it and adds significant effort in system maintenance. (Although this may be mitigated by tools that are designed to work in this style.)