Bottleneck #05: Resilience and Observability

Use multiple zones within a region

For highly critical services (and their data), configure and enable them to run across multiple zones. This should give a bump to your system availability, and increase your resiliency in the case of disruption (within a zone).

Specify appropriate computing instance types and specifications

Business critical services should have computing capacity appropriately assigned to them. If services are required to run 24/7, your infrastructure should reflect those requirements.

Match investment to critical service tiers

Many organizations manage investment by identifying critical service tiers, with the understanding that not all business systems share the same importance in terms of delivering customer experience and supporting revenue. Identifying service tiers and associated resilience outcomes informed by service level agreements (SLAs), paired with architecture and design patterns that support the outcomes, provides helpful guardrails and governance for your product development teams.

Clearly define owners across your entire system

Each service that exists within your system should have well-defined owners. This information can be used to help direct issues to the right place, and to people who can effectively resolve them. Implementing a developer portal which provides a software services catalog with clearly defined team ownership helps with internal communication patterns.

Automate manual resilience processes (within a timebox)

Certain resilience problems that have been solved by hand can be automated: actions like restarting a service, adding new instances or restoring database backups. Many actions are easily automated or simply require a configuration change within your cloud service provider. While in the bottleneck, implementing these capabilities can give the team the relief it needs, providing much needed breathing room and time to solve the root cause(s).

Make sure to keep these implementations at their simplest and timeboxed (couple of days at max). Bear in mind these started out as bandaids, and automating them is just another (albeit better) type of bandaid. Integrate these into your monitoring solution, allowing you to remain aware of how frequently your system is automatically recovering and how long it takes. At the same time, these metrics allow you to prioritize moving away from reliance on these bandaid solutions and make your whole system more robust.

Centralize your logs to be viewable through a single interface

Aggregate logs from core services and systems to be available through a central interface. This will keep them accessible to multiple eyes easily and reduce troubleshooting efforts (potentially improving mean time to recovery).

Define a clear structured format for log messages

Anyone who’s had to parse through aggregated log messages can tell you that when multiple services follow differing log structures it’s an incredible mess to find anything. Every service just ends up speaking its own language, and only the original authors understand the logs. Ideally, once those logs are aggregated, anyone from developers to support teams should be able to understand the logs, no matter their origin.

Structure the log messages using an organization-wide standardized format. Most logging tools support a JSON format as a standard, which enables the log message structure to contain metadata like timestamp, severity, service and/or correlation-id. And with log management services (through an observability platform), one can filter and search across these properties to help debug bottleneck issues. To help make search more efficient, prefer fewer log messages with more fields containing pertinent information over many messages with a small number of fields. The actual messages themselves may still be unique to a specific service, but the attributes associated with the log message are helpful to everyone.

Treat your log messages as a key piece of information that is visible to more than just the developers that wrote them. Your support team can become more effective when debugging initial customer queries, because they can understand the structure they are viewing. If every service can speak the same language, the barrier to provide support and debugging assistance is removed.

Add observability that’s close to your customer experience

What gets measured gets managed.

-- Peter Drucker

Though infrastructure metrics and service message logs are useful, they are fairly low level and don’t provide any context of the actual customer experience. On the other hand, customer notifications are a direct indication of an issue, but they are usually anecdotal and don’t provide much in terms of pattern (unless you put in the work to find one).

Monitoring core business metrics enables teams to observe a customer’s experience. Typically defined through the product’s requirements and features, they provide high level context around many customer experiences. These are metrics like completed transactions, start and stop rate of a video, API usage or response time metrics. Implicit metrics are also useful in measuring a customer’s experiences, like frontend load time or search response time. It's crucial to match what is being observed directly to how a customer is experiencing your product. Also important to note, metrics aligned to the customer experience become even more important in a B2B environment, where you might not have the volume of data points necessary to be aware of customer issues when only measuring individual components of a system.

At one client, services started to publish domain events that were related to the product experience: events like added to cart, failed to add to cart, transaction completed, payment approved, etc. These events could then be picked up by an observability platform (like Splunk, ELK or Datadog) and displayed on a dashboard, categorized and analyzed even further. Errors could be captured and categorized, allowing better problem solving on errors related to unexpected customer experience.

Figure 2: Example of what a dashboard focusing on the user experience could look like

Data gathered through core business metrics can help you understand not only what might be failing, but where your system thresholds are and how it manages when it’s outside of that. This gives further insight into how you might get through the bottleneck.

Provide product status insight to customers using status indicators

It can be difficult to manage incoming customer inquiries of different issues they are facing, with support services quickly finding they are fighting fire after fire. Managing issue volume can be crucial to a startup's success, but within the bottleneck, you need to look for systemic ways of reducing that traffic. The ability to divert call traffic away from support will give some breathing room and a better chance to solve the right problem.

Service status indicators can provide customers the information they are seeking without having to reach out to support. This could come in the form of public dashboards, email messages, or even tweets. These can leverage backend service health and readiness checks, or a combination of metrics to determine service availability, degradation, and outages. During times of incidents, status indicators can provide a way of updating many customers at once about your product’s status.

Building trust with your customers is just as important as creating a reliable and resilient service. Providing methods for customers to understand the services' status and expected resolution timeframe helps build confidence through transparency, while also giving the support staff the space to problem-solve.

Understand the costs of service failure

For a startup, the consequences of not hitting a revenue target this 'quarter' might be different than for a scaleup or a mature product. But as often happens, the initial "new features are more valuable than technical debt" decision becomes a permanent fixture in the organizational culture – whether the actual revenue impact is provable or not; or even calculated. An aspect of the maturity needed when moving from startup to scaleup is in the data-driven element of decision-making. Is the organization tracking the value of every new feature shipped? And is the organization analyzing the operational investments as contributing to new revenue rather than just a cost-center? And are the costs of an outage or recurring outages known both in terms of wasted internal labor hours as well as lost revenue? As a startup, in most of these regards, you've got nothing to lose. But this is not true as you grow.

Therefore, it’s important to start analyzing the costs of service failures as part of your overall product management and revenue recognition value stream. Understanding your revenue “velocity” will provide an easy way to quantify the direct cost-per-minute of downtime. Tracking the costs to the team for everyone involved in an outage incident, from customer support calls to developers to management to public relations/marketing and even to sales, can be an eye-opening experience. Add on the opportunity costs of dealing with an outage rather than expanding customer outreach or delivering new features and the true scope and impact of failures in resilience become apparent.

Manage resilience as a feature

Start treating resilience as more than just a technical expectation. It’s a core feature that customers will come to expect. And because they expect it, it should become a first class consideration among other features. Part of this evolution is about shifting where the responsibility lies. Instead of it being purely a responsibility for tech, it’s one for product and the business. Multiple layers within the organization will need to consider resilience a priority. This demonstrates that resilience gets the same amount of attention that any other feature would get.

Close collaboration between the product and technology is vital to make sure you're able to set the correct expectations across story definition, implementation and communication to other parts of the organization. Resilience, though a core feature, is still invisible to the customer (unlike new features like additions to a UI or API). These two groups need to collaborate to ensure resilience is prioritized appropriately and implemented effectively.

The objective here is shifting resilience from being a reactionary concern to a proactive one. And if your teams are able to be proactive, you can also react more appropriately when something significant is happening to your business.

Requirements should reflect realistic expectations

Knowing realistic expectations for resilience relative to requirements and customer expectations is key to keeping your engineering efforts cost effective. Different levels of resilience, as measured by uptime and availability, have vastly different costs. The cost difference between “three nines” and “four nines” of availability (99.9% vs 99.99%) may be a factor of 10x.

It’s important to understand your customer requirements for each business capability. Do you and your customers expect a 24x7x365 experience? Where are your customers based? Are they local to a specific region or are they global? Are they primarily consuming your service via mobile devices, or are your customers integrated via your public API? For example, it is an ineffective use of capital to provide 99.999% uptime on a service delivered via mobile devices which only enjoy 99.9% uptime due to cell phone reliability limits.

These are important questions to ask when thinking about resilience, because you don’t want to pay for the implementation of a level of resiliency that has no perceived customer value. They also help to set and manage expectations for the product being built, the team building and maintaining it, the folks in your organization selling it and the customers using it.

Feel out your problems first and avoid overengineering

If you’re solving resiliency problems by hand, your first instinct might be to just automate it. Why not, right? Though it can help, it's most effective when the implementation is time-boxed to a very short period (a couple of days at max). Spending more time will likely lead to overengineering in an area that was actually just a symptom. A large amount of time, energy and money will be invested into something that is just another bandaid and most likely is not sustainable, or even worse, causes its own set of second-order challenges.

Instead of going straight to a tactical solution, this is an opportunity to really feel out your problem: Where do the fault lines exist, what is your observability trying to tell you, and what design choices correlate to these failures. You may be able to discover those fault lines through stress, chaos or exploratory testing. Use this opportunity to your advantage to discover other system stress points and determine where you can get the largest value for your investment.

As your business grows and scales, it’s critical to re-evaluate past decisions. What made sense during the startup phase may not get you through the hypergrowth stages.

Leverage multiple techniques when gathering requirements

Gathering requirements for technically oriented features can be difficult. Product managers or business analysts who are not versed in the nomenclature of resilience can find it hard to understand. This often translates into vague requirements like “Make x service more resilient” or "100% uptime is our goal". The requirements you define are as important as the resulting implementations. There are many techniques that can help us gather those requirements.

Try running a pre-mortem before writing requirements. In this lightweight activity, individuals in different roles give their perspectives about what they think could fail, or what is failing. A pre-mortem provides valuable insights into how folks perceive potential causes of failure, and the related costs. The ensuing discussion helps prioritize things that need to be made resilient, before any failure occurs. At a minimum, you can create new test scenarios to further validate system resilience.

Another option is to write requirements alongside tech leads and architecture SMEs. The responsibility to create an effective resilient system is now shared amongst leaders on the team, and each can speak to different aspects of the design.

These two techniques show that requirements gathering for resilience features isn’t a single responsibility. It should be shared across different roles within a team. Throughout every technique you try, keep in mind who should be involved and the perspectives they bring.

Broadly look at your architecture and determine appropriate trade-offs

Either implicitly or explicitly, when the initial architecture was created, trade-offs were made. During the experimentation and gaining traction phases of a startup, there is a high degree of focus on getting something to market quickly, keeping development costs low, and being able to easily modify or pivot product direction. The trade-off is sacrificing the benefits of resilience that would come from your ideal architecture.

Take an API backed by Functions as a Service (FaaS). This approach is a great way to create something with little to no management of the infrastructure it runs on, potentially ticking all three boxes of our focus area. On the other hand, it's limited based on the infrastructure it’s allowed to run on, timing constraints of the service and the potential communication complexity between many different functions. Though not unachievable, the constraints of the architecture may make it difficult or complex to achieve the resilience your product needs.

As the product and organization grows and matures, its constraints also evolve. It’s important to acknowledge that early design decisions may no longer be appropriate to the current operating environment, and consequently new architectures and technologies need to be introduced. If not addressed, the trade-offs made early on will only amplify the bottleneck within the hypergrowth phase.

Enhance resilience with effective error recovery strategies

Data gathered from monitors can show where high failure rates are coming from, be it third-party integrations, backed-up queues, backoffs or others. This data can drive decisions on what are appropriate recovery strategies to implement.

Use caching where appropriate

When retrieving information, caching strategies can help in two ways. Primarily, they can be used to reduce the load on the service by providing cached results for the same queries. Caching can also be used as the fallback response when a backend service fails to return successfully.

The trade-off is potentially serving stale data to customers, so ensure that your use case is not sensitive to stale data. For example, you wouldn’t want to use cached results for real-time stock price queries.

Use default responses where appropriate

As an alternative to caching, which provides the last known response for a query, it is possible to provide a static default value when the backend service fails to return successfully. For example, providing retail pricing as the fallback response for a pricing discount service will do no harm if it is better to risk losing a sale rather than risk losing money on a transaction.

Use retry strategies for mutation requests

Where a client is calling a service to effect a change in the data, the use case may require a successful request before proceeding. In this case, retrying the call may be appropriate in order to minimize how often error management processes need to be employed.

There are some important trade-offs to consider. Retries without delays risk causing a storm of requests which bring the whole system down under the load. Using an exponential backoff delay mitigates the risk of traffic load, but instead ties up connection sockets waiting for a long-running request, which causes a different set of failures.

Use idempotency to simplify error recovery

Clients implementing any type of retry strategy will potentially generate multiple identical requests. Ensure the service can handle multiple identical mutation requests, and can also handle resuming a multi-step workflow from the point of failure.

Design business appropriate failure modes

In a system, failure is a given and your goal is to protect the end user experience as much as possible. Specifically in cases that are supported by downstream services, you may be able to anticipate failures (through observability) and provide an alternative flow. Your underlying services that leverage these integrations can be designed with business appropriate failure modes.

Consider an ecommerce system supported by a microservice architecture. Should downstream services supporting the ordering function become overwhelmed, it would be more appropriate to temporarily disable the order button and present a limited error message to a customer. While this provides clear feedback to the user, Product Managers concerned with sales conversions might instead allow for orders to be captured and alert the customer to a delay in order confirmation.

Failure modes should be embedded into upstream systems, so as to ensure business continuity and customer satisfaction. Depending on your architecture, this might involve your CDN or API gateway returning cached responses if requests are overloading your subsystems. Or as described above, your system might provide for an alternative path to eventual consistency for specific failure modes. This is a far more effective and customer focused approach than the presentation of a generic error page that conveys ‘something has gone wrong’.

Resolve single points of failure

A single service can easily go from managing a single responsibility of the product to multiple. For a startup, appending to an existing service is often the simplest approach, as the infrastructure and deployment path is already solved. However, services can easily bloat and become a monolith, creating a point of failure that can bring down many or all parts of the product. In cases like this, you'll need to understand ways to split up the architecture, while also keeping the product as a whole functional.

At a fintech client, during a hyper-growth period, load on their monolithic system would spike wildly. Due to the monolithic nature, all of the functions were brought down simultaneously, resulting in lost revenue and unhappy customers. The long-term solution was to start splitting the monolith into several separate services that could be scaled horizontally. In addition, they introduced event queues, so transactions were never lost.

Implementing a microservice approach is not a simple and straightforward task, and does take time and effort. Start by defining a domain that requires a resiliency boost, and extract it's capabilities piece by piece. Roll out the new service, adjust infrastructure configuration as needed (increase provisioned capacity, implement auto scaling, etc) and monitor it. Ensure that the user journey hasn’t been affected, and resilience as a whole has improved. Once stability is achieved, continue to iterate over each capability in the domain. As noted in the client example, this is also an opportunity to introduce architectural elements that help increase the general resilience of your system. Event queues, circuit breakers, bulkheads and anti-corruption layers are all useful architectural components that increase the overall reliability of the system.

Regularly chaos test to validate system resilience

Chaos engineering is the bedrock of truly resilient products. The core value is the ability to generate failure in ways that you might never think of. And while that chaos is creating failures, running through user scenarios at the same time helps to understand the user experience. This can provide confidence that your system can withstand unexpected chaos. At the same time, it identifies which user experiences are impacted by system failures, giving context on what to improve next.

Though you may feel more comfortable testing against a dev or QA environment, the value of chaos testing comes from production or production-like environments. The goal is to understand how resilient the system is in the face of chaos. Early environments are (usually) not provisioned with the same configurations found in production, thus will not provide the confidence needed. Running a test like this in production can be daunting, so make sure you have confidence in your ability to restore service. This means the entire system can be spun back up and data can be restored if needed, all through automation.

Start with small understandable scenarios that can give useful data. As you gain experience and confidence, consider using your load/performance tests to simulate users while you execute your chaos testing. Ensure teams and stakeholders are aware that an experiment is about to be run, so they are prepared to monitor (in case things go wrong). Frameworks like Litmus or Gremlin can provide structure to chaos engineering. As confidence and maturity in your resilience grows, you can start to run experiments where teams are not alerted beforehand.

Recruit specialists with knowledge of resilience at scale

Hiring generalists when building and delivering an initial product makes sense. Time and money are incredibly valuable, so having generalists provides the flexibility to ensure you can get out to market quickly and not eat away at the initial investment. However, the teams have taken on more than they can handle and as your product scales, what was once good enough is no longer the case. A slightly unstable system that made it to market will continue to get more unstable as you scale, because the skills required to manage it have overtaken the skills of the existing team. In the same vein as technical debt, this can be a slippery slope and if not addressed, the problem will continue to compound.

To sustain the resilience of your product, you’ll need to recruit for that expertise to focus on that capability. Experts bring in a fresh view on the system in place, along with their ability to identify gaps and areas for improvement. Their past experiences can have a two-fold effect on the team, providing much needed guidance in areas that sorely need it, and a further investment in the growth of your employees.

Always maintain or improve your reliability

In 2021, the State of Devops report expanded the fifth key metric from availability to reliability. Under operational performance, it asserts a product's ability to retain its promises. Resilience ties directly into this, as it’s a key business capability that can ensure your reliability. With many organizations pushing more frequently to production, there needs to be assurances that reliability remains the same or gets better.

With your observability and monitoring in place, ensure what it tells you matches what your service level objectives (SLOs) state. With every deployment to production, the monitors should not deviate from what your SLAs guarantee. Certain deployment structures, like blue/green or canary (to some extent), can help to validate the changes before being released to a wide audience. Running tests effectively in production can increase confidence that your agreements haven’t swayed and resilience has remained the same or better.