Recent Changes

Here is a list of recent updates the site. You can also get this information as an RSS feed and I announce new articles on and Mastodon, X (Twitter), and LinkedIn.

I use this page to list both new articles and additions to existing articles. Since I often publish articles in installments, many entries on this page will be new installments to recently published articles, such announcements are indented and don’t show up in the recent changes sections of my home page.

Governing data products using fitness functions

Thu 05 Sep 2024 09:37 EDT

Decentralized data management requires automation to scale governance effectively. Fitness functions are a powerful automated governance technique my colleagues have applied to data products within the context of a Data Mesh. Since data products serve as the foundational building blocks of a data strategy, ensuring robust governance around them significantly increases the chances of success. Kiran Prakash explains how to do this, starting with simple tests for key architectural characteristics and moving on to leveraging metadata and Large Language Models.

Bliki: Cycle Time

Wed 04 Sep 2024 00:00 EDT

Cycle Time is a measure of how long it takes to get a new feature in a software system from idea to running in production. In Agile circles, we try to minimize cycle time. We do this by defining and implementing very small features and minimizing delays in the development process. Although the rough notion of cycle time, and the importance of reducing it, is common, there is a lot of variations on how cycle time is measured.

A key characteristic of agile software development is a shift from a Waterfall Process, where work is decomposed based on activity (analysis, coding, testing) to an Iterative Process where work is based on a subset of functionality (simple pricing, bulk discount, valued-customer discount). Doing this generates a feedback loop where we can learn from putting small features in front of users. This learning allows us to improve our development process and allows us to better understand where the software product can provide value for our customers. 1

1: It also avoids work being stuck in late activities such as testing and integration, which were notoriously difficult to estimate.

This feedback is a core benefit of an iterative approach, and like most such feedback loops, the quicker I get the feedback, the happier I am. Thus agile folks put a lot of emphasis on how fast we can get a feature through the entire workflow and into production. The phrase cycle time is a measure of that.

But here we run into difficulties. When do we start and stop the clock on cycle time?

The stopping time is the easiest, most glibly it's when the feature is put into production and helping its users. But there are circumstances where this can get muddy. If a team is using a Canary Release, should it be when used by the first cohort, or only when released to the full population? Do we count only when the app store has approved its release, thus adding an unpredictable delay that's mostly outside the control of the development team?.

The start time has even more variations. A common marker is when a developer makes a first commit to that feature, but that ignores any time spent in preparatory analysis. Many people would go further back and say: “when the customer first has the idea for a feature”. This is all very well for a high priority feature, but how about something that isn't that urgent, and thus sits in a triage area for a few weeks before being ready to enter development. Do we start the clock when the team first places the feature on the card wall and we start to seriously work on it?

I also run into the phase lead time, sometimes instead of “cycle time”, but often together - where people make a distinction between the two, often based on a different start time. However there isn't any consistency between how people distinguish between them. So in general, I treat “lead time” as a synonym to “cycle time”, and if someone is using both, I make sure I understand how that individual is making the distinction.

The different bands of cycle time all have their advantages, and it's often handy to use different bands in the same situation, to highlight differences. In that situation, I'd use a distinguishing adjective (e.g. “first-commit cycle time” vs “idea cycle time”) to tell them apart. There's no generally accepted terms for such adjectives, but I think they are better than trying to create a distinction between “cycle time” and “lead time”.

What these questions tell us is that cycle time, while a useful concept, is inherently slippery. We should be wary of comparing cycle times between teams, unless we can be confident we have consistent notions of their stop and start times.

But despite this, thinking in terms of cycle time, and trying to minimize it, is a useful activity. It's usually worthwhile to build a value stream map that shows every step from idea to production, identifying the steps in the work flow, how much time is spent on them, and how much waiting between them. Understanding this flow of work allows us to find ways to reduce the cycle time. Two commonly effective interventions are to reduce the size of features and (counter-intuitively) increase Slack. Doing the work to understand flow to improve it is worthwhile because the faster we get ideas into production, the more rapidly we gain the benefits of the new features, and get the feedback to learn and improve our ways of working.

Further Reading

The best grounding on understanding cycle time and how to reduce it is The Principles of Product Development Flow

Notes

1: It also avoids work being stuck in late activities such as testing and integration, which were notoriously difficult to estimate.

Acknowledgements

Andrew Harmel-Law, Chris Ford, James Lewis, José Pinar, Kief Morris, Manoj Kumar M, Matteo Vaccari, and Rafael Ferreira discussed this post on our internal mailing list

Rewriting Strangler Fig

Thu 22 Aug 2024 11:51 EDT

Two decades ago, I posted that I found that the strangler fig plant was an interesting metaphor for the gradual replacement of a legacy system. I didn’t refer to the metaphor since, but meanwhile it grew a life of its own. Other people increasingly referred to the strangler fig approach to modernization, and traffic to that post steadily increased: currently it gets about 5000 page views a month, one of the more popular pages on this site. So I decided I needed to update that page, and have rewritten it, focusing on the core activities we need to do to make a success of such a venture.

(This summarizes more detailed writing from Ian Cartwright, Rob Horn, and James Lewis.)

Onboarding to a “legacy” codebase with the help of AI

Thu 15 Aug 2024 10:32 EDT

Much of the attention to generative AI in software development is about generating code. But it may have a more useful role in helping us understand existing code. This is especially true for older codebases that are getting hard to maintain (“legacy”) or to improve onboarding in teams that have a lot of fluctuation.

Birgitta Böckeler demonstrates the possibilities here by picking an issue from an open-source hospital management system and exploring how AI could help her deal with it.

Refresh of the PoEAA catalog page

Wed 31 Jul 2024 16:11 EDT

From time to time I take a look at my site analytics to see how much traffic various bits of this site get. When doing this I saw that I continue to get a lot of traffic to the Catalog of Patterns of Enterprise Application Architecture. I put this together not long after writing the book, and it’s rather minimal. Since it still gets traffic I felt it was time for some sprucing up. The content is the same, summaries of the patterns in the book. I’ve changed the main catalog page to include the pattern intents. I’ve also added deep links to the book on oreilly.com, so if you have a subscription to that, it will jump directly to that text. Hopefully these changes will make the catalog page a bit more useful.

Instead of restricting AI and algorithms, make them explainable

Tue 30 Jul 2024 12:10 EDT

There's a lot of discussion about using regulation to restrict the use of AI and other software algorithms. I think that the better regulation would be to ensure that decisions made by software must be explainable.

Testing server calls in generated HTML

Wed 05 Jun 2024 15:00

Matteo Vaccari completes his article on testing template-generated HTML, by looking at how to use TDD with pages that make calls to the server.

Testing the behavior of generated HTML

Thu 30 May 2024 08:42 EDT

In the story so far, Matteo Vaccari has shown how to test the behaviour of the HTML templates, by checking the structure of the generated HTML. That's good, but what if we want to test the behavior of the HTML itself, plus any CSS and JavaScript it may use?

Parameterizing HTML template tests

Wed 29 May 2024 10:44 EDT

Testing templates for generating HTML leads to tests that are very similar. Matteo Vaccari wisely likes to separate the common elements of tests from those that vary. He continues his article to show how he does this by parameterizing the tests. The resulting tests are easier to write, and more importantly, faster to understand and modify.

Prefetching in Single-Page Applications

Wed 29 May 2024 10:24 EDT

Juntao Qiu's completes his set of data fetching patterns for single-page applications. Prefetching involves fetching data before it's called for in the application flow. Although this can mean data is fetched unnecessarily, it reduces latency should the data be needed.

Code Splitting in Single-Page Applications

Thu 23 May 2024 10:26 EDT

Single-Page Applications often require a lot of code to be downloaded to the browser, which can delay a page's initial appearance. Juntao Qiu's next pattern, Code Splitting, describes how this code can be divided up, so that modules are only loaded if they are going to be needed, and the dangers of doing so.

A short note on how I use and render footnotes

Wed 22 May 2024 14:17 EDT

Last week I added a small feature to this website, changing the way it renders footnotes. That prompted me to write this quick note about how I use footnotes, and how that influences the best way to render them.

Testing the contents of generated HTML

Wed 22 May 2024 10:14 EDT

Matteo Vaccari continues his testing of template-generated HTML by describing tests for the contents of that HTML. He shows how to gradually build up the template, using Test-Driven Development in Go and Java.

Using markup for fallbacks when fetching data

Tue 21 May 2024 11:30 EDT

Juntao Qiu's next data fetching pattern looks at how to specify fallback behavior using markup. This allows developers to pull such declarations out of the JavaScript components and into the markup they use while laying out the rest of the page. Juntao's React example shows how this works with the Suspense element, with a similar approach in vue.js.

Test-Driving HTML Templates

Tue 21 May 2024 11:06 EDT

When building a server-side rendered web application, it's valuable to test the HTML that's generated through templates. While these can be tested through end-to-end tests running in the browser, such tests are slow and more work to maintain than unit tests. My colleague Matteo Vaccari has written an article on how to use TDD to test drive these templates using xunit-style tools which can be run easily from the command line or as part of build scripts.

In this first installment Matteo describes how such tests can check the generated HTML for validity, with examples in Java and Go.

Parallel Data Fetching

Wed 15 May 2024 10:47 EDT

The second pattern in Juntao Qiu's series on data fetching is on how to avoid the dreaded Request Waterfall. Parallel Data Fetching queries multiple sources in parallel, so that the latency of the overall fetch is largest of the queries rather than the sum of them.

Data Fetching Patterns in Single-Page Applications

Tue 14 May 2024 09:49 EDT

Juntao Qiu is a thoughtful front-end developer experienced with the React programming environment. He's contributed a couple of useful articles to this site, describing helpful patterns for front-end programming. In this article he describes patterns for how single-page applications fetch data. This first installment describes how asynchronous queries can be wrapped in a handler to provide information about the state of the query.

photostream 131

Sun 12 May 2024 17:39 EDT

photostream 130

Sun 21 Apr 2024 18:21 EDT

Using data replication in legacy displacement

Wed 10 Apr 2024 11:00 EDT

Alessio Ferri and Tom Coggrave complete their article about introducing seams into mainframe systems by looking how we can use data replication. Done well, it can provide a rapid start to a displacement effort, but we must be wary of it coupling new systems to the legacy schema

Creating Seams in a Mainframe's Batch Pipelines

Thu 04 Apr 2024 09:39 EDT

Mainframe processing is often organized into pipelines of batch processes consuming and creating data files. Alessio Ferri and Tom Coggrave show a couple of ways they found to introduce seams into these pipelines, so that processing could be replaced by steps in a replacement platform.

Uncovering Seams in a Mainframe's external interfaces

Tue 02 Apr 2024 09:46 EDT

Alessio Ferri and Tom Coggrave move on to analyzing the internal seams of the application by treating the database as a coarse-grained seam. They describe how they introduced seams into the database readers and writers.

Joining LinkedIn

Thu 28 Mar 2024 12:26 EDT

As the enmuskification of Twitter continues, I’ve increasingly heard that more people are using LinkedIn to keep up with new professional material. So, a couple of weeks ago, I set up my LinkedIn account, so people can follow me on that platform.

I’ve always avoided LinkedIn - I’ve found the whole vibe of connections rather off-putting. I get too much spam from people wanting to connect as it is. But LinkedIn has added a “creator mode”, which encourages people to follow someone for posts rather than the bi-directional connection. It seems to be working reasonably well so far, so I’ve decided that I shall post all updates here to that account too. However I’m still avoiding connections, so please don’t send me a connection request unless we’ve done some substantial work together.

I’m still posting to X (Twitter), but as it steadily deteriorates, it gets less of my attention. If you follow me there, I recommend switching to one of my other feeds if you can.

I don’t particularly like LinkedIn for its feed, as I have little control over it. I never like the site suggestions, preferring as simple feeds of people. Sadly, LinkedIn doesn’t have lists, and pushes everything someone likes into the single connection feed. This leads to Too Many Posts which means I don’t pay attention to much.

Farewell, John Kordyback

Wed 27 Mar 2024 12:48 EDT

John Kordyback, a treasured colleague and friend, died last week, aged 64.

Uncovering Seams in a Mainframe's external interfaces

Wed 27 Mar 2024 10:36 EDT

Alessio Ferri and Tom Coggrave start detailing the seams they explored with two areas of external interfaces. Batch input of files are copied to new implementations while comparing the output of the processing pipelines. API access points can be covered with a proxy and traffic gradually directed to the new implementation.

Uncovering the seams in Mainframes for Incremental Modernisation

Tue 26 Mar 2024 09:32 EDT

Mainframe systems continue to run much of the world's computing workload, but it's often difficult to add new features to support growing business needs. Furthermore the architectural challenges that make them slow to enhance also make them hard to replace. To reduce the risk involved, we use an incremental approach to legacy displacement, gradually replacing legacy capabilities with implementations in modern technology. This strategy requires us to introduce seams into the mainframe system: points in which we could divert logic flow into newer services. In a recent project Alessio Ferri and Tom Coggrave investigated several approaches to introduce these seams into a long-lived mainframe system.

How to capture qualitative metrics

Tue 19 Mar 2024 09:33 EDT

Abi Noda and Tim Cochran complete their article on qualitative metrics by outlining how to capture them effectively. They discuss the mental steps that people go through as they respond to a survey and provide a template to get started when assessing developer experience. A final section looks at how qualitative and quantitative work together: often by starting with qualitative metrics to establish baselines and determine where to focus, followed with quantitative metrics to drill deeper into specific areas.

Code samples for the opening chapter of Refactoring

Fri 15 Mar 2024 11:04 EDT

From time to time people ask me for a copy of the code I used in the opening chapter of Refactoring, so they can follow along themselves. I had Reasons for not providing this code, specifically laziness. Fortunately Emily Bache is more dedicated, and she has set up a github repository - the Theatrical Players Refactoring Kata - with the code, and enough tests to make it reasonable to do the refactoring.

The repository goes further than this, however, in that it includes similar sample code in a dozen languages, including C, Java, Rust, and Python.

She has recently posted a video to her YouTube channel, which outlines why she encourages folks to use this code while they are reading that chapter. Her channel includes a lot of videos on good code technique, and she has a Patreon for readers to support her work.

The Benefits of Qualitative Metrics

Wed 13 Mar 2024 10:36 EDT

Abi Noda and Tim Cochran continue their discussion on using qualitative metrics to assess the productivity of development teams. In this installment they classify qualitative metrics into attitudinal and behavioral metrics. We also see that qualitative metrics allow you to measure things that are otherwise unmeasurable, provide missing visibility, and supply necessary context for quantitative data.

Measuring Developer Productivity via Humans

Tue 12 Mar 2024 09:36 EDT

Measuring developer productivity is a difficult challenge. Conventional metrics focused on development cycle time and throughput are limited, and there aren't obvious answers for where else to turn. Qualitative metrics offer a powerful way to measure and understand developer productivity using data derived from developers themselves. Abi Noda and Tim Cochran begin their discussion by explaining what a qualitative metric is and why we shouldn't reject them for being subjective or unreliable.