High-Water Mark

An index in the write-ahead log showing the last successful replication.

aka: Commit Index

Problem

The Write-Ahead Log pattern is used to recover state after a server crashes and restarts. But a write-ahead log is not enough to provide availability in case of server failure. If a single server fails, then clients won't be able to function until the server restarts. To get a more available system, we can replicate the log on multiple servers. Using Leader and Followers the leader replicates all its log entries to a Majority Quorum of followers. Now, should the leader fail, a new leader can be elected, and clients can mostly continue to work with the cluster as before. But there are still a couple things that can go wrong:

  • The leader can fail before sending its log entries to any followers.
  • The leader can fail after sending log entries to some followers, but before sending it to the majority of followers.

In these error scenarios, some followers can be missing entries in their logs, and some followers can have more entries than others. So it becomes important for each follower to know what part of the log is safe to be made available to the clients.

Solution

The high-water mark is an index into the log file that records the last log entry known to have successfully replicated to a Majority Quorum of followers. The leader also passes on the high-water mark to its followers during its replication. All servers in the cluster should only transmit to clients the data that reflects updates below the high-water mark.

for more details go to Chapter 10 of the online ebook at oreilly.com

This pattern is part of Patterns of Distributed Systems

23 November 2023