Single-Leader Replication: The Contract Behind Replica Lag and Failover

Data Systems

The usual description of leader-follower replication sounds reassuringly simple: send every write to one database node, then copy that change to the other nodes. It is a good starting point, but it hides the decisions that determine whether a user sees their own update, whether a confirmed write survives a failure, and whether recovery makes an outage worse.

In a single-leader system, every write goes to the leader. The leader records the change locally and streams it to followers, which apply the changes in the same order. Reads can come from the leader or from followers. That one ordered write path is the model's strength. The delayed copies are its source of tension.

This is not an infrastructure detail that stops at the API boundary. It becomes visible as a profile edit that vanishes after a refresh, a feed that appears to move backward, or a successful payment that cannot be found during an incident.

Acknowledgement Is a Durability Decision

The first question is not "how many replicas do we have?" It is "when may the system tell the caller that a write succeeded?"

With asynchronous replication, the leader confirms the write after its local commit and sends the change to followers in the background. This keeps write latency low and lets the leader continue even if a follower is slow or unavailable. It also creates a window in which a leader can fail after acknowledging a write that no follower has received.

With synchronous replication, the leader waits for a follower to confirm receipt before responding. If that follower is healthy, the acknowledged write exists in more than one place. The price is direct: a slow or disconnected synchronous follower can stop writes from completing.

Most systems avoid making every follower synchronous. A practical middle ground is semisynchronous replication: require one follower to acknowledge each write while other followers catch up asynchronously. That gives the system one up-to-date copy beyond the leader without turning every lagging replica into a write outage.

The product contract follows the acknowledgement policy:

  • asynchronous acknowledgement optimizes for availability and latency, while accepting a small acknowledged-data-loss window;
  • synchronous acknowledgement narrows that loss window, while making a replica's health part of the write path;
  • a quorum or consensus protocol may offer a stronger contract, but it makes the coordination cost explicit rather than eliminating it.

Calling a write "saved" without knowing which contract backs that word is how a distributed-system tradeoff leaks into product language.

Replica Lag Is a User-Experience Problem

A follower receiving changes asynchronously can be behind for milliseconds, seconds, or longer during recovery and overload. The word eventual does not name a useful deadline.

Suppose a user submits a comment, the write goes to the leader, and the next page load reads from a follower that has not applied the change yet. The comment looks lost even though the system is operating as designed. The remedy is a read-your-writes guarantee: after a user writes data, route reads that might include that data to the leader, a known-caught-up follower, or a replica that has reached the write's logical position.

Two related guarantees matter just as much:

  • Monotonic reads prevent a user from seeing a newer result and then an older one. Sticky routing by user or session is often enough until the chosen replica fails.
  • Consistent-prefix reads preserve causal order. If an application publishes a reply after a question, a reader should not see the reply first because the two writes reached replicas in a different order.

These are not claims that every reader sees the absolute latest value. They are narrower, intentional promises that preserve a coherent experience while keeping reads scalable.

For a JavaScript application, this means an optimistic client state is not a substitute for a backend consistency policy. Optimistic UI can make the immediate interaction responsive, but a reload, another device, or a server-rendered request still needs an answer to: which replica may serve this user, and what freshness must it have?

type FreshnessRequirement =
  | { kind: 'any-replica' }
  | { kind: 'read-your-writes'; minLogPosition: string }
  | { kind: 'leader-only' }

async function getProfile(userId: string, requirement: FreshnessRequirement) {
  return database.read({ userId, requirement })
}

The exact API will differ, but modeling freshness as a requirement is more honest than assuming every read is interchangeable.

Bringing a New Follower Online Is a Coordinated Catch-Up

Copying a live database directory to another machine is not enough. Writes continue while that copy is in progress, so different files can represent different points in time.

The reliable pattern is:

  1. Take a consistent snapshot of the leader at a known log position.
  2. Copy that snapshot to the new follower.
  3. Request the stream of changes that happened after the snapshot position.
  4. Mark the follower eligible for normal traffic only after it catches up.

The snapshot position is essential. It turns an arbitrary bulk copy into a base state plus an ordered suffix of changes. The same pattern makes backups useful for disaster recovery: a periodic snapshot gives you a starting point, and retained logs let a recovered node or a point-in-time restore move forward.

The format of the replication log affects what teams can do operationally. Shipping SQL statements is fragile when expressions are nondeterministic or side effects differ. Shipping a storage engine's write-ahead log tightly couples the leader and followers to the same storage format, which complicates zero-downtime upgrades. A logical, row-oriented change stream is more portable: it describes inserted, updated, and deleted records without exposing low-level page layout. That is why logical replication is often a better boundary for heterogeneous consumers and change-data-capture pipelines.

Failover Is the Moment the Contract Gets Expensive

When a follower fails, it can usually reconnect, request the missing log entries, and catch up. The harder case is a leader failure. A follower must be promoted, clients must send writes to it, and the old leader must return only as a follower.

Every part of that sequence has uncertainty:

  • A timeout cannot distinguish a dead leader from a slow leader or a damaged network path.
  • Promoting too aggressively can produce two leaders, each accepting writes.
  • Promoting a follower that was behind can discard acknowledged asynchronous writes.
  • An old leader that reappears with a stale view can overwrite or conflict with newer data unless it is fenced off.

This is why a sensible failover policy is not merely "switch quickly." It needs a failure detector with carefully chosen timeouts, a rule for selecting an up-to-date candidate, and a mechanism that prevents the previous leader from continuing to accept writes. Faster detection can shorten downtime, but false positives can turn a degraded system into a split-brain incident.

The Design Question to Ask Before Scaling Reads

Leader-follower replication is a strong default when the domain benefits from one ordered write history: account settings, billing records, inventory reservations, and many operational systems do. The architecture becomes dangerous only when teams treat follower reads as a free performance feature.

Before routing reads to replicas, decide:

  • Which screens may show slightly stale data?
  • Which user actions require read-your-writes behavior?
  • Where does causal order matter to comprehension or safety?
  • What does a successful write mean if the leader fails immediately afterward?
  • How will the system detect, promote, and fence a leader during an outage?

Those answers are the real replication design. The nodes and streams merely implement them.

Conclusion

Single-leader replication buys a valuable simplification: one authority orders writes. It does not remove distributed-systems choices. It relocates them to acknowledgement, routing, catch-up, and failover.

Treat those choices as part of the product contract. Then a follower read, an optimistic update, and an incident response all line up with what the system can honestly guarantee.