Replication: Leader-Follower and Multi-Leader

1The core idea#

Replication keeps the same data on more than one machine. You do it for three concrete reasons:

• Survive failure. A node dies and the data is still there on another box.
• Get closer to users. Put a copy near each region so reads are local and fast.
• Spread read load. Fan reads across many machines so a single box does not melt.

The mental model to hold onto: where are writes allowed to happen? That single decision drives everything else. Three answers, three families:

• Leader-follower (also primary-replica, or master-slave in older docs): one node accepts writes. Everyone else copies from it and serves reads only.
• Multi-leader: several nodes accept writes, usually one per region, and they sync changes to each other.
• Leaderless (Dynamo-style, Cassandra): no special writer, clients write to several nodes directly. Worth knowing it exists, but this guide is about the first two.

2How it works#

Leader-follower is a one-way street for writes. A client sends a write to the leader, the leader applies it, then streams the change to followers that replay it in the same order. Reads can go anywhere, writes only go to the leader.

The one knob that matters: when does the leader say "done"?

• Synchronous. The leader waits for at least one follower to confirm before acking. Safer, slower. A same-region sync replica turns a 2ms write into about 4ms (one extra round trip). A cross-region sync replica can turn it into 150ms.
• Asynchronous. The leader acks immediately and ships the change in the background. Fast, but if the leader dies before a write reaches a follower, that write is gone.
• Semi-synchronous (the usual real-world pick). One follower is synchronous, the rest are async. You get durability on at least two machines without waiting on all of them.

Here is the read/write split in a typical leader-follower setup:

Replication ships as an ordered log. This is the mechanism to name out loud:

• The leader writes every change to an ordered log (Postgres WAL, MySQL binlog).
• Followers pull that log and apply it in the same order.
• Ordering is what keeps a follower consistent with the leader.
• A follower tracks the log position it has applied. The gap between the leader's current position and the follower's applied position is exactly your replication lag.

Multi-leader looks different, and the difference is where the pain lives. You put a leader in each region so local writes are fast, then each leader forwards its changes to the others.

3A worked example with real numbers#

Interviewers can smell hand-waving, so make it concrete. Say we are sizing the database tier for a mid-size product.

The numbers to start from:

• 50 million registered users.
• Read-heavy: 20,000 reads/second at peak, 400 writes/second at peak. A 50 to 1 read/write ratio.
• A write takes about 3ms on the leader. A read takes about 2ms on a replica.
• One beefy Postgres box comfortably sustains around 5,000 queries/second before latency climbs.

Sizing the topology:

• One node facing 20,400 queries/second would crush a 5,000-qps box, so spread the load.
• Writes stay on one leader (400 writes/second is well under 5,000, so one leader is plenty).
• Reads needed: 20,000/second. Capacity per replica: roughly 5,000/second. So you need at least 4 read replicas, and I would run 5 for headroom against a dying replica or a spike.
• Final shape: 1 leader for all writes (plus the small share of reads that must be fresh) and 5 async followers soaking up the 20,000 reads/second.

The lag math, honestly:

• Async under normal load lags a few milliseconds, call it 5ms to 20ms. After a write, the change reaches a follower in about 10ms. Usually invisible.
• During a nightly batch job, lag spiked to 40 seconds because the follower was busy applying a flood of log entries.
• For those 40 seconds, any read hitting that follower returned stale data. That is the gap your design has to account for, and section 5 covers how.

Cost sanity check (because "how do I scale reads" has a price tag):

• 6 database instances (1 leader + 5 replicas) instead of 1.
• If each is a db.r6g.2xlarge on RDS at roughly 800 USD per month, that is about 4,800 USD per month for the tier.
• The honest answer to "how do I scale reads" is "throw replicas at it and pay for them," not magic.

4Trade-offs#

The decision table I keep in my head:

Leader-follower's two sore spots are failover and read-your-writes:

• When the leader dies, something has to notice, pick a follower, and promote it. During that window (often 10 to 30 seconds) you cannot write at all.
• Promote a follower that was behind by, say, 200ms of writes, and you lose every write in that 200ms window that never reached it.

Multi-leader's sore spot is conflicts. The moment two regions can write the same key, you own a distributed systems problem. Your options:

• Last write wins (LWW). Keep the write with the highest timestamp, drop the other. Simple, and it silently throws data away. Clocks across machines are never perfectly in sync (tens of milliseconds of skew is normal), so "last" is a lie.
• Application-level merge. On conflict, your code decides. A shopping cart unions both carts, a counter adds both increments. Works when the data has a natural merge rule.
• CRDTs. Data types designed to merge without conflict. Great for counters, sets, collaborative text. Not a fit for arbitrary business logic.

5Where it is used#

Leader-follower is the default for Postgres, MySQL, and the managed services built on them (RDS, Cloud SQL). Reach for it when:

• You have far more reads than writes (most apps: a product page gets read 1000x for every edit). Add read replicas and point the heavy read traffic at them.
• You want a hot standby for failover. One synchronous replica means you can lose the leader without losing data.
• Analytics queries are hammering your database. Run them against a follower so they do not slow down live traffic.

Multi-leader shows up when:

• Users span continents and write latency matters. A user in Singapore writing to a leader in Virginia eats a 200ms+ round trip on every write. A local leader in Singapore drops that to a few milliseconds.
• You have offline-capable clients. A mobile app that works on a plane is basically a leader on the device that syncs when it reconnects. Calendar apps and note apps work this way.
• Two datacenters both need to accept writes and keep running if the link between them drops.

Cousins worth naming:

• Google Docs style collaborative editing (many writers, conflict resolution via operational transforms or CRDTs).
• Cassandra and DynamoDB take the leaderless path, trading a defined ordering for always-on writes.

6Common pitfalls and gotchas#

Replication lag breaking read-your-writes.

• A user updates their profile, the write hits the leader, they reload, and the read goes to a follower 10ms behind (or 40 seconds behind during a batch job).
• They see old data and think it did not save.
• Fixes: route a user's reads to the leader for a few seconds after they write, or track the write log position and only read from a follower that has caught up to it.
• Interviewers love this one because it is so common and so easy to miss.

Assuming failover is free.

• Promoting a follower is not instant and not risk-free. If the old leader was ahead of the follower you promote, those writes vanish.
• Worse is split brain: the old leader comes back, still thinks it is the leader, and now you have two leaders taking writes.
• You need fencing (shoot the old leader in the head, literally the term STONITH) to prevent this.

Monotonic reads violations.

• A user reads from follower A (caught up), refreshes, hits follower B (behind), and sees data go backwards in time.
• Pin a given user to one follower to avoid this.

Treating multi-leader like it has no conflicts.

• Plenty of candidates draw two leaders and move on without ever saying the word "conflict."
• If you draw multiple writers, you must say how you resolve disagreements.

Replication lag under load.

• Lag is usually milliseconds, but a big batch write or a slow follower can push it to seconds or minutes (the 40-second spike from the example was exactly this).
• If your read-your-writes logic assumes lag is always tiny, it breaks exactly when the system is busiest.

7How it shows up in interviews#

Replication almost never gets asked head-on. It hides inside a bigger design:

• "Design a URL shortener," "design Twitter," "design a booking system."
• The moment you say "I'll use a database," a good interviewer asks how you scale reads, what happens when the database dies, and how you serve global users.
• That is the replication conversation, whether or not anyone uses the word.

What they are actually probing:

• Do you default to one leader and know why? (Ordering and no conflicts.)
• Can you scale reads with replicas without hand-waving the lag?
• Do you understand that failover costs availability and can lose data?
• If you propose multi-leader, do you immediately name the conflict problem?

Weak answer: "I'll replicate the database across three nodes for high availability."

• It says nothing. Replicate how? Who writes? What happens on a conflict or a failover?
• An interviewer hears this and starts poking, because you have given them no model to work with. It sounds like a buzzword, not a design.

Strong answer: "Reads dominate here, roughly 50 to 1, so I'll run a single leader for the 400 writes per second and put the 20,000 reads per second across 5 async replicas, with one spare for headroom. Replication is async, so there's lag of about 10ms normally, though it can spike to seconds during batch jobs. For the profile-edit flow where a user expects to see their own change, I'll read from the leader for a few seconds after a write, or track the write's log position and only read from a replica that has caught up to it. If the leader fails, we promote a replica, which means a 10 to 30 second write outage and possibly losing the last few hundred milliseconds of unreplicated writes, so for payment records I'd keep a synchronous replica to make that loss impossible even though it costs a round trip on every payment write."

Why that second answer wins, line by line:

• It names the read/write ratio (50 to 1) and turns it into a count (5 replicas).
• It states the normal lag (10ms) and the failure mode (seconds during batch).
• It names the exact case where lag bites (read-your-writes on profile edits) and gives two concrete fixes.
• It quantifies the failover cost (10 to 30 seconds, last few hundred ms of writes).
• It segments by data type: payments get a sync replica, everything else does not.
• Every claim is a number or a mechanism, not an adjective. That is a staff-level answer.

Knowing when NOT to add a second leader scores as high as knowing how to run one.

One more move that scores points: if the interviewer pushes global writes, talk about sharding by region before reaching for multi-leader. Each user's data lives in one region, so there are no cross-region conflicts to resolve. Showing you avoid complexity when you can is often stronger than showing you can handle it.

Want to see this applied end to end? Try the System Design Question walkthrough on designing a globally distributed social feed, where the read/write split and failover choices decide the whole architecture.