Leader Election

1The core idea#

Leader election is how a group of identical nodes picks exactly one of them to be in charge and keeps that choice stable while machines crash, restart, and lose the network.

Why bother at all.

• Coordinating work when everyone has equal say is miserable.
• Five replicas all accepting writes: two clients update the same row on two replicas in the same millisecond, and now you have a conflict with no clean resolution.
• Funnel all writes through one designated node (the leader) and have the rest follow.
• The leader serializes the work, so there is a single source of truth for ordering.

The mental model I give juniors.

• Leader election is "whose turn is it to hold the talking stick." Only the stick-holder may speak.
• Handing out the stick is easy.
• The hard part is agreeing the stick-holder dropped it, when you cannot see their hands.

Three properties that matter:

• Safety: at most one leader at a time. Two leaders (a "split brain") is the thing that corrupts data.
• Liveness: eventually someone gets elected, so the system does not freeze forever.
• Stability: the leader should not flap. Re-electing every few seconds is almost as bad as having none.

2How it works#

Nearly every practical scheme is the same pattern: a leader proves it is alive by renewing a short-lived lease, and if the lease expires, someone else may grab it.

A lease is a lock with a deadline.

• The leader holds it for, say, 10 seconds and must renew before expiry.
• Stop renewing (crashed, frozen, partitioned away) and the lease lapses.
• A follower then claims it.

You almost never build this by hand.

• Lean on a coordination service that already solved consensus: ZooKeeper, etcd, or Consul.
• These run a consensus protocol (ZAB for ZooKeeper, Raft for etcd and Consul) across a small cluster of 3 or 5 nodes.
• Your application nodes do not run consensus themselves. They ask the service to hold the lock.
• The service's own majority quorum guarantees only one holder.

A few things worth calling out:

• Node 2 is not polling frantically. Most clients register a watch and get notified the instant the lease frees up.
• The renewal interval beats the TTL on purpose, so a single slow renewal is survivable.
• The crash-to-new-leader gap is about one TTL. Unavoidable with this design.

Why the coordination service itself does not get two leaders.

• It uses quorums. To grant the lease, a majority of its 3 or 5 nodes must agree.
• Two different majorities of the same cluster must overlap in at least one node.
• That shared node refuses to say yes twice.
• Majority overlap is the math trick underneath basically all of this.

The raw algorithms worth naming: Raft and Paxos.

• Raft bakes leader election right into the protocol.
• Nodes sit idle until a randomized election timeout fires (typically 150ms to 300ms), then bump a term number and ask peers for votes.
• Win a majority, you lead.
• The randomized timeout is the clever bit: it makes two candidates starting at the exact same instant unlikely, so split votes are rare.

A worked example with real numbers#

"Roughly one TTL" hides a lot, so let me make it concrete.

The setup:

• A nightly billing batch across 5 app servers, elect one to run it.
• An etcd cluster of 3 nodes, a lease TTL of 10 seconds, a renewal every 3 seconds.

The failover math:

• The leader renews at t=0, t=3, t=6, t=9, each resetting the 10-second countdown.
• Crash at t=10 (one second after the t=9 renewal) means the lease, pushed out to t=19, expires 9 seconds later.
• Worst case is close to a full TTL (10s). Best case is near zero if the crash lands right as the lease was about to lapse.
• Average is about half a TTL plus notification latency: call it 5s to 6s here.

What a tighter SLA costs:

• A billing SLA of "start within 30 seconds of schedule" leaves comfortable headroom with a 10s TTL.
• Asked for 2s failover? TTL drops to 2s, renewals jump from every 3s to maybe every 600ms (roughly 5x the renewal traffic against etcd).
• And you are now one 2.5s GC pause away from a false failover.

3Trade-offs#

The decisions you actually make in a design review:

Lease TTL length.

• Short TTL: fast failover, but more renewal traffic and more risk of a false failover during a GC pause or brief network blip.
• Long TTL: fewer false alarms, but you sit leaderless longer when a real crash hits.
• I usually start around 10 to 15 seconds and tune from there.
• If someone proposes a 1-second TTL, ask what a stop-the-world garbage collection pause does to it.

Run your own vs use a service.

• Rolling your own Raft is a great way to spend six months finding edge cases etcd fixed years ago.
• Use the service unless leader election is literally your product.

Dedicated coordination cluster vs embedded consensus.

• A separate ZooKeeper/etcd cluster is one more thing to run, patch, and page on.
• Embedding Raft inside your service (CockroachDB, or Kafka's KRaft mode) removes that dependency.
• But it couples your availability to getting consensus right yourself.

That last row is underrated.

• You can do real leader election with a single conditional write to DynamoDB.
• Write your node ID only if the current lease is expired, using a condition expression.
• No ZooKeeper, no Raft, just one atomic compare-and-set against a table you already have.
• For a lot of systems that is exactly the right amount of machinery.

4Where it is used#

This pattern is everywhere once you know to look for it.

• Kafka elects a controller broker that manages partition assignment and metadata. Older versions used ZooKeeper; newer ones use KRaft, an embedded Raft.
• Kubernetes control plane components (the scheduler, the controller manager) run active-standby. Only the elected leader acts; the standbys wait, holding a lease object in etcd.
• PostgreSQL high-availability setups use tools like Patroni, which stores leader state in etcd or Consul to decide which replica is primary.
• CockroachDB and TiDB use Raft per data range, so leadership is not one global thing but thousands of tiny elections, one per shard.
• Any cron-like job you do not want running twice. A nightly billing batch across five app servers elects one to run it. Running billing twice is a great way to double-charge customers.

5Common pitfalls and gotchas#

Split brain is the one that corrupts data.

• A GC pause or network partition makes the old leader go quiet.
• The system elects a new leader.
• The old leader wakes up, still believing it is leader, and writes. Now you have two writers.
• The fix is fencing tokens: every leadership change hands out a monotonically increasing number.
• Old leader has token 33, new one has 34. Storage rejects any write carrying a token lower than the highest it has seen.
• So when zombie-leader-33 wakes and tries to write, storage says no.
• This is the single most important detail most candidates miss, and naming it unprompted is a strong interview signal.

Trusting clocks across machines.

• Do not compare timestamps between nodes to decide who is leader. Clocks drift.
• Use the coordination service as the single arbiter and measure the lease duration locally.

TTL shorter than your worst pause.

• TTL is 5 seconds but your JVM occasionally pauses for 8.
• You will get spurious failovers under load, exactly when you can least afford them.

The herd problem.

• A leader dies and 500 followers all watch the same znode.
• Naive implementations wake all 500 and they stampede the service.
• ZooKeeper's recipe: each node watches only the one directly ahead of it in line, so a single node reacts.

Forgetting the odd-number rule.

• Consensus clusters want an odd node count: 3, 5, 7.
• A 4-node cluster tolerates the same single failure as a 3-node one but costs more and has more ways to fail.
• The majority is 3 either way, so the 4th node buys you nothing.

Assuming the leader knows it lost.

• A partitioned leader often has no idea. It thinks everything is fine.
• Your design must assume the old leader keeps acting until proven otherwise.
• That assumption is the whole reason fencing exists.

6How it shows up in interviews#

Interviewers rarely ask "explain leader election" directly. It surfaces inside bigger questions: design a distributed scheduler, a message queue, a database with replication, a rate limiter with a single counter owner.

What they are probing:

Do you know when you even need it?

• Weak: "I'll put a leader in front of everything so there's one source of truth." This signals you treat coordination as free. It is not: every lease renewal is traffic, every failover is downtime.
• Strong: "We only need single ownership for writes; reads can go to any replica, so I will elect a leader for the write path only." Knowing the cost of coordination and avoiding it where you can is senior thinking.

Can you handle the failure case?

• The follow-up is always some version of "what if the leader pauses but does not die."
• Weak: "The others detect it and elect a new one." That is only half the system, and the dangerous half: you just described how a second leader gets created without saying how you stop the first one.
• Strong: "The others elect a new leader after the lease lapses, and I use fencing tokens so the old leader's late writes get rejected, because a paused leader does not know it was replaced. Token 33 keeps writing, storage only honors 34 and above."

Do you reach for the right tool?

• Weak: "I will implement Raft." Three words that say you are about to spend your design budget rebuilding a solved problem and probably botching the edge cases on the whiteboard.
• Strong: "I will use etcd or ZooKeeper for the lease, because getting consensus right myself is months of work and they have been hardened for a decade." Knowing not to build it is a plus, not a cop-out.

Can you reason about the numbers?

• Expect "how long is your failover?"
• Weak: "Pretty fast, a second or two." No basis given, so it reads as a guess.
• Strong, tied to your TTL: "A 10-second TTL means up to roughly 10 seconds of write unavailability on a crash, closer to 5 to 6 seconds on average once you account for when the last renewal landed plus watch-notification latency. If the product needs tighter, I will shorten the TTL and accept more renewal traffic and a higher false-failover rate." Concrete trade-off, stated in both directions. That earns the senior checkmark.

The failure case is not the leader dying. It is the leader that does not know it died.

For a full walkthrough, try the related System Design Question on designing a distributed job scheduler, where leader election decides which worker claims each run.