Distributed Locks

1The core idea#

Start with what a distributed lock actually promises, because the mental model decides everything downstream.

The hotel-key picture.

• A distributed lock lets processes on different machines agree that only one holds a resource at a time.
• Picture one bathroom key at a hotel front desk: you cannot enter unless you hold the physical key.
• The key is the coordination point. Everyone trusts the desk, not each other.

Why a mutex does not translate.

• In one process a mutex works because threads share memory and atomic instructions like compare-and-swap.
• Across machines you have none of that: no shared memory, no shared clock, and a network that drops, delays, and reorders.
• So the lock lives somewhere both parties can reach and trust: a strongly consistent store (ZooKeeper, etcd, Redis), sometimes a plain database row.

The model I want you carrying into the room: a lock is a lease.

• You are not holding a lock forever. You are renting exclusivity for a bounded window, say 30 seconds.
• The whole difficulty comes from one brutal fact: the service can think you are dead when you are just slow.
• A 200ms GC pause, a network hiccup, a 4 second live-migration, and your lease expires while you still believe you own it.
• Two clients now think they hold the same lock. That is the failure mode that keeps people up at night, and I keep coming back to it.

2How it works#

The acquire protocol is short. The release protocol is where people cut themselves.

Acquire: set-if-absent with a TTL.

• A client asks the lock service to set a key only if it does not already exist, with a time-to-live attached.
• In Redis that is one atomic command: SET lock_key random_value NX PX 30000.
• NX means "only if not set." PX 30000 means "expire in 30 seconds."
• Command succeeds: you hold the lock. Command fails: someone else has it.

The TTL is the safety net.

• If your client crashes while holding the lock, the key expires on its own and the next client grabs it.
• Without a TTL, one dead process blocks everyone forever.

Release is more dangerous than it looks.

• You cannot just delete the key. By release time your lease may have expired and someone else may own the lock.
• Deleting then would stomp on their lock.
• So you store a random value on acquire, and on release you delete only if the stored value still matches yours.
• That check-and-delete must be atomic, which is why real code uses a small Lua script, not a read-then-delete round trip.
• A read-then-delete has a window between read and delete where the lease can expire and get reassigned, and you delete the wrong owner's key.

Here is the acquire path and the dangerous gap:

The fix for split ownership is a fencing token.

• Every grant hands out a number that only ever goes up: 33, then 34, then 35.
• The client passes that token to the actual resource it is protecting, say a storage service.
• The resource remembers the highest token it has seen and rejects anything lower.
• So when paused Client One wakes with token 33 and tries to write, the storage already saw 34 from Client Two and says no.
• The slow client gets fenced out. This is the single most important idea in the topic, and most candidates have never heard of it.

3A worked example with real numbers#

Numbers make the pause problem concrete, so here is one I have actually debugged.

The setup.

• A nightly billing job must charge each customer exactly once.
• The service runs on 10 pods for availability.
• The job takes about 90 seconds on a normal night.

First attempt, naive. Each pod tries SET billing_lock <uuid> NX PX 120000, a 120 second TTL to cover the 90 second runtime with margin. Pod 3 wins and starts charging. Then the timeline that bit us:

• t = 0s: Pod 3 acquires the lock, TTL expires at t = 120s.
• t = 85s: Pod 3 hits a stop-the-world GC pause plus a noisy-neighbor CPU throttle on the node. It freezes.
• t = 120s: the lock TTL expires. Redis frees the key. Pod 3 is still frozen and has no idea.
• t = 121s: Pod 7 runs SET ... NX, succeeds, and starts the billing job from the top.
• t = 140s: Pod 3 thaws, believes it still holds the lock, and resumes charging the remaining customers.

The blast radius.

• Every customer charged after t = 121s got billed twice.
• With 50,000 customers and the pause hitting at roughly 70% through the run, that is about 15,000 duplicate charges.
• That is a Sev 1 and a refund fire drill, not a rounding error.

Now add fencing.

• The lock service issues a monotonically increasing token with each grant.
• Pod 3 got token 41. Pod 7 got token 42.
• The payment ledger records the highest token it has committed, now 42.
• When Pod 3 thaws and sends a charge stamped with token 41, the ledger sees 41 < 42 and rejects it.
• Zero duplicate charges, even though two pods genuinely believed they held the lock at the same time.

The lesson in one line.

• The lock reduced contention. Only the fencing token at the resource made the system correct.

4Trade-offs#

The real decision is where the lock lives. Each choice buys you something and charges you something else.

The Redlock argument, which you should be ready to have.

• The idea: run five independent Redis nodes and require a majority (three of five) to grant, so one node failing does not lose your lock.
• It sounds safer.
• The counterargument, the one I expect a strong candidate to raise: Redlock relies on timing assumptions distributed systems do not honor.
• If a client pauses long enough, its lease expires regardless of how many nodes agreed, and you are back to two owners.
• Going from 1 node to 5 does not fix the pause problem. Only fencing tokens fix the pause problem.

The deeper trade-off: strong locks cost availability.

• A consensus system needs a quorum.
• A network partition that cuts off the quorum stalls acquisition.
• That is CAP showing up in your lock: you are choosing consistency over availability.
• Usually the right call for a lock, but say it out loud so the interviewer knows you made it on purpose.

5Where it is used#

Name real uses, because it shows you have seen locks in the wild, not just in a blog post.

Leader election is the biggest one.

• A cluster of workers all try to grab one lock, and the winner becomes the leader that runs the cron job or the rebalancer.
• Kafka's older controller election leaned on ZooKeeper for exactly this.
• Kubernetes controllers use a lease object in etcd so only one instance reconciles at a time while the rest sit on standby.

Cron deduplication.

• This is the billing example from section 3: a service on ten pods, but the nightly job must run once.
• A lock keyed to the job name, with a TTL slightly longer than the runtime, means only one pod fires it.
• Fencing is the real backstop.

Resource serialization in payments and inventory.

• "Only one process may decrement the stock for SKU 12345 right now."
• Honestly, here I would push back in an interview and suggest a conditional DB update or an atomic counter instead of a lock.
• The database already gives you the atomicity for free.
• Reaching for a distributed lock when a single UPDATE ... WHERE quantity > 0 would do is a small red flag.

The canonical references.

• HDFS and Google's Chubby.
• Chubby was built as a coarse-grained lock service and ended up used mostly for leader election and as a highly available config store.
• That tells you something: people want locks less often than they want a reliable place to agree on one small fact.

6Common pitfalls and gotchas#

These are the ones that separate a candidate who read about locks from one who has run them.

The TTL guessing game.

• Too short and your lease expires mid-job while you are still working.
• Too long and a crashed holder blocks everyone for ages.
• There is no perfect number, as the 120 second versus 35 second pause showed.
• The mature answer: a lease you renew. A heartbeat extends the TTL every few seconds while you are alive, so a healthy long job keeps its lock but a dead one releases quickly.

Releasing a lock you no longer own.

• Delete the key without checking your value and you can release someone else's lock.
• Always do compare-value-then-delete atomically with a Lua script.

Assuming clocks agree.

• Lock TTLs depend on time, and clocks drift between machines, often by tens of milliseconds and more after an NTP correction.
• A client with a fast clock can think its lease is still good when the server already expired it.
• Never reason about lock validity using your own wall clock alone.

Forgetting the pause.

• GC pauses, container throttling, and VM migrations can freeze your process for seconds.
• During that freeze the lock expired and someone else took it.
• No fencing means your design is wrong for correctness-critical work.
• Say "what if the process pauses right here" to yourself at every step.

Treating the lock as the source of truth.

• The lock coordinates access, but the resource must still defend itself.
• Fencing tokens live at the resource, not at the lock.
• If you cannot change the resource to check tokens, you cannot build a truly safe lock around it, and you should be upfront about that limit.

7How it shows up in interviews#

Interviewers rarely ask "explain distributed locks" directly. It surfaces inside a larger design: a job scheduler, a payment system, a dedup layer. They wait to see whether you reach for a lock, and then whether you understand what you reached for.

A weak answer.

"I'll use a Redis lock with SET NX and a TTL, and whichever pod gets it runs the job."

• That is the starting line, not the finish. It shows you know the command but not the failure model.
• Ask "what if that pod pauses for 40 seconds" and the weak answer either goes quiet or says "make the TTL longer," which just moves the cliff.

A strong answer, same billing prompt.

"A single Redis SET NX PX lock picks which pod runs, that handles contention. But a lock alone cannot guarantee exactly-once charging, because a GC pause or VM migration can let the lease expire mid-job while the pod is frozen, and a second pod then starts. So the lock service hands out a monotonic fencing token, and the payment ledger rejects any charge carrying a token lower than the highest it has committed. The lock is an optimization to cut contention. The token at the ledger is what actually makes it correct. If this needs to survive a Redis failover cleanly, I would put the lock on etcd and accept that during a partition I lose the ability to acquire, because a double charge is far worse than a brief stall."

Why the strong answer wins (it is not vocabulary).

• It names the two reasons to lock: efficiency versus correctness, and picks the mechanism to match.
• It raises the pause problem and fencing before the interviewer pulls them out.
• It questions whether a lock is even the right tool when a conditional update or idempotency key fits better.
• It is honest about the availability cost.
• Say all of that and I relax, because now I know you have been burned by this before.

The lock picks a winner. The fencing token keeps the losers from writing.

For a full walkthrough, see the related System Design Question on designing a distributed job scheduler, where locking, leader election, and fencing all come together under real load.