The CAP Theorem

1The core idea#

Start with why the problem exists at all. The trouble begins the moment you copy data across machines.

• Picture a service running on more than one machine, with data copied across those machines so no single box going down takes you offline.
• That copying is the whole point. It is also where the trouble starts, because now two machines can disagree.

The three properties CAP talks about.

• Consistency (C): every read sees the most recent write. Write balance = 50, read a microsecond later, you get 50, not the old 40.
• Not the same C as in ACID. ACID's C is about honoring database constraints inside a transaction. CAP consistency means all nodes agree on one single value right now, which database people call linearizability.
• Availability (A): every request to a live, reachable node gets a non-error response. Not "fast," just an answer that is not a failure. A node that is up but returns a 503 or hangs forever is not available by this definition.
• Partition tolerance (P): the system keeps working even when the network drops messages between nodes. One group of servers can no longer talk to another, and neither side knows if the other is dead or just unreachable.

The famous line is the trap.

• "You can only have two of the three" is misleading. Here is the mental model interviewers actually want.
• In any real distributed system, partitions will happen: cables get cut, a top-of-rack switch reboots, a cloud zone loses peering for 90 seconds, a bad BGP change blackholes a region.
• You do not get to opt out of partitions. P is not a choice. It is the weather.

2How it works#

Make the partition concrete, because the abstraction is where people get fuzzy.

The setup.

• Two replicas of a user's account balance, one in a US datacenter and one in Europe, kept in sync.
• A client writes balance = 50 to the US node.
• The transatlantic link goes dark.
• A second client, routed to Europe, asks for the balance. Europe still thinks it is 40, and Europe has no way to ask US "hey, did anything change?"

The two honest options.

• Choose consistency (CP): Europe refuses to answer, or blocks, because it cannot confirm it has the latest value. The client gets an error or a timeout. Correct data, but the node is effectively down for that request. You sacrificed availability.
• Choose availability (AP): Europe cheerfully returns 40. The client gets an answer, but it is wrong. You sacrificed consistency. When the link heals, the two nodes reconcile and agree on 50.

Reconciliation is where AP systems earn their keep.

• Last-writer-wins, vector clocks, or CRDTs (conflict-free replicated data types) decide how to merge divergent copies once the partition clears.
• CP systems avoid that problem entirely by never letting the copies diverge, usually through a quorum protocol like Raft or Paxos where a majority of nodes must acknowledge a write before it counts as committed.

3Trade-offs#

Here is the decision laid out the way I think about it at a whiteboard.

The trade-off nobody tells juniors about: latency.

• This is the PACELC extension, and dropping it in an interview makes you look sharp.
• PACELC says: if there is a Partition, choose between Availability and Consistency (that is plain CAP). Else, during normal operation, choose between Latency and Consistency.
• That "else" branch is the one that actually bites you day to day. Keeping replicas perfectly in sync means a write waits for other nodes to acknowledge before it returns.
• Across regions that round trip can be 80 to 150 ms. Relax consistency and you answer from the local replica in under 5 ms.
• Most of the time there is no partition at all, so the real everyday question is "how much latency will I pay for how much consistency?"

A concrete feel for it with real numbers. A write from a US East user to replicas in US East, Europe, and Asia:

• Synchronous cross-region write (CP, strong): returns only after a quorum acknowledges. US East to Europe is roughly a 90 ms round trip, US East to Asia closer to 180 ms. Waiting on 2 of 3 replicas lands around 120 ms per write.
• Asynchronous local write (AP, eventual): commits to the local US East replica in about 3 ms and returns; replication to Europe and Asia happens in the background over the next few hundred milliseconds.

So which do you eat?

• For a social feed, 3 ms and occasionally-stale wins every single time. Nobody files a bug because their friend's like showed up 400 ms late.
• For a bank ledger, you eat the 120 ms, because a double-spend is a lawsuit and 120 ms is nothing next to that.

The partition is the dramatic case, but latency is the bill that arrives every single day.

4Where it is used#

You pick a point on this spectrum every time you choose a datastore, whether you realize it or not. Naming the right store with the right reason is a fast way to signal you have done this before.

Lean CP when correctness beats uptime.

• etcd and ZooKeeper store cluster config, leader election, and locks. They run Raft and ZAB respectively. If the coordination layer served stale data, two nodes might both think they are the leader, and now you have split-brain writing conflicting data to your real database. These refuse writes on the minority side rather than risk it. Kubernetes stores its entire cluster state in etcd for exactly this reason.
• Relational databases on a single primary (Postgres, MySQL with synchronous replication) behave CP-ish: one source of truth, and if it is unreachable, writes stop until a new primary is promoted.
• Google Spanner is the interesting one. It uses synchronized atomic clocks (TrueTime) to offer strong consistency across regions, and it leans CP. It accepts a small write latency and brief unavailability during partitions as the price for not handing out wrong data. Spanner is the existence proof that you can get very close to "consistent and globally available," but you pay with GPS and atomic clocks in every datacenter, which most of us cannot buy at the hardware store.

Lean AP when a stale answer beats no answer.

• Cassandra and DynamoDB (in eventually-consistent modes) take your write and serve reads from whatever replica is closest, even mid-partition. A shopping cart showing a slightly old item count beats a cart that throws a 500 during a network blip and sends the customer to a competitor.
• DNS is the oldest AP system around. Records propagate slowly, governed by TTLs, and you might hit a stale one for minutes, but DNS basically never tells you "sorry, unavailable." The whole internet depends on that choice.

5Common pitfalls and gotchas#

Each of these is a line I have watched sink an otherwise-strong loop.

• Treating P as optional. The classic blunder. Say "I'll build a CA system" and you are really saying "I'll build a single-node system," because the moment you have two machines talking over a network, partitions are possible. Any interviewer who knows the material will pounce, and they will be right to.
• Thinking CAP is a system-wide label. A real product mixes both. The payments path is CP, the recommendation feed is AP, the notification count is AP, the auth session store is CP. One letter for an entire architecture is a junior move. Reason per data flow.
• Confusing eventual consistency with "eventually correct, no effort." Eventual consistency promises replicas converge if writes stop. Converging correctly under conflicting concurrent writes is real engineering: you need merge rules, and bad ones silently drop data. Last-writer-wins quietly throws away a conflicting update based on a timestamp. For a like counter that is a lost like, annoying but survivable. For inventory that is overselling the last unit to two customers, and now someone's order gets cancelled.
• Assuming CP means "always consistent, always up." CP gives up availability during partitions. If your business genuinely cannot tolerate any downtime, pure CP will hurt. Sometimes the right answer is AP with careful conflict resolution. Amazon built Dynamo as AP precisely because a cart that is always writable mattered more to revenue than a cart that is always perfectly consistent.
• Forgetting the latency cost outside partitions. This is why PACELC matters. Teams pick a strongly consistent cross-region setup, then act surprised when p99 write latency is 200 ms and the mobile app feels sluggish. The consistency was not free even when the network was perfectly healthy. You paid for it on every write, all year.

6How it shows up in interviews#

Interviewers rarely ask "define CAP." They weave it into a design.

• You say "I'll use a database to store orders," and they ask "what happens if that region loses network to the other region?"
• That is CAP, dressed up as an innocent follow-up.
• If you hear a question shaped like "what if these two nodes cannot talk," your CAP reasoning should fire immediately.

What they are actually probing.

• Do you know partitions are not optional?
• Can you tie C-versus-A to the business requirement in front of you?
• Do you understand the normal-operation latency cost, not just the failure case?
• Can you reason about different parts of the same system differently?

The answer that lands.

Candidate: Partitions will happen, so the real choice is C versus A during one. For the payment ledger I want CP: I'd rather reject a charge for a few seconds than double-charge someone, so I'll use a quorum-based store and accept that the minority side rejects writes. For the activity feed I want AP: a slightly stale feed is fine, so I'll serve from the nearest replica and let it converge with last-writer-wins. And even with no partition, I'm choosing to pay roughly 120 ms on cross-region writes for the ledger, which PACELC frames as the latency-versus-consistency call, versus about 3 ms for the feed.

Interviewer: Good. Now what decides whether a stale read is actually acceptable on the feed?

Why that answer works.

• It hits the theory (P is given), the per-flow reasoning (ledger CP, feed AP), a concrete number (120 ms versus 3 ms), and PACELC.
• That is the difference between "understands the slide" and "has made this call for real."
• One more move that lands well: when they describe a scenario, ask "is a stale read acceptable here, and for how long?" You are forcing the requirement that decides C versus A, which is exactly what you would do on the job before writing a design doc.

For a hands-on application of these ideas, see the System Design Question walkthrough on designing a globally distributed key-value store, where choosing CP versus AP per operation is the crux of the whole problem.