Fault Tolerance and Redundancy: A System Design Guide

If you remember five things

The whole topic in one glance. Everything below is the defense of these five calls.

Redundancy is not fault tolerance. Having a spare means nothing until something detects failure and routes around it. A replica no client is configured to use is dead weight.
The failover gap is the real downtime. The seconds between "primary died" and "replica promoted" are what the user feels. Shrinking that gap is most of the engineering.
Say RPO and RTO out loud. How much data you can lose (RPO) and how long recovery can take (RTO) turn a vague "fault tolerant" into numbers an interviewer can score.
Match the nines to the business. Four nines is 52 minutes of outage a year and a different architecture than three nines. Do not buy nines the business did not ask for.
Redundancy can cut both ways. A flapping health check or a split brain turns your safety net into the outage. The automation around redundancy fails more often than the hardware.

1The core idea#

Start with the mental model I give every junior engineer: everything fails. The only question is whether the user notices.

Disks fail. Machines reboot. Network links drop packets.
A deploy ships a bug. A data center loses power because someone tripped over the wrong thing.
The question is never "will a component fail" but "when it fails, does the user notice, and for how long."

Two words people constantly mix up:

Redundancy. You have more than one of something. Two servers instead of one, three copies of the data, a spare link. It is a means.
Fault tolerance. The system keeps working correctly when a component fails. It is the outcome.

Redundancy is a means, not the end

You can have three copies of your data and still fall over, because nothing knows how to fail over to the spare. I have watched teams proudly point at a "replica" that no client was ever configured to use. That is redundancy with zero fault tolerance. Dead weight.

One more distinction worth burning in: fault tolerance vs graceful degradation.

Fault tolerance: the system stays up through a failure, often with no visible blip.
Graceful degradation: the system loses a feature but keeps the core working.
Example: recommendations go down on a shopping site, and you still show the catalog and checkout. The sale still happens.
A good design uses both. You cannot afford full redundancy on every feature, so you decide which ones are allowed to degrade.

2How it works#

The machinery breaks into a few reliable patterns. Learn to name each one on sight.

Eliminate single points of failure (SPOFs).

Walk your request path and at each hop ask "if this one thing dies, what happens?"
If the answer is "everything stops," that is a SPOF.
The fix is almost always a second one, plus a way to route around the dead one.

Redundancy comes in two flavors.

Active-active: all copies serve traffic at once, a load balancer spreads requests, and losing one means the rest absorb the load. No warm-up, but every node must handle shared state carefully.
Active-passive (failover): one node serves, a standby waits, and on failure you promote the standby. Simpler to reason about, but you pay for idle hardware and the failover takes time.

Detection plus failover. Redundancy is useless without something watching.

Health checks ping each node every few seconds.
Miss two or three in a row and the node gets pulled from rotation.
For databases, a leader election (think a consensus protocol) picks a new primary when the old one goes quiet.

Replication keeps data redundant, and the sync choice is the whole trade.

Synchronous: wait for the copy to acknowledge before confirming the write. Zero data loss, higher latency.
Asynchronous: confirm first, copy after. Fast, but you lose the last few writes if the primary dies mid-flight.

Here is a typical request path built to survive a node loss.

flowchart LR A[Client] --> B[Load Balancer] B --> C[Service A] B --> F[Service B] C --> D[(Primary DB)] F --> D D --> G[(Replica DB)] C --> E[(Cache)] F --> E

A path with no single box that kills it. The load balancer health-checks both services, so if Service A goes quiet, traffic shifts to Service B with no user impact. The primary replicates to a replica, so a dead primary means you promote the replica. Watch the load balancer itself: run it as one box and you just moved the SPOF, which is why in practice it runs redundant too (multiple instances behind a floating IP or DNS).

Now the same failover over time.

sequenceDiagram participant C as Client participant LB as Load Balancer participant P as Primary DB participant R as Replica DB C->>LB: Write request LB->>P: Forward write P--xLB: No response primary down LB->>R: Promote replica R->>LB: Now primary LB->>C: Write succeeds

The failover gap is the whole story. Everything between "no response, primary down" and "now primary" is real downtime the user feels. Shrinking that gap is most of the engineering, and it is what the numbers in the next section are built to expose.

3A worked example: putting real numbers on it#

Abstract patterns do not land in an interview. Numbers do. Let me walk a concrete one I would draw on the whiteboard.

A checkout service doing 2,000 writes per second at peak.
Each write is one row (an order) of about 1 KB.
A failed checkout costs roughly $40 in lost revenue on average.
The business wants 99.99% availability on the payment path.

That is enough to drive every decision below.

Step 1: translate 99.99% into a time budget.

99.99% of a year (525,600 minutes) means about 52 minutes of downtime per year.
For comparison: 99.9% is about 8.8 hours a year, 99.999% is about 5.3 minutes a year.
So four nines gives us roughly 52 minutes of total outage across all incidents for the whole year. That is the constraint.

Step 2: cost the failover gap.

If the primary dies and promotion takes 45 seconds, we drop 2,000 writes/sec for 45 seconds: about 90,000 attempted writes fail or queue.
Say 10% are live checkouts that abandon: 9,000 lost orders times $40 is roughly $360,000 per incident.
Now "shave the failover from 45s to 10s" is not a nerdy detail. It is a ~$280,000 decision.

Pause before the next step: at 2,000 writes/sec, your async replica lags just 2 seconds when the primary dies. How many paid orders can you lose, and does that change your replication choice?

Reveal the answer

Up to 4,000 committed orders live only on the primary when it dies, gone on failover. At $40 each that is $160,000 vanishing, plus customers charged for nothing. That single number is why the payment path takes synchronous replication even though every write now pays a cross-AZ round trip. On a low-value, high-volume path (logs, feeds) you would happily keep async. Let the cost of a lost write pick the mode.

Step 3: pick replication from the RPO target.

At 2,000 writes/sec, async replication lagging even 2 seconds means up to 4,000 committed orders sitting only on the primary when it dies.
Those are paid orders lost on failover: $160,000 gone, plus customers charged for nothing.
So on the payment path I choose synchronous replication to a replica in a second availability zone: RPO near zero.
Cost: a cross-AZ round trip (call it 1 to 2 ms) added to every write. Paying 2 ms per write to never lose a paid order is an easy trade.

Step 4: size the app tier for N+1.

One app instance handles 500 writes/sec, and we need 2,000/sec, so 4 instances at 100% load. No headroom when one dies.
So I run 6 instances across 2 AZs (3 per AZ).
Lose a whole AZ and the 3 survivors still cover 1,500/sec, with burst headroom left.
That is the N+1 (really N+2) rule: provision so you survive the loss of the largest single fault domain.

Step 5: do the downtime math on the whole path.

Synchronous replication (RPO near 0), a 45-second automated failover (RTO 45s), stateless app instances behind a redundant load balancer.
A single clean database failover costs 45 seconds.
We can afford about 69 of those a year before we blow the 52-minute budget.
So a handful of real incidents is fine, but a flapping health check that triggers a failover every hour would wreck us.
The lesson: the automation matters more than the redundancy.

~52 min

The whole yearly outage budget at four ninesEvery failover, every bad deploy, every flapping check spends from this one pool. It is why 45-second failovers are fine in small numbers but a health check that flaps hourly is fatal. Remember this number; it reframes "add more redundancy" into "spend the budget wisely."

This is the level of specificity that separates a hire from a maybe. You are not reciting patterns, you are doing arithmetic that shows you understand the business cost of each choice.

4Trade-offs#

This is the part interviewers actually care about. Redundancy is never free, and every choice gives something up.

More nines cost more than linearly.

Going from 99.9% (about 8.8 hours a year) to 99.99% (about 52 minutes) is not 10% more work.
It is often a different architecture, a second region, and a much bigger on-call burden.
Match the target to the business: a payments system needs those nines, an internal analytics dashboard does not.

Sync vs async replication is the classic. Here is the shape of it.

Choice	You get	You give up	Reach for it when
Synchronous replication	Zero data loss on failover (RPO near 0)	Higher write latency, writes stall if replica is slow	Paid orders, auth, anything where losing a committed write is unacceptable
Asynchronous replication	Fast writes, replica can be far away	Lose the last few seconds of writes on crash	Analytics, feeds, logs, cross-region copies where a small RPO is fine
Active-active	Full capacity, instant failover	Hard shared-state problems, conflict resolution	Stateless tiers, or data stores built for multi-writer
Active-passive	Simple, clean promotion	Idle hardware, slower failover	Single-writer databases where correctness beats cost

RPO and RTO are the two numbers to say out loud.

RPO (Recovery Point Objective): how much data you can afford to lose, measured in time.
RTO (Recovery Time Objective): how long recovery can take.
Sync replication pushes RPO toward zero. A hot standby pushes RTO toward seconds.
Both toward zero means a big bill. In the worked example, sync replication bought RPO near 0 and the 45-second promotion was our RTO.

From the interviewer's chair

When you say "fault tolerant," I am waiting for the two numbers that make it real. Give me RPO and RTO and tie them to a dollar risk, and I relax. Stay at the word "fault tolerant" with no numbers behind it and I start probing, because it usually means you have read about this but never watched a failover at 3 a.m.

Redundancy can make you less reliable if you are sloppy.

Every replica, every failover script, every health check is more code that can be wrong.
A bad health check that flaps will yank healthy nodes out of rotation and cause the outage it was meant to prevent.
I have caused an outage this exact way. The redundancy was fine. The automation around it was the bug.

Correlated failures defeat redundancy.

Three servers in the same rack on the same power strip are not three independent failures. They are one.
Three replicas running the same buggy build all crash on the same poisoned input.
True redundancy needs independence: different racks, different availability zones, sometimes different software versions.

Redundancy only counts when the failures are independent. Same rack, same build, same bug is one failure wearing three hats.

5Where it is used#

You see these patterns everywhere once you know to look.

Databases. Managed relational databases offer a primary with one or more replicas and automatic failover. Promotion typically takes tens of seconds to a couple minutes, which is why serious shops keep read replicas warm.
Load balancers and web tiers. Stateless app servers behind a load balancer are the easy win. They hold no state, so you run ten and lose a few with no drama. This is why "keep your services stateless" gets repeated so often: it makes redundancy trivial.
Object storage and message queues. These store each piece of data across multiple machines or zones by default, often three copies. You get fault tolerance without thinking about it, which is the whole selling point.
Multi-AZ and multi-region. Across availability zones protects you from one data center failing. Across regions protects you from a whole region failing, at the cost of cross-region latency and a much harder consistency story.
Graceful degradation in product features. Feed ranking down? Serve a reverse-chronological feed. Fraud scoring timing out? Fall back to a cheaper rule and flag for later review. The core path survives.

Rule of thumb for how much to spend:

Reach for heavy redundancy when downtime has direct, measurable cost: checkout, auth, payments, anything users pay for.
Go lighter when a short outage is annoying but survivable.

6Common pitfalls and gotchas#

These are the ones interviewers love, and the ones that bite in production.

The untested failover.

Your standby is only as good as your last failover test.
Never actually killed the primary in production (or a realistic staging setup)? You do not have fault tolerance, you have a theory.
Teams that run game days and chaos drills find the broken DNS TTL, the stale credential, the replica that was 40 minutes behind.

Hidden SPOFs.

The database is redundant, but there is one config service everything reads at startup. Or one DNS provider. Or one certificate that expires.
The SPOF is usually not the thing you drew. It is the thing you forgot to draw.

Split brain.

In active-passive, the old primary comes back thinking it is still primary while the promoted replica also thinks it is primary.
Now two primaries take writes, the data diverges, and you cannot cleanly merge it.
This is why real systems use a quorum or a fencing mechanism, so a majority has to agree who leads.

Thundering herd on recovery.

A failed node comes back or a cache cold-starts, and every client reconnects or every request misses the cache at once.
The recovering component gets crushed again.
Jittered backoff and gradual traffic ramp-up save you here.

Confusing replication with backups.

A replica faithfully copies your mistakes. Delete a table on the primary and the replica deletes it too, instantly.
Replication is for availability. Backups are for recovery from bad data.
You need both, and they are not the same thing.

Health checks that lie.

A check that only confirms the process is alive keeps routing to a node that is up but cannot reach its database.
Check the thing that actually matters to the request, not just the pulse.

Red flag that fails you here

Treating redundancy as a magic word. "I'll add a replica so it's fault tolerant" with no mention of how failure is detected, how long promotion takes, or what the client does during the switch. The interviewer hears someone who read about redundancy but never carried a pager. Worse is adding more of it when asked to improve reliability, instead of asking what the uptime target even is.

7How it shows up in interviews#

Interviewers probe fault tolerance to see if you think past the happy path. The moment you draw a single database, expect "what happens when that database dies?" Have the answer ready before they ask.

What they are listening for:

Do you name SPOFs unprompted and remove them?
Do you distinguish availability from durability from consistency?
Can you put numbers on it: RPO, RTO, how many nines, how long failover takes?
Do you know redundancy has a cost, and can you justify the spend against the business need?

Weak answer: "I'll add a backup server and a replica database so it's fault tolerant."

This is a box with no mechanism.
It says nothing about detection, failover time, data loss, or what the client does during the switch.
An interviewer hears a candidate who has read about redundancy but never carried a pager.

Strong answer (notice how every claim carries a number or a limit):

Candidate: The app tier is stateless behind a load balancer, so I run six instances across two availability zones and survive losing a whole AZ. The database is a primary with a synchronous replica in a second AZ, so RPO is near zero. That matters because at 2,000 writes per second, an async lag of even 2 seconds risks losing 4,000 paid orders on failover. Automatic failover promotes the replica in roughly 45 seconds, which is our RTO, and at 99.99% we have about a 52-minute yearly budget, so that is fine for a handful of incidents but not for a flapping check. The load balancer itself runs redundant so it is not a new SPOF. For non-critical features like recommendations, I degrade gracefully instead of failing the whole request. I would run a failover drill before trusting any of this.

The difference is everything:

The weak answer is a noun ("a backup").
The strong answer is mechanisms, numbers, and limits: it names the failover time, ties the replication choice to a dollar-shaped risk, bounds the error budget, and admits what has not been tested.
That honesty reads as senior, not weak.

One move that always lands:

When asked to improve availability, do not just pile on redundancy. Ask what the actual uptime target is first.
Designing for five nines (5.3 minutes a year) when the business needs three (8.8 hours a year) is over-engineering, and a sharp interviewer will respect you for catching it.

Follow-ups the interviewer will ask (and your answers)

"Your primary database just died. Walk me through what happens."

Health checks miss two or three in a row and mark the primary down. A leader election or the managed service promotes the synchronous replica in the second AZ, roughly 45 seconds. During that gap writes fail or queue, so the client retries with backoff. Because replication was synchronous, RPO is near zero: no committed order is lost. Once promoted, the replica is the new primary and a fresh standby spins up behind it.

"Sync replication adds latency to every write. Why pay that?"

On the payment path the alternative is worse. At 2,000 writes/sec an async lag of 2 seconds risks 4,000 committed orders gone on failover, about $160,000 plus customers charged for nothing. A 1 to 2 ms cross-AZ round trip per write is cheap next to that. On a non-paid path like logs or feeds I would keep async, because losing a few seconds of writes there costs nothing.

"The old primary comes back after promotion. What stops split brain?"

Fencing plus quorum. The promotion is only valid if a majority agrees, and the old primary is fenced off (STONITH, a revoked lease, or a load balancer that refuses to route to it) before the replica takes writes. When the old node returns it rejoins as a replica, not a second primary, so it can never accept writes and diverge the data.

"Your health check is flapping and triggering failovers. Why is that dangerous?"

Each failover spends from the 52-minute yearly budget and pulls healthy nodes out of rotation. A check flapping hourly can trigger more failovers than the budget allows and cause the outage it was meant to prevent. The fix is a check that tests what the request needs (can it reach its database?), plus hysteresis: require several consecutive misses to eject and several hits to re-admit.

"How is a replica different from a backup? I have a replica, am I covered?"

No. A replica copies your mistakes instantly: drop a table on the primary and it is gone on the replica too. Replication protects against a node dying. Backups protect against bad data: a snapshot you can restore to a point before the mistake. You need both, and interviewers watch for candidates who conflate them.

What you can now explain in the room

Why redundancy without detection and failover is dead weight, not fault tolerance.
How to translate a nines target into a yearly outage budget, and back into architecture.
Why the failover gap, not the redundancy, is the real downtime, and how to cost it in dollars.
How to pick sync vs async replication from an RPO target and the price of a lost write.
How to size a tier for N+1 so it survives losing a whole fault domain.
The failure modes of redundancy itself: split brain, correlated failures, flapping checks, and replica-is-not-backup.

Want to see these patterns in a full design? Try the System Design Question walkthrough on building a highly available URL shortener, where failover and replication choices drive the whole architecture.

← Previous

Back-of-the-Envelope Estimation

Idempotency

Get a prep plan built around your Fault Tolerance and Redundancy interview

Tell us the role and your interview date. We will send back a dated, day by day plan. Free.

Thanks. Check your inbox, your Fault Tolerance and Redundancy prep plan is on the way.