Circuit Breaker and Bulkhead

1The core idea#

Start with the failure, not the pattern. The whole value of these two tools is clearer once you have watched a service die.

• A payment service calls a fraud-scoring API. Normally that call takes 40ms.
• One day the fraud API starts hanging: no error, no refused connection, just dead air for 30 seconds before timing out.
• Each request now holds a thread for 30 seconds instead of 40ms, and traffic keeps coming.
• Within a minute every thread in the pool is parked waiting on a dead API.
• The payment service stops answering anyone, including the 95% of requests that never needed fraud scoring.

Put real numbers on it so the failure cannot be hand-waved.

• The payment service runs 200 threads and normally serves 400 requests per second, because each finishes in ~40ms (200 threads / 0.040s = 5000 req/s of headroom, comfortable).
• The fraud call now hangs for 30s. At even 50 incoming requests per second that touch fraud, you consume 50 threads/s and each stays stuck for 30s.
• After 200 / 50 = 4 seconds, every thread is parked. From second 5 on, the service answers zero requests, fraud-related or not.
• One dependency you call on 5% of requests just took 100% of your traffic to the floor in under 5 seconds.

The circuit breaker solves the first half: stop calling something that is clearly broken.

• It wraps a remote call and watches the failure rate.
• When failures cross a threshold, it "trips" and immediately rejects calls for a cooldown window instead of letting them hang.
• Think of the breaker in your house: when there is a short, it pops open so the wiring does not catch fire. Same idea, same name.

The bulkhead solves the second half: contain the damage so one broken dependency cannot starve everything else.

• The name comes from ships. The hull is divided into sealed compartments so a hole in one does not flood the whole vessel.
• In software, you give each downstream dependency its own isolated pool of threads or connections.
• If the fraud API goes dark and fills up its 20 reserved threads, the other 180 threads stay free to serve everything else.

Breaker: stop making the bad call. Bulkhead: make sure the bad call can only hurt its own corner. You almost always want both.

2How it works#

A circuit breaker is a small state machine with three states. Learn to draw it from memory.

• Closed is normal. Calls flow through. The breaker counts outcomes over a rolling window (say the last 50 calls or the last 10 seconds). If the failure rate stays under the threshold, nothing happens.
• Open means it tripped. Once failures cross the line (for example, more than 50% of the last 20 calls failed), the breaker short-circuits. Every call fails instantly with a "circuit open" error, no network request made. The breaker starts a cooldown timer, usually 5 to 30 seconds.
• Half-open is the recovery probe. When the cooldown expires, the breaker lets a small number of trial calls through. If they succeed, it flips back to closed. If they fail, it snaps back to open and waits again.

The open state is the whole point: a fast, cheap failure instead of a slow, expensive timeout. Half-open stops you from slamming a recovering service with full traffic the instant it comes back up.

The bulkhead is less about state and more about resource partitioning. The cleanest way to picture it: instead of one shared pool of 200 threads for all outbound calls, you carve out dedicated pools per dependency.

Replay the worked example with both patterns in place and watch the numbers change.

• Fraud gets a bulkhead of 20 threads and a per-call timeout of 200ms. The fraud API hangs.
• The first 20 concurrent fraud calls grab the 20 reserved threads; every call after that is rejected instantly, because the bulkhead is full.
• Those 20 stuck threads each hit the 200ms timeout and register as a failure.
• The breaker trips at 50% failures over the last 20 calls, so after ~20 timed-out calls (roughly 4 seconds, not 30) it flips to open.
• From there, fraud calls fail in microseconds with "circuit open," freeing the bulkhead entirely. The other 180 threads never budged.
• The payment service keeps serving its ~400 req/s the whole time, dropping only the fraud check on the ~5% of requests that needed it.

The two patterns stack in practice.

• A single outbound call gets wrapped in a bulkhead (its own thread or connection budget) and a circuit breaker (its own failure tracking).
• Libraries like Resilience4j, and historically Netflix Hystrix, give you both as decorators around a call.
• You will also see bulkheads done as a semaphore (a simple permit counter) instead of a separate thread pool, when you do not want the overhead of extra threads.

3Trade-offs#

The hard part is not the concept, it is the tuning. Every parameter is a guess about failure behavior, and a bad guess hurts.

Breaker failure threshold is a tightrope.

• Too low and it trips on normal noise. Every service has a baseline error rate. If 2% of calls naturally fail and you trip at 5% over a window of 10 calls, one unlucky burst opens the circuit and denies service to healthy traffic.
• Too high and the breaker never trips when you need it, so you get the cascading failure you were trying to prevent.

Cooldown window is the same balancing act.

• Short cooldown means fast recovery but you risk hammering a service that is still struggling.
• Long cooldown means you stay degraded well after the dependency healed.

Bulkhead sizing has its own tension.

• Too small and you reject traffic the dependency could actually have handled, hurting throughput on a good day.
• Too large and the bulkhead is not really isolating anything; one dependency can still consume most of your capacity.

The biggest conceptual trade-off: these patterns do not make failures go away.

• They convert slow, total, cascading failure into fast, partial, contained failure.
• You are choosing to fail some requests immediately and loudly so that most requests keep working.
• That is a good trade almost every time, but you have to own it. When the breaker is open, something is genuinely not working, and you need a fallback plan for those rejected calls.

Threads versus semaphore is a real fork, not a detail.

• A dedicated thread pool per dependency gives you true timeout isolation, because the calling thread can walk away and let the pool thread die.
• A semaphore bulkhead is cheaper but the caller still blocks, so you only get concurrency limiting, not timeout protection.
• Know which one you are describing when you say "bulkhead" in the room.

4Where it is used#

Reach for this anywhere a service calls another over the network and the caller cannot afford to wait forever.

• API gateways and edge services are the classic home, since they fan out to dozens of backends.
• Netflix built Hystrix precisely because one slow recommendation service should not blank out the whole home screen. They wrapped every backend call in a breaker with a fallback: show a generic row instead of personalized picks.

Service meshes moved a lot of this out of application code.

• Envoy, the proxy behind Istio and many gateways, has circuit breaking and connection-pool limits built into the sidecar.
• You get breakers and bulkheads without changing your app, configured as policy.
• The catch: the mesh breaker works at the connection and request level and does not know your business-specific fallback logic, so you often still want an in-app breaker for the calls that need a smart fallback.

You reach for the pattern when all three hold:

• The dependency is remote and can fail independently.
• A timeout there would block a scarce resource (threads, connections, memory).
• You have a sensible thing to do when the call fails.
• If a call is purely internal, in-process, and cannot hang, you do not need a breaker; you would just be adding moving parts.

Databases are an interesting case.

• People slap breakers on DB calls, but the better first move is usually a connection pool with a hard checkout timeout, which is itself a bulkhead.
• The pool caps how many requests can wait on the DB, so a slow database rejects fast instead of pinning every app thread.
• Add a breaker on top if you need the fast-fail-and-fallback behavior.

5Common pitfalls and gotchas#

Most failures here come from skipping a step, not from the concept. These are the ones that bite in production and in interviews.

• No timeout under the breaker. The number one mistake. A breaker counts failures, but if the underlying call has no timeout, it never registers a "failure," it just hangs. The breaker sits happily in closed state while every thread drowns. Set an aggressive per-call timeout first; the breaker tracks those timeouts as failures. No timeout, no protection.
• Retrying into an open-ish system. Retries and breakers fight each other if you are not careful. A retry storm (every client retrying 3 times during a blip) can triple load on a struggling service right when it is weakest. If you retry, use a limited count with backoff and jitter, and put the breaker in front so retries stop the moment it opens.
• One shared breaker for many dependencies. If you wrap all outbound calls in a single breaker, a failure in service A trips the circuit for service B too. Breakers must be scoped per dependency, ideally per dependency-plus-operation.
• Breaker but no bulkhead. The breaker trips only after it has seen enough failures. During that detection window, calls are still hanging and eating threads. With a 30s hang and no bulkhead, a 200-thread pool exhausts in ~4 seconds, but a breaker tuned to 20 observations might need those same seconds just to decide. The bulkhead holds the line while the breaker makes up its mind.
• No fallback. An open circuit returns an error fast, which is good, but if your code treats that error like any other 500 and bubbles it to the user, you have just made failure faster, not better. Decide up front what "degraded" looks like.
• Health that lies. A breaker is only as good as its failure signal. If your downstream returns HTTP 200 with an error body, or returns stale data that looks fine, the breaker will not trip. Make sure the thing you count as a failure actually reflects a broken dependency.

6How it shows up in interviews#

Interviewers reach for this the moment you have drawn a few services calling each other. The trigger is usually some version of: "What happens if this downstream service gets slow?" If your answer is "we would add retries," you have walked into the trap, because naive retries make it worse.

The contrast I would coach anyone on.

• Weak answer: "I would add a circuit breaker so if the service is down we stop calling it." It names the pattern and stops. It cannot say what trips the breaker, what the states are, what happens to blocked requests, or what the user sees.
• Strong answer: walks the hang, puts numbers on the thread exhaustion, names all three mechanisms (timeout, bulkhead, breaker), and lands on the user experience.

Here is the strong answer in full, the way I would say it in the room:

Candidate: First the failure mode. The fraud API hangs for 30 seconds instead of its usual 40ms. With a shared 200-thread pool and 50 fraud calls a second, I exhaust every thread in about 4 seconds and the whole payment service goes dark, even for requests that never touch fraud. So I would do three things. One, a tight per-call timeout, maybe 200ms, so a hang counts as a failure fast. Two, a bulkhead: a dedicated pool of 20 threads just for fraud, so even during the detection window a stuck fraud API can only eat its own 20 threads, not all 200. Three, a circuit breaker scoped to fraud, tripping at 50% failures over the last 20 calls, so after a few seconds it opens and fails those calls in microseconds with a 5 to 30 second cooldown and a half-open probe before it trusts the API again. And the fallback matters most: when the circuit is open, we skip the fraud score and queue the transaction for async review, so the user still checks out. We traded a 30-second total outage for a 4-second partial degradation nobody noticed.

The detail that separates senior candidates: naming the fallback.

• Interviewers want to hear what the user experiences when the circuit is open.
• "The product page drops the personalized recommendations row and shows a bestseller list from cache. The page still loads."
• That shows you are thinking about the product, not just the plumbing.

Other things they will probe.

• The difference between the two patterns: breaker = stop calling, bulkhead = contain resource usage.
• Why timeout comes first: no timeout, breaker never sees a failure.
• Where the breaker lives: in-app library versus service mesh sidecar, and the fallback trade-off between them.
• How you pick the threshold: measure the baseline error rate and set the trip point comfortably above it, over a window big enough to ignore noise.

If you can draw the three-state machine, explain the half-open probe, and give a concrete fallback, you are answering at the level they are hiring for. Bonus points for saying these patterns trade total slow failure for partial fast failure, and that you accept some rejected requests as the cost of staying up.

For a worked example of this in a larger design, see the System Design Question walkthrough on building a resilient API gateway.