Saga Pattern

1The core idea#

Hold one mental model and the rest falls out of it.

• ACID transaction: all or nothing, held under a lock until the end.
• Saga: a chain of committed steps, each of which already happened, plus a documented way to take each one back.
• A saga is a sequence of local transactions. Each step updates one service's own database, then triggers the next step.
• If step 4 fails, you do not roll back the way a database would. You run compensating transactions that undo steps 3, 2, and 1 in reverse.

Why that difference bites harder than it looks.

• Every step commits immediately and is visible to the rest of the system. There is no global lock.
• So between step 1 and step 5, the world can see partial state: money is captured but the item has not shipped.
• That is why sagas give you eventual consistency, not the strict consistency a single database handed you for free.

The travel analogy makes the compensation concrete.

• You reserve a flight, then a hotel, then a car.
• If the car fails, you cancel the hotel and cancel the flight.
• Each cancel is a real operation with its own side effects (maybe a fee), not a magic rewind.
• That cancel is the compensating transaction, and designing good ones is the whole game.

2A worked example with real numbers#

Make it concrete before the theory piles up. Here is the order saga I actually ran, with the real five steps, the real latencies, and the money that moves. Say a customer buys a $79 pair of headphones.

The happy path.

• End to end is about 725 ms.
• That is the number I quote in an interview, and it is dominated by payment and shipping, the two external calls.

Now walk a failure.

• Steps 1 and 2 commit: the order exists and $79 is captured.
• Step 3 hits a warehouse that just sold its last unit, so inventory returns out of stock.
• The saga compensates in reverse: step 3 reserved nothing, so it has no compensation to run, then step 2 refunds the $79, then step 1 marks the order CANCELLED.
• The refund call is 200 ms, so from the failure at step 3 the money is released inside roughly 250 ms if the refund succeeds first try.

The number the business actually cares about: the inconsistency window.

• Between the payment capture finishing (end of step 2) and the refund completing, the customer has been debited $79 for an order they will not get.
• Happy case: that window is about 250 ms.
• Bad case: the refund call to the payment provider times out and we retry with backoff, so it can stretch to 30 to 60 seconds.
• A cron sweeps stuck sagas every minute to catch the ones that fall through.
• When someone asks "who sees the inconsistency and for how long," this is the answer: the customer, on their card statement, for a fraction of a second normally and up to a minute in the tail.

Scale it to see why the refund path matters.

• At 500 orders per second, with a 0.5% inventory-failure rate, that is roughly 2 to 3 refund compensations firing per second.
• The refund path has to be as reliable as the capture path, because it runs constantly, not rarely.

3How it works#

Two ways to coordinate the steps. You have to know both cold.

Choreography: no central brain.

• Each service listens for events and reacts by doing its work and publishing the next event.
• Order publishes OrderCreated. Payment hears it, charges the card, publishes PaymentCaptured. Inventory hears that, reserves stock, publishes StockReserved. And so on.
• The flow lives in the wiring between services, not in any one place.

Orchestration: a central coordinator owns the flow.

• The orchestrator tells each service what to do and waits for the reply.
• It owns the state machine: "Call payment. Got success? Call inventory. Got failure? Call payment-refund."
• The logic sits in one place you can read top to bottom.

Here is an orchestrated order saga, happy path and one failure:

The orchestrator must persist state after every step. This is not optional.

• If the coordinator crashes right after payment captured but before it records that fact, on restart it must know exactly where it was so it does not double-charge.
• The saga state ("we are at step inventory, pending") lives in a durable store, not in memory.

The choreography version of the same flow has no orchestrator box. Each service is a node reacting to messages:

One rule holds for both styles: compensation runs backward.

• If you captured payment, reserved stock, then failed at shipping, you release the stock first, then refund the payment.
• Reverse order, every time, because later steps may depend on earlier ones.

4Trade-offs#

The big decision is choreography versus orchestration. I have argued both sides in real design reviews, so here is the honest version.

My rule of thumb.

• More than four steps or any real branching, I reach for orchestration.
• Being able to look at one state machine and answer "where is order 8842 stuck and why" is worth the central dependency.
• I have watched engineers lose an afternoon tracing a choreographed flow across five services' logs to find one dropped event.
• The five-step order saga above is exactly why I orchestrated it: four of those steps have a failure branch that fans out to a different compensation.

The thing you give up with sagas in general, either style, is isolation.

• In ACID, the "I" means no other transaction sees your half-done work. Sagas have no isolation.
• Between steps, other operations can read and act on intermediate state.
• Classic problem: a balance is debited in step 2, a second saga reads that balance in step 3 and decides on it, then step 2's saga rolls back. The second saga acted on money that no longer exists.

You handle this with application-level countermeasures.

• Semantic locks: mark a record "pending" so others know not to trust it yet.
• Commutative updates: design operations so order does not matter, like increment instead of set.
• Reread values before committing.
• These are not free, and getting them right is 80% of the hard work in a real saga.

5Where it is used#

Anywhere a single business action spans services that each own their data.

• Order fulfillment, the textbook case. Place order, charge, reserve inventory, assign a courier, notify. Five services, five databases, one user action. E-commerce platforms run exactly this, and it is the example above.
• Travel and ticketing. Flight plus hotel plus car, each from a different provider with its own cancellation rules. Compensations here are genuinely hard because a cancel might cost money or be time-windowed.
• Banking and money movement. Transfers between institutions, loan disbursement, multi-step KYC onboarding. Compensation is often not "undo" but "post a correcting entry," which is cleaner because ledgers already think in reversible entries.
• Account signup across systems. Create the user, set up billing, send to the CRM, grant the default workspace. If billing setup fails, tear down the half-created account.

When I reach for it.

• I have a cross-service operation, I cannot use a shared database, and the business can tolerate brief windows of inconsistency.
• If the business truly cannot tolerate any visible intermediate state, a saga is the wrong tool and you need to rethink the service boundaries. That is a legitimate answer in an interview.

When I actively avoid it.

• Anything that fits in one local transaction. Do not build a distributed saga for an operation you could do with one COMMIT.
• I have seen teams add an orchestrator for a two-step flow inside a single bounded context. That is complexity with no payoff.

A saga is a business decision wearing a technical costume: it only makes sense when the business can live with visible partial state.

6Common pitfalls and gotchas#

The sharp edges, each one a thing interviewers listen for.

• Compensations that cannot fully undo. You sent the confirmation email in step 5; you cannot un-send it. The move is to send a correction ("your order was cancelled"), not pretend it never happened. Design compensations as "make it right," not "make it never happened."
• Non-idempotent steps. Buses deliver at least once, so your "capture payment" handler will get the same message twice someday. If it is not idempotent, you double-charge that $79. Every step and compensation needs an idempotency key, usually the saga ID plus step name, checked before doing work.
• Compensation itself fails. The refund times out. You cannot just give up, you captured $79. Retry with backoff, and if it still fails after the budget runs out, the saga parks in a "needs human" state with an alert. Build the dead-letter path from day one.
• Losing saga state on crash. The coordinator state must be persisted transactionally with whatever action it just took. The outbox pattern (write the state change and the outgoing event in the same local transaction, then a relay publishes the event) is the standard fix.
• The lost update from missing isolation. Two sagas touch the same record, one rolls back, the other already acted on the stale value. This is the sharpest edge and the one juniors never see coming.
• Cyclic events in choreography. A's event triggers B, B's triggers A under some condition, and you have an infinite loop in production. Another reason I lean orchestration for anything non-trivial.
• No end-to-end visibility. Without a correlation ID threaded through every step and log line, debugging a failed saga across five services is misery. Thread one saga ID everywhere.

7How it shows up in interviews#

The prompt is usually a transactional flow across services: "Design checkout," "Design a ride-booking flow," "Design fund transfer between banks." The interviewer wants to see you reach for a saga without being told, then reason about the failure cases.

What they probe:

"Why not two-phase commit?" Weak answer: "2PC is slow." True but shallow, and it tells the interviewer you memorized a one-liner. Strong answer: "2PC holds locks across every participating service for the full duration of the transaction, and it blocks if the coordinator dies mid-flight, so a single slow or crashed node freezes everyone. That kills availability and throughput at scale. A saga trades strict consistency for availability and drops the distributed locks entirely, each step commits locally and we clean up with compensations. That trade is the whole reason sagas won in microservices." Notice the strong version names the mechanism (cross-service locks, coordinator blocking) and the trade, not just the symptom.

"Choreography or orchestration, and why?" Weak answer: "I'd use orchestration." Names one with no reasoning, so I have to drag the justification out of you. Strong answer: "This checkout flow is five steps with a failure branch at nearly every step, so I'd orchestrate it. One state machine lets me answer 'where is order 8842 stuck' from a single row, and adding a step is a one-place edit. The cost is the orchestrator becomes a dependency I have to run highly available, so I'd put it behind at least two replicas with its state in a durable store." Picks one, justifies with step count and branching and debuggability, and admits the tradeoff.

"What happens if step 3 fails?" Weak answer: "We roll back." Red flag, because there is nothing to roll back to, the earlier steps already committed. Strong answer: "There's no rollback, the capture at step 2 already happened. The orchestrator runs compensations in reverse: step 3 reserved nothing so it's a no-op, step 2 refunds the $79, step 1 marks the order cancelled. Each compensation is idempotent and keyed by saga ID, and if the refund fails I retry with backoff and park the saga for a human if it exhausts the budget."

"How do you handle the message being delivered twice?" Idempotency keys. If you cannot answer this instantly, it signals you have not run this in production.

"Where's the inconsistency window and who sees it?" This separates the people who have only read about sagas from the people who have operated one. Weak answer: "There's eventual consistency somewhere in there." Strong answer: "Between payment capture at step 2 and the refund completing, the customer has been debited $79 for an order they won't receive. Normally that's about 250 ms. If the refund call times out and we retry, it can reach 30 to 60 seconds, and a cron sweeps stuck sagas every minute to catch stragglers. The customer could see the pending charge on their statement in that tail."

Keep it concrete. Numbers help: "payment capture is 200 ms, if inventory fails we fire the refund async and the window is usually under 2 seconds, with a cron sweeping stuck sagas every minute." That is an engineer who has lived this, not read a blog post.

Next, try the System Design Question walkthrough on designing a checkout and order-fulfillment service, where the saga pattern earns its keep step by step.