The Outbox Pattern

1The core idea#

The one sentence to carry in: you cannot do a reliable atomic write across two different systems, so stop trying.

• Your database does not know about Kafka. Kafka does not know about your Postgres transaction.
• There is no shared commit between them, so any code that writes to one and then publishes to the other has a gap where exactly one of the two can succeed.
• This is the dual-write problem. No amount of retry logic or try/catch cleverness closes it, because the failure can happen between the two calls, after the first one already committed.

The move: take the second system off the critical path.

• Instead of writing your business row and then publishing an event, write the business row and the event into the same database, in the same transaction.
• The event goes into a plain table, usually called outbox.
• Both writes hit the same database, so they commit together or roll back together. There is no middle state.
• You turned a two-system problem into a one-transaction problem, and your database already solves one-transaction problems perfectly with ACID.

Then publish out of band.

• A separate process reads unpublished rows out of the outbox table and ships them to Kafka (or SNS, or RabbitMQ, or wherever).
• If that process crashes, nothing is lost: the event is sitting safely in the table.
• When it comes back up, it reads the table again and keeps going from where it left off.

2How it works#

Walk through placing an order. The service receives "create order," opens a transaction, and inside that one transaction does two inserts: one into orders, one into outbox.

• The outbox row holds the event payload as JSON, a type like OrderCreated, a destination topic, and a published_at column that starts null.

BEGIN;
INSERT INTO orders (id, user_id, total_cents, status)
VALUES ('ord_9a3', 'usr_42', 4999, 'CONFIRMED');

INSERT INTO outbox (id, aggregate_id, type, topic, payload, published_at)
VALUES ('evt_7c1', 'ord_9a3', 'OrderCreated', 'orders', '{"orderId":"ord_9a3","total":4999}', NULL);
COMMIT;

At commit, you are safe.

• The order exists and the intent to publish exists, bound together atomically.
• If the commit fails, neither row exists, the client gets a 500, and it retries.
• There is no version of the world where the order exists but the event does not.

Now the relay moves events from the table to the broker. People also call it the message relay or the publisher. Two common ways to build it:

• Polling (the simple way). A worker runs SELECT ... FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100 every second or so, publishes each event, then marks the row by setting published_at = now(). Easy to build, easy to reason about, and honestly fine for most teams.
• Log tailing (the better-at-scale way). A tool like Debezium reads the database write-ahead log directly and streams every new outbox row into Kafka. No polling query hammering your primary, lower latency, and it rides the same log the database already keeps for replication, so you reuse something that exists instead of adding read load.

So the delivery guarantee is at-least-once, not exactly-once.

• The relay can publish and then crash before marking the row, so it resends on restart.
• Do not fight it. Put a stable ID on every event and make consumers idempotent, so a duplicate is a no-op.

3A worked example with real numbers#

Make "eventually" concrete, because interviewers love to pin you on it. Say you run a checkout service at 500 orders per second at peak, each order emitting one event.

Latency (how long an event waits before it leaves the table).

• A polling relay on a 1 second interval with LIMIT 100 drains up to 100 rows per tick. At 500 events/sec that is 500 new rows per second but only 100 pulled, so one worker falls behind immediately.
• Fix it with the interval, the limit, or more workers. Bump to a 200 ms interval with LIMIT 200 and one worker clears 1,000/sec with headroom.
• A typical event now waits 0 to 200 ms before it leaves the table, call it ~100 ms average publish lag.
• With CDC via Debezium tailing the WAL, the same event is in Kafka in under 50 ms, because there is no poll wait: the relay reacts to the commit itself.

Table growth (why pruning is not optional).

• 500 events/sec times 86,400 seconds is 43 million rows per day.
• At ~500 bytes of payload plus indexes per row, that is around 21 GB per day landing in one table.
• Leave the reaper off for a month and you have 1.3 billion rows, and a published_at IS NULL scan that took 2 ms now takes seconds.

Reaper sizing (holding the table to a sane size).

• Delete published rows older than 1 hour and steady state is 500 times 3,600, about 1.8 million rows, not a billion.
• Delete in batches (say 5,000 rows per statement, every few seconds) so the deletes do not take a long lock and stall the relay reads.
• On Postgres those deletes create dead tuples fast, so autovacuum has to keep up or the table bloats even though the live row count is small.

That is the whole shape: single-digit to low-hundreds of milliseconds of lag, tens of millions of rows a day, a reaper that holds the table to low millions. Walk an interviewer through those three numbers and you have proven you understand the pattern operationally, not just conceptually.

4Trade-offs#

Nothing here is free. The core cost: you traded strong, synchronous consistency for eventual consistency plus duplicate handling. The event will arrive, but a little late (the 50 to 200 ms above) and maybe twice.

The main decision inside the pattern is how you run the relay.

My default: start with polling.

• One worker and one query carries you to a few thousand events per second before the constant query pressure on the primary matters.
• Reach for CDC only when latency has to be sub-100 ms or the poll load itself is a problem, not before.
• Adding Debezium and Kafka Connect to a team that has never run them is a new on-call rotation, and that cost is easy to underweight in a design review.

The table grows forever unless you clean it.

• Every event you ever emit lands there: 43 million rows a day at 500 events/sec, as counted above.
• You need a reaper that deletes or archives published rows, tuned so the deletes do not fight the relay reads or bloat the table.
• On Postgres, heavy churn means you care a lot about autovacuum, and many teams move the outbox to its own tablespace or even its own database to isolate that churn from the business tables.

Publishing is now coupled to your database availability and write path.

• If the database is down, you are not taking writes anyway, so that part is acceptable.
• But a slow outbox insert now slows your business transaction, because they share the commit.
• In practice the insert is cheap (a single row with a primary key), but under a hot lock or a bloated table it stops being cheap.

The bigger alternative: Listen to Yourself / event sourcing.

• There the event log is the source of truth and the database is a projection, which flips the whole model.
• It is powerful and a much larger commitment: every read rebuilds from events, and every schema change becomes an event-versioning problem.
• Outbox is the pragmatic middle: keep your normal relational model, bolt on reliable publishing. For most teams that is the right call.

5Where it is used#

Any time a state change in one service must reliably trigger work in another, and you cannot tolerate the two drifting apart, this fits.

Concrete cases.

• E-commerce checkout writes the order and emits OrderPlaced so the warehouse, email service, and loyalty system all react.
• A payments service records a successful charge and publishes PaymentCaptured so the subscription activates exactly once.
• A signup flow writes the account row and emits UserRegistered to kick off a welcome sequence.
• In all of these, a lost event means a customer paid and got nothing, which is the kind of bug that ends up in a postmortem with your name on the timeline.

It is the backbone of the Saga pattern.

• A saga is a chain of local transactions across services, coordinated by events.
• Each step needs its event to actually get published or the saga stalls halfway, leaving an order paid for but never shipped.
• Outbox is how you make each hop reliable. If asked to design order fulfillment across order, payment, and inventory services, outbox is the mechanism carrying the saga's messages, and naming it unprompted is a strong signal.

It is baked into frameworks too.

• Debezium markets outbox as a first-class use case.
• Plenty of large systems run exactly this: a boring table and a CDC pipe.
• The pattern is unglamorous, which is precisely why it is everywhere.

6Common pitfalls and gotchas#

Publishing inside the transaction, or right after.

• The whole point is to not touch the broker during the business transaction.
• If someone "optimizes" by calling kafka.send() right after commit in the same request, you are back to the dual-write bug, because the process can die in that gap.
• Keep the relay separate and asynchronous.

Assuming exactly-once.

• It is at-least-once. If consumers are not idempotent, duplicates corrupt state: a card charged twice, two welcome emails.
• Stamp a unique event ID and dedupe on the consumer, usually with a seen-IDs table or a unique constraint.
• This is the single most common thing juniors get wrong here.

Losing ordering.

• If AccountDebited and AccountCredited arrive out of order, balances go negative mid-flight and trip fraud alarms.
• If order matters, publish from the outbox in id order and key Kafka messages by the aggregate (for example account ID) so they land on the same partition and stay ordered.
• Events for different accounts can still spread across partitions for throughput.

Two relays double-publishing.

• Run two polling workers without coordination and both grab the same unpublished rows, so every event ships twice.
• Use SELECT ... FOR UPDATE SKIP LOCKED, or a single-leader relay, or partition the work by id range.
• Do not just hope only one instance is running, because the day it scales to two is the day it breaks.

Forgetting to prune.

• Covered above, but it bites teams months later when the table is 2 billion rows and every relay query has gone slow.
• Ship the reaper in the first PR, not the first incident.

Serializing the wrong thing.

• Store the event as a serialized Java or C# object, change the class later, and old rows will not deserialize, so the relay throws on real events.
• Store a language-neutral payload (JSON, Avro, or protobuf) with a schema version field so old rows stay readable after the code moves on.

7How it shows up in interviews#

Interviewers reach for this the moment your design says "save X, then publish an event." The follow-up is almost always: "what happens if the service crashes right after the database write but before the message is sent?" That question is the outbox question.

Interviewer: Your service writes the order, then publishes an event to Kafka. What happens if it crashes right after the DB write but before the publish?

Candidate: Then the order exists and the event never fires, and the two systems disagree forever. That is the dual-write problem. The fix is to write the event into an outbox table in the same transaction as the order, so they commit atomically, and have a separate relay publish from that table.

Interviewer: Fine. Does that give you exactly-once delivery?

Candidate: No, at-least-once. The relay can publish to Kafka and then crash before it marks the row published, so on restart it resends. I make every consumer idempotent on a stable event ID so a duplicate is harmless.

The contrast that separates candidates.

• A weak answer stops at the name: "I will use an outbox table so the write and the event are atomic." True, but it says you read the pattern, not ran it. They keep probing and now you are improvising.
• A strong answer keeps going unasked and front-loads the hard parts.

"I insert the business row and an outbox row in one transaction, so they commit together. A relay then publishes from the table. It is at-least-once, not exactly-once, because the relay can publish and then crash before marking the row, so every consumer must be idempotent on a stable event ID. For the relay I start with polling on a 200 ms interval, which gives about 100 ms of publish lag, and move to Debezium CDC for sub-50 ms if latency or DB load demands it. At 500 events/sec the table grows about 43 million rows a day, so I need a reaper trimming published rows older than an hour, which holds it to about 1.8 million rows. If ordering matters I publish in id order and key Kafka by the aggregate ID. And I guard against two relays with SELECT FOR UPDATE SKIP LOCKED."

• Same pattern, wildly different signal. The second answer names the delivery guarantee, the latency numbers, the growth math, the ordering fix, and the concurrency guard before anyone asked. That is a candidate who has operated this.

Next, try the System Design Question walkthrough on designing an order-processing system with the Saga pattern, where the outbox becomes the thing carrying every message between services.