Event-Driven Architecture

1The core idea#

Start with the mental model, because everything else hangs off it.

• In a normal request-response system, Service A calls Service B and waits for an answer. A knows B exists, knows its address, and cares whether B succeeds.
• That is a command: "do this thing, now, and tell me it worked."
• Event-driven architecture flips the direction of knowledge. Service A does its work and announces a fact: "order 8821 was placed." It does not know who listens. It does not wait.
• Maybe three services react, maybe zero, maybe seven next quarter. A just fired the event and moved on.

That flip, from "call this service" to "announce this fact," is the entire idea. Everything else is plumbing.

The payoff is loose coupling.

• The order service does not import the email service, the analytics service, or the loyalty-points service.
• It emits OrderPlaced and those three teams subscribe on their own.
• You can add a fourth consumer next month without touching the order service at all.
• That is the thing everyone falls in love with.

2How it works#

Three roles, one broker in the middle.

• Producers emit events.
• A broker holds and routes them (usually Kafka, RabbitMQ, AWS SNS/SQS, Google Pub/Sub, or similar).
• Consumers react on their own schedule.

Two broad shapes of broker, and the difference matters.

• Queue model (RabbitMQ, SQS): a message is delivered, a worker processes it, then it is acknowledged and gone. Think of a to-do list that empties out. Good when one worker among many should handle each message.
• Log model (Kafka): events are appended to an ordered log and kept for a retention window, say 7 days. Consumers track their own position (offset). A new consumer can start from the beginning and replay history, and the same event can be read by many independent consumer groups. That replay ability is a big deal, and I come back to it.

Now the part that separates real answers from hand-waving: how do you atomically update your database and publish an event?

• Say the order service writes the order row and then publishes OrderPlaced. Two writes, two systems.
• If the DB commit succeeds but the broker publish fails (process crashes in between), you have an order with no event. Downstream, nobody knows it happened.
• This is the dual-write problem, and it is the single most important mechanic in this topic.

The standard fix is the transactional outbox.

• In the same database transaction that writes the order, you insert a row into an outbox table. Both commit together or neither does.
• A separate process, the relay, reads the outbox and publishes to the broker, marking rows as sent.

An upgrade I mention when the interviewer pushes.

• Instead of polling the outbox table, use change-data-capture (Debezium tailing the Postgres write-ahead log) so the relay reacts to committed rows with near-zero lag instead of a polling interval.
• Same guarantee, lower latency, no extra load on the primary.

3A worked example with real numbers#

"It scales" means nothing without arithmetic, so let me make it concrete.

• A mid-size store doing 2 million orders per day.
• Daily peak of roughly 120 orders per second.
• A Friday evening rush pushes it to 600 per second for about 20 minutes.

Sizing the broker first, to show it is a non-issue.

• One OrderPlaced event fans out to 5 consumers: email, inventory, fraud scoring, loyalty points, analytics.
• At peak that is 600 x 5 = 3,000 consumer deliveries per second.
• Each event is a fat JSON payload of about 2 KB, so sustained peak throughput is 3,000 x 2 KB = 6 MB/s.
• That is trivially inside what a single Kafka broker handles (a modest broker does 50 to 100 MB/s).

Now the part that justifies the whole pattern: the checkout call itself.

Sizing the disk, so retention is not hand-waved.

• At 2 KB per event and 2M orders/day, raw order-event volume is 2M x 2 KB = 4 GB/day.
• With a 7-day Kafka retention window and replication factor 3, that is 4 GB x 7 x 3 = 84 GB for this one topic. Cheap.
• That retention is what lets a new analytics consumer replay a week of history, or a crashed consumer resume from its offset.

4Trade-offs#

This is where interviews are won. Loose coupling is not free. Here is the honest ledger.

The three trade-offs I push candidates hardest on.

• Strong consistency for eventual consistency. When the order service emits OrderPlaced and returns success, inventory may not be decremented yet. For a few hundred milliseconds, or a few seconds under load, the system is inconsistent. Usually fine. Sometimes not: if two customers can both "reserve" the last item because the event has not propagated, that is a real bug, and you may need a synchronous check at that exact point.
• Easy debugging for operational complexity. One failed request gives you one stack trace. In an event system, a single user action can spawn a tree of events across six services. You need correlation IDs stamped on every event and distributed tracing to reconstruct the story. Without that, you are reading six log files guessing the order. I have lived this. It is miserable when you did not plan for it.
• Predictable load for async buffering. This one is actually a win, so say it out loud. We saw it in the example: a 10x spike that would topple a synchronous database gets absorbed as a 240,000-event backlog that drains in 10 minutes. You pay in latency instead of downtime. That is often exactly the trade you want.

5Where it is used#

A few real shapes so this is not abstract.

• E-commerce checkout. One OrderPlaced event drives email confirmation, inventory reservation, fraud scoring, loyalty points, warehouse notification, and analytics. These teams ship independently, and nobody wants checkout to synchronously wait on loyalty points. This is exactly the worked example above.
• Ride-hailing and food delivery. A driver's location update is an event flowing to the rider's map, the ETA engine, and the dispatch system. High volume, many consumers, order matters per driver. A textbook Kafka use case.
• Bank and ledger systems. Event sourcing, a cousin of this pattern, stores every change as an immutable event: AccountOpened, MoneyDeposited, MoneyWithdrawn. The current balance is just the sum of the events, so you get a perfect audit trail for free and can rebuild state at any past point. Regulators love it. The cost: reading current state means replaying events, so you keep a materialized view on the side.

When would I reach for it?

• When multiple independent consumers care about the same thing.
• When producer and consumer scale differently.
• When you want to absorb spikes.
• When you genuinely need the history of what happened.

When would I not?

• When the caller needs an immediate answer to proceed (a password check, a payment authorization the user is staring at).
• When strong consistency is non-negotiable and the extra moving parts are not worth it.

Please do not add Kafka to a two-service CRUD app.

6Common pitfalls and gotchas#

• Treating "at-least-once" like "exactly-once." Most brokers guarantee a message is delivered at least once, which means sometimes twice. If your consumer charges a card or decrements stock on each delivery, a duplicate costs real money. Fix: idempotency. Give each event an ID, record processed IDs, skip anything you have already seen. Exactly-once end to end is extremely hard, and what people usually mean is "at-least-once plus idempotent consumers." Say it that way.
• No dead-letter queue. A message that keeps failing (bad data, a bug) will retry forever and block the ones behind it, or get silently dropped. Route poison messages to a dead-letter queue after N attempts so a human can look while the rest of the pipeline keeps moving.
• Ignoring ordering. Kafka only guarantees order within a partition. Spread one user's events across partitions and they can process out of order, so you might apply MoneyWithdrawn before MoneyDeposited. Partition by a stable key (user ID, account ID) when order matters for that entity.
• Fat events vs thin events. A thin event says "order 8821 changed, go look it up," which is simple but makes every consumer hammer the order service with lookups, recreating tight coupling through the back door. A fat event carries the full payload, no callback needed, but payloads get large and can go stale. In the worked example I chose 2 KB fat events precisely so the five consumers never call back. I lean toward events carrying enough to be useful on their own, and reserve thin events for genuinely huge payloads.
• Schema changes with no plan. The producer adds a required field, old consumers choke. Use a schema registry and only make backward-compatible changes (add optional fields, never remove or repurpose one). This bites teams constantly once more than two services share an event.

7How it shows up in interviews#

Expect event-driven architecture inside larger prompts: "design a notification system," "design a food-delivery order flow," "design a ticket-booking system." The interviewer rarely wants a textbook definition. They want judgment.

Here is the weak-versus-strong contrast I score against. One more thing that always lands: show you know the limits. Saying "I would keep the payment authorization synchronous because the user needs an answer before the page can proceed" is a stronger signal than any place you do apply the pattern.

Next, try the System Design Question walkthrough on designing a notification system, where the outbox, fan-out, and idempotency ideas from this guide all come together in one problem.