Publish-Subscribe

1The core idea#

Pub-sub is how parts of a system talk without knowing about each other.

• A publisher sends a message.
• One or more subscribers receive it.
• Neither side holds a direct reference to the other.
• A broker (also called a message bus or event bus) sits in between and routes each message to whoever asked to hear about that kind of message.

The mental model I use: a bulletin board, not a phone call.

• Phone call (request-response, the pattern most APIs use): you dial a specific number and wait for that specific person to pick up.
• Bulletin board (pub-sub): you pin a note under a topic like "orders" and walk away.
• Anyone who cares about "orders" reads it on their own schedule.
• You have no idea who read it, when, or even how many readers there are.

That one property is the whole point. The publisher not knowing or caring who consumes is called decoupling, and it is what you are really buying when you reach for pub-sub.

Two words you will hear thrown around:

• Topic: a named channel. Publishers write to it, subscribers listen to it. Think payments.completed or user.signup.
• Fan-out: one message landing in many subscribers. Publish an order.placed event once, and the inventory service, the email service, and the analytics pipeline all get their own copy.

2How it works#

Here is the flow when a user places an order and three downstream services need to react.

The write path is short on purpose.

• The order service publishes one order.placed message to the broker.
• The broker stores it durably in a log, then delivers a copy to each subscriber.
• The order service is done the moment the broker acknowledges the write.
• It does not wait for inventory to decrement or for the email to send.
• Checkout stays fast even if the email provider is slow. That is the decoupling paying off.

Now the part people skip over: delivery over time. Subscribers are rarely all healthy at the same instant.

Why this gives you at-least-once.

• The broker holds the message until the subscriber acks "processed cleanly."
• If the subscriber crashes mid-processing and never acks, the broker redelivers.
• So the same message can arrive twice. Plan for it (more on that pain below).

Two broad flavors of broker, and knowing the difference is interview gold:

• Log-based (Kafka, AWS Kinesis, Google Pub/Sub, RabbitMQ with durable queues). Messages are stored on disk and retained for hours or days. A slow or offline consumer catches up later by reading from where it left off. Kafka tracks this position as an offset.
• Fire-and-forget (Redis Pub/Sub, raw WebSocket fan-out). The broker pushes to whoever is connected right now. An offline subscriber misses the message entirely. No replay. Fast and simple, but lossy.

3A worked example with real numbers#

Let me make this concrete instead of hand-wavy. Say we run a mid-size e-commerce site during a flash sale.

The setup.

• Baseline traffic: 500 orders/sec.
• Peak: the sale drives 50,000 orders/sec for a 3-minute burst.
• Each order.placed event is roughly 1 KB of JSON.
• Four services subscribe: inventory, payments reconciliation, warehouse routing, and the confirmation email.

What the broker absorbs during that 3-minute burst (write side).

• Incoming write volume: 50,000 msgs/sec x 1 KB = 50 MB/sec into the broker.
• Over 180 seconds that is 9 million messages and about 9 GB on disk before replication.
• Kafka default replication factor of 3 means roughly 27 GB of disk churn across the cluster for the burst.
• A single modern broker node handles 50 MB/sec of writes without breaking a sweat, so even a 3-node cluster has huge headroom.

The fan-out side is what actually sizes the cluster.

• Four subscriber groups each read the full stream.
• Read volume is 4 x 50 MB/sec = 200 MB/sec out of the broker at peak.
• That read number, not the write number, is what you plan capacity around.

Here is the key move: the email service is the slow consumer.

• It can only send 2,000 emails/sec through our provider.
• During the burst it receives at 50,000/sec but drains at 2,000/sec.
• Without a broker, checkout would block on that 2,000/sec ceiling and 96 percent of orders would time out.
• With the broker, checkout still returns in under 200 ms because it only waits on the 50 MB/sec write, and the email backlog simply grows.

The broker's job is to turn a hard failure into a soft, recoverable delay.

4Trade-offs#

The honest pitch is decoupling and absorbing spikes. The honest cost is that you turned one synchronous call into a distributed system with its own failure modes.

What you gain:

• Independent scaling. The analytics service can be slow and backed up without touching checkout latency.
• Spike absorption. During the flash sale, 50,000 orders/sec land in the broker and drain into consumers at whatever rate they handle. The broker becomes a shock absorber.
• Easy extension. Want to add a fraud-check service next quarter? It subscribes to order.placed. Zero changes to the order service.

What you give up:

• Simple debugging. With a direct API call, a stack trace tells the whole story. With pub-sub, a message left service A and something went wrong "somewhere." You now need correlation IDs and tracing to follow one event across five services.
• Instant consistency. The subscriber reacts milliseconds to seconds later, so your system is now eventually consistent. If a user expects to see their new post immediately but the feed service has not consumed the event yet, you have a product problem to design around.
• Exactly-once as a fantasy. You will get duplicates. Plan for it.

Choosing between the two broker styles comes down to this table:

My default rule:

• If losing a message would make someone angry (payments, orders, audit logs), use a log-based broker.
• If the data is cheap and ephemeral (someone is typing, live viewer count), fire-and-forget is fine and much cheaper to run.

5Where it is used#

Some concrete places I have either built or seen up close:

• Order processing. An e-commerce checkout publishes order.placed. Inventory, payments reconciliation, warehouse routing, and the confirmation email all subscribe. The checkout API returns in under 200 ms because it only waits on the broker write, not on all four downstream jobs.
• Activity feeds. When you follow someone and they post, that post event fans out so each follower's feed can be updated. Twitter-style feed systems live and die on fan-out design.
• Metrics and logging. Services publish events to a topic, and a stream processor reads them to build dashboards, trigger alerts, and load a data warehouse. Analytics being 30 seconds behind is totally acceptable here, which makes pub-sub a natural fit.
• Cache invalidation. A service updates a product's price, publishes product.updated, and every node holding a cached copy subscribes and evicts its stale entry.
• Cross-service notifications. A payment clears, and that single event wakes up loyalty points, receipt generation, and a push notification, none of which the payment service should know about.

Reach for pub-sub when:

• One event has many parties that care about it.
• The producer and consumer run at different speeds.
• You want to add consumers later without touching the producer.

Do not reach for it when:

• It is a single caller needing a single immediate answer (a user clicking "get my balance"). Just make the API call.
• Forcing async on a request that needs an instant reply only adds latency and complexity.

6Common pitfalls and gotchas#

Treating delivery as exactly-once.

• It is not. A consumer that processes a message then crashes before acking will see that message again on restart.
• Fix: make consumers idempotent. Processing the same order.placed twice must not charge the card twice or send two emails.
• Usual mechanism: a dedup key. Store processed message IDs in a table or cache and skip ones you have already handled.

Ignoring ordering.

• Kafka only guarantees order within a single partition, not across the whole topic.
• Route order.placed and order.cancelled for the same order to different partitions and the cancel can arrive first.
• Fix: key messages by something stable (like order_id) so all events for one order land in the same partition and stay ordered.

No dead-letter plan.

• Some message will always fail: malformed, or a bug trips on it.
• If your consumer just retries forever, it blocks the whole partition and nothing moves. This is a poison message.
• Fix: a dead-letter queue. After N failed attempts, shove the message aside for a human to inspect and keep the pipeline flowing.

Unbounded consumer lag.

• If producers outrun consumers, the broker backlog grows.
• In the flash sale, the email service built an 8.64 million message backlog on purpose, which was fine because it drained in 96 minutes and retention was 7 days.
• The danger is lag that never recovers. On a log-based system you hit retention limits and silently drop old messages, or run out of disk.
• Fix: always monitor consumer lag (how far behind the latest offset each consumer is). A rising lag graph that never flattens is your early warning that a consumer is permanently underprovisioned.

Fan-out amplification.

• One celebrity with 10 million followers posts once, and you have generated 10 million feed writes.
• Naive fan-out-on-write melts under this.
• Fix: real systems blend strategies. Fan-out on write for normal users, fan-out on read (pull) for the handful of huge accounts.

Schema drift.

• The producer adds a required field, old consumers break.
• Fix: use a schema registry, or at least version your events and only make backward-compatible changes (add optional fields, never rename or remove).

7How it shows up in interviews#

Interviewers use pub-sub to test whether you understand asynchronous systems, not whether you can name Kafka. Here is what they probe, with the weak answer and the answer that actually lands.

"Why not just call the service directly?" The question is checking if you know why you are adding a broker.

• Weak: "Kafka is scalable." (Names a tool, explains nothing, and would have checkout block on a 2,000/sec email cap.)
• Strong: "The checkout path should not block on the email service, which caps at 2,000 emails/sec. At a 50,000/sec peak, a direct call would time out 96 percent of orders. Decoupling keeps user-facing latency under 200 ms and lets me add consumers later. The cost is eventual consistency and harder debugging, which I handle with tracing and idempotent consumers."

"A consumer was down for an hour. What happens?" This separates people who have run these systems from people who have read about them.

• Weak: "The messages are lost."
• Strong: "With a log-based broker, nothing is lost. The consumer resumes from its last committed offset and catches up. If it fell an hour behind at 2,000/sec, that is 7.2 million queued messages, and it drains them using spare capacity once it is back. I would watch consumer lag during recovery and make sure retention is longer than my worst expected downtime, say 7 days."

"How do you prevent processing a message twice?" They want to hear idempotency and at-least-once.

• Weak: "I will use exactly-once delivery." (Promises a guarantee the broker cannot give end to end.)
• Strong: "I assume at-least-once, so I make the consumer idempotent. I check a unique message ID against a dedup store before any side effect, or I design the operation to be naturally idempotent, like setting a status to PAID rather than incrementing a counter."

"How do you guarantee ordering?"

• Weak: "Kafka keeps everything in order." (Wrong, it is per-partition only.)
• Strong: "I partition by a stable key like order_id so all events for one order stay ordered. I accept that cross-key ordering is not guaranteed, and I question whether I even need global ordering, because I usually do not."

The single best move in these rounds is to state the guarantees out loud: "This gives me at-least-once delivery and per-partition ordering, so my consumers will be idempotent and I will key by order_id." That one sentence, paired with a real number like the 96-minute email catch-up, signals you have actually operated these systems, not just drawn boxes on a whiteboard.

Next up, try the System Design Question walkthrough on building a notification service, where pub-sub fan-out and dead-letter handling are the whole game.