Message Delivery Semantics

1The core idea#

Start with the one failure that drives everything, because every guarantee is just a different answer to it.

The problem in one sentence. The network can drop any packet at any time, and you can never tell "my message was lost" apart from "my message arrived but the reply was lost."

That ambiguity is the whole game. When a producer sends and hears nothing back, it has two moves:

• Give up, and risk losing the message.
• Retry, and risk sending it twice.

You cannot get safety and no-duplicates for free. That is a direct consequence of unreliable networks, not a gap in your tooling.

So we name three guarantees:

• At-most-once. Delivered zero or one times. Never duplicated, sometimes lost. The producer sends and forgets.
• At-least-once. Delivered one or more times. Never lost, sometimes duplicated. The producer retries until it gets an ack.
• Exactly-once. Takes effect once and only once. The most misunderstood of the three.

A mental model that sticks: mailing a check.

• At-most-once: drop it in the mailbox, never check if it arrived.
• At-least-once: mail a fresh check every week until the vendor confirms, and pray they do not cash two.
• Exactly-once: at-least-once plus the vendor writing down your check number so they refuse to cash the same one twice.

That last part is the key insight. Exactly-once is almost never about putting a message on the wire a single time. It is about making the effect happen once even when the message shows up twice. That shifts the burden off the transport and onto the consumer.

Exactly-once is a property of your effects, not your network.

2How it works#

Walk the at-least-once flow in detail, because it is the one you will build 90% of the time.

The happy path.

• A producer writes a message to the broker.
• The broker persists it to disk and sends an ack.
• The consumer pulls the message, does its work, then acks the broker so the message can be deleted.
• If any ack is lost or times out, the sender retries.

That is at-least-once working exactly as designed. The consumer did real work, the broker never heard about it, so the broker replays. Nobody is broken. You just processed id 42 twice.

To climb from at-least-once to effectively-once, add idempotency on the consumer.

• Give each message a stable ID.
• Before processing, check whether you have seen that ID.
• Seen it: skip the work and just ack.
• New: process it, and record the ID in the same transaction as the side effect.

At-most-once is the boring one.

• The producer sends with no retry, or the consumer acks before doing the work.
• Fast, simple, lossy.
• Fire a UDP packet and move on.

3A concrete worked example: a wallet debit service#

Abstract guarantees are easy to nod along to and hard to apply, so ground it in numbers from a system you would actually design.

The setup.

• A wallet service debits user balances when an order is placed.
• 2,000 orders per second at peak, each debit $12 on average.
• A queue sits between the order service (producer) and the wallet service (consumer).

Now inject the failure.

• The broker redelivery timeout is 30 seconds.
• The wallet consumer takes about 80ms to debit, but under a GC pause or a brief database stall it occasionally takes 35 seconds.
• When a debit crosses the 30-second visibility timeout, the broker decides the message was never handled and redelivers it.
• The original worker is still running, finishes its debit, and a second worker now debits the same order.

Put a number on the blast radius.

• Suppose 0.5% of messages cross the 30-second timeout during a bad database window lasting 10 minutes.
• At 2,000 per second that is 1.2M messages in the window.
• 0.5% of those duplicate, so 6,000 double-debits at $12 each.

Here is the same path with an idempotency key, which costs almost nothing. The order service stamps each order with a key like debit:order:88431. The wallet consumer runs this inside one transaction:

BEGIN;
INSERT INTO processed_debits (idempotency_key)
VALUES ('debit:order:88431')
ON CONFLICT (idempotency_key) DO NOTHING;
-- rows affected: 1 means first time, 0 means duplicate
UPDATE wallets
SET balance = balance - 1200   -- cents
WHERE user_id = 7781 AND <rows affected above = 1>;
COMMIT;

What that does.

• First delivery: the INSERT succeeds, the UPDATE fires, balance drops by $12.
• Redelivery: the INSERT hits the unique constraint and affects 0 rows, so the UPDATE is skipped and the worker just acks.
• Both the dedup key and the balance change live in one transaction, so there is no window where one commits without the other.

What did it cost?

• One extra table, one unique index, one INSERT per message.
• At 2,000 per second that is 2,000 extra small writes per second, which a single Postgres primary handles without noticing.
• Measured latency went from about 80ms to about 83ms per debit.

The trade in one line: pay 3ms to turn a $72,000 incident into a non-event, every single time. A near-free consumer-side check converts cheap, durable at-least-once transport into exactly-once effects.

4Trade-offs#

Every step up in guarantee costs you latency, throughput, or complexity. There is no free lunch.

The decisions underneath the table:

• At-most-once trades durability for speed. You accept that some messages vanish. For sensor readings arriving every 100ms, losing one is noise, and paying for retries plus dedup on that firehose is wasteful.
• At-least-once trades simplicity for durability. You will see duplicates, period, so your consumers must tolerate them. Forget this and you ship a bug that only appears under failure: double charges, double emails, double inventory decrements. The wallet example is that bug with a dollar figure attached.
• Exactly-once trades throughput and portability for correctness. True end-to-end exactly-once across independent systems needs coordination (two-phase commit or a transactional outbox). Coordination is slow and couples your components. The honest version most teams ship is at-least-once delivery with idempotent consumers, which buys exactly-once effects without the distributed-transaction tax.

The rule of thumb I use: pick the weakest guarantee that keeps the system correct.

• Correctness is defined by the side effect, not by the message.
• Logging a view event? At-most-once is fine.
• Debiting a wallet? You need effectively-once, which means idempotency keys on the write path.

5Where it is used#

Map the guarantees onto real systems, because interviewers love a candidate who knows what the brokers actually default to.

• Kafka defaults to at-least-once. You can get at-most-once by committing offsets before processing. Kafka also offers exactly-once within its own boundary: the idempotent producer (a producer ID plus sequence numbers so the broker drops duplicate writes) and transactions that tie a consume-process-produce loop into one atomic unit. Watch the boundary: that guarantee holds inside Kafka. The moment you write to an external database, you are back to needing your own idempotency.
• SQS standard queues are at-least-once with no ordering. SQS FIFO adds ordering and a 5-minute dedup window keyed on a dedup ID. The catch is in the window: FIFO deduplicates within 5 minutes, not forever. A retry that lands at minute six is a duplicate.
• RabbitMQ is at-least-once with manual acks, or at-most-once with auto-ack (the broker considers the message delivered the instant it hands it over).
• Payment and order systems almost always run at-least-once transport with a client-supplied idempotency key. Stripe is the textbook case: you send an Idempotency-Key header, and retrying the same create-charge call returns the original result instead of charging twice. That is at-least-once on the wire turned into exactly-once effect at the API. The wallet SQL above is the same pattern behind your own endpoint.

The selection rule, restated for the room:

• Stronger semantics when the side effect is irreversible or expensive to undo: money, inventory, emails, provisioning.
• Weaker semantics when the data is high-volume and individually cheap: metrics, logs, clickstream, cache invalidation hints.

6Common pitfalls and gotchas#

These are the specific ways people lose the guarantee they thought they had.

• Acking before you finish the work. This silently downgrades you to at-most-once. Crash after the ack but before completing, and that message is gone with no resend. Ack only after the work is durably done.
• Assuming "exactly-once" on the box means your effects are safe. A broker's exactly-once mode protects its own log. Your database write, your email send, your call to a third party: none of that is covered. The dedup has to live where the side effect lives.
• Dedup windows that are too short. FIFO SQS forgets after 5 minutes. A dedup table pruned nightly forgets after a day. A delayed retry or a replayed partition that outlives your window sails through as a duplicate. Size the window to your worst-case retry horizon, including manual replays during incidents.
• Non-atomic dedup. Writing the result and the dedup key in two separate transactions reintroduces the double-effect bug in the gap between them. One transaction, or nothing. In the wallet example, splitting the INSERT and the UPDATE into two commits is exactly how you lose the guarantee.
• Ignoring poison messages. A message that always fails to process, under at-least-once, retries forever and can block a partition or burn your consumer in a tight loop. You need a retry cap and a dead-letter queue.
• Confusing ordering with delivery. They are separate axes. At-least-once says nothing about order. If your logic depends on order, you need per-key partitioning on top of the delivery guarantee, and even then duplicates can arrive out of order relative to each other.

7How it shows up in interviews#

Interviewers use this topic to separate people who have operated systems from people who have only read about them. The tell is whether you reach for "exactly-once" as magic or decompose it.

What they probe:

• "The consumer processes the message, then crashes before acking. What happens?" They want: the broker redelivers, so the consumer must be idempotent.
• "How do you actually get exactly-once?" Strong: you do not get it from the transport in general. You get exactly-once effects with at-least-once delivery plus an idempotency key and a transactional write. Weak: "I'll enable exactly-once mode."
• "Where does the dedup key live and when do you write it?" They are checking whether you know it must be atomic with the side effect.

Weak vs strong, side by side.

Even stronger: pull in the worked numbers. "If a debit takes longer than the 30-second redelivery timeout, the broker redelivers and I double-debit. At 2,000 per second, a 0.5% timeout rate over a 10-minute bad window is 6,000 double-debits at $12, so $72,000 wrong. The fix is a unique idempotency key written in the same transaction as the balance change, which added about 3ms. I will pay 3ms to kill a $72,000 incident." Naming the dollar figure and the latency cost moves you from "knows the vocabulary" to "has carried the pager."

And close the loop on why the trade-off exists. Say "the ambiguity is that a lost ack looks identical to a lost message, so the sender has to choose retry or drop." Showing you understand why, not just what, is the staff-level finish.

For a full walkthrough, see the System Design Question on building an idempotent payment processing pipeline.