Dead Letter Queues

1The core idea#

Start with the mental model, then name the failure it prevents.

The setup.

• You have a queue. Consumers pull messages off it and do work.
• Most of the time the work succeeds and the message goes away.
• Some messages are poison and fail every time.

What "poison" looks like in practice.

• The payload is malformed JSON that fails to deserialize.
• It references a user_id that was deleted an hour ago.
• A downstream service returns a 500 every single time for that one record, because of a bug that only triggers on that exact shape of data.

Leave a poison message on the main queue and one of two bad things happens.

• It blocks the queue behind it (in strict-ordering systems).
• It gets retried forever, burning CPU and spamming error logs while never succeeding.

A dead letter queue (DLQ) is the fix.

• A separate queue where you send messages that have failed too many times.
• You stop retrying them on the hot path and set them aside so a human or a batch job can look later.
• The main queue keeps flowing. The bad message is preserved, not dropped.
• The whole trick in one line: fail loud, fail isolated, do not lose data, do not block the line.

2How it works#

The mechanics are simple once you see where the retry count lives.

The basic loop.

• Every message carries a delivery or receive count.
• When a consumer fails, it either nacks the message or lets the visibility timeout expire instead of acking.
• The message becomes visible again and gets redelivered. The broker tracks how many times this has happened.
• Once the count crosses a threshold you set (say 5), the broker moves the message to the DLQ instead of back onto the main queue.

Where the retry count actually lives (per broker).

• The count is in message metadata, not your database.
• SQS calls it ApproximateReceiveCount.
• RabbitMQ uses a mix of the x-death header and TTL or reject behavior.
• Kafka has no native DLQ, so you build it: the consumer catches the exception, publishes the record to a separate orders.DLT topic, and commits the offset so the main partition advances.

The DLQ message should carry context, not just the raw payload.

• Wrap the original message with the source queue, the last exception, a timestamp, and the final retry count.
• Without that, someone staring at the DLQ three days later has no idea why anything landed there.

Here is roughly what that envelope looks like in practice:

{
  "original_queue": "orders-processing",
  "failed_at": "2026-11-03T04:12:55Z",
  "receive_count": 5,
  "last_error": "DownstreamTimeout: payments-svc 504 after 2000ms",
  "payload": { "order_id": "ord_9f3a", "user_id": "usr_771", "amount_cents": 4999 }
}

The DLQ is itself just a queue, which is the point.

• Attach a consumer to it.
• Set up a cron that replays messages back to the main queue after a fix ships.
• Export its contents to object storage for an audit trail.
• It is not a black hole. It is a holding area with its own lifecycle.

Here is the flow over the life of one poison message:

3A worked example with real numbers#

Reason with numbers, not vibes. Interviewers reward it.

The setup.

• An order-processing pipeline. Steady state is 2,000 orders per second through SQS into a fleet of consumers that call a downstream payments-svc.
• Retry policy: maxReceiveCount = 5 with exponential backoff.
• Visibility timeout: 30 seconds.

The bug.

• payments-svc throws a divide-by-zero and returns 500 on any order where amount_cents is exactly 0 (a free-trial promo).
• About 0.1% of orders hit this path, so 2 orders per second are poison.
• The other 1,998 per second are fine.

Walk the math on one poison message.

• It is delivered, fails, waits out the visibility timeout, comes back, fails again, 5 times before it dead-letters.
• With backoff that is roughly 30s + 60s + 120s + 240s + 480s, call it about 15 minutes of churn per bad message before it lands in the DLQ.
• During that window it consumes a consumer slot 5 times.
• Across all poison traffic: 2 per second times 5 attempts is 10 extra consumer invocations per second burned on garbage, plus 10 error logs per second, plus 10 failed calls per second into an already-unhappy payments-svc.

With the DLQ, the same bug becomes a bounded blip.

• After 15 minutes each poison message exits the hot path.
• Steady state settles at roughly 2 messages per second accumulating in the DLQ: about 7,200 per hour, 172,800 per day.
• Main queue latency for the 1,998 good orders per second stays flat, because poison traffic is capped at 5 attempts and then gone.
• The DLQ-depth alarm fires at minute 16, on-call sees last_error: divide by zero on amount_cents=0, ships a fix, and replays the 172,800 messages through a throttled redrive at, say, 500 per second so payments-svc does not get re-overloaded. That replay drains in under 6 minutes.

Same bug, two outcomes.

• Without the DLQ: a self-inflicted outage and a buried root cause.
• With it: a bounded blip, a precise error message, and a clean recovery.
• That contrast is the entire value proposition, and it is the thing to say out loud in an interview.

4Trade-offs#

Almost every decision here is about retry policy and what you do once a message is dead.

Decision 1: how many retries before dead-lettering.

• Too low and you DLQ transient failures that would have passed next attempt, like a 2-second database blip.
• Too high and a genuinely poisoned message wastes resources and delays the signal that something is broken.
• I usually start at 3 to 5 attempts with exponential backoff, then tune based on what actually shows up in the DLQ.

Decision 2: retry on the same queue, or use a separate retry queue.

• Immediate in-place retries hammer a downstream service that is probably already struggling.
• A staged approach uses retry queues with increasing delays (5s, 30s, 5m) before the DLQ.
• More moving parts, but far kinder to a flaky dependency.

Decision 3: what the DLQ actually means for your data.

• For a payment event, a DLQ message is a real financial action that did not happen, so it needs a human and an SLA.
• For an analytics event, you might be fine draining the DLQ to cold storage and moving on.

5Where it is used#

Every broker exposes the same idea with different plumbing.

Amazon SQS has first-class DLQ support.

• Create a second queue, set it as the main queue's dead letter target, set maxReceiveCount.
• After that many failed receives, SQS moves the message automatically.
• This is probably the most common DLQ you will see in the wild.

RabbitMQ does it with dead letter exchanges.

• Mark a queue with an x-dead-letter-exchange argument.
• Rejected or expired messages get republished to that exchange, which routes them to your DLQ.
• Flexible, a little fiddly to configure correctly.

Kafka has no built-in DLQ because of its log model.

• You cannot just pull one record out of a log.
• Frameworks like Kafka Connect and Spring Kafka give you a dead letter topic pattern: on failure, produce the bad record to a .DLT topic and move the consumer offset forward.
• Same idea, built in application code.

When to reach for one.

• Whenever a message failing is not safe to ignore and not something you want retried into the ground.
• Order processing, email and notification pipelines, webhook delivery (a customer's endpoint is down for an hour), payment reconciliation, and any event-driven flow where losing an event has real consequences.
• If your messages are pure best-effort fire-and-forget telemetry, a DLQ may be more machinery than you need.

6Common pitfalls and gotchas#

These are the ones that turn a DLQ from a safety net into a second failure.

• No alerting on the DLQ. The most common mistake by far. The DLQ fills up quietly and nobody notices until a customer complains. Put a metric and an alarm on DLQ depth. If DLQ depth > 0 for more than a few minutes on a critical queue, someone should get paged.
• Non-idempotent consumers. If your consumer charges a card, then fails on the next line before acking, the retry charges the card again. Retries and DLQs assume processing is safe to repeat. Make work idempotent first, usually with a dedupe key, or retries become their own incident.
• Dropping the failure context. Moving the raw message to the DLQ without the exception, source queue, or timestamp means every investigation starts from zero. Wrap the message with metadata on the way in.
• No replay path. You fix the bug, deploy, and now have 12,000 messages stuck in the DLQ with no tooling to put them back. Build the replay path early, even if it is just a small script. And replay carefully: dump everything back at once and you can re-trigger the same overload that caused the failures.
• DLQ for transient failures. If a short database outage sends thousands of messages to the DLQ, your retry policy is too aggressive or your backoff is missing. The DLQ should mostly catch genuinely broken messages, not a dependency that was down for 90 seconds.
• The DLQ itself overflowing. It is a queue with its own retention, often 14 days on SQS. If messages sit past retention, they vanish. The side table has a size limit too.

7How it shows up in interviews#

Interviewers rarely ask "what is a dead letter queue" directly.

It comes up as a follow-up.

• You sketch any async or event-driven design and they ask: "what happens when this consumer fails?" or "what if this message can never be processed?"
• That is your cue.

Here is the contrast that decides the signal:

Weak answer: "We retry the message."

• The interviewer waits. You add, "a few times, and if it keeps failing we log it."
• No end state. It quietly assumes failures are always temporary.
• No isolation story, so one bad record can still block or flood the queue.
• No way anyone learns the retry gave up. "We log it" means the data is gone and nobody will see that line among the noise.
• This is the answer of someone who has read about queues but never been paged by one.

Strong answer: isolation, bounded retries, observability, recovery, and idempotency in one breath.

"I'd retry with exponential backoff up to 5 attempts. If it still fails, I'd move it to a dead letter queue so it doesn't block the main queue or get retried infinitely. I'd alarm on DLQ depth so we know within a minute or two, and I'd keep the original payload plus the exception and source queue so we can debug and replay after a fix. Replay would be throttled, say 500 per second, so I don't re-trigger the overload that caused the failures. And since retries only work if processing is safe to repeat, the consumer uses a dedupe key on an idempotency store."

• That answer is obviously written by someone who has operated this.

Expect these probes:

• "Why not just retry forever?" Poison messages, wasted resources, and in ordered systems, head-of-line blocking.
• "What about ordering?" In a strict FIFO queue, a stuck message blocks everything behind it. The DLQ lets the line keep moving, but you need to reason about whether out-of-order processing is acceptable for that data.
• "How do messages get out of the DLQ?" This separates people who have operated systems from people who have only drawn boxes. Talk about manual review, a replay job, and throttled redrive so you do not re-overload the system.
• "What makes retries safe at all?" Idempotency. If they ask this and you have not mentioned it, bring it up yourself.

A DLQ is an operational commitment, not a config flag.

The strongest signal you can send is treating it that way. Mention the alarm, the on-call runbook, and the replay tooling. That is what tells an interviewer you have been the person paged at 3am by a queue full of messages nobody was watching.

For a walkthrough that puts this to work, see the System Design Question on building a reliable webhook delivery system, where retries, backoff, and a DLQ are the heart of the design.