Idempotency: A System Design Guide

If you remember five things

The whole concept in one glance. Everything below is the defense of these five calls.

At-least-once is free, exactly-once is built. The network hands you duplicate delivery for nothing. Exactly-once effects are something you construct, and idempotency is the main tool.
One key per intent, reused across retries. The client mints the key once and sends the same one on every retry of that one logical operation. A fresh key per attempt buys you nothing.
Atomic insert, not check-then-write. A unique constraint on the key decides the race, so two simultaneous duplicates cannot both charge the card.
Store the full result, with a TTL. Save the body and status code, not a boolean, so a retry replays the exact response. Expire keys past your real retry horizon (24 hours is typical).
Guard every side effect. The DB write is only half of it. The email, the queue publish, the external charge all need the same gate, or the retry resends them.

1The core idea#

Start from a definition you can say in one breath, then sort the HTTP verbs the way interviewers like.

An operation is idempotent if doing it once and doing it ten times leaves the system in the same state.

That is the whole definition.
The result the caller sees might differ slightly. The observable state of your data does not drift with each repeat.

The verbs, sorted (interviewers love this framing):

GET is idempotent and safe. Reading a product page a hundred times changes nothing.
PUT is idempotent. "Set the user's email to a@b.com" lands on the same state no matter how many times you send it.
DELETE is idempotent. "Delete order 42" twice still leaves order 42 deleted.
POST is the troublemaker. "Create a charge for $50" run twice creates two charges. Two real dollars gone.

The hard problem is always the create-or-mutate with side effects: charging a card, sending an email, shipping a package, incrementing a counter.

The mental model to carry in

Assume every request will be delivered at least once, possibly many times, and never assume "exactly once" is free. The network gives you at-least-once for cheap. Exactly-once is something you build, and idempotency is the main tool you build it with.

Why the duplicate even happens:

The server processes the request, and the response gets lost on the way back.
A load balancer times out at 30 seconds and the client retries.
A mobile user taps "Pay" twice because the spinner froze.
A message queue redelivers because the consumer crashed before it acked.

In every case the server did the work, but the caller does not know, so it tries again. Idempotency lets the server recognize "I have seen this exact intent before" and return the original outcome instead of redoing the side effect.

2How it works#

The standard pattern is the idempotency key: the client generates a unique token (usually a UUID) and attaches it to the request, typically as an Idempotency-Key header. The server uses that key to deduplicate.

The detail most people get wrong

The client generates the key once and reuses the same key across all retries of that one logical operation. Mint a fresh UUID on every retry and you have gained nothing. The key identifies the intent, not the attempt.

The server-side flow:

flowchart TD A[Client] --> B[Load Balancer] B --> C[Payment Service] C --> D[(Idempotency Store)] C --> E[(Payments DB)] D -->|key seen return saved result| C D -->|key new acquire lock| C

The hot path. The service checks the idempotency store before it touches the Payments DB. A seen key short-circuits to the saved result and the card is never charged again; a new key acquires the lock and does the real work.

Three outcomes when the key arrives:

Key not seen before. Insert it with status "in progress," do the real work, save the response body and status code against the key, then return. The insert is atomic (a unique constraint on the key column), so two concurrent requests race and exactly one wins.
Key seen and completed. Return the saved response. Do not touch the card. Do not send the email. Replay what happened the first time.
Key seen but still in progress. The concurrent-duplicate case. Return a 409 Conflict, or make the second request wait on the lock. Stripe returns an error here telling you a request with that key is already in flight.

The sequence over time, when a response is lost and the client retries:

sequenceDiagram participant C as Client participant S as Service participant K as Idempotency Store C->>S: POST charge key abc S->>K: insert abc in progress S->>S: charge card once S->>K: save result for abc S-->>C: 200 response lost C->>S: POST charge key abc retry S->>K: lookup abc K-->>S: completed with saved result S-->>C: 200 same result no new charge

The retry replay. The card is charged exactly once (step 3). The lost 200 triggers a retry with the same key, and the second time the service returns the stored result without ever touching the card again.

Mechanics that matter in a real build:

Scope the key. Unique per endpoint and per account, not globally, or one customer's key could collide with another's. Store the dedup row keyed on (account_id, endpoint, idempotency_key).
Store the full response, not just a flag. On replay you want the identical body and status, including the resource ID you created. Store only "done" and the retry cannot learn what order ID it got.
Set a TTL. You do not keep keys forever. Stripe keeps them for 24 hours. Most retries happen within seconds to minutes, so a window of 24 hours to 7 days is plenty, then purge.
Make it transactional where you can. The ideal: insert the key, do the business write, and save the result in one DB transaction, so either everything commits or nothing does. When the side effect is an external call (a card charge), you cannot wrap it in your DB transaction, so you lean on the provider's own idempotency plus careful ordering.

From the interviewer's chair

The cleanest answer is often no key at all. "Set balance to 100" is idempotent; "add 50 to balance" is not. If you can express a mutation as a declarative set-to-this-state rather than a relative delta, you sidestep the whole machinery. Reach for natural idempotency first, and only build the key store when a true create with side effects forces your hand.

Pause before you read on: two duplicate requests with the same key land on two different servers in the same millisecond. If each server does "SELECT to check, then INSERT," what happens?

Reveal the answer

Both SELECTs see "not found," both proceed, and the card gets charged twice. Check-then-write has a race that defeats the entire point. The fix is a single atomic insert against a unique constraint (or a lock on the key): one write wins and does the work, the other gets a duplicate-key error and replays or returns 409. Reaching for the atomic insert unprompted is the concurrency signal interviewers want.

A concrete worked example with real numbers#

Make it physical. Say you run a checkout API doing 2,000 charge requests per second at peak, each charge averaging $50. Mobile clients sit on flaky LTE, and the load balancer cuts any request past 30 seconds. Measured in production, about 3% of charge requests get retried because the response was lost or the LB timed out. That is 60 retries per second.

Without idempotency, every retry is a second charge:

60 duplicate charges per second times $50 is $3,000 per second of wrongly billed money.
That is $10.8M per hour of peak traffic sitting in your dispute and refund pipeline.
Even if only a tenth of retries land a duplicate on lucky timing, you still face $1,080,000 per peak hour of chargebacks, support tickets, and trust damage.

~$1M/hr

The failure mode you are buying out ofConservative duplicate-charge cost at peak with no idempotency. This is why payments teams treat the key store as non-negotiable, and why the 0.5 ms tax below is the easiest trade you will ever make.

Now add the key. Sizing the store:

2,000 writes per second of new keys at peak.
Each row is roughly: key (36 bytes for a UUID), status, HTTP status code, and a serialized response body around 1 KB. Call it 1.1 KB per row.
At 2,000 per second that is 2.2 MB/s, about 190 GB per day if you kept everything.
With a 24-hour TTL you hold at most about 173 million rows, roughly 190 GB resident, and it stays flat because expiry drops yesterday's rows as fast as today adds them.

The hot-path cost is one extra round trip:

On Redis that is sub-millisecond, call it 0.5 ms added to a charge that already takes 200 ms end to end, a 0.25% latency tax.
The retry path is even cheaper: the store returns the saved result, you skip the card charge entirely, and the client gets its original 200 back in a few milliseconds instead of re-running the full flow.
So the common retry gets faster and safer at the same time.

3Trade-offs#

The decisions cluster around how you dedupe and how strict you need to be.

Approach	How it works	Pro	Con	Reach for it when
Natural idempotency	Use PUT style set-to-state, or upserts keyed on a natural ID	No extra storage, no key management	Not possible for true creates with side effects	The operation is a state assignment
Idempotency key + store	Client sends key, server dedupes	Works for any POST, replays exact result	Needs a store, TTL, locking, more code	Payments, orders, anything with external side effects
Dedup at the queue	Consumer tracks processed message IDs	Protects async pipelines from redelivery	Dedup window is finite, not forever	Event-driven and streaming systems
Database unique constraint	Natural key collides on duplicate insert	Dead simple, DB enforces it	Only covers the write, not external calls	You control the ID and have no external side effect

The strictness question comes down to consistency vs latency:

For a hard guarantee, serialize on the key with a lock or a unique insert, and concurrent duplicates either wait or get a 409. That adds the ~0.5 ms store round trip on the hot path. For a payments API, worth it every time.
For a low-stakes event (track this analytics ping), a cheaper best-effort dedup with a Redis set and a short TTL is fine, and you accept the rare double-count in a dashboard nobody bills against.

Storage and cleanup are not optional:

Every key is a row. From the example, 2,000 per second is 173 million rows a day.
You need a TTL and a purge job, or a table that bloats until a query falls off a cliff.
Redis with native expiry is a common home. A Postgres table with an indexed created_at and a nightly delete also works, but at 173M rows a day partition by day so you can drop whole partitions instead of running mass deletes.

Exactly-once delivery is a lie. At-least-once delivery plus idempotent processing is the truth, and it produces exactly-once effects.

Across a distributed system, "exactly-once delivery" does not exist.
What you actually get is at-least-once delivery plus idempotent handlers.
Saying this crisply signals you have done the real thing, not just read about it.

4Where it is used#

The pattern shows up anywhere a costly side effect meets a retrying caller.

Payment APIs. Stripe, PayPal, and Square all expose an idempotency key on charge and payment-intent creation. The canonical example. A double tap or a client retry must never bill twice.
Message queues and stream processors. Kafka consumers, SQS, and friends deliver at-least-once by default. Consumers dedupe on a message ID, or make the downstream write idempotent (upsert on the event's primary key) so redelivery is harmless.
Order and checkout systems. "Place order" is a POST with real inventory and money behind it. Keys stop the double order when a mobile connection flaps during submit.
Internal RPC and service-to-service calls. Any service that retries on timeout (most of them, via retry middleware) needs the downstream to tolerate the retry. Your retry logic and the callee's idempotency are two halves of one guarantee.
Webhooks. Providers retry when your endpoint does not 200 fast enough. You dedupe on the event ID they send, so you do not process the same payment.succeeded twice.

The rule of thumb: the moment a request causes an irreversible or costly side effect and some layer in the path retries, you need idempotency between them.

5Common pitfalls and gotchas#

Each of these is a real way to ship a bug that bills a customer twice.

Client regenerates the key per retry. The single most common bug. Mint the key once per logical operation, before the first attempt, and hold it across retries. Not inside the retry loop.
Checking then writing without a lock. "SELECT to see if the key exists, then INSERT" has a race. Two duplicates both see "not found" and both charge the card. Use an atomic insert with a unique constraint, or a lock.
Only storing a boolean. Store "processed: true" but not the response and the retry cannot return the original resource ID. Store the full result.
No TTL, or a TTL that is too short. Too long and the table bloats. Too short and a legitimate retry an hour later sneaks past the dedup and double charges. Match the window to your realistic retry horizon.
Idempotent write, non-idempotent side effect. Your DB upsert is safe, but you also fire an email every time the handler runs. The retry resends the email. Guard every side effect, not just the primary write.
Reusing a key for a different payload. A client reuses a key but changes the amount from $50 to $80. A strict server hashes the request body and compares; if it differs, return a 422 rather than silently replaying the old result, because the key promised "same intent."
Assuming the message broker gives exactly-once. It does not. Design the consumer to be idempotent regardless of what the broker claims.

6How it shows up in interviews#

Idempotency is rarely the whole question. It is the detail that separates a senior answer from a junior one inside a bigger design: payments, ticket booking, a URL shortener's create path, a distributed job queue.

What interviewers probe:

"A user clicks pay twice, what happens?" They want idempotency key, dedup store, atomic check, replay the saved result.
"Your service retries on timeout. Is that safe?" They want you to connect your retry policy to the callee's idempotency. If the callee is not idempotent, your retry is a bug generator.
"Two duplicate requests arrive at the same millisecond on two servers." The concurrency probe. Weak answers do read-then-write. Strong answers reach for a unique constraint or a distributed lock so exactly one wins.
"How long do you keep the keys?" Tests whether you have thought about storage and cleanup, not just the happy path.

Fails the round

The weak answer

What they say: "I will check if the charge already exists and skip it if so."
What is missing: No race handling, no stored result, no TTL, no split between the DB write and the external charge.
What happens next: Interviewer asks "two requests at the same millisecond, both see nothing, now what?" and the candidate is stuck.
The read: Read about idempotency, never shipped it.

Passes the round

The strong answer

What they say: Key per operation reused across retries, atomic insert on a unique key scoped to account and endpoint.
The race: Win the insert, do the work in a transaction, store full body and status; lose it, replay or 409.
The gaps: 24-hour TTL, ~173M rows a day with droppable daily partitions, provider key for the external charge.
The read: Named the race, scope, storage, TTL, external-call gap, and the numbers. Shipped it.

The strong answer takes about 30 seconds and lands because it names the external-call gap too: the card charge cannot live inside your DB transaction, so you pass an idempotency key to the payment provider and order it so the provider's dedup covers the window between your write and their charge. The extra store lookup costs about 0.5 ms, nothing against the million-dollar-per-hour double charge it prevents.

Follow-ups the interviewer will ask (and your answers)

"Two duplicate requests hit two servers in the same millisecond. Who wins?"

Exactly one, by construction. Both try an atomic insert against a unique constraint on the key. One commits and does the work; the other gets a duplicate-key error and either replays the stored result or returns 409 if the first is still in progress. No read-then-write, so there is no window for both to proceed.

"The card charge is an external call you cannot wrap in your DB transaction. How do you stay correct?"

Order it and pass a key downstream. I insert my idempotency row, call the provider with its own idempotency key, then record the result. If I crash between the charge and my write, the retry re-sends the same provider key, the provider dedupes, and I record the one real charge. The provider's idempotency covers the gap my transaction cannot.

"How long do you keep keys, and what does the store cost?"

A TTL matched to the real retry horizon, typically 24 hours, up to 7 days. At 2,000 new keys per second and ~1.1 KB per row, that is about 173M rows and ~190 GB resident, held flat by expiry. In Postgres I partition by day so cleanup is a partition drop, not a mass delete. In Redis I lean on native expiry.

"A client reuses a key but changes the amount from $50 to $80. What do you do?"

Reject it. I hash the request body and store the hash with the key. A matching key with a different body is a contract violation, so I return 422 rather than replaying the old $50 result or silently charging $80. The key promised the same intent, and this request broke that promise.

"Your Kafka consumer gets the same message twice. Safe?"

Only if I made it safe. Brokers deliver at-least-once, not exactly-once, so I dedupe on the message or event ID, or I make the downstream write an upsert keyed on the event's primary key so redelivery is a no-op. I never trust the broker's exactly-once claim; the idempotent handler is what actually guarantees exactly-once effects.

What you can now explain in the room

Why at-least-once delivery is free and exactly-once effects are something you build with idempotency.
Why the key must be minted once per intent and reused across every retry.
Why check-then-write races, and how an atomic insert on a unique constraint settles it.
Why you store the full response with a TTL instead of a boolean that never expires.
How to size the key store (around 173M rows a day, 190 GB flat) and clean it with partitions or native expiry.
How to cover an external charge your DB transaction cannot reach, using the provider's own idempotency key.

Want to see this applied end to end? Try the System Design Question walkthrough on designing a payment system, where idempotency keys sit right on the critical path.

← Previous

Fault Tolerance and Redundancy

Latency vs Throughput

Get a prep plan built around your Idempotency interview

Tell us the role and your interview date. We will send back a dated, day by day plan. Free.

Thanks. Check your inbox, your Idempotency prep plan is on the way.