Design a Payment System

1Clarifying the requirements#

Pin down which slice of "payments" you own before drawing a single box. The scope changes the architecture, so treat it as load-bearing. I ask four questions and then stop, because asking twenty signals stalling.

Candidate: Are we building the full stack including the card networks, or are we the service that sits between a merchant app and an external gateway like Stripe or a bank?

Interviewer: You are the internal payment service. You own charging customers, recording the money, and talking to one or more external payment service providers. You do not implement the card network.

Candidate: Is this pay-in only, or do we also handle payouts to sellers, refunds, and disputes?

Interviewer: Pay-in and refunds for sure. Assume payouts are a separate service, but your ledger has to support them later.

Candidate: One currency or many? And do we need to be strict about exactly-once charging?

Interviewer: Multi-currency. And yes, exactly-once is the whole point. A retry must never charge twice.

Candidate: Last one. Who is our client? Is it our own checkout page calling us, or untrusted third-party merchants with their own servers?

Interviewer: Assume our own first-party checkout and a trusted internal mobile app. You do not need full multi-tenant merchant isolation today, but keep the API clean enough that we could add it.

What those answers just told me:

• The hard parts are idempotency, the ledger, and reconciliation with the external provider.
• I can skip merchant onboarding, fraud scoring, and the card-network protocol.
• "Keep it clean enough for merchants later" means draw service boundaries I would not have to tear up.
• "Exactly-once is the whole point" is the interviewer handing me the thesis of the design.

Functional requirements

• Create and execute a payment for an order (authorize then capture, or a combined sale).
• Support multiple payment methods (card, wallet, bank transfer) through one or more external providers.
• Record every money movement in an immutable ledger.
• Handle refunds, full and partial, against a prior payment.
• Process asynchronous status updates from providers via webhooks.
• Reconcile our records against the provider's settlement files daily.

Non-functional requirements

• Correctness over availability. A payment that is "maybe charged" is worse than one that cleanly failed.
• Exactly-once charge semantics even under client retries, network partitions, and our own crashes.
• Durable. Once we say "paid," it stays paid. No data loss.
• p99 latency under about 800 ms for the synchronous authorize step (the provider call eats 200 ms to 500 ms of that).
• Auditable. Full history of state changes: who did what, when.
• PCI scope minimized. We do not store raw card numbers; we tokenize through the provider.

2Back-of-the-envelope estimation#

Get the scale in your head, because here it decides what you do NOT build. One number reframes every later decision, and I will flag it when I reach it.

Users and traffic.

• 50 million daily active users, but only a fraction pay on a given day.
• Say 5 million successful payments per day.
• 5,000,000 over 86,400 seconds is about 58 payments/second average.
• Peak is spiky (flash sales, paydays), so plan for 10x: roughly 580 payments/second.

Writes (one payment is many writes).

• A payment is not one write: create intent, authorize, capture, webhook updates, ledger entries. Call it about 8 writes per payment.
• Peak write QPS to the datastore is about 580 x 8, so roughly 4,600 writes/second.

Reads.

• Status checks, order pages, admin dashboards, and reconciliation jobs run maybe 10x the write rate.
• About 5,000 to 6,000 reads/second, bursty during reconciliation.

Storage.

• A payment plus its ledger entries and state history is around 4 KB.
• 5M x 4 KB = 20 GB/day, about 7.3 TB/year.
• Money data lives long, so over 7 years of retention that is roughly 50 TB total. Big, but it is cold archival storage, not hot.

The number that reframes everything.

• 4,600 writes/second at peak into a single primary.
• That is comfortably inside what one well-tuned Postgres primary handles (tens of thousands of simple writes/second on decent hardware).
• No sharding, no NoSQL, no distributed transaction engine required. This is not a web-scale problem.

Why I say it out loud.

• It licenses every later decision: I will trade some latency and some write amplification for guarantees.
• It tells the interviewer I know a flash sale is a hot-row problem, not a QPS problem.
• A candidate who spends ten minutes on consistent hashing here has misread the room.

3Core APIs#

Keep these small and boring on purpose. Boring is a compliment for a payment API. The client never sends a raw card number; it tokenizes with the provider's SDK first and hands us a token.

POST /v1/payments
Idempotency-Key: 2f1c9a7e-...-order-889213
Content-Type: application/json

{
  "order_id": "ord_889213",
  "amount": 4999,
  "currency": "USD",
  "customer_id": "cus_1042",
  "payment_method_token": "pm_tok_abc123",
  "capture": true
}

Two choices worth defending in the amount field alone:

• Minor units, never floats. 4999 means cents. A float cannot represent 0.10 exactly, and that rounding error compounds into real cents missing from the ledger.
• Client-supplied token. The raw card never touches us, so the big-compliance bytes stay out of our systems.

The response returns our ID and the resolved status:

{
  "payment_id": "pay_7Xk2",
  "status": "succeeded",
  "amount": 4999,
  "currency": "USD",
  "created_at": "2026-11-03T10:22:14Z"
}

The rest of the surface stays small:

GET  /v1/payments/{payment_id}
POST /v1/payments/{payment_id}/refunds
POST /v1/webhooks/psp/{provider}
GET  /v1/payments/{payment_id}/ledger

Two things to say out loud about this surface:

• Refunds carry their own Idempotency-Key. A refund is a money movement too, and a double-refund is just a double-charge pointed the other way.
• The Idempotency-Key header is the single most important field in the design. If you remember one thing from this answer, make it that header. I come back to it hard in the deep dive.

4High-level design#

The spine is a thin API layer, a stateful Payment Service that owns the state machine, an append-only Ledger, an async worker tier that talks to providers and retries, and a reconciliation pipeline.

The flow I care about most.

• The request comes in and we check idempotency first.
• We persist an intent in PENDING transactionally with a reserved ledger entry.
• We call the provider, then flip to SUCCEEDED and write the final balanced ledger lines.
• If anything blows up between those steps, a background worker reconciles the true outcome from the provider instead of guessing. Guessing is how you lose money.

Why a relational store.

• For money I want ACID, not eventual consistency.
• Our volume does not force NoSQL on us, so I do not pay its costs.
• NoSQL would make the double-entry invariant far harder to enforce.

Why the queue.

• It decouples the slow, flaky provider call from the user-facing request where it can.
• It drives retries and webhook processing.
• It gives at-least-once delivery, not exactly-once, so every worker action is idempotent and keyed. I design as if every message arrives twice, because eventually it will.

One nuance on the synchronous path.

• The first authorize call is in-band: the user is waiting, so that one provider call sits on the request thread under the 800 ms budget.
• Everything after (capture retries, webhook application, slow-path ledger finalization) rides the queue, so the user is never blocked on it.

5Data model#

Three core tables, kept narrow and append-heavy, each picked for its access pattern.

payments (the state of a payment):

• The unique constraint on psp_charge_id is what makes a reconciliation backfill idempotent: re-inserting the same provider charge is a no-op, not a duplicate.

ledger_entries (append-only, never updated or deleted):

• Double-entry. Every transaction is a set of entries whose debits equal credits, grouped by txn_id. A $49.99 charge might be debit customer_cash 4999, credit psp_clearing 4999.
• The invariant I defend in review. For any txn_id, sum(debits) equals sum(credits). If that ever breaks, money was lost or invented. I enforce it at write time and verify it continuously in reconciliation, because an invariant nobody checks is just a comment.
• One currency per transaction. A USD charge and its EUR fee are two balanced transactions linked by payment_id, never one unbalanced one. FX conversion gets its own explicit pair so the gain or loss is visible in the books.

idempotency_keys (the exactly-once anchor):

• The primary key does the mutual exclusion, the request_hash catches a client reusing a key with a different body, and response_body lets a retry replay the exact original answer.

6Deep dives#

6.1 Exactly-once charging with idempotency#

This is the heart of the problem and where I spend most of my time. The client will retry. Networks will drop the response after we already charged. Get this wrong and a double-charge on a real card is the single most expensive bug this system can ship.

• The client generates one idempotency key per logical payment attempt, tied to the order, not regenerated per retry.
• That last part is load-bearing: if the client makes a fresh key on every retry, idempotency is worthless.

The details that actually matter:

• Insert before you charge. We write the idempotency key with a unique constraint before calling the provider. If two requests race, exactly one insert wins; the loser reads the in-progress row and waits or returns "still processing." The database does the mutual exclusion, so I do not reach for a distributed lock when a unique index will do. A unique index is already consistent, already durable, with no second system to keep alive.
• Pass the same key down to the provider. Every serious gateway supports this. So even if our process dies after calling the provider but before recording the result, the retry sends the same key, the provider recognizes it, and returns the original charge instead of making a new one. Our guarantee is only as strong as the provider's, so we lean on theirs explicitly. This is the step junior candidates miss: they protect their own layer and forget the money moves one layer down.
• Store and check the request hash. If a client reuses a key with a different body (a bug on their side), we reject with a 422 rather than silently returning the wrong result. A key that maps to two different charges is a landmine, so we refuse rather than guess.

The dangerous window, handled explicitly.

• The gap between "provider charged" and "we recorded it" is where a crash leaves the idempotency row stuck IN_PROGRESS.
• A sweeper job picks up stale in-progress rows (say older than 30 seconds) and asks the provider "what happened to key K?" to resolve the true state.
• We never assume; we ask the source of truth. The sweeper plus the pass-through key is what turns "probably once" into "exactly once in effect."

6.2 The state machine and durable workflow#

A payment is a small state machine, and I draw it explicitly because illegal transitions are where double-captures and ghost refunds come from. Most money bugs are really an illegal state transition nobody guarded.

Transitions only happen through guarded writes.

• To move AUTHORIZED to SUCCEEDED I use a conditional update: UPDATE payments SET status='SUCCEEDED' WHERE payment_id=? AND status='AUTHORIZED'.
• If zero rows change, someone already moved it and I stop.
• This is optimistic concurrency, and it saves me from a second worker capturing the same authorization. No locks, no leases, just a WHERE clause the database evaluates atomically.

The long-running coordination is a workflow, not a request.

• Authorize, wait, capture, maybe retry: I run it as a durable async process where each step is committed.
• If the worker dies, another worker resumes from the last committed step by reading the payment's current state.
• State lives in the database, not in memory. That is the whole trick to surviving crashes mid-payment: kill any worker at any instant and a replacement reconstructs exactly where to pick up by reading one row.

Webhooks feed the same machine.

• A charge.succeeded webhook flips AUTHORIZED to SUCCEEDED.
• Webhooks arrive out of order and more than once, so each carries an event ID, I dedupe on it with a unique constraint, and I apply it as a guarded transition.
• An out-of-order webhook that tries an illegal move is simply ignored because the guard fails. A charge.succeeded that lands after a refund.succeeded cannot un-refund the payment: the guard rejects it and I move on.

6.3 The ledger and reconciliation#

The payments table tells you the state of a payment. The ledger tells you where the money is. I keep them separate on purpose, because they answer different questions and the ledger is the legal record.

The atomic write is the payoff for choosing a relational store.

• When a charge succeeds, the status flip and the balanced ledger entries commit in the same database transaction.
• Either both happen or neither does. There is no window where we say "succeeded" but have no ledger record.
• This single atomic write is exactly why I picked real transactions over eventual consistency.

Reconciliation is the safety net for everything the provider did that we did not directly observe. Once a day (and streaming where the provider supports it) the provider sends a settlement file: every charge, refund, and fee they actually processed.

Three kinds of breaks, and I name all three because interviewers love this:

• In ours, not theirs. We think we charged, they have no record. Likely we created an intent and never actually hit the provider, or a webhook lied. Flip to the true state, which usually means failing a payment we optimistically marked pending.
• In theirs, not ours. They charged, we missed it. Usually a dropped webhook. We backfill the payment and ledger from the settlement row, keyed by psp_charge_id so the backfill itself is idempotent.
• Amount mismatch. Fees, FX, or partial-capture differences. We post an adjusting ledger entry so the books balance. We never edit the original entries because the ledger is append-only; a correction is a new balanced transaction, not a mutation.

The point, and a cheaper check I run far more often.

• Reconciliation, not the live path, is the final arbiter of correctness. The live path is allowed to be wrong temporarily; the daily reconcile is not.
• Anything we cannot auto-resolve opens a case for a human.
• I also run a cheap internal check well more often than daily: sum every account across the ledger and confirm it nets to zero. If it does not, something upstream violated double-entry and I want to know in minutes, not at the next settlement file.

7Bottlenecks and trade-offs#

The provider is the bottleneck and the single point of failure.

• It is slow (hundreds of ms), occasionally down, and rate-limited.
• Defenses: timeouts with retries using exponential backoff and jitter, a circuit breaker so we fail fast when it is clearly down, and a secondary provider so we can route around an outage.
• The cost of a second provider: it doubles reconciliation and tokenization complexity, because each has its own settlement format, charge IDs, and token vault. So I add one only when uptime or cost actually demands it.

Hot rows during flash sales.

• A normal payment has no shared hot row; each payment is independent.
• The hot spots are shared counters: a seller's running balance, a promo code's remaining uses.
• I do not update a single balance row on the synchronous path, because 580 concurrent writers fighting over one row serializes the whole flash sale.
• I append a ledger entry and compute balances from the ledger, with periodic snapshots so reads do not scan all of history. Append scales; update-in-place on a hot row does not.

Database write amplification.

• Eight writes per payment at peak is fine for one primary Postgres for a long time (about 4,600 writes/second is well within one primary).
• When it stops being fine, I shard by customer_id (or order_id) because a payment and its ledger entries always belong to one customer, so cross-shard transactions stay rare and the atomic status-plus-ledger write stays inside one shard.
• I resist sharding until the numbers force it, since it complicates the double-entry invariant checks that would then run per shard and roll up.

Consistency versus availability.

• I chose consistency. During a partition where I cannot reach the database or the provider, I would rather reject the payment than risk a charge I cannot record.
• A declined payment is a retry; a lost one is a support ticket, a refund, and a reputation hit.
• I say this explicitly because it is the defining trade-off of the whole system, and an interviewer who hears you name it clearly stops worrying about whether you understand the problem.

Exactly-once is a lie we engineer around.

• There is no true exactly-once across a network.
• What we actually build is at-least-once delivery plus idempotent, keyed operations, which produces exactly-once effects.
• Every worker, webhook, and retry is idempotent. If a candidate claims true exactly-once delivery, that is where I push, and the good ones correct themselves to "exactly-once effects" before I finish the question.

PCI scope.

• Keeping raw card data out of our systems via provider tokenization is a deliberate trade: we give up some control and take a dependency.
• In return the compliance surface shrinks enormously, from full PCI DSS Level 1 down to a much smaller SAQ. Worth it almost always.

Treat money as untrusted, external, and slow, and let the reconcile (not the live path) decide what is true.

8Summary#

The decisions that matter:

• Not a scale problem, a correctness problem. About 4,600 writes/second fits on one primary, so every guarantee gets spent on partial failure, not throughput.
• Idempotency keys everywhere, inserted with a unique constraint before the external call and passed through to the provider, turning at-least-once plumbing into exactly-once effects.
• A relational store with real transactions so the status flip and the balanced ledger entries commit atomically. No window where "succeeded" has no ledger record.
• Double-entry, append-only ledger as the financial source of truth, with balances derived, never updated in place.
• An explicit state machine with guarded transitions, run as a durable workflow so a crash mid-payment resumes instead of double-charging and an out-of-order webhook cannot un-refund a payment.
• Consistency over availability, because a failed payment is recoverable and a lost one is not.
• Reconciliation as the final arbiter, with a continuous nets-to-zero ledger check so a broken invariant surfaces in minutes.

For how specific companies frame this, see our related guides on payment and fintech interviews at Stripe, Square, and PayPal.