Design a UPI / Digital Wallet Payment System

1Clarifying the requirements#

Pin down what "payment" even means here before drawing a box. UPI and a closed-loop wallet are two different beasts with two different trust models, and guessing wrong costs you the whole 45 minutes.

Candidate: Are we the wallet provider holding balances ourselves, or are we the UPI switch routing authorization messages between banks without ever holding funds? Those are completely different problems. One is a ledger, the other is a stateless router with settlement.

Interviewer: Build the wallet provider. You hold user balances in a ledger. But you also need to top up from a bank account over UPI and pay out to merchants. So you talk to external rails.

Candidate: Got it. So internally it is a closed-loop ledger that I fully control, and the edges touch external banks and PSPs that I do not. Do we need peer-to-peer transfers between two of our own users, or only user-to-merchant payouts?

Interviewer: Both. P2P and P2M.

Candidate: One more on scope, the one that shapes everything. What is our correctness bar? I am going to assume three hard invariants: money is never created or destroyed, every transfer is all-or-nothing, and a retried request must not double-charge. If those three hold, the rest is engineering.

Interviewer: Yes, all three. And it has to feel instant to the user, under a second.

Candidate: Last thing, what should I explicitly not build? I want to spend my time on the money core, not KYC screens.

Interviewer: Skip fraud models, KYC onboarding, and the mobile UI. Leave hooks for fraud but do not design it.

What those answers just told me:

• The interview is about correctness under failure, not throughput. Nobody fails this on TPS; they fail it on the two-leg crash.
• "Closed-loop ledger I control, edges I do not" splits the system cleanly: strong consistency inside, reconciliation at the bank boundary.
• Three invariants named out loud (no money created, all-or-nothing, no double-charge) are the thesis the interviewer just handed me.

Functional requirements

• Fund a wallet from an external bank account (top up) and withdraw to one.
• Transfer value between two internal wallets (P2P and P2M), atomically.
• Idempotent payments: the same client request, retried, moves money exactly once.
• A queryable, immutable transaction history per user.
• Reconciliation against external banks and PSPs.
• Reversals and refunds as first-class ledger entries, never in-place edits.

Non-functional requirements

• Correctness over availability for the ledger. We never show a wrong balance.
• P99 under 800 ms for an internal transfer.
• Durability: a committed payment survives node loss. Zero acknowledged-then-lost transactions.
• Horizontal scale on users and transaction volume.
• Auditability. Every rupee is traceable to a double-entry pair.

Explicitly out of scope (name it so you stay honest about what 45 minutes covered):

• Fraud scoring models, KYC onboarding flows, the mobile UI.
• I leave an event hook so a fraud model can subscribe later, but I do not design the model.

2Back-of-the-envelope estimation#

Get the shape of the load in your head, because the numbers pick the architecture, not your taste. One pairing drives everything, and I will flag it when I reach it.

Users and traffic.

• 100M registered users, 40M daily active. That tracks with a mid-size national wallet.
• Each active user does roughly 3 money-moving actions a day (one P2M coffee, one P2P split, one bill): 40M x 3 = 120M transactions/day.
• 120M / 86,400 seconds is about 1,400 write TPS average.
• Peak bites. Festival days, salary day, and the 8 PM dinner rush push 8x to 12x average. Design the hot path for roughly 15,000 write TPS peak, with headroom above.

Reads.

• Balance checks and history scrolls run maybe 10x writes: about 15K average, 150K peak read QPS.
• Those are cacheable and can be served a second stale. Writes cannot.

Storage.

• Each transaction is a few hundred bytes of ledger data. With double-entry (two rows per transfer), indexes, idempotency records, and audit metadata, call it about 1 KB effective per transaction. That is a blended number, not the raw row.
• 120M/day x 1 KB = 120 GB/day, about 44 TB/year raw.
• With 3x replication and a 7-year financial retention rule, plan for petabyte-scale cold storage plus a much smaller, much hotter recent tier.

Bandwidth.

• Payloads are a few hundred bytes. Money systems are not bandwidth bound; they are correctness and contention bound.

3Core APIs#

Keep these small and boring on purpose. A payment API earns trust by being predictable.

POST /v1/transfers
Idempotency-Key: 9f1c3a7e-2b44-4d9a-a1f0-7c2e5b8d1234
Content-Type: application/json

{
  "from_wallet": "wlt_8812",
  "to_wallet": "wlt_4457",
  "amount_minor": 50000,
  "currency": "INR",
  "note": "lunch"
}

{
  "transfer_id": "txn_01HF9...",
  "status": "COMPLETED",
  "from_balance_minor": 112500,
  "created_at": "2026-11-03T09:14:22Z"
}

The rest, listed tersely:

POST /v1/topups        # pull from external bank over UPI
POST /v1/withdrawals   # push to external bank
GET  /v1/wallets/{id}/balance
GET  /v1/wallets/{id}/transactions?cursor=...&limit=50
POST /v1/transfers/{id}/reverse

Two choices I will not let slide, and I will say why in the room:

• Amounts are integer minor units (paise), never floats. 0.1 + 0.2 is not 0.3 in IEEE 754, and in a ledger that rounding error becomes a real missing rupee that reconciliation will scream about.
• The Idempotency-Key header is required on every mutating call, validated for presence at the gateway. A payment API without a mandatory idempotency key is a bug waiting to charge someone twice on the first flaky network.

4High-level design#

The spine is a double-entry ledger.

• Every money movement is two entries that sum to zero: a debit on one account, a credit on another.
• Balances are derived from the sum of entries, not stored as a mutable number I overwrite in place.
• This is the single most important decision in the system, and I defend it hard in 6 below.

The write path stays deliberately lean.

• Requests hit an API gateway that does auth, request validation, idempotency-key presence checks, and rate limiting.
• They land on a stateless Payment Service, scaled horizontally to absorb the 15K peak.
• The service runs the idempotency check first, then the transfer: same-shard is one local ACID transaction, cross-shard is a saga (6.1).
• It emits exactly one event per move. Idempotency check, ledger transaction, one event, done.

External rails are async by necessity.

• Top-ups and withdrawals go through a PSP Adapter that talks to banks and reconciles asynchronously.
• I do not control the bank's latency and will not block a user on it.

Reads are a separate path on purpose.

• Balance and history reads go through a cache and read replicas, so they never contend with the write path for row locks.
• The event bus feeds history, reconciliation, notifications, and the fraud hook.
• If the notification service is down, money still moves; the event just sits in Kafka until the consumer catches up.

5Data model#

Use the right store for each access pattern. The ledger and the history log are different jobs, and forcing one database to do both is a mistake in either direction.

The ledger is relational and ACID, full stop.

• I want foreign keys, strong transactions, and a CHECK that forbids a negative balance.
• I want to assert invariants inside the database rather than trusting every caller.
• A money ledger is the textbook case for SQL, and I will not be talked into eventual consistency for the authoritative balance.

accounts (one row per wallet, holds the materialized balance for fast reads)

ledger_entries (the source of truth, append only)

transfers (the business-level operation)

Why the balance is a cache of truth, not the truth.

• accounts.balance_minor must always equal the sum of that account's ledger_entries, and I update it inside the same transaction that writes the entries.
• The entries are the real source of truth. If the materialized balance and the entry sum ever disagree, the entries win and I alert.
• The balance_after snapshot on each entry lets an auditor replay any single account's history row by row without re-summing the whole table.

Why history lives somewhere else.

• Transaction history is write-once, read by one user, time-ordered newest-first, huge volume, zero updates.
• It goes to a wide-column store (Cassandra or DynamoDB), partitioned by account_id and clustered by time descending, so the newest page is a single contiguous read.
• SQL where I need cross-row atomic debit-credit; NoSQL where I need cheap append and infinite scroll at 150K peak read QPS. A wide-column store cannot give me an atomic pair; a single SQL box cannot cheaply serve a decade of scrolls for 100M users.

6Deep dives#

6.1 The two-leg transfer: atomicity and the cross-shard saga#

This is the signature hard part, where the interview is won or lost. A transfer is one debit and one credit that must both happen or neither. The whole difficulty is that the two legs may live on two machines with no shared transaction.

Same-shard is easy, so dispatch it fast.

• Both wallets live on one database shard.
• Open one transaction, lock both account rows in a consistent order (always lowest account_id first, which kills deadlocks because every concurrent transfer grabs locks in the same order).
• Check the sender has funds, write both ledger entries, update both materialized balances, commit.
• One ACID transaction. Nothing clever needed.

Cross-shard is the hard case, and I am deliberately not using 2PC.

• Sender on shard A, receiver on shard B, no single transaction spans them.
• Two-phase commit holds locks on both shards across a blocking coordinator for the whole round trip.
• If the coordinator dies after prepare but before commit, those rows stay locked and that slice of the system stalls until a human intervenes.
• At 15K TPS on the hot path, a blocking coordinator is a guaranteed outage waiting for its moment. I give up isolation before availability.

So I run a saga with a reserve-then-commit shape.

How the legs play out.

• Step one reserves funds on the sender: a PENDING debit that reduces available balance but is not yet final. If the sender lacks funds, I fail right here and nothing was ever touched on B, so there is nothing to undo.
• Step two commits the credit on B.
• Step three finalizes the debit on A.
• If step two or three fails, I run the compensating action: release the reservation on A as a new ledger entry (a credit that cancels the pending debit), never an in-place undo. The audit trail shows exactly what happened and when, including the failure.

The subtle part: every leg must be idempotent.

• The Payment Service can crash between any two steps, and a recovery worker replays the saga from its last persisted state.
• I store the saga's progress (which step is done) in the transfers row and key each leg by transfer_id plus leg name.
• Replaying "credit on B" when B already credited is a no-op lookup, not a second credit.
• Without that, crash recovery would itself double-spend, which is the exact bug we are trying to prevent.

6.2 Idempotency and double-spend prevention#

Two failure modes get conflated constantly. Pull them apart first, because they live at different layers and need different fixes.

Double-submit is the client retrying the same logical request.

• The phone sends the transfer, the network times out before the response arrives, the app retries automatically.
• Without protection, the user pays twice for one tap.
• The fix is the Idempotency-Key: on the first request I insert a row keyed on that value in a dedicated idempotency store, carrying the request fingerprint and a PENDING status, in the same path as the transfer.
• On a retry with the same key I find the existing row and return the stored result instead of moving money again.
• The insight I say plainly: the idempotency record must be durable before I acknowledge the request. Ack before the key is persisted and a crash-then-retry double-charges, defeating the entire mechanism.

Guarding the race of two concurrent requests with the same key.

• Put a unique constraint on the key column.
• The second insert fails the constraint; that thread waits briefly and reads the winner's result rather than running its own transfer.
• The database does the hard coordination for me, instead of a distributed lock I would have to reason about.

Double-spend is a different animal entirely.

• The same user spends the same balance twice through two concurrent but genuinely distinct transfers.
• User has 500, fires two separate 500 transfers at the same instant (two devices, or a fast double-action). Both read balance 500, both pass the funds check, both debit, and now the balance is -500 and we created money out of nothing.
• Idempotency keys do not help here at all, because these are two different requests with two different keys.

The fix lives in the ledger, not the application.

• Either take a row lock on the sender account inside the transaction (SELECT ... FOR UPDATE), which serializes concurrent debits so the second transfer reads the already-reduced balance and fails the funds check.
• Or use optimistic concurrency with the version column and retry on conflict.
• I prefer the row lock on the hot debit path, because conflicts on a single busy account are common and optimistic retries would just thrash and burn CPU re-reading.
• As a final backstop, the CHECK constraint forbids a negative balance, so even a logic bug upstream physically cannot create money; the transaction aborts.

6.3 Reconciliation with external rails#

Internal transfers I control end to end. External top-ups and withdrawals I do not. The bank or PSP is the authority on whether money actually moved, and they are frequently slow, occasionally wrong, and sometimes silent. Designing as if the bank always answers promptly is how you lose real money.

The timeout is the case that matters most.

• A top-up call can return success, failure, or a timeout with no answer at all.
• On a timeout I must not assume either outcome: assuming success lets a failed top-up become spendable, and assuming failure strands a user whose money did leave their bank.
• So I record the top-up as PENDING and let a reconciliation worker settle the truth.

How the worker settles the truth.

• It pulls the PSP's settlement file or status API on a schedule and matches each pending record against the bank's record of truth.
• Three buckets fall out of the match:

The rule I enforce hard.

• Never credit a user's spendable balance on an optimistic external top-up before the bank confirms.
• I can show "pending" in the UI so the user sees activity, but spendable funds only move on confirmation.
• That one rule shuts down the classic attack: top up, immediately spend, then the top-up fails at the bank, leaving us eating the loss.
• Reconciliation is not cleanup that runs when someone remembers. It keeps us solvent, so I staff and monitor it like production, with paging alerts on any unmatched record older than a threshold (say 30 minutes), because an unmatched record is either a bug or someone draining us.

7Bottlenecks and trade-offs#

What breaks first: a hot account.

• A big merchant receiving thousands of payments a second becomes a single-row contention point, because every credit takes a row lock on that one account_id and they serialize.
• Fix: shard the merchant's balance into N sub-accounts (say 16) and credit a randomly chosen one, then sum across them for the real balance.
• One hot row becomes 16 warm rows, cutting lock contention by roughly 16x.
• The "true balance" read becomes a fan-in sum across sub-accounts, which I cache and refresh on write, so the merchant dashboard stays fast.

Single points of failure: the idempotency store and the event bus.

• If the Redis holding idempotency keys goes down, I lose my double-submit guarantee, so I back it with a durable store (or co-locate the idempotency record in the same SQL transaction as the ledger write) rather than treating it as a best-effort cache.
• Kafka I replicate across brokers with a minimum in-sync-replica count of 2, so an acknowledged event is not lost if one broker dies.
• Consumers (history, reconciliation) are idempotent and track their own offsets, so at-least-once delivery is fine: a replayed event just upserts into history or re-runs a match that finds nothing new.

The consistency call I made on purpose.

• The ledger is strongly consistent, and I reject a write before I record a wrong one.
• Everything downstream (history, balance cache, notifications, analytics) is eventually consistent, fed off the event bus, so a user might see their transaction list lag the actual ledger by a second.
• CAP forces a choice when a ledger shard partitions, and I choose consistency over availability there.
• Availability that serves a wrong balance is strictly worse than a brief honest error, because a wrong balance either loses us money or loses the user's trust, and both are unrecoverable.

The cost of my choices.

• The saga adds latency (an extra round trip on the cross-shard path) and real operational complexity versus one giant unsharded SQL database.
• If the business fit on a single primary with read replicas, I would start there and skip sharding and sagas entirely, and I would be right to.
• The sharded saga only earns its keep past the point where one node cannot hold peak write TPS, which is exactly our 15K number.
• Premature sharding is how teams buy a distributed-systems problem they did not need to have.

A payment system is allowed to be briefly slow or honestly unavailable. It is never allowed to be confidently wrong about money.

8Summary#

The decisions I will defend in the debrief:

• Double-entry, append-only ledger. Balances are derived, entries are immutable, every rupee traces to a debit-credit pair. This is the foundation that makes audit, reversal, and reconciliation sane instead of a forensic nightmare.
• SQL for the ledger, NoSQL for history. Match the store to the access pattern. Atomic debit-credit needs transactions; a decade of per-user scrolls needs cheap append.
• Mandatory idempotency keys, committed durably before the acknowledgement, cleanly separating double-submit (fix at the API layer) from double-spend (fix at the ledger layer) and solving each where it lives.
• Saga with reserve-then-commit for cross-shard transfers, not two-phase commit, trading isolation latency for throughput and availability on purpose, with idempotent legs so crash recovery cannot double-spend.
• Reconciliation as a first-class, monitored system, with the hard rule that external funds never become spendable before the bank confirms.
• Strong consistency on the ledger, eventual everywhere else, because a wrong balance is the one failure a payment system is not allowed to have.
• If I had more time: fraud and velocity-limit hooks on the event stream, a cold-storage tiering plan for the 7-year retention, currency conversion as its own balanced ledger pair rather than a lossy inline multiply, and the exactly-once effects on the saga recovery worker so a replayed transfer provably runs each side effect once.

For how a real payments company structures this interview and what they probe on, see our related guide on payment platform system design loops.