Latency vs Throughput

1The core idea#

Here is the mental model I use. Picture a coffee shop.

Latency is one customer's wait.

• It is the clock from "I order" to "I have my drink."
• A flat white that takes 90 seconds has a latency of 90 seconds, full stop.

Throughput is drinks out the door per minute, across all baristas.

• Three baristas pulling shots in parallel might push out 20 drinks a minute.
• Each individual drink still takes its own 90 seconds. Parallelism did not make any single drink faster.

They move for different reasons, and that is the whole lesson.

• Add baristas and throughput climbs, while a single drink is exactly as slow as before.
• Buy one faster machine and a single drink speeds up, while drinks-per-minute barely moves if one person is still making them.

In software terms:

• Latency is milliseconds for one operation. "p99 read latency is 40ms" means 99 of 100 reads finish within 40 milliseconds.
• Throughput is requests per second (RPS), queries per second (QPS), or sometimes bytes per second. "The service handles 12,000 RPS" is a throughput number.

One word that causes half the confusion: "fast."

• A system can be "fast" because each request is snappy (low latency).
• Or "fast" because it chews through a huge batch quickly (high throughput).
• Nail down which one you mean before you say it in a room.

2How it works#

Trace a single request to see where latency comes from, then watch throughput stack on top of it.

Latency is additive. Every hop adds to the bill.

• Network time to reach the load balancer.
• The load balancer picking a backend and forwarding.
• The service doing its work: parsing, auth, business logic.
• A cache lookup: a hit is under 1ms, a miss falls through.
• On a miss, a database query: 5ms to 50ms depending on indexes and load.
• The response traveling all the way back.

The only levers on latency: make a step faster or delete a step.

• Five steps at 10ms each is 50ms no matter how beefy the servers are.
• This is exactly why caching matters so much: a cache hit deletes the database round trip from the critical path.

Throughput is a different game: how many pipelines run at once without stepping on each other.

• Add workers or servers so more requests run in parallel.
• Batch, grouping many small operations into one trip. A database writing 1,000 rows in one batch does far more per second than one writing them singly, even though each row now waits longer.
• Pipeline, starting request B while request A is still blocked on I/O.

Here is the time view of a batched write so the tension is visible:

A worked example: the same service, two ways#

Numbers make this click, so run one all the way through. A service writes rows to a database. Each round trip to the database costs 5ms of latency, and the database can physically do at most 10,000 writes per second no matter how you send them.

Option A: one write per request.

• Each client write is its own database trip, so a single write's latency is a clean 5ms.
• A thread doing 5ms per trip manages 1 / 0.005 = 200 writes per second.
• To reach the 10,000 writes/second ceiling you need 10,000 / 200 = 50 threads in parallel, each paying the full 5ms.
• Scorecard: 5ms per write, 50 threads burned, capped at the database's 10,000 writes/second.

Option B: batch 500 writes per trip.

• One trip now carries 500 rows. Call the write cost 10ms for the whole batch (5ms transfer plus a bit of server work).
• Throughput per thread jumps to 500 / 0.010 = 50,000 writes/second, so one or two threads saturate the database.
• But latency: an item arriving as a batch starts filling waits for 499 more items plus the flush. With a 20ms flush window its end-to-end latency is roughly 20ms + 10ms = 30ms, not 5ms.

Read the scorecard honestly. Same hardware. Batching bought a 6x throughput-per-thread gain and way less concurrency to manage, and it cost a 6x worse per-write latency (5ms to 30ms). If a human waits on each write, Option A wins. If it is a firehose of analytics events, Option B wins and the 30ms is invisible.

3Trade-offs#

The real decisions come down to a few recurring patterns.

A few things worth saying out loud:

You do not optimize both at once for free.

• When an interviewer says "improve this system," your first move is to ask which number they care about.
• A fraud check on every card swipe needs low latency: a human is waiting.
• A nightly report over a billion rows needs throughput: nobody cares if it takes 40 minutes as long as it finishes.

Queues decouple the two.

• Put a queue between producers and consumers and the producer gets low latency (fire and forget).
• Consumers grind through the backlog at whatever throughput they can manage.
• The cost: the end-to-end result is now delayed and eventually consistent. This is the whole reason async architectures exist.

Throughput has a ceiling set by your slowest shared resource.

• If every request hits one database that maxes out at 8,000 QPS, no pile of app servers gets you past 8,000.
• Find the bottleneck first. Adding capacity anywhere else is wasted money.

Latency under load is the number that bites you.

• A system that answers in 20ms idle can balloon to 2 seconds near its throughput ceiling, because requests start queuing.
• Always talk about latency at a given load, never in a vacuum.

4Where it is used#

Concrete places this shows up.

Payment and checkout flows: latency wins.

• A user staring at a spinner during checkout abandons the cart.
• Keep the synchronous path short: validate, authorize, return.
• Push anything that can wait (receipt email, analytics) to a queue and do it later.

Log and metrics pipelines: throughput wins.

• You are shipping millions of events, and each event's individual latency does not matter.
• Batch aggressively, flush every 5 seconds or every 10,000 events, and compress hard.
• One pipeline might accept 500,000 events/second while each event sits in a buffer for a couple of seconds. Totally fine.

Search autocomplete: latency wins, brutally.

• If suggestions do not appear in under about 100ms they feel broken and users ignore them.
• Serve from memory, often a cache or an in-process trie, and never touch a slow backend on the hot path.

Video encoding or ML training: throughput wins.

• You care about jobs finished per hour, not how long any single frame or batch takes.
• Run huge parallel batches on GPUs and keep them saturated.

If a human is actively waiting, optimize latency. If a machine is grinding through a backlog, optimize throughput.

5Common pitfalls and gotchas#

Quoting averages instead of percentiles.

• Average latency lies. If 99 requests take 10ms and one takes 3 seconds, your average is about 40ms, which sounds fine and hides a disaster.
• Talk in percentiles: p50 (median), p95, p99. The p99 is what your unluckiest 1% feel, and at scale that 1% is a lot of people.
• At 10,000 RPS, p99 is 100 slow requests every second. Say "p99" in a room and you sound like you have run production systems.

Confusing high throughput with low latency.

• A system can have massive throughput and awful latency at once.
• The batch pipeline is the classic: 500,000 events/second with 2 seconds of per-event latency.
• Do not assume a "fast" system is fast in the way this problem needs.

Ignoring the tail under load.

• Everything looks great in a demo with one user.
• The interesting behavior is at 80% of capacity, where queues build and tail latency explodes.
• Reason about the loaded state, not the idle one.

Forgetting that batching adds delay you must bound.

• Flush only when the buffer is full and a quiet period parks items for a long time.
• Always add a time-based flush too: "flush when full OR every 500ms," so latency has a ceiling.
• This is the exact gap flagged in the worked example.

Thinking more servers always helps.

• They help throughput only up to the bottleneck.
• If the real limit is a single primary database or a lock, more app servers just pile up more waiting requests and make latency worse.

Measuring throughput without a latency budget.

• It is easy to push throughput up by letting latency rot.
• "We hit 50,000 RPS" means nothing if p99 was 9 seconds at that point.
• The honest number is "50,000 RPS while keeping p99 under 200ms."

6How it shows up in interviews#

Interviewers probe this to see whether you treat systems as trade-offs and make deliberate choices instead of reaching for "just add more servers."

Here is what the probing sounds like and how to handle it.

"How would you make this faster?"

• A trap worth catching. Ask back: lower latency per request, or more requests per second?
• Strong candidate pauses and clarifies. Weak one starts adding servers, which only helps throughput and might not touch the latency problem at all.

"What is your p99 target?"

• They want a number tied to the use case.
• Strong: "User-facing search, p99 under 150ms. Background indexing job, no latency target, I care about a million documents per hour."
• Weak: "Fast as possible." Tells them nothing and shows no judgment.

"This endpoint is slow under load, what do you do?"

• Strong answer walks the request path to find where time goes: CPU, slow query, lock, network. Then picks a fix matching the cause: cache the hot read, add an index, move work off the synchronous path into a queue.
• Weak answer immediately says "scale horizontally" without knowing if the bottleneck is even scalable that way.

"Walk me through the trade-off of your design."

• They want you to volunteer what you gave up.
• Strong: "I batched the writes to hit our throughput target. The cost is each write takes up to 500ms longer, which is fine because this is an analytics path where nobody is waiting."
• Naming the cost out loud is what earns the point.

Weak answer vs strong answer#

The prompt: "We have a write service capped around 10,000 writes per second on one database, and we need to ingest a lot more. What do you do?"

Candidate (weak): I would add more app servers and scale horizontally so we can handle more writes.

That fails twice. It never asks whether anyone is waiting on each write, and it throws servers at a bottleneck that is the single database, so the extra servers just queue up behind the same 10,000 writes/second ceiling and latency gets worse, not better.

Candidate (strong): First, is a human waiting on each write, or is this a firehose we just need to absorb? If it is ingest, I would batch, say 500 writes per database trip. From the math, that takes one thread from 200 writes per second to around 50,000, so one or two threads saturate the database instead of 50 threads. The cost is per-write latency goes from about 5ms to roughly 30ms with a 20ms flush window, which is fine for ingest. I would bound that with a flush-when-full-OR-every-500ms rule so a quiet period cannot park an item indefinitely, and I would hold myself to a target like 50,000 writes per second while keeping p99 under 200ms.

That picks the right metric, names the bottleneck, shows the arithmetic, and states the cost it accepted. That is the whole rubric in one breath.

The signal they are reading: can you pick the right metric for the problem, reason about the bottleneck, and state the trade-off plainly? Do that and you look like someone who has shipped and operated systems, not just read about them.

One last move for the whiteboard. Point at the synchronous path and say "this is my latency-critical path, I keep it short." Then point at the queue and say "this absorbs the throughput and runs async." That single gesture shows the interviewer you get it.

Next, try the System Design Question walkthrough on designing a URL shortener, where the read path is pure latency and the write path is a throughput problem, and you have to balance both in one design.