Lambda and Kappa Architectures: A System Design Guide

If you remember five things

The whole answer in one glance. Everything below defends these five calls.

Fast and correct fight. Streaming gives you a number in milliseconds but drops late events and double-counts on retries. Batch is dead accurate but hours behind. Both architectures exist to resolve that tension.
Lambda is two kitchens, one plate. A batch layer recomputes the authoritative truth from raw events, a speed layer approximates the recent window, and the serving layer merges them. The batch run heals the fast path's errors every cycle.
Kappa is one kitchen, cook it again. No batch layer. The event log is the source of truth, and to recompute history you replay the log through the same streaming code into a fresh table, then cut reads over.
The real trade is pain you prefer. Lambda spends engineering time keeping two definitions of a metric in sync. Kappa spends storage and a long replay window. That choice is most of the decision.
Default to Kappa in 2026. Modern stream engines handle exactly-once state, event-time windows, and watermarks well enough that the original reason for Lambda is mostly gone. Add a batch layer only when a specific requirement forces it.

1The core idea#

Both architectures exist to solve one problem: you have a firehose of events, and you want two answers that pull in opposite directions.

The setup.

A stream of events: ad clicks, meter readings, payment attempts, page views.
Picture 50,000 events per second, roughly 4.3 billion a day.
You want results right now: how many clicks in the last 30 seconds.
You also want results that are exactly right: the true total for yesterday, recomputed with late data and bug fixes folded in.

Fast and correct are at war.

A streaming system that returns a number in 200ms drops some late events and occasionally double-counts on a retry.
A batch job that reads every record from cold storage and recomputes from scratch is dead accurate, but it finishes at 6am for yesterday's data.
You cannot have both from one simple pipeline, so each architecture makes a different bet about how to get both.

Lambda's bet: run both and merge.

Keep a batch layer that reprocesses the full history and produces the authoritative truth.
Keep a speed layer that approximates the recent window the batch has not caught up on.
Merge them at query time.
The batch layer eventually overwrites the approximation, so errors in the fast path heal themselves within a cycle.

Kappa's bet: delete the batch layer.

Two codebases for one computation will drift and ruin your week.
Treat everything as a stream. The event log is the source of truth.
Need to recompute history? Replay the log through the same streaming code.
One pipeline, one set of logic, no second definition to keep in sync.

Lambda is two kitchens, one plate. Kappa is one kitchen, cook it again if you got it wrong.

2How it works#

Draw both quickly, then spend your words on where they bite.

Lambda#

Three layers, each with a clear job.

Batch layer. Owns the master dataset: an immutable, append-only log of raw events in something cheap and durable (S3, HDFS, a data lake). On a schedule, usually hourly or daily, a batch job (historically MapReduce, now Spark) chews through all of it and writes precomputed batch views into a serving store.
Speed layer. Consumes the same events in real time through a stream processor (Flink, Spark Streaming, Kafka Streams) and maintains real-time views that only cover the window the batch layer has not caught up on yet.
Serving layer. Answers queries by merging the batch view with the real-time view. Once a batch run completes for a time window, its real-time view is discarded, because the authoritative version now exists.

flowchart LR A[Event Source] --> B[Message Log] B --> C[Batch Layer] B --> D[Speed Layer] C --> E[(Batch Views)] D --> F[(Realtime Views)] E --> G[Serving Layer] F --> G G --> H[Query API]

Lambda's two paths from one log. The same events flow into both the batch and speed layers. Notice the serving layer is where they reunite: it merges the authoritative batch view with the approximate real-time view, and the real-time view is thrown away once batch catches up.

Why this is elegant: it self-heals.

Say your speed layer has a bug and overcounts clicks by 3% today. Annoying, but temporary.
Tonight the batch job recomputes from the raw master dataset, which never had the bug, and the correct number replaces the bad one.
The raw log is sacred and immutable, so you can always rebuild from truth.

Kappa#

Kappa collapses the three layers into one.

No batch layer at all.
You keep a durable, replayable log. In practice that means Kafka with long retention or tiered storage, or Pulsar, or Kinesis.
A single stream processing job reads the log and produces the serving views.

flowchart LR A[Event Source] --> B[Durable Log] B --> C[Stream Processor] C --> D[(Serving Views)] D --> E[Query API]

Kappa's single path. One stream processor, one output. There is no second system to reconcile, which is the whole point. The cost of that simplicity hides in the word "durable": the log has to retain and replay all the history you might ever recompute.

Reprocessing is the clever trick.

To fix a bug or change the computation, you do not patch the old output.
You start a second instance of the job, point it at offset zero of the log, and let it chew through all of history into a fresh output table.
When it catches up to the present, you flip reads to the new table and retire the old one.
Same code path for "live" and "historical," because there is only one code path.

Pause before you read on: in Kappa, you never run a separate batch job, yet you still need to recompute two years of history after a logic change. How?

Reveal the answer

You replay. The live job is just the streaming code reading new offsets. To recompute history, you point a second copy of that same code at offset zero of the durable log, write into a new output table, and let it race forward until it catches the present, then cut reads over. "Batch" in Kappa is just "the stream, replayed from the start." That is why the log must be retained and replayable, and why retention cost is the thing that bites.

3A worked example with real numbers#

Make it concrete with a metering system: counting billable API calls per customer at 50,000 events per second, feeding both a live usage dashboard and the monthly invoice run.

On Lambda.

The speed layer (Flink) keeps a per-customer running count for the current hour in RocksDB state, updated in about 150ms per event batch, and the dashboard reads it.
The batch layer (a Spark job) runs every hour over the raw events in S3 and writes the authoritative per-customer-per-hour totals to a serving table.
At 14:20 the dashboard shows customer A at 812,400 calls for the 14:00 hour.
At 15:00 the Spark job finishes the 14:00 window, finds 811,900 (the speed layer had double-counted 500 calls on a consumer retry), and overwrites it.
The invoice reads only batch totals, so the customer is billed 811,900, not the inflated number.
Cost of this correctness: two implementations of "a billable call" (one in Flink, one in Spark) and a Spark cluster running hourly over about 180 million events per window.

On Kappa.

One Flink job reads the Kafka log and maintains the same per-customer counts, written to a serving table every few seconds.
The dashboard and the invoice both read that one table.
To fix the double-count, I change the job to dedupe by event id, spin up a second instance against offset zero, and let it rebuild.
Replaying 30 days at 50,000 events per second is 129.6 billion events.
At 200,000 events per second (4x live, because I gave it 4x the partitions and task slots), that is about 180 hours serially, so I widen to 20 parallel partitions and bring it down to roughly 9 hours, then cut reads over.
No second codebase, but a real retention and compute bill: 30 days of that log is on the order of 10 to 15 TB that Kafka has to keep replayable.

~9 hrs

Kappa's hidden cost, made realWall-clock time to replay 30 days (130 billion events) through a widened 20-partition job to fix one metric definition. This is the number candidates forget. Name it out loud, with the retention bill behind it, and you sound like you have actually run Kappa.

Same requirement, two different failure modes.

Lambda spends engineering time keeping two definitions in sync.
Kappa spends storage and a long replay window.
Which pain you prefer is most of the decision.

4Trade-offs#

The one trade-off to anchor on

Lambda buys flexibility and bulletproof correctness at the cost of maintaining the same logic twice. Kappa buys operational simplicity at the cost of a stream engine strong enough to carry the whole load, plus a log you can afford to retain and replay. Everything below is a consequence of that one sentence.

The "logic twice" problem is not theoretical.

On a Lambda system I worked on, every metric was defined once in Scala for the Spark batch job and again in a streaming DSL for the speed layer.
They drifted.
A PM would ask why the real-time dashboard said 1.2M and the morning report said 1.18M.
The honest answer: two engineers had interpreted "active session" slightly differently.
We spent real hours reconciling numbers that should have been identical by construction.

Dimension	Lambda	Kappa
Codebases for one metric	Two (batch + stream)	One (stream only)
Correctness of historical data	Very high, recomputed from raw	High, depends on replay + engine
Operational burden	Heavy, two systems to run	Lighter, one pipeline
Reprocessing a logic change	Rerun batch job	Replay log from offset zero
Storage cost	Cheap raw log + view stores	Log must retain full history (pricier)
Latency of fresh data	Seconds via speed layer	Seconds via the single stream
Best when	Heavy batch analytics, complex historical recompute	Event-centric, near-real-time, simpler aggregations

A few sharp edges worth naming.

Kappa leans hard on log retention. If your log keeps only 7 days and you need to recompute 2 years, the replay story falls apart unless you have tiered storage that spills old segments to object store. That feature exists now and it is good, but it is a cost and an operational commitment, not a free lunch.
Lambda wastes compute. You process the same recent events in both layers. In the metering example, those 180 million events per hour flow through both Flink and Spark. For many workloads that redundancy is cheap insurance. For some it is a meaningful bill.
Kappa's reprocessing is only as fast as the stream job can replay. Replaying 2 years through a job tuned for live throughput can take many hours (the 9-hour replay above) and will compete for cluster resources with live traffic. Plan for that or you will take down the live view while fixing the historical one.

5Where it is used#

Lambda came out of the big-data era, when stream processors were immature and you genuinely could not trust them for correctness.

The classic shape: a clickstream or IoT analytics platform. Raw events land in a data lake, a nightly Spark job builds the source-of-truth aggregates, and a thin speed layer gives the dashboard a live-ish number for the current hour.
If your org already has a mature batch stack (a warehouse, scheduled Spark, a BI layer) and you are bolting on real-time, you often land in Lambda whether you planned to or not.

Kappa fits event-driven businesses where the aggregations are not insanely complex.

Real-time fraud scoring, live personalization, usage metering for billing, operational monitoring.
If you are already all-in on Kafka or Pulsar and your team thinks in streams, Kappa is the natural destination.
Modern engines handle exactly-once semantics, event-time windowing, and watermarks for late data well enough that the original reason Lambda existed ("you cannot trust the stream") is mostly gone.

My honest default for greenfield in 2026: Kappa, unless a requirement forces otherwise.

The requirements that force a batch layer are real: heavy ML feature backfills over years of data, a computation so expensive it only makes sense as a bulk batch job, or a regulatory need to recompute from immutable raw records in a way auditors trust.
Those push you back toward Lambda, or toward a hybrid where a warehouse (a lakehouse table) serves the batch role.

6Common pitfalls and gotchas#

Each of these is a place I have watched a candidate (or a real system) fall over.

Treating the master dataset as mutable. The whole Lambda guarantee rests on raw events being immutable and append-only. The moment someone edits or deletes a record from the master log to "fix" something, you have lost the ability to rebuild from truth. Model corrections as new events, never as edits.
Forgetting about late and out-of-order data. Events arrive late: a phone was offline, a retry fired twice, a clock was skewed. Your speed layer needs event-time processing and watermarks, not processing-time, or your windows will be subtly wrong. This is the single most common thing juniors miss.
Idempotency and exactly-once wishful thinking. "Exactly-once" in a stream engine usually means exactly-once state updates inside the engine, not exactly-once side effects on the outside world. If your job writes to an external database or sends an email, you still need idempotent writes keyed by event id. The 500 double-counted calls in the metering example came from exactly this gap.
Underestimating Kappa replay cost. Teams adopt Kappa for simplicity, then discover that reprocessing 18 months of data to fix a metric takes 9 hours and saturates the cluster. Size the reprocessing path deliberately: a parallel replay with more partitions and a temporary burst of compute is how the 180-hour serial replay above came down to 9.
Picking Lambda by default because it is the "safe" textbook answer. In an interview this reads as dated. The field moved. Defending two full codebases needs a reason now.

Red flag that fails you here

Saying "exactly-once" as if it solves correctness, then going quiet when I ask what happens when your job writes to an external invoice table and the task restarts mid-batch. Exactly-once state inside the engine does not make your external writes idempotent. If you cannot key those writes by event id, you have not actually closed the double-count hole.

7How it shows up in interviews#

Nobody says "compare Lambda and Kappa." It is smuggled in through a design prompt, and you are expected to reach for the right shape and justify it.

"Design a real-time analytics dashboard."
"Build a metering system for API usage."
"Design fraud detection over a transaction stream."

From the interviewer's chair

The prompt is bait. I want to see if you name the fast-versus-correct tension on your own and pick a shape with a reason, not recite "Lambda has two layers." The moment you say "I would start with Kappa because one codebase, and add batch only if we need auditable backfills," you have told me you have shipped this, not just read about it.

What I probe for, with the weak and strong versions.

Do you understand why Lambda exists? Weak: "Lambda has a batch layer and a speed layer." Strong: "Lambda exists because you want both low latency and perfect correctness, and the batch layer heals the approximations the speed layer makes. If the speed layer double-counts 500 calls today, tonight's batch job recomputes from the raw log and overwrites the bad number. The cost is maintaining the same logic twice, which is why Kappa tries to kill the batch layer."
Do you know the modern context? Weak: recommending Lambda for everything because it is what the book shows. Strong: "I would start with Kappa for the single codebase, and only add batch for expensive historical backfills or auditable recomputation from raw records."
Can you handle correctness details? The follow-ups go to late data, event-time versus processing-time, watermarks, idempotent writes, and how reprocessing works. Saying "we replay the log from offset zero into a new output table, catch up, then cut reads over" shows you get Kappa rather than having memorized a diagram.
Do you talk about cost and operations? Strong candidates mention log retention cost (that 10 to 15 TB of replayable Kafka), replay compute, and the human cost of two drifting codebases (the 1.2M versus 1.18M argument). That lived-experience signal is what senior interviewers hunt for.

The trap to avoid: treating these as sacred patterns you must pick between.

The best answer is often "neither, exactly."
A warehouse-backed near-real-time pipeline, or a streaming-first design with the data lake as cold storage for occasional backfill, solves most problems without a formal Lambda build.
Name the pattern, then show you can see past it.

Follow-ups the interviewer will ask (and your answers)

"Your speed layer double-counts on a retry. Is the customer billed wrong?"

On Lambda, no: the invoice reads only batch totals, and tonight's batch job recomputes from the immutable raw log that never had the bug, overwriting the inflated number. The dashboard was briefly wrong, the bill is right. On Kappa, there is one table, so I fix it by deduping on event id in the stream job and replaying from offset zero into a fresh table, then cutting reads over.

"Events arrive late and out of order. How do your windows stay correct?"

Process on event time, not processing time, and use watermarks to decide when a window is complete enough to emit while still allowing a bounded grace period for stragglers. Processing-time windows would bucket a 10-minutes-late event into the wrong window and quietly corrupt the totals. This is the detail that separates people who have run streaming from people who have read about it.

"Exactly-once solves correctness, right?"

Only inside the engine. Exactly-once in Flink or Kafka Streams means exactly-once state updates within the pipeline. The moment you write to an external database or fire an email, you need idempotent writes keyed by event id, because a task restart can replay the last batch of side effects. That gap is exactly where the 500 double-counted calls came from.

"You chose Kappa. Now I need to recompute two years to fix a metric. Walk me through it."

Spin up a second instance of the same job against offset zero of the durable log, writing to a new output table, and let it race forward to the present. The cost is retention and compute: the log must hold two years replayable (tiered storage spilling old segments to object store), and the replay competes with live traffic. I size it deliberately: widen partitions and task slots so replay runs at several times live throughput, budget the wall-clock hours, then cut reads over. For 30 days that was roughly 9 hours on 20 partitions.

"When would you refuse both and build something else?"

When a warehouse-backed near-real-time pipeline covers it with less baggage, or when the real requirement is heavy ML feature backfills over years of data or auditor-trusted recomputation from raw records. In the first case I skip the formal Lambda build; in the second I accept a batch role, often a lakehouse table, rather than a bespoke batch layer. The pattern name is a starting point, not a cage.

What you can now explain in the room

Why fast and correct fight, and how Lambda and Kappa each resolve that tension.
Lambda's three layers, and precisely how the batch run heals the speed layer's errors.
Kappa's replay trick: recompute history by rerunning the one codebase from offset zero into a fresh table, then cutting reads over.
The real trade-off: Lambda pays in two drifting codebases, Kappa pays in retention and a long replay window.
The correctness traps: mutable master data, processing-time windows, and exactly-once that stops at the engine boundary.
Why Kappa is the sane 2026 default, and the specific requirements that still force a batch layer.

Want to see this applied end to end? Try the System Design Question walkthrough on designing a real-time analytics pipeline, where these trade-offs decide the whole architecture.

← Previous

Change Data Capture (CDC)

MapReduce

Get a prep plan built around your Lambda and Kappa Architectures interview

Tell us the role and your interview date. We will send back a dated, day by day plan. Free.

Thanks. Check your inbox, your Lambda and Kappa Architectures prep plan is on the way.