Design a Notification Service

1Clarifying the requirements#

Narrow the problem before drawing a box. "Notification service" can mean ten different systems, and a pure OTP sender looks nothing like a marketing platform. Each answer below removes a whole branch of the design tree, so these questions are load-bearing, not throat-clearing.

Candidate: When you say notifications, do you mean push only, or email and SMS too?

Interviewer: All three. Push, email, SMS. Assume we might add WhatsApp later.

Candidate: Are these triggered by product events like "order shipped," or can users also schedule them?

Interviewer: Mostly event-triggered from other internal services. Some scheduled, like a daily digest.

Candidate: Do we care about ordering? If I send "your code is 1234" then "your code is 5678," the second has to win.

Interviewer: Good catch. Per-user ordering matters for some types. Best-effort globally.

Candidate: And delivery guarantee: at-least-once or exactly-once?

Interviewer: At-least-once is fine, but you must dedupe so the user doesn't see duplicates.

What those answers just told me:

• "WhatsApp later" means the channel layer is pluggable, not three hardcoded senders.
• "Machines, not browsers" changes the API shape: idempotency keys matter more than CSRF tokens, and a scheduler sits off to the side, not in the hot path.
• "Per-user, not global ordering" is the gap between a strict ordered log (expensive) and a cheap last-write-wins check (good enough). I come back to this in 6.2.
• "At-least-once with user-facing dedup" is the whole game. It is also a trap: promise exactly-once across third-party providers later and you contradict this and lose the room.

Functional requirements

• Accept notification requests from internal services over an API.
• Support three channels (push, email, SMS), extensible to more.
• Let each user set preferences (opt out of marketing, quiet hours, preferred channel).
• Template support, so callers send order_shipped plus data, not raw strings.
• Scheduled and recurring sends (daily digest).
• Track delivery status: queued, sent, delivered, failed, bounced.

Non-functional requirements

• High write throughput, bursty. A marketing blast can be millions in minutes.
• Low latency for transactional sends (an OTP must land in a couple of seconds).
• At-least-once delivery, deduped so users never see a repeat.
• Reliable: if a provider fails, retry and fail over, never drop.
• Respect rate limits, both ours and the downstream providers'.

One non-goal I name out loud, because naming it buys trust:

• We are not promising guaranteed delivery with a hard SLA.
• A user's phone off for three days, push token expired: that notification dies, and that is correct behavior.

2Back-of-the-envelope estimation#

Get the scale in your head, because it decides the architecture. One number drives everything, and I will flag it when I reach it.

Users and steady load.

• 50 million daily active users.
• Each gets about 4 notifications a day, so 200M notifications/day.
• Over 86,400 seconds that is about 2,300 notifications/second average.

200,000,000 / 86,400 sec ≈ 2,300 notifications/sec average

• If 2,300/sec were the real load, this would be a boring single-service design. It is not.

The spike (this is the one that matters).

• Notifications are spiky. A promo push to all users goes out in a 10-minute window.
• 30M pushes in 600 seconds is 50,000 notifications/second at peak.

30,000,000 / 600 ≈ 50,000 notifications/sec peak

• Peak-to-average is about 22 to 1. You cannot provision steady-state and hope.
• You must absorb a 22x spike without touching the latency of the OTP a user is staring at while trying to log in.

Writes, reads, and the provider ceiling.

• Design for about 50k writes/sec peak, 2-3k steady.
• Reads (status lookups, preference checks) are lighter, call it 10k/sec. This is a write-heavy, fan-out-heavy system.
• We can produce 50k/sec, but a typical email provider contract caps you near 5k/sec. That gap is 10x, and the queue is what bridges it.
• The bottleneck is not our CPU, it is the vendor's ingress.

Storage.

• Each record holds recipient, channel, template, payload, status, timestamps: about 500 bytes after trimming.
• 200M/day x 500 bytes = 100 GB/day.

200M/day * 500 bytes = 100 GB/day

• Keep full records 30 days then archive: about 3 TB hot storage, easy to hold partitioned, with old rows tiered to cheap object storage.

3Core APIs#

The internal-facing API is the important one. Callers are other services, not browsers, so I optimize for safe retries and clear priority, not human ergonomics.

POST /v1/notifications
Content-Type: application/json
Idempotency-Key: order-9931-shipped

{
  "user_id": "u_4482",
  "template": "order_shipped",
  "channels": ["push", "email"],
  "data": { "order_id": "9931", "eta": "Nov 5" },
  "priority": "transactional"
}

{
  "notification_id": "ntf_a1b2c3",
  "status": "accepted"
}

Three design choices worth defending:

• The Idempotency-Key is how callers get safe retries. If the order service POSTs twice because of its own retry, we treat it as one.
• priority routes OTPs into a separate lane ahead of marketing.
• The response is accepted, not sent. The API returns the instant work is durably queued, usually single-digit milliseconds. Make it block until the provider confirms and you have welded your p99 to a flaky third party: the first provider slowdown takes down ingestion for everyone.

A status read and a preference update round it out:

GET /v1/notifications/ntf_a1b2c3
PUT /v1/users/u_4482/preferences

For bulk, I would not accept 30M individual POSTs.

• I expose a separate campaign endpoint that takes a segment, not a recipient list.
• It hands the expansion to a fan-out worker. More on that in 6.1.

4High-level design#

The spine is standard, so draw it fast:

• An ingestion API validates and drops work onto a queue.
• Workers resolve preferences and templates.
• Channel-specific senders own the provider relationships.
• Transactional and bulk ride separate queues so a blast never delays an OTP.

Write path.

• A service calls POST /notifications.
• The API does cheap validation, checks the idempotency key, writes an "accepted" row, and publishes to the right queue (transactional or bulk).
• It returns immediately. Everything after is async.

Worker path. For each message a worker:

1. Resolves the user's preferences. Opted out of this category? Drop it. In quiet hours? Delay or downgrade.
2. Picks channels based on preference and what the caller asked for.
3. Renders the template with the data.
4. Checks the dedup cache. Already sent this exact thing recently? Skip.
5. Hands the rendered message to the right channel sender.

Channel senders own the provider relationship: auth, rate limits, retries, failover between providers, and status updates as providers ack or report delivery via webhooks.

Why a queue in the middle. Two reasons, both straight out of the estimation:

• It absorbs the 50k/sec spikes so senders drain at the 5k/sec the email provider actually allows. Without the buffer, that 10x gap turns into dropped messages.
• It gives us at-least-once: the queue re-delivers any message a worker crashes on before acking, so a worker dying mid-send never silently loses a notification.
• The cost of that guarantee is duplicates, which is exactly why 6.2 exists.

5Data model#

Use a few stores, each picked for its access pattern. The notification log and the preferences table are different beasts, so I do not force them into one database.

notifications (high-write log in a partitioned store: Cassandra-style, or Postgres partitioned by time):

• The log is append-heavy, write-once-update-a-few-times, and time-ordered, which is exactly what a wide-column store with a time clustering key is built for.

user_preferences (small, relational, read on every single send, so cache it hard):

user_devices (for push):

Templates live in their own table keyed by template plus locale plus channel, holding the body with placeholders.

• Keeping them out of code means product can change copy without a deploy.
• A typo fix in an email subject line does not require a release and a worker restart, which in practice is the feature that saves you at 11pm before a campaign.

One indexing note I volunteer unprompted:

• Delivery webhooks arrive keyed by provider_msg_id, not by notification_id.
• In Cassandra that needs its own lookup path: a secondary table (by_provider_msg_id -> notification_id), because you do not get free secondary indexes on a wide-column store without paying for them.

6Deep dives#

This is where the interview is won. Three hard parts: fan-out, dedup, and provider failure. I spend most of my time here.

6.1 Fan-out: the marketing blast problem#

The hard part is not sending one notification. It is sending 30 million when someone clicks "send campaign."

• Enqueue 30M messages synchronously and the API falls over: 30M inserts in one request is both a timeout and a memory bomb.
• Worse, it blocks the ingestion tier that OTPs share.

So a bulk campaign is a two-stage job.

• The caller submits one campaign request with a target segment.
• A fan-out worker reads that segment in pages (say 10k users at a time).
• For each page it enqueues individual messages onto the bulk queue.
• At 10k per page, 30M is 3,000 pages streamed out steadily instead of one lethal burst.

Two protections baked in:

• The bulk queue is rate-limited on the consumer side so we never exceed the email provider's 5k/sec ceiling from the estimation.
• Bulk lives on a totally separate queue from transactional, so your OTP is never stuck behind a promo backlog. That separation is the single most important design choice in this whole system.

6.2 Dedup and idempotency#

At-least-once means duplicates will happen.

• A worker sends a message, then crashes before acking the queue.
• The queue redelivers. Now two workers might send the same push.
• We promised the user never sees a dupe, so we need a guard.

I use a dedup cache in Redis, claimed atomically right before the send.

• Before a sender fires, it runs SET dedup:{hash} 1 NX EX 86400, where the hash is user_id + template + channel + a content fingerprint.
• If the key already exists, the set fails and we skip the send.
• The key lives 24 hours, which covers any realistic retry window without blocking a legitimately repeated notification tomorrow.
• The NX is doing the real work: it makes the check-and-claim a single atomic operation, so two workers racing on the same redelivered message cannot both win. Exactly one gets the key, exactly one sends.

This is a different layer from the caller-side idempotency key.

• The idempotency key dedupes at ingestion: if the order service POSTs twice, the API sees the same key and returns the existing notification_id without re-enqueuing.
• The dedup cache dedupes right before the provider call.
• Two layers (front door and last step) because a dropped message here means a user gets buzzed twice and trust erodes fast.

The ordering subtlety in the atomic claim, worth saying out loud:

• You claim the dedup key before the provider call, but the provider call can still fail after you own the key.
• Treat that key as "sent" and a transient provider error becomes a permanently lost message.
• So the sender claims the key, calls the provider, and on a hard failure either releases the key or lets the retry path re-claim under a retry-aware hash.
• I bias toward the user never seeing a dupe, and I accept the rare lost transactional message gets caught by the caller's own retry with a fresh idempotency key.

Per-user ordering, done cheaply (the "newest wins" case):

• For OTPs where the newest wins, I do not enforce a global order.
• I keep a per-user last_sent_seq in Redis and a sender discards any message whose sequence is below it.
• This satisfies "best-effort global, strict per-user newest-wins" without the cost of a totally ordered partition per user.

6.3 Provider failure and retries#

Treat every provider as unreliable, because over a year it provably is.

• Email and SMS providers go down, rate-limit you, or silently drop messages.
• The channel sender owns surviving that, not the caller.

Backoff with jitter, then failover:

• Exponential backoff with jitter, so a provider blip does not turn into a retry stampede. Without jitter, thousands of failed messages retry at the same instant and you take down your own recovering provider.
• After a few attempts on the primary, fail over to a secondary. Multi-provider is worth the complexity for SMS and email, where single-vendor outages are real and common.

Circuit breakers stop you from wasting latency on a dead endpoint:

• Wrap each provider in a circuit breaker. If the primary throws errors past a threshold, trip it and send straight to the secondary for a cooldown window.
• The breaker is the difference between a 30-second provider outage costing you 30 seconds of latency versus a full retry timeout on every one of 50k queued messages.

Dead-letter queue and the bounce-suppression loop:

• If everything fails, the message lands in a dead-letter queue. We do not lose it, we alert, and we can replay once the provider recovers.
• Delivery status comes back async: providers call our webhook when an email bounces or a push is delivered. The handler looks up the row by provider_msg_id and updates status.
• Hard-bounced emails get suppressed so we stop sending to dead addresses.
• That suppression is not a nice-to-have: keep hammering dead addresses and the provider's reputation system tanks deliverability for every other user, so the bounce feedback protects the whole channel.

6.4 Quiet hours and preference enforcement#

The deep dive most candidates skip, and the one I like to push on, because it is where correctness meets product.

Quiet hours are category-aware, never blanket:

• A worker has a transactional OTP and a user whose quiet hours are 10pm to 7am local. Do you hold the OTP until 7am? No.
• Quiet hours apply to marketing and social, never to transactional.
• So the preference check is category-aware, and the worker evaluates quiet hours only for the categories allowed to delay.

The timezone math is the trap:

• Quiet hours are stored in the user's local time with a timezone field, so the worker converts "now" into the user's local time before comparing against quiet_start and quiet_end.
• It handles the wrap-around case where quiet_start 22:00 is greater than quiet_end 07:00.
• A notification landing inside quiet hours for a delayable category is not dropped, it is re-enqueued with a visibility delay until the window opens.
• That is why the scheduler and the real-time path share the same queue primitive.

7Bottlenecks and trade-offs#

The dedup cache is a single point of pressure.

• Every send hits Redis. At 50k/sec that is a lot of ops.
• If Redis is down, do we send without dedup or block? I fail open for transactional (better a rare dupe than a missed OTP) and fail closed for bulk (a dupe marketing blast is a disaster).
• Shard the cache by user_id to spread load and avoid one hot instance.
• The fail-open versus fail-closed split is a decision I say out loud, because it shows the two lanes have opposite failure preferences.

Hot providers and rate limits.

• We generate 50k/sec; the email provider might accept 5k/sec.
• The queue plus consumer-side rate limiting saves us: buffer and drain at the provider's pace.
• The trade-off is latency for bulk, which is fine. Nobody needs a promo in 200ms.
• The transactional lane never hits this wall because its volume is a tiny fraction of that 50k spike.

Preference and template lookups per message.

• At peak that is about 50k preference reads/sec.
• Cache preferences in Redis with a short TTL and bust on update.
• Templates are nearly static, so cache them aggressively in-process on the workers, which turns a template lookup from a network hop into a map read.

Exactly-once is a trap.

• Promise exactly-once across unreliable third-party providers and you are overselling. You physically cannot know if an SMS gateway delivered after a timeout.
• So we commit to at-least-once plus dedup, and we are honest that the dedup window is finite at 24 hours.
• That honesty is what I am listening for.

Scheduled sends.

• The daily digest needs a scheduler. I keep a separate table of scheduled jobs and a cron-like dispatcher that enqueues them when due, rather than cramming scheduling into the hot path.
• The dispatcher reuses the same fan-out worker for a digest that targets a segment, so scheduling is just "enqueue the fan-out job at time T" rather than a second parallel system.

Observability and SLOs.

• SLOs as measured targets: transactional send p99 under a couple of seconds end to end, bulk measured on drain time, not per-message latency.
• Watch: queue lag per lane, dedup-cache op rate and error rate, provider error rate and breaker state, bounce rate per channel, dead-letter depth.
• When a number moves, I already know which lever it maps to.

The biggest thing I defend in review.

• Separating transactional and bulk into isolated lanes (queues, workers, even rate-limit budgets).
• It is not the flashiest idea, but it is what keeps a bad marketing push from taking down your login OTPs.

Isolate the lanes first; a blast that can starve an OTP is a design that fails the one moment that matters.

8Summary#

The decisions that matter:

• Two physical lanes. Transactional (low latency) and bulk (high throughput), physically separated so one can never starve the other. This is the answer to the 22x peak-to-average spike and the single most important call.
• Async spine. The API validates and enqueues, workers and senders do the heavy lifting off the hot path. It returns accepted in milliseconds, never waits on a provider.
• Paged fan-out. A dedicated worker pages a segment out with a durable cursor, so a crash resumes instead of re-sending from scratch. Never 30M synchronous inserts.
• At-least-once with two dedup layers. Idempotency key at ingestion, atomic content-hash claim before the provider call. The queue guarantees delivery, the data structure guarantees the user sees it once.
• Provider reliability. Backoff with jitter, multi-provider failover, circuit breakers, a dead-letter queue, and a bounce-suppression loop that protects channel deliverability.
• Honest guarantees. At-least-once, deduped, not exactly-once, with a finite 24-hour dedup window named explicitly.

For how specific companies probe this one, see our related guides on designing a rate limiter, a message queue, and a URL shortener for patterns that show up across these company interviews.