Webhooks

1The core idea#

A webhook flips the direction of a normal API call. Instead of your service calling someone and waiting, you register a URL and the provider calls you.

• Normal API: your code asks Stripe "has this payment succeeded yet?" on a loop.
• Webhook: Stripe calls your /webhooks/stripe endpoint the instant the charge clears and hands you the whole event.
• The mental model: "don't call us, we'll call you" for software.

The alternative is polling, and the cost difference is brutal.

• Polling is a consumer asking "anything new?" on a timer.
• Say 50,000 clients each poll every 10 seconds for an event that fires once an hour.
• That is 5,000 requests per second, and 99.99% of them return "nothing new."
• Webhooks invert that: zero traffic until there is actually something to say, then one POST.

Polling is pull on a timer. Webhooks are push on an event.

2How it works#

The lifecycle has three phases. Keep them straight and the rest falls out.

• Registration. The consumer tells the provider a callback URL and which events it cares about (say payment.succeeded and payment.refunded). Dashboard setting or API call. The provider stores it as a subscription.
• Delivery. On a matching event, the provider serializes it (almost always JSON) and POSTs to the registered URL, with a signature header, an event ID, and usually a timestamp.
• Handling. The consumer verifies the signature, does the minimum work to safely accept the event, returns a fast 2xx, and processes the real logic asynchronously.

Here is the delivery flow for a typical payment webhook:

The internal shape of a resilient receiver.

• The receiver is thin: verify, dedupe, enqueue, ack.
• The worker is where the real work happens.
• Split this way, the worker scales, retries, and is monitored independently of the hot path.

3A worked example: Stripe payments at 500 orders per minute#

"Ack fast, process async" stays abstract until you size it, so let me put real numbers on it.

• Checkout peaks at 500 completed payments per minute, roughly 8.3 events per second.
• Each fulfillment does three things: a database write (10 ms), a Stripe refetch to confirm the amount (120 ms), and an email send (300 ms).
• That is about 430 ms of real work per event.

Processing inline.

• At 8.3 events per second with 430 ms each, you need at least 4 concurrent handlers just to keep up with the mean.
• Any spike pushes tail latency past Stripe's 20-second timeout.
• Stripe then retries the slow ones with backoff over up to 3 days.
• Those retries land as duplicates while the originals are still running: one spike, a duplicate storm.

Ack fast and queue.

• The receiver verifies the HMAC (about 0.2 ms) and writes one dedup row (about 2 ms), then enqueues and returns 200.
• Total receiver time is roughly 3 ms, so a single modest instance handles 8.3 events per second with 99.9% headroom.
• The 430 ms of real work moves to workers draining the queue.
• Steady state needs ceil(8.3 x 0.43) = 4 workers, and you can scale to 8 during the flash sale without touching the receiver.
• The queue absorbs the spike, Stripe never sees a slow response, so it never retries.

Sizing the dedup store.

• 500 events per minute with a 24-hour retention window is 720,000 keys.
• At roughly 100 bytes per key (event ID plus a timestamp), that is about 72 MB in Redis. Trivial.
• TTL of 24 hours: Stripe's retry window is 3 days, but duplicates in practice cluster within minutes, and a backstop idempotent write covers the rest.

4Trade-offs#

The headline call is webhooks versus polling. Inside "use webhooks" there are more.

What you give up with webhooks: control and simplicity.

• You now operate a public HTTP endpoint that must be up, fast, authenticated, and idempotent.
• You are at the mercy of the provider's retry policy.
• If your endpoint is down during a burst, you depend entirely on the provider replaying them, and not every provider does that well.

What you give up with polling: freshness and money.

• You either accept lag, or you poll aggressively and burn compute on both sides.

A third option worth naming: long polling, server-sent events, or WebSockets.

• These keep a connection open so the server can push to a client that cannot expose a public URL (a browser, a mobile app).
• Webhooks are the right tool for server-to-server push where the consumer owns a stable, reachable endpoint.
• If the consumer is a phone behind a NAT, webhooks do not fit. Reach for a persistent connection or a push notification service.

One trade-off inside delivery semantics: webhooks are at-least-once by nature.

• Providers retry on failure, and retries plus network uncertainty mean you will receive the same event twice eventually.
• You cannot get exactly-once over HTTP.
• You get at-least-once delivery plus idempotent processing, and together those approximate exactly-once effects.
• Anyone who claims exactly-once webhook delivery is hand-waving.

5Where it is used#

Webhooks are everywhere once you look.

• Payments. Stripe, PayPal, and friends send payment.succeeded, subscription.cancelled, charge.refunded. The webhook is the source of truth for fulfillment, not the browser redirect, because the user can close the tab before the redirect fires.
• Version control and CI. GitHub POSTs to your CI system on every push and pull request, so the pipeline starts a build without polling the repo.
• Chat and collaboration. Slack and Discord deliver message events and slash-command invocations to your bot's endpoint.
• Communications. Email and SMS providers fire delivered, bounced, opened so you can update contact status.
• E-commerce. Order platforms notify your inventory system when an order is placed or shipped.

When I reach for a webhook.

• The event is rare relative to how often I would otherwise poll.
• Latency matters.
• The consumer is a server I control with a stable URL.
• If I am reacting to things in another system, webhooks are the default.
• If I am observing my own internal services, I lean toward a message queue or event bus instead: inside my own walls I do not need HTTP callbacks, I have Kafka or SQS.

6Common pitfalls and gotchas#

These are the things that actually break in production.

• No idempotency. The classic bug. A retry delivers payment.succeeded twice and you fulfill the order twice. Fix: every event carries a unique ID (Stripe calls it evt_...). Store processed IDs in a dedup table or a Redis set with a TTL, and make the "mark processed" write atomic with the work itself. Check "have I seen this ID?" before doing anything.
• Processing inline instead of acking fast. The most common design miss I see in interviews. Verify, enqueue, return 2xx in under a second. Do the work in a worker.
• Skipping signature verification. Your URL is public. Anyone who learns it can POST fake payment.succeeded events and get free product. Providers sign each payload with a shared secret (HMAC-SHA256 over the raw body). Recompute and compare before trusting anything. Verify against the raw bytes, not a re-serialized object, because JSON key reordering changes the hash.
• Replay attacks. An attacker captures a valid signed request and sends it again. The signature alone does not stop this, because it is a genuine signature. Include a timestamp in the signed payload and reject anything older than, say, five minutes, combined with your idempotency check.
• No retry or dead-letter strategy on your side. The provider retries delivery, but once you accept and enqueue, you own reliability. If the worker fails, you need retries with backoff and a dead-letter queue, or events quietly vanish.
• Ordering assumptions. Webhooks are not ordered. You might get subscription.updated before subscription.created from retries and parallel delivery. Do not assume order. Use the event timestamp or refetch current state from the provider's API when order matters.
• Silent death. An endpoint returning 500 for an hour, or an expired cert, can stop deliveries with no alarm. Monitor delivery success and alert on a drop. Many providers offer a replay or event-list API: know whether yours does before you need it.

7How it shows up in interviews#

Webhooks usually appear as a piece of a larger design: "design a payment system," "design a CI/CD pipeline," "design a notification service." The interviewer is checking whether you understand event-driven communication and its failure modes.

The weak answer.

• "The provider calls my API when something happens and I update my database."
• Why it fails: no retries, no duplicates, no verification, no async split. It treats the webhook as request/response and assumes the happy path.

The strong answer.

• Register a signed webhook. On receipt, verify the HMAC-SHA256 against the raw body and reject timestamps older than five minutes.
• Check the event ID against a Redis dedup store, enqueue, and return 200 in a few milliseconds.
• A pool of workers processes it idempotently with exponential backoff retries and a dead-letter queue.
• Do not assume ordering: for state-sensitive events like subscription changes, refetch from the provider.
• Monitor delivery rate so a silent outage pages me.
• Why it wins: a concrete failure for every happy-path step, a specific mitigation for each, and a clear picture of where the time budget goes.

Next, try the System Design Question walkthrough on designing a payment processing system, where webhook reliability is the whole game.