API Gateway

1The core idea#

A gateway is a single front door for a set of backend services. Clients talk to one endpoint, say api.yourcompany.com, and the gateway decides where each request goes.

The mental model: it is the front desk of a large office building.

• Visitors do not wander the hallways hunting for the right room. They check in at one desk, show ID, and get sent to floor 7.
• The desk logs who came in and turns away people who should not be there.
• The gateway plays that exact role for your API traffic.

It owns the cross-cutting concerns, the things true for every request.

• Authentication, rate limiting, TLS termination, request logging, routing.
• Without a gateway you copy that logic into every service, and six months later three of them handle auth slightly differently and you have a hole.
• I have debugged exactly that: two services accepted an expired token because each team wrote its own 15-line JWT check and one forgot the exp claim.

2How it works#

A request hits the gateway and runs a chain of steps before it reaches any service. The order is deliberate: reject bad traffic as cheaply as possible, so auth and rate limiting come before the expensive routing and transformation.

Why the order matters.

• Rejecting at step 2 costs you a signature check.
• Rejecting at step 5 means you already paid for routing and a downstream connection.

A typical path through the gateway:

1. TLS termination. Decrypt HTTPS once. Internal hops run plain HTTP or lighter mutual TLS, so each service skips the handshake cost.
2. Authentication and authorization. Validate the token or API key. A common pattern: verify a JWT signature, then pass the decoded user ID downstream in a header so services trust the gateway instead of re-checking.
3. Rate limiting and quotas. Drop the request if this client has blown past its budget. Usually a 429 with a Retry-After.
4. Routing. Match path or host to a backend. GET products goes to catalog, POST orders goes to the order service.
5. Request and response transformation. Rewrite headers, strip internal fields, maybe collapse several backend calls into one response.
6. Forward to the backend (through a load balancer), then shape the response on the way out.

The full lifecycle of one request is the version I sketch when someone says "show me the flow":

3A worked example: sizing the gateway tier#

Make it concrete. Say we front a shopping API.

The numbers to anchor on:

• 50,000 requests per second at peak, Black Friday style.
• Request plus response averages about 4 KB, so roughly 200 MB/s through the gateway.
• Each gateway box is 4 vCPU and comfortably handles 8,000 req/s with TLS termination and routing before CPU crosses 70 percent.

Sizing for load, then for failure:

• Raw math: 50,000 / 8,000 = 6.25, round up to 7 instances to carry the load.
• But 7 is the number where losing one box pushes the rest over the line.
• Spread across 3 availability zones to survive one going dark: lose a zone with 3 per zone (9 total) and the surviving 6 cover 48,000 req/s, uncomfortably close.
• I would rather run 4 per zone, 12 total, and sleep. The extra boxes cost maybe 150 dollars a month each. Cheap insurance against a 2am page.

The auth decision is the one that actually moves latency. Two options:

• Call the auth service per request. An extra network hop. Even at 1 ms in-zone, at 50,000 req/s the auth service must also do 50,000 validations per second, and its p99 now rides on top of yours. One slow GC pause there and every request in the system stalls.
• Verify the JWT locally at the gateway using the auth service's public key, cached and refreshed every few minutes. Signature verification is pure CPU, roughly 0.1 ms, no network call. At 50,000 req/s that is maybe 5 percent of a core per instance, and it removes an entire failure dependency from the hot path.

Rate limiting has the matching gotcha, and the fix has a cost.

• Shared counters go in Redis with atomic INCR plus EXPIRE. At 50,000 req/s that load is well within one node's reach.
• But Redis is now a dependency on the hot path. Cache the decision for a short window and fail open on a Redis outage rather than blocking all traffic.

What a strong answer sounds like. Not "add a gateway," but "9 to 12 instances across 3 zones, local JWT verify at 0.1 ms, Redis-backed limits with fail-open." Numbers signal you have run one.

4Trade-offs#

The honest pitch and the honest cost.

• Pitch: the gateway centralizes the annoying stuff and gives you one place to enforce policy.
• Cost: you just added a hop that every single request passes through. If it is slow, everything is slow. If it is down, everything is down.
• So treat it like critical infrastructure: several instances behind a load balancer, stateless so any instance serves any request, and disciplined about what logic lives there.

A few decisions you will actually face:

On that last row, aggregation, the math is real but so is the cost.

• A mobile app on a 150 ms round trip does not want 8 calls to render one screen. Serially that is 1.2 seconds of pure network time.
• A gateway or BFF fans out to 8 services in parallel inside the data center and returns one payload in roughly 150 ms plus the slowest downstream. Genuine win.
• But now the gateway handles partial failures, per-downstream timeouts, and merging. More code in a place where bugs hurt everyone. Reach for it when the latency win is real, not by default.

5Where it is used#

Pretty much every large API-driven company runs a gateway, even if they do not call it that.

• Netflix popularized the BFF pattern. TV, phone, and browser have different needs, so they front them with device-specific edge layers instead of one gateway serving everyone.
• Public API products like Stripe or Twilio put a gateway in front to enforce API keys, per-key rate limits, and usage metering for billing. The gateway is literally how they count what to charge you, so every request through it is also a billing event.
• Kubernetes shops often run Envoy or an Ingress controller as the gateway. Service mesh setups (Istio, Linkerd) push some concerns down to sidecars, but you still want an edge gateway for north-south traffic coming in from outside the cluster.

When I reach for one in a design:

• The moment external clients need to reach multiple services and I want consistent auth, rate limiting, and a stable public contract.
• A client hits api.site.com/v2/orders and never learns there are four services behind it, so I can refactor freely. That decoupling is worth a lot.

When I skip it:

• A single monolith with one public endpoint does not need one yet.
• Do not add infrastructure to solve a problem you do not have.

6Common pitfalls and gotchas#

Making it a single point of failure.

• One instance, no redundancy. Run at least two, behind a load balancer, across zones.
• As the worked example showed, I would rather run 12 than the bare minimum 7. Table stakes.

Stuffing business logic in.

• Covered above, and it is the most common failure I see. The gateway is plumbing.
• Keep decisions about what your product does inside your services.

Auth that trusts a header with no backstop.

• If the gateway strips and re-sets an X-User-Id header, make sure clients cannot spoof it and services cannot be reached directly from outside.
• Network policy should force all external traffic through the gateway.
• Otherwise someone curls your order service directly with a forged X-User-Id: 1 and acts as any user.

Timeout and retry storms.

• The gateway retries a slow service, which is already overloaded, which makes it slower, which triggers more retries.
• Use timeouts, circuit breakers, and capped retries with backoff. A gateway without a breaker turns a slow service into a dead one.
• Concretely: a 200 ms timeout, at most one retry, and a breaker that trips after say 50 percent errors in a 10-second window.

Per-instance rate limit counters.

• If limits live in local memory, a client allowed 1,000 req/s can do 1,000 times your instance count.
• Push counters to Redis, as the example walked through.

No per-route limits.

• One expensive endpoint (a big search query) gets the same treatment as a cheap health check.
• Set limits and timeouts per route, not globally. Search might get a 2-second timeout and 100 req/s, health checks get 50 ms and no real limit.

Versioning pain.

• Because the gateway is your public contract, a breaking change here breaks every client at once.
• Plan for /v1 and /v2 coexisting and a real deprecation path.

7How it shows up in interviews#

Interviewers probe the gateway to check whether you understand operational reality, not just boxes and arrows.

Treat the gateway as a real service with its own scaling and failure modes, not a magic box labeled "gateway."

Next, try the System Design Question walkthrough on designing a URL shortener, where the gateway, rate limiting, and caching decisions all show up together in one problem.