Load Balancing

1The mental model (what this really is)#

A load balancer is a traffic cop between clients and a pool of identical servers. Requests come in, it picks which server handles each one, and it hides the fact that there are many servers behind a single address. That is the surface. Here is the model that actually matters.

A load balancer turns a set of fragile individual machines into one reliable virtual machine.

• Any single server can die, be deploying, or be overloaded, and the system keeps serving.
• The LB is what makes "just add more servers" actually work.

Two jobs, always:

• Distribute load so no single server melts while others sit idle.
• Route around failure so a dead or sick server stops getting traffic.

Here is the picture:

One more framing: load balancers live at different layers, and the layer changes what the box can do.

• Layer 4 (transport): routes by IP and port. Fast, cheap, dumb. It sees TCP/UDP, not HTTP, so it cannot read a URL path or a cookie. A single L4 box forwards packets at near line rate, think millions of connections per second, because it barely inspects them.
• Layer 7 (application): reads the actual HTTP request. It can route /api to one pool and /images to another, terminate TLS, rewrite headers, and retry failed requests. The cost is real: each request burns CPU to parse headers and run the TLS handshake, so an L7 box handles maybe tens of thousands of requests per second per core instead of millions of packets.

2How to recognize it (the signals in a problem)#

Bring up a load balancer the moment the design has more than one server doing the same job. Specific triggers:

• "Millions of users" or "high availability." You cannot get HA from one box. A single server at four nines is still one fuse that takes everyone down when it blows.
• You drew one web server and the interviewer asks "what happens when this dies?" That is your cue.
• Zero-downtime deploys. Rolling deploys need something to drain traffic from the box you are restarting.
• Spiky traffic. Flash sales, viral moments, Monday 9am logins.
• You said "horizontally scale the service." Scaling out is meaningless unless something spreads traffic across the new instances.

The tell that you get it:

• You place the LB and immediately say what it health-checks and how it fails over.
• Not just "put a load balancer here."

3The template (how it works, with pseudocode)#

Every load balancer is the same loop: pick a healthy server, send the request, watch for trouble.

def route(request, pool):
    healthy = [s for s in pool if s.is_healthy]
    if not healthy:
        return error(503)   # fail loud, don't send to dead boxes

    server = ALGORITHM.pick(healthy, request)
    try:
        return server.handle(request, timeout=T)
    except (Timeout, ConnError):
        server.mark_suspect()
        # L7 can safely retry idempotent requests on another server
        if request.is_idempotent:
            return route(request, [s for s in healthy if s != server])
        raise

The ALGORITHM.pick is where the famous choices live:

# Round robin: simple, assumes every request costs the same
def round_robin(healthy, _):
    i = next_counter() % len(healthy)
    return healthy[i]

# Least connections: better when request durations vary a lot
def least_connections(healthy, _):
    return min(healthy, key=lambda s: s.active_connections)

# Weighted: bigger boxes get more traffic
def weighted(healthy, _):
    return random.choices(healthy, weights=[s.capacity for s in healthy])[0]

# Consistent hashing: same key -> same server (sticky, cache-friendly)
def consistent_hash(healthy, request):
    return hash_ring.lookup(request.session_key)

And the health check that powers everything:

def health_loop(pool):
    while True:
        for s in pool:
            ok = probe(s, path="/healthz", timeout=2s)
            s.consecutive_fails = 0 if ok else s.consecutive_fails + 1
            s.is_healthy = s.consecutive_fails < 3   # don't flap on one blip
        sleep(5s)

That consecutive_fails < 3 detail is what makes you sound experienced.

• A single failed probe should not eject a server, or one slow GC pause takes a box out and triggers a cascade.
• With a 5 second interval and a 3-failure threshold, a truly dead box leaves the pool in about 15 seconds.
• That is the right trade between fast ejection and not overreacting to one hiccup.

4A worked example (walk through one problem step by step)#

Prompt: Design the backend for an online ticketing site. Big concerts sell out in minutes.

Here is how I would reason out loud, with numbers, because numbers turn hand-waving into a design.

First, size the spike.

• Steady state is 2,000 requests per second.
• A show goes on sale at noon and we get 50x for ten minutes, so 100,000 requests per second.
• One web server comfortably handles 1,000 req/s before latency climbs.
• Steady state needs 2 servers, so keep maybe 4 for headroom. The sale needs 100,000 / 1,000 = 100 servers.
• That gap, 4 to 100, is the whole problem in one line: I need both fast scale-out and even distribution. One web server is a non-starter.

Pick the layer: L7, not L4.

• I want to route /api/checkout differently from /browse.
• I want TLS terminated at the edge.
• I want the LB to retry a failed idempotent GET on another box.
• That intelligence is worth the extra CPU.

Pick the algorithm: least connections, not round robin.

• During a sale, some requests are cheap (a browse that returns in 20ms) and some are slow (a checkout that hits the DB and payment and takes 400ms, 20x longer).
• Round robin spreads request count evenly, so it would happily pile three slow 400ms checkouts onto one unlucky server while its neighbor burns through cheap browses and idles.
• Least connections tracks live in-flight work, so it steers new requests toward the box with the lightest load.
• With a 20x spread in request cost, that difference is the gap between p99 staying flat and one server's queue blowing up.

Now the part most people skip: sessions.

• When a user is mid-checkout, do they need to hit the same server every time? No, and that is a design win.
• Keep the web tier stateless and push session state to Redis.
• Any server can handle any request, the LB stays simple, and losing one server does not log anyone out.
• Store session in server memory instead and you are forced into sticky sessions, which wreck even distribution the moment traffic spikes.

Health checks.

• Each server exposes /healthz that checks it can reach Redis and the DB.
• The LB probes every 5 seconds, ejects after 3 consecutive failures.
• During a deploy, mark a box "draining," let it finish in-flight requests, then pull it. No dropped checkouts.

One LB is a single point of failure.

• Run the load balancer itself in a pair (active plus standby) sharing a virtual IP, or use the cloud's managed LB which is already redundant across availability zones.
• Say this out loud: "I made the servers redundant, now let me make the thing in front of them redundant too."

Scaling the spike: pre-warm, do not react.

• Autoscaling watches CPU and request latency, but it takes a minute or two to boot new boxes.
• Booting 96 of them reactively while traffic is already at 100,000 req/s means the first wave of users eats errors.
• My spike is instant and the start time is known, so for a scheduled sale I pre-warm: scale to 100 boxes at 11:45, fifteen minutes before noon, not reactively at 12:00:01.
• This is the detail that separates someone who has run a real launch from someone who read about autoscaling.

That walk took the design from "a server" to "a system that survives a 50x spike from 2k to 100k req/s and a server death mid-sale." That is the interview win.

5Variations and gotchas#

• Sticky sessions (session affinity). The LB pins a client to one server via a cookie or source IP. Sometimes necessary (legacy apps with in-memory state), but it fights even distribution and makes one server's death painful for its pinned users. Default to stateless servers so you never need it.
• Consistent hashing. Route the same key to the same server so caches stay warm. Critical for distributed caches and sharded stores. The "consistent" part means adding or removing one server only reshuffles 1/N of keys, not all of them. With 10 servers, adding an 11th moves roughly 1/11 of keys, about 9 percent, instead of a plain hash % N which would remap nearly all of them. Expect this in cache and sharding questions.
• Global load balancing (GSLB / DNS / Anycast). Across regions you do not use a single LB. You use DNS or Anycast to send users to the nearest healthy region, then a regional LB spreads load inside it. Two layers.
• Thundering herd on health. If all servers fail health checks at once (shared DB down), the LB has nowhere to route and returns 503 for everyone. A failing dependency can look like "all servers unhealthy." Know the difference between a sick server and a sick dependency, because ejecting all 100 boxes when Redis blips just turns a dependency outage into a total outage.
• Retry storms. The LB retries a slow request, the client retries too, and now a struggling backend gets 3x the load and dies harder. Retries need budgets and backoff. One rule of thumb: cap retries at something like 10 percent of total requests so a bad moment cannot triple your traffic. Retrying is not free.
• Connection draining. Always mention it for deploys. Stop sending new requests, let existing ones finish within a timeout (commonly 30 to 300 seconds), then remove the box.

6Trade-offs (the stuff that earns the round)#

The meta-point to say out loud:

• A load balancer adds a hop, so it adds latency (often 1 to 5ms per request for an in-region L7 box) and is itself something that can fail.
• You accept that cost to buy availability and scale.
• Name the cost, do not pretend it is free. That honesty is what senior engineers do.

7Weak answer vs strong answer#

Same prompt, "put a load balancer in front of your web tier." Here is the contrast an interviewer actually scores.

Weak answer: "I'll add a load balancer so traffic gets spread across the servers. I'll use round robin because it's simple and balances evenly."

• Names the box and one algorithm, then stops.
• Ignores failure, ignores the LB being a SPOF.
• Picks round robin without checking whether request costs vary.
• Says nothing about state or deploys. An interviewer hears pattern-matching.

Strong answer: "I'll put an L7 load balancer in front of a stateless web pool. L7 so I can terminate TLS, route /api separately from static assets, and retry idempotent reads on another box. I'll use least connections rather than round robin because my checkout requests are roughly 20x slower than browses, and round robin would pile slow requests on one server. The LB probes /healthz every 5 seconds and ejects a box after 3 straight failures so one GC pause doesn't flap it. The LB itself is a SPOF, so it's an active-standby pair on a virtual IP, or a managed cloud LB spread across AZs. Session state lives in Redis so any box serves any request, which keeps distribution even and survives a server death. For deploys I drain connections before pulling a box. The costs I'm accepting: an extra hop of a few ms, and a Redis round trip per request for session lookup."

• Same thirty seconds of airtime, roughly.
• The difference: the strong answer names a reason, a number, and a cost for every choice.

Juniors place the box. Seniors defend it.

8Mistakes that cost people the round#

• Drawing the box and moving on. "Add a load balancer" with no algorithm, no health check, no failover. Reads as pattern-matching, not understanding.
• Forgetting the LB is a SPOF. You carefully made 10 redundant servers, then funneled them through one load balancer. Interviewers pounce on this.
• Defaulting to sticky sessions because "the user needs the same server." Usually wrong. Push state out and keep servers stateless.
• Confusing L4 and L7. Claiming an L4 balancer can route by URL path. It cannot. It never sees the URL.
• Ignoring the failure path. Only talking about distribution, never about what happens when a server goes sick mid-request.
• Retries without limits. Saying "the LB just retries" without mentioning backoff or budgets, which is how you turn a small problem into an outage.

9Your practice path#

Work these in order. Do each one out loud, start to finish, placing the LB and defending every choice.

1. Design a URL shortener. Warm-up. Put an LB in front of a stateless read-heavy service. Practice saying "stateless, so round robin is fine here."
2. Design a rate limiter. Forces the sticky-vs-shared-state conversation, because counters must be shared or the LB will split a user's requests across boxes.
3. Design an e-commerce checkout / ticketing site. The spike plus sessions plus draining story from section 4. This is the full exercise.
4. Design a distributed cache (like a mini Redis cluster). Where consistent hashing earns its keep.
5. Design a globally available service (like a CDN or multi-region API). Now layer GSLB / DNS / Anycast on top of regional LBs. The two-tier story.

If you can do all five cleanly, you are ready for the load balancing portion of any L4/L5 interview.

10Where this shows up#

• Load balancing appears in essentially every system design round at Amazon, Google, Meta, Microsoft, Netflix, Uber, and basically any company that serves traffic at scale.
• It is rarely the whole question. It is the foundation layer the interviewer expects you to lay down early and correctly so you can build the interesting parts on top.
• The L5/senior bar is specifically the trade-off talk: L4 vs L7, stateless vs sticky, how you fail over, how you handle a spike.

Want to see which companies lean hardest on this and how they phrase it? Check the company guides to spot where load balancing shows up round by round.