Service Discovery

1The core idea#

The problem in one sentence: in a dynamic system, the thing you want to call does not live at a fixed address.

The old world was static, so a hardcoded list never went stale.

• Three app servers with known IPs, written into a config file.
• Maybe a load balancer with a stable DNS name in front.
• Boxes lived for months. Nothing moved.

The new world moves several times a minute.

• payments runs as 12 pods right now, maybe 30 in ten minutes when a sale starts and the autoscaler reacts to CPU crossing 70 percent.
• Each pod gets a fresh IP when it boots and loses it when it dies.
• A rolling deploy replaces every instance over a few minutes.
• So "where is payments?" has no permanent answer. At peak it changes several times a minute as pods come and go.

Service discovery answers that question in real time. The mental model: a phone book that updates itself.

• Services register when they come up: "I am payments, I live at 10.2.4.19 port 8080, and I am healthy."
• Callers look up the name payments and get back a current list of healthy addresses.
• When a pod dies, its entry disappears, so callers stop sending it traffic.

Three moving parts make this work.

• The service registry. A database of what is running where. This is the source of truth.
• Registration. How instances get into the registry and get removed when they die.
• Resolution. How a caller turns the name payments into a real address it can connect to.

Everything else is a variation on these three.

2How it works#

Trace a single request end to end so the pieces click.

Registration and health, on the pod side.

• A new payments pod boots and tells the registry "I exist, here is my address."
• The registry stores it and starts checking health every few seconds, usually by hitting a GET health endpoint.
• Answer 200 and the pod stays in the pool. Miss two or three checks and the registry evicts it.

Resolution, on the caller side.

• orders wants to charge a card, so it asks "give me a healthy payments instance."
• It gets an address, picks one, and sends the request.

Now the fork in the road: who does the lookup and the picking? This is the single biggest design decision in service discovery, and interviewers will push on it.

Client-side discovery. The caller is smart.

• orders queries the registry itself, gets the full list of healthy payments instances, and runs its own load-balancing to pick one.
• Netflix's Eureka plus the old Ribbon library is the classic example.

• The win: no extra network hop. The caller can also prefer same-zone instances to cut latency and cross-zone transfer cost (on AWS that cross-AZ transfer runs about 1 to 2 cents per GB, which adds up at millions of calls).
• The cost: every service in every language now needs discovery logic baked in. A Java, Go, Python, and Node fleet means you maintain that client four times and fix the same bug four times.

Server-side discovery. The caller is dumb.

• orders sends to a stable address, and something in the middle (a load balancer or proxy) does the lookup and routing.
• Think an AWS load balancer in front of a target group, or a Kubernetes Service where kube-proxy handles it.

• The win: orders stays simple and language-agnostic. It hits one DNS name. All the discovery smarts live in one place you can upgrade independently.
• The cost: an extra hop, and that proxy is now a thing you have to keep highly available.

A third pattern worth knowing: the service mesh (Istio, Linkerd).

• A sidecar proxy, usually Envoy, runs next to every service instance.
• The app talks to localhost, and the sidecar handles discovery, load balancing, retries, and mTLS.
• It is basically client-side discovery, but the smarts live in a sidecar instead of your code, so you get the language-agnostic benefit too.
• You pay in operational weight and a little latency per hop (Envoy typically adds roughly 1 to 3 ms per call, which matters if you chain ten services in one request path).

3A worked example with real numbers#

Abstract talk about staleness never lands until you put numbers on it, so size a concrete setup and show where the failure window actually is.

The setup.

• payments runs 20 pods behind client-side discovery.
• Each pod re-registers and renews its lease every 30 seconds.
• The registry evicts a pod after 3 missed renewals, so worst-case eviction for a silently dead pod is 90 seconds (3 times 30).
• Callers refresh their cached instance list every 30 seconds too.
• Traffic into orders is 2,000 requests per second, spread evenly, so each pod takes about 100 requests per second.

Walk the window after a hard crash (kernel panic, no graceful deregister).

• From crash to eviction: the registry still lists the dead pod for up to 90 seconds, waiting out 3 missed renewals.
• On top of that, callers refresh only every 30 seconds, so a caller holds a stale list for up to another 30 seconds after eviction.
• Worst case, a caller routes to the dead pod for roughly 120 seconds.

Lever one: retries.

• If orders retries once against a different instance on a connection failure, almost all 12,000 requests succeed on attempt two. The user never sees it.
• So the real cost is not 12,000 failures, it is 12,000 extra retries plus a little latency.
• This only holds with a fast connect timeout (say 200 ms), so the client gives up on the dead pod quickly instead of hanging 30 seconds on a TCP timeout.

Lever two: shrink the window.

• Drop renewals to every 10 seconds and evict after 2 misses, and worst-case eviction falls from 90 seconds to 20 seconds.
• The price is 3 times the registration chatter: 20 pods renewing every 10 seconds is 2 renewals per second instead of about 0.67.
• At 20 pods that is nothing. At 5,000 pods the chatter becomes real registry load, which is exactly the trade you will be asked to reason about.

Lever three: active health checks can beat the lease timeout entirely.

• If the registry probes GET health every 5 seconds and evicts after 2 failures, it notices a dead pod in about 10 seconds instead of 90, independent of the renewal interval.
• This is why serious setups combine both: leases catch "pod vanished and stopped renewing," active probes catch "pod is up but broken" faster.

Discovery data is always a little wrong. You can compute exactly how wrong, then design retries and timeouts to survive that window.

4Trade-offs#

The honest answer to "which approach is best" is "depends on your fleet, your team, and how much operational complexity you can stomach." Here is how to reason about it.

A few trade-offs deserve more than a table cell.

Consistency versus availability.

• Your registry is a distributed database, so CAP applies.
• Consul, etcd, and ZooKeeper lean CP: a consensus protocol (Raft or similar) means they refuse to serve data rather than serve stale data during a partition.
• Eureka famously leans AP: it keeps serving whatever it last knew, even if the list is a little stale.
• For discovery I usually want AP. If the registry is briefly unsure who is healthy, I would rather callers try a possibly-dead instance and fail fast then retry than have the registry go dark and take down every service at once.
• A stale address costs one retry (12,000 cheap retries in the example above). An unavailable registry costs you the whole system.

Push versus pull.

• How fast does the caller learn an instance died?
• Poll every 30 seconds and you can send traffic to a dead box for up to 30 seconds after eviction, as the worked example showed.
• Push-based updates (the registry streams changes) are faster but more complex.
• Most systems use a cache plus a background refresh, so you accept a window of staleness. This is why retries and fast connect timeouts matter so much.

DNS as the lowest common denominator.

• You can do discovery with plain DNS. Kubernetes gives every service a DNS name.
• It is dead simple and works in every language with zero libraries.
• The catch is caching and TTLs: clients and resolvers cache aggressively, sometimes ignoring your TTL entirely, so changes can take a while to propagate.
• Classic DNS also carries no health or weighting. A fine default, as long as you know its limits.

5Where it is used#

You have almost certainly used this without noticing.

• Kubernetes. The one most people meet first. Its Service object plus internal DNS is server-side discovery: you call payments.default.svc.cluster.local and kube-proxy routes you to a live pod. The registry is etcd, updated by the control plane as pods come and go. For most teams on Kubernetes, discovery is "already solved," which is a big reason people adopt it.
• AWS. A load balancer in front of a target group (server-side), plus Cloud Map for name-based discovery. Health checks and deregistration are managed for you.
• Netflix. Built the pattern everyone references: Eureka as the AP registry, with client-side load balancing. It fit their scale and their mostly-JVM fleet, where maintaining one discovery client in one language was cheap.
• HashiCorp Consul. The popular standalone choice when you are not all-in on one cloud or orchestrator. Discovery plus health checking plus a key-value store, and it plays nicely across VMs and containers.

When would I reach for explicit service discovery at all? The moment instances become ephemeral.

• Three long-lived VMs? A load balancer with a static config is fine, and adding a registry is overengineering.
• The need shows up when you autoscale, when you deploy by replacing instances, or when you have enough services that maintaining address lists by hand becomes a source of outages.
• That is the trigger.

6Common pitfalls and gotchas#

Each of these has caused a real incident somewhere.

Shallow health checks.

• Your health endpoint returns 200 because the web server is up, but the service cannot reach its database.
• The registry thinks the instance is healthy and keeps sending it traffic that fails.
• Health checks need to reflect whether the instance can actually do its job.
• But be careful: if the check pings the database and the database hiccups, every instance fails at once and you drop the entire service. Separate "am I alive" (liveness) from "can I take traffic right now" (readiness).

Thundering herd on registry failure.

• Registry goes down, every client panics, retries, and re-queries at once, and you hammer the registry into staying down.
• Clients must cache the last known good list and keep using it when the registry is unreachable.
• A client that cannot function without a live registry has made the registry a hard dependency for the whole fleet. That is the opposite of what you wanted.

Slow deregistration.

• An instance crashes hard and never tells the registry it is leaving, so you depend entirely on leases and health checks to notice.
• A 30-second renewal with 3 allowed misses is a 90-second eviction. A 10-second check with 3 failures is 30 seconds of sending traffic into a black hole.
• Tune the interval, and make callers fail fast and retry another instance rather than hanging.

Trusting DNS TTL.

• You set a 5-second TTL expecting quick failover, but a JVM or a resolver in the middle caches the record for much longer.
• Traffic keeps going to a dead address.
• If you rely on DNS, test what your actual clients cache. Do not assume they respect your TTL.

Forgetting the retry budget.

• Discovery data is eventually consistent, so some requests will hit dead instances. That is fine if you retry.
• But blind retries during an incident amplify load and can turn a small problem into a cascading failure: if every one of 2,000 requests per second retries 3 times, you just tripled traffic at the worst possible moment.
• Pair retries with timeouts, a cap on attempts, and ideally a circuit breaker.

7How it shows up in interviews#

When I ask about service discovery, I am checking whether you understand that addresses are not stable in a modern system, and whether you know the real trade-offs rather than one memorized pattern.

Interviewer: How does service A find service B?

Candidate: Not by IP, since that box gets replaced on every deploy. A registers against a registry on startup, or a managed layer like a Kubernetes Service does it. A resolves the name B to a current list of healthy addresses.

Interviewer: Client-side or server-side, and why?

Candidate: Depends on the fleet. Polyglot fleet, so I lean server-side or a mesh to avoid maintaining a discovery client in four languages and fixing the same bug four times.

Interviewer: What happens when the registry goes down?

Candidate: Clients keep serving from their cached last-known-good list and degrade gracefully. The registry is AP, so a blip costs a retry, not the whole fleet.

What I am really listening for across those answers.

• Reach for a registry or managed discovery the moment I mention a box getting replaced in a deploy.
• Tie the client-vs-server call to the actual fleet, not to a favorite tool.
• Caching and graceful degradation when the registry dies, not "everything breaks." This one separates the strong candidates fast.
• Health checks and leases, the liveness-versus-readiness distinction, and a real number for the staleness window.

The answer that lands: "Instances register and renew a lease, say every 30 seconds with eviction after 3 misses, health-checked on top. Callers resolve the name to a healthy instance and cache the list, refreshing every 30 seconds. I lean server-side or a mesh for a polyglot fleet. The registry favors availability over consistency so a blip does not take everyone down, and I pair it with a 200 ms connect timeout and one retry to a different instance, because worst case a caller routes to a just-dead pod for up to about two minutes. At 100 requests per second per pod that is a bounded number of retries, not an outage."

Next, try the System Design Question walkthrough on designing a URL shortener or a ride-sharing backend, where service discovery quietly underpins how every internal call finds its target.