Observability: Logs, Metrics and Traces

1The core idea#

Observability is your ability to ask questions about what your system is doing from the outside, without shipping new code to answer each one.

• If answering "why did user 8831 fail checkout?" needs a new log line and a redeploy, you are not observable, you are debugging in production.
• The classic framing uses three signals, sometimes called the three pillars.
• They are most powerful when they link to each other, not when you have all three but disconnected.

Metrics: numbers measured over time.

• Request rate, error count, CPU percent, queue depth.
• Cheap because you aggregate as you go: a counter says "4,210 requests in the last 10 seconds" instead of keeping every request.
• A single counter costs a few bytes per scrape no matter how many requests it counted.
• Great for dashboards and alerts. Bad at telling you why a specific user got an error, because that user already vanished into the count.

Logs: discrete events with detail.

• "User 8831 failed checkout because the payment provider returned a 502 at 14:03:22."
• Rich, searchable, human readable.
• Expensive at scale because you keep every line, and a busy service writes millions per minute.
• One line is cheap. A billion lines a day is a budget problem.

Traces: one request as it travels across services.

• A checkout might touch the API gateway, cart, inventory, payments, and a database.
• A trace stitches those hops into one timeline, so you see payments ate 1,900 ms of a 2,100 ms request.
• This is the signal that saves you when slowness hides in one of 30 microservices.
• Without it, "the API is slow" means opening 30 dashboards and guessing.

2How it works#

Every signal follows the same shape: your code emits data, an agent or library collects it, something ships it to a backend, and you query it. The differences are in volume and how much you aggregate before storage.

The collection paths differ by signal.

• Metrics get pulled or pushed on a schedule: a collector scrapes each instance every 10 to 15 seconds, reads counters and gauges, writes them to a time-series database like Prometheus or a hosted equivalent.
• Logs get written to stdout or a file, picked up by a shipping agent, streamed to a log store that indexes them for search.
• Traces pass a shared trace ID through every call. Each service records a span (its own slice of the work) tagged with that ID, and a backend reassembles the spans into one tree.
• OpenTelemetry is the common wire format for all three now, worth naming because it lets you swap backends without re-instrumenting.

Here is the collection path for a single user request in a small service mesh.

The linkage interviewers love: the trace ID in your logs.

• The trace ID generated at the gateway should also land in your logs.
• When you log "payment failed," you attach trace_id=abc123.
• Now an alert on your error metric leads to the trace, the trace shows the slow span, the span clicks through to the exact log lines for that request.
• Without it, you are grepping blind across a 7-day log index hoping you picked the right time window.

One more mechanic that matters: sampling.

• You cannot afford to trace every request on a busy system.
• Trace a fraction (say 1 percent), or use tail-based sampling: keep all the slow or failed traces, drop the boring successful ones.
• Metrics stay complete because they are already aggregated.
• Logs you often sample or filter too. The next section shows why that is not a nice-to-have.

A worked example: the cost of logging at 10k requests per second#

Real numbers on why sampling is not optional. Say your service does 10,000 requests per second, each request writes 5 log lines across its hops, and each line is about 1 KB once you include timestamp, trace ID, and fields.

• Lines per second: 10,000 x 5 = 50,000 lines/sec
• Bytes per second: 50,000 x 1 KB = 50 MB/sec
• Per day: 50 MB/sec x 86,400 sec = ~4.3 TB/day
• At a common managed-logging price of $0.50 per GB ingested: 4,300 GB x $0.50 = ~$2,150/day, about $780,000/year, just to ingest. Storage and query cost sit on top.

That number ends the "just log everything" conversation. Now apply three levers.

1. Drop to INFO sampling at 10 percent, keep 100 percent of WARN and ERROR. If 2 percent of lines are warnings or errors, you keep 0.02 + 0.10 x 0.98 = ~0.118, roughly 12 percent of volume. 4.3 TB drops to ~0.5 TB/day, about $93k/year to ingest.
2. Tier retention. Hot searchable storage for 7 days, then cheap object storage for the next 83. You still keep 90 days total, but the expensive indexed tier shrinks by roughly 10x.
3. Trace at 1 percent. A 5-span trace at ~5 KB per request would be 10,000 x 5 KB = 50 MB/sec at full capture. Sample to 1 percent and it is 0.5 MB/sec, ~43 GB/day, trivial next to the logs.

Same system, from $780k/year of logging down to under $100k, and you still have every error plus a representative slice of everything else. That is the concrete answer that separates a staff candidate from someone reciting the three pillars.

3Trade-offs#

The real decision is almost never "should I have observability." It is "where do I spend my budget," because all three signals cost money, and at scale the cost is brutal. I have seen a logging bill larger than the compute bill it was watching, the kind of thing that gets a VP into your sprint planning.

Trade-off one: cardinality, the sharpest of the three.

• Cardinality is the number of unique combinations of label values on a metric.
• Tag a metric with user_id and 10 million users, and you just created 10 million time series.
• Each series carries its own memory and index overhead, so a time-series DB that happily holds 100,000 series falls over at 10 million: memory blows up, queries crawl, the collector itself drops data.
• Rule: metrics get low-cardinality labels (region, status code, endpoint name, a dozen values each); high-cardinality detail goes in logs and traces.
• Getting this wrong is how teams take down their own monitoring during the incident they most needed it.

Trade-off two: retention versus cost.

• Metrics compress well because they are regular numeric series, so 13 months is cheap enough to compare this Black Friday to last year's.
• Logs are the opposite: 30 days of raw logs for a high-traffic service can cost more than the service it watches.
• Common answer: hot storage for 7 days where search is fast, then cheap cold object storage for 90 days where retrieval is slow but you rarely need it.

Trade-off three: sampling versus completeness.

• Sample traces too aggressively and the one rare bug you are chasing got dropped, so you are back to adding log lines and redeploying.
• Do not sample and your bill and your trace backend both explode.
• Tail-based sampling is the grown-up answer: keep every failed or slow trace, drop most of the fast successful ones.
• Cost to name out loud: it buffers spans until the request finishes before deciding to keep them, which eats collector memory and adds complexity.

4Where it is used#

Every serious production system runs all three. Where you lean depends on the shape of the system.

Monolith with a database: barely needs traces.

• One process, one obvious path.
• Metrics plus good structured logs do almost everything.
• Spend effort on a few key metrics (latency percentiles, error rate, DB connection pool usage) and clean logs.
• A full tracing backend here is effort you could spend elsewhere.

Microservice architecture: where traces earn their keep.

• When a request fans out across a dozen services, "the API is slow" is useless without a trace to show which hop ate the time.
• Teams running large service meshes treat distributed tracing as non-negotiable for exactly this reason.

Batch and data pipelines: lean on metrics and logs.

• Records processed per minute, lag behind real time, failed-record counts.
• A stuck pipeline shows up as a flat throughput metric long before anyone reads a log, so a single "records processed > 0" alert catches most stalls.

Mobile and frontend: flip the usual balance.

• You cannot scrape a phone every 10 seconds.
• You lean on events and crash reports shipped opportunistically when the app has a network, which look more like logs than metrics.

The reach-for-it rule.

• The more moving parts and network hops between a user action and the result, the more you invest in tracing.
• The more a single node does heavy local work, the more metrics and logs on that node matter.

5Common pitfalls and gotchas#

Each of these is a quick way to lose the room, and naming them unprompted is a quick way to win it.

• Logging everything at INFO. A service logging 10 lines per request at a million requests a day is a budget fire and a needle-in-haystack problem at once. Log decisions and errors, not every step. Use levels honestly.
• High-cardinality metric labels. Never put a user ID, request ID, email, or full URL path with IDs in it as a metric label. That detail belongs in logs and traces, which are built for it. This is the single most common way teams break their own monitoring.
• Averaging latency. The average response time is a lie. If 95 percent of requests take 50 ms and 5 percent take 5 seconds, your average is about 300 ms, which looks fine while one in twenty users suffers. This is the exact trap that let my 3 AM incident hide. Alert on percentiles, p95 and p99, not the mean.
• Unstructured logs. log.info("something happened " + value) is unsearchable at scale. Emit structured logs (key-value or JSON) so you can filter by status=500 AND region=eu instead of guessing at a regex under time pressure.
• No trace ID in logs. The most common reason an incident takes two hours instead of ten minutes. If logs and traces do not share an ID, you cannot cross between them, and you lose the whole benefit of having both.
• Alerting on causes instead of symptoms. An alert on "CPU is high" pages you for something possibly harmless (a nightly batch job). Alert on what the user feels: error rate up, latency up, orders not completing. This is symptom-based alerting and SLOs, where you page on error-budget burn rather than raw infrastructure noise.
• Treating the monitoring stack as always-on. Your observability backend is itself a system that can fail or get overwhelmed, often during the exact incident you need it, because incidents spike the data volume. Ask who watches the watcher, and keep at least one dead-simple health signal outside the main stack.

6How it shows up in interviews#

Observability usually shows up as a follow-up, not the main prompt. You design a URL shortener or a payment system, and the interviewer asks, "Okay, it is live and users say it is slow. How do you find the problem?" This is a test of whether you have operated software, not just drawn it.

Weak answer.

"I would check the logs and add some monitoring. Maybe set up alerts and a dashboard."

• Vague in every direction.
• Does not say which signal answers which question.
• Reaches for logs first, the most expensive and slowest way to find where something is slow.
• "Add monitoring" says nothing about what to monitor or what it costs.
• The answer of someone who has read about observability but not lived it.

Strong answer.

"My p99 latency metric alerts first, because the mean would hide a slow tail. I open the dashboard, slice by region and endpoint to see if it is global or one hot path. Say it is checkout. I pull a sampled trace for a slow checkout and read the span timeline. If payments is eating 1,900 ms of a 2,100 ms request, I have my culprit service. Then I jump from that trace ID straight to the log lines for that request to read the actual error or slow query. If I had tagged user_id on the metric I would have taken down my own monitoring, so that detail lives in the trace and logs instead."

• Walks the signals in order: metric to trace to log.
• Names the cardinality trap before being asked.
• Shows the three tools are linked by a shared ID.

What interviewers probe.

• Cardinality. Say "I'd tag the metric with user ID" and expect a follow-up. Knowing why it is dangerous (10 million users becomes 10 million time series) is a quick senior signal.
• Cost. "You log every request at a million requests a day. What does that cost and what would you cut?" They want the actual math ($0.50/GB ingested adds up fast), then sampling, levels, retention tiers, structured logs. The worked example in section 2 is exactly this question.
• Alerting philosophy. Can you tell a symptom (users see errors) from a cause (a disk is 80 percent full)? Mention SLOs and error budgets if you can.
• The difference between the three. Many candidates blur logs and traces. Saying crisply "metrics tell me something is wrong, traces tell me where, logs tell me why" lands well.

The weak answer is always vague. The strong answer names which signal answers which question and what it costs. Say the trade-off before they ask for it.

Next, try the System Design Question walkthrough on designing a URL shortener, and practice bolting a real observability plan onto it before the interviewer asks.