CI/CD Pipelines

1The core idea#

CI/CD is two ideas glued together. Keep them separate in your head, because interviewers notice when you blur them.

CI, continuous integration: catch problems early.

• Every push triggers a machine to build the code, run the tests, and tell you within minutes whether it is broken.
• The model: instead of ten engineers each hoarding two weeks of changes and then fighting a brutal merge, everyone integrates small changes into the shared branch constantly.
• Small merges rarely hurt. Big ones always do, because conflicts and behavioral surprises compound with the size of the diff.

CD means one of two things.

• Continuous delivery: every change that passes CI is packaged and ready to ship, but a human clicks release.
• Continuous deployment: pass the gates and it goes to prod automatically, no human in the loop.

The difference is a real decision, not trivia.

• A payments company runs continuous delivery with a manual approval, because a bad release costs money and sometimes breaks a regulation.
• A content site runs full continuous deployment and ships fifty times a day, because the worst case is a cosmetic bug for a few minutes.
• Match the gate to the cost of being wrong.

2How it works#

A pipeline is a sequence of stages. Each one passes and hands off, or fails and stops everything.

Order stages by cost per unit of signal.

• Cheap, high-signal checks run first; expensive ones run last.
• No point spending eight minutes spinning up a test environment for code that will not compile.
• A 2 second lint that catches 30 percent of mistakes earns its place before a 5 minute integration suite.

A worked example, because this is where most answers go vague.

Say my team pushes 40 commits a day to one service. Each run has these stages:

The number that matters most is pre-merge feedback.

• Build plus unit plus integration: 90 + 120 + 300 = 510 seconds, about 8.5 minutes.
• That is what a developer stares at before they can merge, so it drives their behavior.
• The post-merge stages (registry push, staging, canary, rollout) add another 340 seconds, but those run after the developer has moved on. They cost patience, not focus.

Now the capacity math, the part most candidates skip and most interviewers love.

• With 4 CI runners, each pre-merge run occupies a runner for 8.5 minutes.
• One runner does 60 / 8.5 = ~7 runs/hour, so 4 runners do ~28 runs/hour.
• Over an 8-hour day that is ~224 runs of headroom against 40 commits plus reruns. Comfortable.
• Let integration tests creep from 300s to 900s and per-run time jumps to 18.5 minutes. Each runner now clears ~3.2 runs/hour, 4 runners give ~13/hour, and on a busy afternoon with flaky reruns you hit a queue and everyone waits.
• Queue time is invisible on the dashboard but very real to the engineer watching a spinner.

The fix is usually not "add runners."

• Split that 300s integration suite into 4 parallel shards and wall-clock drops to ~75s plus overhead.
• Pre-merge feedback becomes 90 + 120 + 90 = 300 seconds, 5 minutes, and each runner frees up far faster.
• Parallelism buys you speed and throughput from the same hardware. Adding runners only buys throughput, and it costs money every month.
• Sharding buys both for the price of a one-time test-suite change. That reasoning separates a strong answer from a shopping list of tools.

A principle I would tattoo on a junior: build once, promote the same artifact through every environment.

• Tag the image with the commit SHA so any running instance traces back to a line of code.
• The moment you rebuild separately for staging and prod, "works in staging" means nothing, because the thing you tested is not the thing you shipped.
• Different base image, different dependency resolution, different build timestamp: any of those can make the prod binary behave unlike the one that passed the gates.

One more distinction that makes interviews interesting: deploy is not release.

• Pushing an artifact to prod is not the same as exposing it to users.
• Good pipelines separate deploy (put the version on servers) from release (send traffic to it).
• That separation is what enables canaries and blue-green, and it is the single idea that unlocks most zero-downtime questions.

3Trade-offs#

The meat of a good answer lives here. The main decision is how you release a new version to live traffic, and every option trades speed against blast radius.

Things to understand, not memorize:

Rolling updates: the Kubernetes default, for good reason.

• Cheap, no extra fleet, no downtime.
• The catch: during the rollout you run two versions at once, so code and schema both have to tolerate that.
• If v2 renames a column while v1 is still live, v1 throws errors.
• This is why schema changes must be backward compatible and staged. The twenty-minute outage up top was exactly this, and no deploy strategy saves you from it. Only a staged migration does.

Blue-green: instant rollback, at a price.

• Rollback is pointing traffic back at the old environment, so it is near-instant.
• The price is running two full copies during the switch: a 100-node fleet becomes 200 nodes for the duration of the cutover.
• It does not fix migrations, since both colors usually share one database.
• Blue-green protects you from a bad binary, not a bad schema.

Canary: my pick for anything high-risk.

• Route a sliver, say 5 percent, watch error rate and p99 latency for ten minutes, widen only if metrics stay clean.
• Concrete trigger: baseline 5xx is 0.1 percent, so if the canary pushes 5xx past 0.5 percent I auto-abort and drop it.
• That burns 5 percent of users for a few minutes instead of all of them.
• The cost is complexity: you need real metrics and a traffic-splitting layer at the load balancer or mesh. A canary with no automated check is just a slow manual deploy wearing a fancy name.

The other big trade-off: test depth versus pipeline speed.

• The answer is not "test everything on every push." It is a pyramid.
• Thousands of fast unit tests per commit, fewer integration tests, a handful of expensive end-to-end and load tests that run on merge to main or nightly, off the critical path.
• Push the slow, low-frequency-bug tests off the path a developer waits on, and keep the fast, high-frequency-bug tests right in front of them.

4Where it is used#

Nearly every serious team runs some form of this; the shape changes with the stakes.

Tools you can name (to show breadth):

• GitHub Actions and GitLab CI for orchestration.
• Jenkins in older or on-prem shops.
• Argo CD and Flux for GitOps on Kubernetes.
• Spinnaker for multi-cloud at scale.
• Name them, but spend your words on strategy. The interviewer cares which one and why, not that you have heard of five.

A large web company, stateless API.

• Rolling or canary through Kubernetes, with rollback wired to error-rate alarms, shipping hundreds of times a day.
• That volume is only possible because the pipeline, not a human, does the work.
• A human in the loop caps you at maybe a few deploys a day; automation is what gets you to hundreds.

A bank or healthcare system.

• Same mechanics, but continuous delivery: mandatory approval gates, audit logs of who approved what, change-freeze windows around quarter close.
• Regulation makes the human signature non-negotiable.
• The smart move is to keep the pipeline fully automated right up to the approval step, then record who clicked.

GitOps, the pattern worth knowing.

• The desired state of prod lives in a Git repo as declarative config, and a controller like Argo CD reconciles the cluster to match.
• A deploy becomes a merge, a rollback becomes a git revert, and your audit log is your git history.
• The quiet benefit is drift detection: if someone hand-edits prod, the controller notices the cluster no longer matches the repo and pulls it back.

Mobile, a special case worth raising.

• You cannot deploy to a phone instantly; app-store review sits in the way and can take a day or more.
• So CD shifts toward feature flags: ship the code dark inside an already-approved app and flip it on server-side later.
• That decoupling of deploy from release is everywhere once you look, and raising it unprompted signals you have actually shipped to more than one kind of target.

5Common pitfalls and gotchas#

Rebuilding per environment.

• Test artifact A, ship artifact B, debug a ghost.
• Build once, promote the same image. If you remember one line from this guide, make it this one.

Migrations coupled to code deploys.

• If a migration and the code that needs it ship together, you cannot roll back code without the schema being wrong.
• Expand then contract: add the new column, deploy code that writes both, backfill, deploy code that reads the new one, then drop the old column in a later release.
• Four or five small safe steps beat one big risky one.

Slow pipelines.

• Once CI crosses ~10 minutes, people batch changes to dodge the wait, which brings back the big-merge problem CI existed to kill.
• Parallelize and cache dependencies aggressively. Pipeline speed is a behavioral lever, not just a convenience.

Flaky tests.

• A test that fails 2 percent of the time, run 40 times a day, fails about once a day for no reason.
• People hit retry reflexively, and a real failure gets retried away and ships.
• Quarantine flaky tests fast; they rot trust in the whole suite, and a suite nobody trusts gets ignored.

No automated rollback.

• A canary with no metric check and no auto-revert is theater.
• If a human must notice the dashboard, acknowledge the page, and revert by hand, your recovery time is the length of someone's lunch.
• Wire the abort to the same metric that defines healthy.

Secrets in the pipeline.

• Runners hold deploy credentials, which makes them a target.
• Secrets live in a managed store, injected at runtime, scoped tight, never printed in logs.
• Never bake API keys into an image, because an image is forever and so is a leaked key inside it.

6How it shows up in interviews#

CI/CD rarely gets its own hour. It shows up as a follow-up: "you have designed this service, now how do you deploy a new version without downtime?" or "a deploy just broke prod, walk me through what happens." That question separates people fast, because you cannot fake having been on call.

What interviewers probe:

• Can you release with zero downtime? (They want rolling, blue-green, or canary, and why you would pick one.)
• How do you roll back? (Fast, automated, tied to metrics.)
• How do you handle a schema change during a deploy? (Expand and contract.)
• How do you know the deploy is healthy? (Error rate, p99 latency, saturation, not just a 200 on a health check. A health check tells you the process is up, not that it is serving correct answers.)

Weak answer.

• "I would set up a CI/CD pipeline with Jenkins that runs the tests and auto-deploys to prod, and if something breaks we roll back."
• That is naming a tool and a definition. No strategy, no numbers, and "roll back" hides the hardest problem, which is what happens to the database.

Strong answer.

• "Every merge builds one immutable image tagged with the commit SHA, runs unit then integration tests in about 5 minutes with integration sharded 4 ways."
• "The same image promotes through staging to a canary at 5 percent of prod traffic. I watch 5xx and p99 for ten minutes; baseline 5xx is 0.1 percent, so past 0.5 percent I auto-abort and drop it, otherwise I ramp to 100 percent."
• "The one thing I will not auto-deploy is a destructive migration. I split those into expand-and-contract across two releases, so a code rollback never leaves me with missing data."

Why the second answer wins: it names the risk, picks a strategy with a reason, gives a concrete rollback trigger tied to a real number, and handles the database unprompted. You do not need my exact numbers, you need to show you know which numbers exist and why they move the decision.

Match deploy rigor to blast radius: a human gate for payments, auto-deploy and no canary for an internal dashboard.

Next, try the System Design Question walkthrough on designing a zero-downtime deployment for a high-traffic API to put these trade-offs to work.