Deployment Strategies: Blue-Green and Canary

1The core idea#

Start with the mental model. You have a version serving real users, and you want to replace it. The scary moment is the switchover, because a broken new version broadcasts that breakage.

The shared question. Blue-green and canary are two answers to one thing: how do I cut over without betting the whole business on the new code being correct?

Blue-green keeps two full environments.

• Blue is the current live version taking all traffic.
• Green is the new version, fully deployed and warmed up, taking zero real traffic.
• You test green in isolation, then flip a switch (load balancer or DNS) and all traffic moves to green at once.
• Blue stays alive and idle. If green misbehaves, you flip back in seconds because blue never went anywhere.

Canary is gradual.

• Deploy the new version alongside the old, then send it a tiny slice of traffic first. Think 1%.
• Watch error rates, latency, and business metrics.
• If it looks healthy, bump to 5%, then 25%, 50%, 100%, pausing to observe at each step.
• The name is the coal-mine canary: a small thing that dies first so the miners get a warning.

2How it works#

Both strategies need the same foundation: a routing layer that controls which users hit which version. That is almost always a load balancer, a service mesh, or sometimes DNS.

Blue-green mechanics: two complete stacks, one clean cutover.

• Green runs health checks and maybe synthetic test traffic, but no real users.
• You run smoke tests against green directly.
• When they pass, you change one config value and the arrow flips: all traffic goes to green, blue becomes the idle standby.

Canary mechanics: a weighted split you adjust over time.

A real rollout steps up over time, and a tool drives it.

• Nobody sits clicking buttons. Argo Rollouts, Flagger, or Spinnaker automate the stepping and, more importantly, the automatic rollback.
• You define a threshold (say error rate under 0.5% and p99 latency under 300ms) and the system rolls back on its own if the canary breaches it, no human paged at 3am.

3A worked example: rolling out a payments API#

"1% canary" means nothing until you do the arithmetic, so make it concrete.

The setup.

• A payments API serving 20,000 requests per second, which is 1.2 million requests per minute.
• SLO: error rate under 0.5% and p99 latency under 300ms.
• You are shipping a change to the fraud-scoring path, and it has a bug that rejects 3% of legitimate charges.

Blue-green math.

• You run 200 instances in blue. Going blue-green means standing up 200 more in green, so you pay for 400 instances during the deploy, a clean 2x.
• Warm green for 10 minutes with mirrored traffic, run smoke tests, flip at the load balancer in about 2 seconds.
• The 3% bug now hits 100% of users instantly: at 20,000 rps that is 600 failed charges per second.
• On-call sees the alarm and flips back to blue, say 90 seconds from flip to rollback.
• Blast: roughly 54,000 wrongly rejected charges in that window.

Canary math.

• Put 1% on the canary: 200 rps, about 2 instances' worth, so infra during deploy is 202 instances, roughly 1.01x instead of 2x.
• The 3% bug now affects 6 failed charges per second, and only on the 1% slice.
• Automated analysis watches the canary's checkout-success-rate against stable, sees it drop from 99.4% to 96.4%, and auto-rolls-back after a 5-minute bake, before you ever reach 5%.
• Blast: roughly 1,800 wrongly rejected charges, fix shipped with no human touching it.

The signal is trustworthy fast.

• 1% of 1.2M requests per minute is still 12,000 requests per minute. That is a huge statistical sample.
• This is why high-traffic companies love canary: the slice is tiny as a fraction but enormous in absolute terms.

4Trade-offs#

This is where interviews actually live. Neither strategy is "better." They optimize for different things.

Blast radius versus rollout speed is the sharpest trade.

• Blue-green flips everyone at once, so a test miss hits 100% of users for the handful of seconds until you flip back. The instant flip-back is the saving grace.
• Canary never exposes more than your current step, so a bug that slips tests only hits 1% while you watch. The cost is that it is slow, and a human or a metrics pipeline has to actually be watching.

Cost is the other big one.

• Blue-green doubles your compute during a deploy: 400 instances instead of 200 in the worked example. On a large fleet that is real money.
• Canary adds only a few extra instances, 202 in the same example, so it is much cheaper on infra.

5Where it is used#

Blue-green shines for clean, auditable, reversible cutovers when you can afford the double infra.

• Major framework upgrades, a runtime bump, or a release where "all users on the same version" matters for consistency.
• Common in regulated environments where you want a known-good version sitting ready at all times.
• A comfortable fit when deploys are infrequent but high-stakes.

Canary is the default for high-traffic consumer services that deploy many times a day.

• At millions of requests, even a 1% slice is a statistically meaningful sample, so you get fast, trustworthy signal.
• 1% of a billion requests is still ten million requests, plenty to catch a regression.
• The natural match for microservices where you deploy one service at a time and want tight blast-radius control.

A pattern I like: use both.

• Blue-green for the infrastructure-level switch, with a canary-style gradual ramp inside the green environment before you send it everything.
• You get instant rollback plus graduated exposure.

Feature flags are a cousin worth naming.

• They operate at the application layer, not the deployment layer.
• You ship the code to everyone dark, then turn the feature on for 1% of users via config.
• This decouples "deploy the binary" from "release the feature," a genuinely powerful separation.
• Mature teams run all three: blue-green or rolling for the binary, canary for the rollout, feature flags for the feature.

6Common pitfalls and gotchas#

The database is the real problem.

• Both diagrams showed two app versions sharing one database, and that is the honest picture.
• New code often wants a schema change. If green adds a column and you flip, then flip back to blue, blue may not understand the new schema.
• Fix is expand-contract: deploy a schema both versions can read and write (expand), deploy the new code, then later remove the old columns (contract) once no version needs them.
• Never ship a breaking schema change in the same release as the code that depends on it. This one rule prevents a huge class of rollback disasters.

Rollback is not free if state changed.

• You can flip app servers back in seconds. You cannot un-send an email, un-charge a card, or un-write a row the new version already persisted in a new format.
• In the payments example, the 1,800 wrongly rejected charges do not un-reject themselves on rollback. Someone has to reconcile them.
• "Instant rollback" is only instant for stateless layers. Say so in an interview, it shows maturity.

Cold caches and cold JVMs.

• When you flip to green, its caches are empty and its connection pools are cold. Latency can spike for a few minutes even though nothing is "wrong."
• This is exactly why the worked example warmed green for 10 minutes with mirrored traffic before the flip.
• Warm the new environment first, or your clean deploy will look like an incident.

Canary sample bias.

• Route canary traffic to only one region or one user segment and your 1% is not representative.
• A canary that only sees North American daytime traffic tells you nothing about the batch job that runs in Asia at midnight.
• Make the slice random and representative.

Watching the wrong metrics.

• Error rate and latency are table stakes, but some bugs are silent: a checkout button that renders fine but submits the wrong price.
• A canary that only watches CPU and 500s will happily promote a revenue-destroying bug.
• In the payments example, the signal that caught the bug was checkout-success-rate dropping from 99.4% to 96.4%, not a spike in 500s. Watch business metrics too.

Forgetting to actually retire the old version.

• With blue-green, teams flip to green and leave blue running "just in case" for weeks, paying double.
• In the example that is 200 idle instances burning money.
• Set a clear window, verify, and tear down.

7How it shows up in interviews#

When I ask about deployment in a system design round, I am checking whether you think about failure, not just the happy path. A lot of candidates design a beautiful system and then hand-wave the release. That is the gap I am probing.

Weak answer:

"Blue-green has two environments and canary is gradual. I'd probably use blue-green so I can roll back fast."

• True on the definitions, but a textbook recital.
• Names no metric, ties the choice to nothing, and misses that a payments bug hits 100% of users at the flip.
• This tells me you memorized the terms but never ran a rollout that went wrong.

Strong answer:

"This is a payments API doing about 20,000 requests per second, so I'd canary, not blue-green. I'd start at 1%, which is still around 12,000 requests per minute, enough signal to trust. I'd gate promotion on error rate under 0.5%, p99 under 300ms, and critically on checkout-success-rate compared against the stable fleet, because a fraud-model regression shows up as rejected legit charges long before it shows up as 500s. Automatic rollback via Argo Rollouts if any threshold breaches during a 5-minute bake. I'd avoid blue-green here mainly because a bad flip hits 100% of charges at once, and the 2x infra, roughly 400 instances, buys me a fast rollout I don't actually need for a service that can ramp over 45 minutes. One caveat: rollback doesn't undo charges already rejected, so I'd pair this with reconciliation on the payments ledger."

• Picks a strategy and justifies it with the specific situation.
• Names real metrics and thresholds, does the blast-radius math out loud, and flags the stateful-rollback limit.
• That is staff-level reasoning.

Tie the deployment choice to the system's requirements, not to habit. The signal the interviewer wants is that you read the room.

For a worked example, see the System Design Question walkthrough on designing a CI/CD pipeline with safe, automated rollouts.