Canary Deployment Checklist: Decide When to Expand or Recover

Canary Deployment Checklist: Decide When to Expand or Recover

By The Incident Challenge Team | 2026-11-08T16:21:16Z

A canary deployment checklist should establish what the first rollout tests, which signals compare the candidate with the working version, what stops expansion, and whether the recovery path is still compatible with the system's state.

Use the checklist below as a review aid for your own release. The worked numbers are fictional examples, not recommended production thresholds.

Google's canarying releases chapter discusses evaluating a limited release with meaningful signals and a representative population. A small rollout provides useful evidence when the observations cover the behavior the change can affect.

Before the rollout: write the decision you need to make

State the change, its expected effect, and the observation that would contradict the hypothesis. For example: a new request-routing setting should preserve successful checkout while changing which backend handles a subset of requests.

Then record the release boundaries:

Check Evidence to record
Artifact identity Executable and configuration revisions under review
Candidate population Who or what receives the new behavior
Comparison group A working population serving comparable operations
Customer outcome The operation that should continue to succeed
Stop condition The signal, scope, and owner that halt expansion
Recovery route The action and compatible state needed to restore service

Assign a decision owner. If nobody can say who pauses the rollout, an alert alone does not complete the process.

Check coverage against the actual change

Create a matrix of the paths the candidate must exercise. A fictional checkout release might need coverage of warm and cold caches, newly started instances, existing sessions, and requests that create asynchronous work.

For a configuration migration, inspect whether both old and new input formats are represented. For a routing change, inspect the relevant network topology. For a background worker, include enough time to observe useful completion and retries.

Record missing coverage explicitly. If a required path receives no observations, the review has an unanswered question. Increasing the number of healthy requests on another path does not answer it.

Cloudflare's June 2022 outage is a useful case for discussing differences between rollout groups.

Compare candidate and control without losing the denominator

Consider this invented observation window:

Population Requests Failures Failure rate
Control 9,500 19 0.2%
Candidate 500 15 3.0%
Combined 10,000 34 0.34%

The combined rate hides how unevenly failures are distributed. The candidate's observed failure rate is fifteen times the control's in this example.

That ratio alone does not establish the cause. Check whether the populations serve comparable operations, whether the candidate saw an unusual traffic mix, and whether the measurement window is appropriate. Use the result to investigate the difference and apply the service's agreed stop conditions.

For client releases, Google Cloud's release-canary discussion describes identifying the client version in requests so its behavior can be measured separately.

A comparison also needs an absolute check. If both populations depend on a failing shared component, similar results may coexist with an unacceptable customer experience.

Include the work that outlives the request

For a release that creates jobs or writes records, inspect the resulting work. A successful HTTP response may be followed by a failed worker, a duplicate task, or an incompatible record.

Add an observation that follows an operation through its relevant stages. A synthetic upload, for example, might need an accepted request, a persisted job, a completed transformation, and a retrievable result.

Choose a window that can observe those outcomes. Document what remains untested if a delayed path cannot finish during the initial stage. Avoid promoting the candidate simply because the dashboard has no failures yet.

Confirm the rollback target before expanding

Write the state the earlier version expects: schema, configuration, message format, and downstream contracts. Then compare that expectation with what the candidate can already have changed.

The database migration rollback plan provides a worked example where the correct recovery target depends on the migration phase. Use a prepared incident runbook for the release procedure itself.

Copy the promotion record

Candidate revision:
Comparison revision:
Change and expected behavior:
Observed population, volume, and time window:
Required paths covered:
Coverage still missing:
Customer outcome and useful-work results:
Candidate/control differences investigated:
Absolute service-health check:
Rollback target and compatibility evidence:
Decision: expand / hold / recover
Decision owner, timestamp, and next checkpoint:

The record should explain why the evidence supports the next stage. A deployment command's exit status belongs in that record, alongside the behavior observed after it ran.

Rehearse a hold decision

In a tabletop, provide a healthy aggregate graph and the fictional population table above. Let the responder request the missing traffic breakdown and delayed-work observations. Ask them to state what would permit promotion and what would make them recover instead.

Score the quality of the decision and the verification plan. Completion speed is useful context, but it cannot show whether the responder recognized missing coverage.

Use the engineering game-day plan to run the discussion. For a prepared investigation with a runtime configuration change and a deploy, try a past Incident Challenge.