What is canary deployment? Benefits, process, and best practices

A canary deployment limits the first blast radius of a release, but only a deliberate decision process makes it safer.

A canary deployment exposes a new software version to a small portion of production traffic while the stable version continues serving everyone else. If the canary behaves as expected, the team increases exposure in stages until the new version reaches all intended users. If it fails a predefined check, the team pauses or rolls back before the problem reaches the full population.

The name comes from the canaries once used as early warning signals in coal mines. In software, the early warning comes from real production telemetry: error rates, latency, resource consumption, customer behavior, support signals, and any guardrail specific to the change.

That controlled exposure is the main benefit. It is not a guarantee that a release is safe. A tiny, unrepresentative cohort can miss a serious problem; weak observability can hide it; and an irreversible database change can make rollback impossible. A useful canary therefore combines traffic control, representative selection, explicit health criteria, and a rehearsed recovery path.

How a canary deployment works

Most canary deployments maintain two production versions at once:

• The stable version serves the majority of traffic.
• The canary version serves an intentionally limited cohort.
• A router, service mesh, load balancer, orchestration platform, or feature flag decides which version or behavior each request receives.
• Monitoring compares the canary with the stable baseline.
• Promotion rules expand traffic, while failure rules stop or reverse the rollout.

Google Cloud describes a canary as a progressive rollout that splits traffic between the deployed and new versions before full release. Its deployment tooling models percentage phases followed by a stable phase. Amazon ECS uses a two-step canary: a small initial traffic shift, an observation period, and then the remaining shift if the revision passes.

Those implementations show an important point: canary is a strategy, not one exact schedule. A team might use 5% -> 25% -> 50% -> 100%, deploy first to one region, enable a feature for internal accounts, or route only a subset of requests to a new service. The right shape follows the failure mode and the routing controls available.

Deployment and release can be separate controls

Some canaries operate at the infrastructure layer. Two application revisions run side by side, and the platform shifts network traffic. Kubernetes Deployments provide the underlying replica and rollout machinery, while a service mesh or a controller such as Argo Rollouts can manage weighted traffic and analysis stages.

Other canaries operate at the feature layer. The code can be deployed to every server while a flag exposes the behavior to a limited set of users. GrowthBook's guide to feature flags in CI/CD explains how flags decouple code deployment from feature release.

These controls solve related but different problems. An infrastructure canary is appropriate when the new binary, container, dependency, or runtime might affect every request. A feature flag is useful when a specific capability can be isolated in code and targeted by user or account. Teams often use both: first validate the new service revision operationally, then expand a flag-gated feature to increasingly broad cohorts.

Benefits of canary deployment

Canary deployment changes a release from one large, irreversible moment into a series of smaller decisions.

Smaller initial blast radius

If a problem affects 2% of traffic rather than 100%, fewer users encounter it while the team diagnoses and reverses the change. That matters most for failures that pre-production tests cannot reproduce: unusual data shapes, production load, third-party behavior, regional differences, or long-tail device combinations.

The real benefit depends on containment. A canary that writes corrupt data to a shared database can harm stable users even if it receives little traffic. Before rollout, identify shared state, background jobs, queues, caches, and schema changes that escape the nominal traffic percentage.

Evidence from the production environment

Staging environments rarely reproduce production volume, traffic mix, integrations, and user behavior. A canary lets teams observe the release under real conditions while retaining a stable comparison.

That comparison is stronger than asking whether a dashboard merely looks normal. If stable latency rises at the same time as canary latency, a shared incident may be responsible. If only canary requests regress, the new revision becomes the leading explanation. Cohorts still need to be comparable, but concurrent baselines help separate release effects from background changes.

Faster recovery without abandoning delivery speed

Canaries let teams deploy frequently without making every deployment an all-user exposure event. When rollback is automated and the stable version remains available, recovery can be a routing change rather than an emergency rebuild.

This is one reason to keep deployment separate from release. GrowthBook's release management guide treats deployment as moving code into production and release as deciding who can access it. That separation gives product and engineering teams another control surface after the pipeline succeeds.

Safer learning about product outcomes

Operational health answers, "Did the release break the system?" It does not answer, "Did the release improve the product?" If a change is intended to influence activation, conversion, retention, or model quality, teams may pair the canary with a controlled experiment.

The distinction matters. GrowthBook's discussion of canary releases for AI models separates defensive monitoring from evidence about user outcomes. A model can produce no new errors and still make the experience worse. The same reasoning applies to ranking systems, checkout changes, onboarding flows, and pricing experiences.

A practical canary deployment process

A production-ready canary starts before traffic moves. The team must decide what will be exposed, what success looks like, and how recovery works while the release is still easy to change.

1. Define the scope and failure modes

Write down the release boundary. Is the canary a container revision, API endpoint, worker, database client, feature, model, or combination? Map the systems it reads and writes.

Then list plausible failures:

• Higher request errors or timeouts.
• Increased latency or infrastructure cost.
• Data corruption or incompatible writes.
• Degraded downstream services.
• Broken behavior for a device, region, plan, or account type.
• Worse product or business outcomes despite healthy infrastructure.

The failure modes determine the metrics, cohort, observation window, and rollback design. A memory leak needs time to appear. A checkout error may appear quickly but deserves a very low tolerance. A cache or schema change may need a forward-fix plan because routing alone will not undo it.

2. Choose a representative first cohort

The first cohort must balance safety and information. Too large increases exposure; too small produces noisy metrics and may never exercise rare paths.

Random percentage routing is useful when requests are interchangeable. Stable hashing by user or account is better when a person must receive a consistent experience across sessions. An internal or opt-in cohort can uncover obvious usability problems, but it may not represent customer permissions, data volume, integrations, or devices.

For multi-tenant SaaS, avoid routing individual requests from the same customer unpredictably when versions can produce inconsistent state. Account-level assignment may be safer. For regional systems, a single low-risk region can be a useful first stage only if its architecture and traffic resemble later regions.

Document exclusions. Administrators, regulated tenants, unusually large accounts, and users in the middle of a critical workflow may need special treatment.

3. Establish the stable baseline and guardrails

Record normal behavior before rollout. At minimum, consider:

• Request success and error rates.
• Latency percentiles, not only averages.
• CPU, memory, queue depth, connection use, and saturation.
• Dependency errors and timeouts.
• Logs and traces associated with the new version.
• A product guardrail such as completed checkouts or successful activations.
• Support contacts or client-side errors not visible in server metrics.

Use version, cohort, and region dimensions so canary and stable traffic can be compared. OpenTelemetry's guidance on service telemetry covers the relationship among metrics, logs, and traces; the practical requirement is that every signal can identify the revision or flag exposure that produced it.

Set numerical thresholds where possible. "Watch latency" is not a release rule. "Pause if canary p95 latency is more than 15% above stable for ten minutes with at least 5,000 requests" is testable. Add an absolute ceiling so a stable baseline that is already unhealthy does not normalize a bad result.

4. Verify rollback before exposure

Rollback must include more than switching traffic. Confirm that:

• The stable artifact is still deployable and running.
• Flags or routing rules can be changed by the on-call team.
• Database changes are backward compatible.
• New writes can be read safely by the old version.
• Queued jobs and caches will not preserve the failure.
• The rollout owner has permission to execute recovery.

Test the mechanism in a lower environment or with a synthetic release. If rollback takes 30 minutes, the canary percentage and thresholds must reflect 30 minutes of continued exposure.

5. Start small and observe a complete window

Deploy the canary, run smoke tests, and hold at the initial stage long enough to see meaningful behavior. The observation window should cover the failure mode rather than an arbitrary timer.

A high-volume API may reveal error and latency regressions in minutes. Billing, batch processing, daily retention, or memory accumulation needs longer. Include enough cycles to exercise scheduled jobs and traffic peaks. Do not promote solely because a minimum timer expired; require both sufficient exposure and passing checks.

6. Expand in deliberate stages

Each promotion should be a new decision. A possible schedule is:

This is an example, not a default. A low-volume product may need a larger starting percentage to generate data. A critical financial path may need a smaller cohort, stricter thresholds, and manual approval. As exposure grows, watch for nonlinear failures such as connection exhaustion, cache churn, rate limits, and lock contention.

7. Complete, roll back, and document

At 100%, continue observing. The stable comparison disappears, and problems can emerge only at full load. Retain the previous revision until the recovery window closes.

Record the release, cohort, stage timings, metric results, alerts, decision, and any manual intervention. A successful canary is not just one that reached 100%; it is one that improves the next rollout. False alarms should refine thresholds. Missed failures should add signals or change cohort design. Slow recovery should become reliability work.

Canary deployment best practices

The mechanics are straightforward. Operational discipline is where canaries succeed or fail.

Automate checks without hiding ownership

Automated analysis can pause or roll back faster than a human watching dashboards. Google Cloud supports analysis within canary phases, and Argo Rollouts can query metrics providers during promotion. Automation is most useful when thresholds, minimum sample requirements, and failure behavior are explicit.

Still assign a release owner. Someone must interpret ambiguous signals, coordinate incidents, and decide whether a technically healthy release is acceptable for users.

Keep cohorts sticky and observable

A user who alternates between versions can see inconsistent behavior and contaminate measurements. Use deterministic assignment where identity exists, and log the chosen version or flag value with the event. GrowthBook's feature flagging guide for AI models shows why assignment data becomes essential when a rollout also evaluates outcome quality.

Separate operational promotion from experiment conclusions

A canary can be promoted operationally once reliability guardrails pass, but a product experiment may need more observations to estimate impact. Conversely, a statistically promising experiment should not override a reliability failure.

Use two decision sets:

• Release health: errors, latency, saturation, data integrity, and incident signals.
• Product evidence: primary outcome, uncertainty, practical effect, and experiment guardrails.

This prevents "nothing broke" from becoming evidence of value and prevents a favorable conversion fluctuation from excusing an unsafe system.

Design schema and state changes for coexistence

During a canary, old and new versions run simultaneously. Database migrations should follow an expand-and-contract pattern: add backward-compatible structures, deploy code that can tolerate both shapes, migrate data, then remove the old structure only after the rollback window.

The same rule applies to event schemas, messages, APIs, and cached objects. If the new version emits data that the stable version cannot handle, the real blast radius is larger than the routing percentage.

Limit concurrent changes

When multiple releases, configuration changes, and experiments overlap, attribution becomes difficult. Keep a change calendar for critical systems, attach version metadata to telemetry, and pause unrelated rollouts during an incident. A canary is most informative when the stable comparison differs mainly in the change being evaluated.

Treat rollback as a normal outcome

Teams sometimes keep a weak canary running because rollback feels like failure. That defeats the strategy. Predefined thresholds make reversal an expected control action, not a judgment on the people who shipped the code.

The useful question after rollback is whether the system contained harm and produced diagnostic evidence. If it did, the canary worked.

Canary, rolling, blue-green, and feature-flag releases

These patterns are often combined, but they control different things.

Choose based on the unit of risk. Use infrastructure routing for runtime and artifact changes. Use flags for isolated behavior and audience control. Use blue-green environments when environment parity and rapid reversal matter. Use rolling replacement when gradual instance turnover is sufficient. A mature pipeline may use all four at different layers.

A canary is a controlled question, not a smaller launch

A good canary asks a precise question: can this release serve a representative slice of production traffic without crossing our reliability, data-integrity, or product guardrails? The rollout stages create time and containment to answer it.

Before moving traffic, define the cohort, baseline, observation window, promotion thresholds, and rollback path. During the rollout, compare canary and stable behavior, expand only when checks pass, and preserve consistent assignment. Afterward, document what the release taught the team.

Feature flags can extend that control beyond infrastructure by separating deployment from audience exposure. To manage targeted and percentage-based releases alongside experimentation, explore GrowthBook feature flags.