Feature flagging for AI models: how to safely roll out changes
Swapping a model ID or tweaking a system prompt doesn't touch your codebase — which means your CI/CD pipeline has no record of it, no way to track it, and nothing to roll back when it degrades in production.
That's the gap that catches AI teams off guard. Feature flags for AI models aren't just a nice-to-have on top of your existing deployment process. They're the control plane that makes model configs, prompt templates, and hyperparameters as safe to change as any other piece of production code.
This article is for engineers, PMs, and data teams who are shipping AI features and want a practical system for doing it safely — without slowing down. Whether you're swapping foundation models, iterating on prompts, or trying to measure whether a new model actually performs better for your users, the same flag infrastructure handles all of it. Here's what you'll learn:
- Why AI rollouts fail in ways that staging environments and test suites can't catch
- What belongs behind a feature flag — model IDs, prompt templates, and LLM hyperparameters bundled as a single JSON payload
- How to run a staged canary rollout using percentage-based traffic splits and user attribute targeting
- How to turn a canary into a controlled experiment that produces real evidence for model promotion decisions
- How to set up instant rollback, automated guardrails, and approval workflows for model and prompt changes so bad configs never reach most of your users
The article moves in that order — from the case for this infrastructure, to what you put behind flags, to how you roll out and measure, to how you govern and revert. Each section builds on the last, and by the end you'll have a complete picture of how feature flags become the operational backbone of a safe AI release process.
Why AI model rollouts demand a different kind of infrastructure
If your team already runs a mature CI/CD pipeline with staging environments, automated tests, and deployment gates, you might reasonably ask: why isn't that enough for AI? The answer cuts to something fundamental about how AI systems fail — and it's different enough from traditional software that it demands a different class of tooling.
AI failures are probabilistic, not deterministic
When a software bug ships to production, it's reproducible. Given the same input, you get the same broken output. You can replicate it in staging, trace it to a commit, and revert. The feedback loop is tight and the failure mode is legible.
AI failures don't work that way. A model or prompt change might succeed on 999 user inputs and fail catastrophically on the thousandth — not because of a logic error, but because of the way model parameters interact with an edge case that no one anticipated.
As Statsig frames it, AI model parameters are "often interdependent, and those relationships only become apparent when real users encounter edge cases." One tweak can shift many behaviors in surprising ways. That's not a bug in the traditional sense. It's a probabilistic failure that your test suite has no reliable mechanism to catch.
This distinction matters enormously for how you think about release infrastructure. Traditional QA gates — unit tests, integration tests, staging environments — are necessary but not sufficient. They test known inputs. Real users generate unknown ones.
Production is the only real test environment
The uncomfortable reality for any team shipping AI features is that offline evaluation and staging environments are approximations. They test the inputs you thought to include. Production tests everything else.
The team at Helix, an image recognition platform, discovered this directly. When adding new image classifications to their model, their core question was: "How do we know if they will work in the real world with real users and return the correct classification?" The answer wasn't a better staging environment — it was controlled exposure to a specific subset of real users before any broader rollout.
Production exposure wasn't optional; it was the only meaningful test. The only question was whether that exposure would be controlled or uncontrolled.
This is the defining characteristic of AI risk that makes it categorically different from shipping a new UI component or a refactored API endpoint. You cannot fully validate an AI system before real users touch it. You can only control how many real users touch it, and which ones, and what you're measuring while they do.
Why CI/CD alone can't solve this
Here's the specific gap that catches teams off guard: prompt changes, model version updates, and hyperparameter adjustments often don't involve any code change at all. You might update a system prompt in a config file, swap a model ID in an environment variable, or adjust temperature through an API parameter — none of which produces a git commit that your deployment pipeline can track or revert.
When that change degrades in production, there is no deployment to roll back. The rollback mechanism has to exist at the configuration layer, not the deployment layer. Feature flags provide exactly that: runtime control over application behavior without touching the deployment pipeline.
LaunchDarkly's framing captures the principle cleanly — flags make individual features, not deployments, the unit of control. GrowthBook's documentation puts the gap even more bluntly: "Don't Let Vibe Coding Become Vibe Shipping." The tools that help you build AI features don't automatically give you the infrastructure to ship them safely.
The business risk of skipping this infrastructure
The stakes of an uncontrolled AI rollout aren't just technical. A hallucinating documentation bot, a content moderation model that misclassifies at scale, or a recommendation system that degrades user trust — these carry reputational and sometimes ethical consequences that a slow API response or a broken form field typically don't.
At the same time, the velocity pressure is real. New foundation models ship constantly, and teams that can't safely test them in production are forced into a bad choice: move slowly and fall behind, or move fast and accept uncontrolled exposure.
Feature flags resolve that tradeoff. They're the infrastructure layer that makes it possible to move quickly and maintain a kill switch — not as competing priorities, but as the same mechanism. That's why treating them as optional for AI deployments is the wrong frame. For any team running AI features in production, they're the floor, not the ceiling.
Models, prompts, and hyperparameters all belong behind a flag
Once you accept that runtime configuration control is the gap CI/CD can't close, the next question is concrete: what exactly belongs behind a flag? When engineers first start thinking about feature flags for AI, they usually think about the model — which version of GPT-4 or Claude is active.
That's the right instinct, but it captures maybe a third of what actually needs to be controlled. Every configurable dimension of an LLM integration is a potential source of behavioral change in production, and every one of them belongs behind a flag. The practical question is: what exactly does that mean, and what does it look like in code?
Model identity and provider configuration
The modelId field is the most fundamental thing to put behind a flag. It determines which model handles a request, and swapping it — from Claude 3 Haiku to Claude 3 Sonnet, or from GPT-4 Turbo to GPT-4o — is a meaningful behavioral change that should be controlled, not deployed.
LaunchDarkly's AI model flag templates, released in GA in June 2024, are built around exactly this pattern: a JSON flag carries the modelId alongside inference parameters, and the application reads it at request time.
The DevCycle/Helix case study makes the operational benefit concrete. Helix built model selection directly into their image recognition microservice using a feature flag, and new model candidates were added as flag variations by the product manager — no engineering deployment required. That detail matters: when introducing a new model is a flag configuration change rather than a code change, the iteration cycle compresses dramatically.
Prompt templates as flaggable configuration
Prompt templates are not static strings. They are configuration that directly determines model behavior, and they should be versioned and controlled with the same rigor as model IDs.
LaunchDarkly treats prompts as a distinct flaggable category — separate from the model flag — which signals that the industry has recognized prompts as a first-class configuration concern rather than an implementation detail.
A prompt flag can carry the full system prompt, the user prompt structure, and any few-shot examples as a string or structured JSON field. This matters operationally because prompt changes are among the most frequent modifications AI teams make in production, and without a flag, every iteration requires a deployment cycle. It also matters for governance, which is addressed later in this article — but the short version is that a prompt change is effectively a code change, and it should be treated as one.
LLM hyperparameters — temperature, max_tokens, top_p, top_k
These are the inference-time knobs that control output behavior: temperature governs creativity versus determinism, max_tokens caps response length, top_p controls nucleus sampling, and top_k limits the candidate token pool. LaunchDarkly explicitly names all four as parameters that belong in model flags, with the note that parameter naming conventions vary by provider — Anthropic and OpenAI use slightly different field names for equivalent concepts.
Changing any of these without a flag creates an uncontrolled variable in production. Adjusting temperature in isolation, for example, can interact unexpectedly with a top_p value that was calibrated for a different temperature range — a combination that may have worked well in offline testing but behaves differently when real users generate the full distribution of inputs.
Bundling everything into a structured JSON payload
That interdependency is the strongest argument for encoding the entire LLM configuration as a single multivariate JSON flag rather than managing model ID, prompt, and hyperparameters as separate flags. When configuration is split across multiple flags, interaction effects become difficult to reason about — you can't be certain which combination of values is actually active for a given user at a given moment.
The industry pattern, confirmed across LaunchDarkly, ConfigCat, and GrowthBook's documentation, is a single JSON flag payload that bundles the full configuration. A minimal example looks something like this:
{
"modelId": "claude-3-sonnet-20240229",
"provider": "anthropic",
"temperature": 0.7,
"maxTokens": 1024,
"topP": 0.9,
"systemPrompt": "You are a helpful assistant specialized in..."
}GrowthBook's JSON flag type is explicitly designed for this kind of complex, multi-value configuration. GrowthBook's JSON flag type supports JSON Schema validation on flag values, which means teams can enforce that every LLM config payload contains required fields — modelId is always present, temperature is always a float within a valid range — before the configuration reaches production. That's a meaningful safety layer for teams with compliance requirements or customer-facing AI features.
The flag value is evaluated at request time, so a prompt change, model swap, or hyperparameter adjustment takes effect immediately across all running instances. No deployment, no restart, no coordination across engineering teams. The configuration becomes a runtime variable, which is exactly the kind of control that AI deployments require.
Canary releases and targeted rollouts: controlling blast radius at every stage
Deploying a new AI model to production is not a binary event. The question isn't whether to flip a switch — it's how to design a progression that limits blast radius at every step while preserving the ability to reverse course instantly. Percentage-based traffic splitting and user attribute targeting are the two mechanisms that make this possible, and feature flags are the infrastructure layer that ties them together.
Why feature flag rollouts differ from infrastructure-level canaries
The term "canary release" gets applied to two fundamentally different patterns, and the distinction matters for AI. A traditional canary deployment routes traffic at the infrastructure layer — a load balancer sends a fraction of requests to machines running the new version. That works when the change is a new binary.
But when the change is a model ID, a prompt template, or an endpoint configuration, there's no new binary to deploy. The change is a config value, and the correct control mechanism is an application-layer feature flag, not a load balancer rule. Feature flags make individual configurations — not deployments — the unit of control, which means rollback is a flag toggle rather than a container redeployment.
Percentage-based canary mechanics
The mechanics of a percentage-based rollout are straightforward: a flag routes X% of traffic to the new model configuration and the remainder to the current one. What makes this safe is deterministic user assignment. GrowthBook uses MurmurHash3 to hash user identifiers into consistent bucket assignments — the same user always resolves to the same model variant, preventing the kind of mid-session inconsistency that would corrupt evaluation data and confuse users.
A staged ramp schedule — 10%, then 25%, 50%, 75%, and finally 100% — gives teams defined checkpoints to assess metrics before advancing. Adjusting the overall exposure percentage mid-rollout is safe; reshuffling split ratios between variants is not, because it risks users switching model assignments.
User attribute targeting strategies
Percentage-based rollouts control how many users see the new model. Attribute targeting controls which users see it first, and that distinction is what makes a canary genuinely risk-stratified rather than just small.
The logical progression sequences exposure by risk profile: internal employees carry zero customer risk and provide a first functional validation pass; opted-in beta users are tolerant of rough edges and generate useful early signal; premium or paid-tier users offer higher-value feedback before broader exposure; geographic targeting limits regional blast radius before a global rollout.
In B2B contexts, organization-level targeting ensures all users within a tenant get the same model — critical for consistency in enterprise accounts. GrowthBook supports AND/OR targeting logic across attributes like subscription tier, geography, company ID, and custom properties, with reusable Saved Groups for audience segments that appear across multiple flags. This lets teams define a "beta cohort" once and reference it across every model rollout without rebuilding the targeting rule each time.
The kill switch as a required component
At every stage of the ramp-up sequence, the flag must be instantly reversible. This is not an optional feature — it is the architectural requirement that justifies the entire staged approach. If a model starts producing degraded outputs at 25% exposure, the correct response is to turn the flag off, not to file a deployment ticket.
GrowthBook's Safe Rollouts capability takes this further by combining the ramp schedule with automatic guardrail monitoring: if a key metric regresses beyond a defined threshold, the system can trigger rollback without waiting for manual intervention. That matters specifically for AI, where a quality regression might surface gradually across thousands of requests before it's visible in aggregate dashboards.
What a responsible rollout sequence actually looks like
A concrete progression, grounded in the mechanics above, looks like this: start at 0% with the flag forced on only for internal employees via attribute targeting — validate that the model responds correctly and latency is acceptable. Expand to 5–10% targeting the beta cohort, monitoring error rates and response quality. At 25%, broaden to a specific subscription tier or geography and begin watching user-facing outcome metrics.
From there, advance through 50%, 75%, and 100% using a scheduled ramp with guardrail thresholds defined before each step. The critical discipline is specifying, before each advance, what metric threshold would trigger a hold or rollback — not after a problem surfaces, but as a precondition for moving forward. That pre-commitment is what separates a controlled rollout from an optimistic deployment.
From canary to controlled experiment: measuring which AI model actually performs better
A controlled experiment and a canary release are not the same thing, and conflating them is one of the more expensive mistakes a team can make when rolling out AI model changes. A canary answers one question: does this break anything? A controlled experiment answers a different question entirely: does this actually improve outcomes that matter?
For AI model selection, the distinction is critical. A new model can pass a canary without throwing errors, without degrading latency, without producing a single visible hallucination in the sample traffic — and still be the wrong model for your users. Safety is necessary, but it is not sufficient.
The conceptual pivot: from safety check to quality check
The DevCycle/Helix image recognition rollout described earlier in this article is a clean example of the canary pattern. Exposing a new model to internal users and a beta cohort via email-based targeting confirms the model works under real conditions. It does not tell you whether the model is statistically better than the incumbent at the outcomes your product depends on.
Once you have confirmed a model does not break things, you still need evidence it is better before promoting it to 100% of traffic. That evidence has to come from production, not from internal benchmarks. As Statsig has observed, real-time evaluation in production consistently reveals gaps that offline checks miss — which is exactly why the experiment has to happen in the wild, with real users, under real conditions.
Landon Smith, Head of Post-Training at Character.AI, describes this directly: "GrowthBook has been an invaluable tool for Character.AI, helping us develop our models into a great consumer experience. We can compare different modeling techniques from the perspective of our users — guiding our research in the direction that best serves our product." That framing — from the perspective of our users — is the defining characteristic of experiment-driven model selection. Internal evals measure what the model does. Experiments measure what the model does to users.
Instrumenting metrics at two layers
Running a rigorous model comparison experiment requires instrumentation at two distinct layers, and both matter.
The first layer is user-facing outcome metrics: engagement, task completion rate, session length, satisfaction signals, conversion. These are the metrics that justify model decisions to product leadership and business stakeholders. "Our internal eval preferred Model B" is not the same argument as "Model B produced a statistically significant improvement in task completion rate across a large, representative user sample." The experiment produces the latter.
The second layer is LLM-level operational metrics: latency per request, cost per inference, refusal rate, error rate. A model that improves engagement while tripling inference cost may not be the right choice at your traffic volume. These operational metrics function as guardrails — they prevent a model that wins on quality from being promoted despite being economically or operationally unsustainable. Treating them as guardrail metrics in your experiment framework means a degradation surfaces as a warning before it becomes a production incident.
Warehouse-native analysis for the cost-quality tradeoff
The specific challenge with AI model experiments is that the data you need lives in two places: inference cost data comes from your LLM provider, and user outcome data comes from your product. Joining and analyzing those together is a SQL problem, not a dashboard problem.
GrowthBook's warehouse-native experimentation connects directly to Snowflake, BigQuery, Redshift, ClickHouse, Databricks, and similar environments — the analysis runs as SQL queries against data where it already lives, without requiring a separate data pipeline or moving raw user-level data out of your environment. This matters for teams with data residency requirements and for teams that simply want their experiment results to reflect the same source of truth as their product analytics.
This architecture also matters for experiment runtime. Variance reduction techniques like CUPED can meaningfully shorten the time required to reach statistical significance, which matters when inference costs are accumulating throughout the experiment window. Shorter experiments mean lower cost exposure and faster decisions — both of which are meaningful when you are running model comparisons continuously as new model versions become available.
Connecting results to the promotion decision
A defensible model promotion decision has a specific shape: statistically significant improvement on at least one user-facing outcome metric, no degradation on guardrail metrics, and a documented record of the experiment that produced the decision. That record is what makes model selection auditable and repeatable over time — not just for the current decision, but for the institutional knowledge that accumulates across many model iterations.
The mechanical bridge back to flag infrastructure is direct. The same feature flag used to run the canary becomes the experiment assignment mechanism. When the experiment concludes in favor of the new model, the flag promotion to 100% traffic is the final step in a process that started with a targeted rollout and ended with statistical evidence. The flag is not just a deployment tool — it is the control plane for the entire model selection lifecycle.
Instant rollback, kill switches, and governance for AI changes in production
When a bad model config reaches production, the clock starts immediately. Every request served by a degraded prompt or misconfigured temperature setting is a user interaction that may produce a harmful, embarrassing, or simply wrong response.
The problem with deploy-based rollback in this scenario isn't just that it's slow — it's that there's nothing to redeploy. The code hasn't changed. The model config lives outside your codebase, and if it wasn't stored behind a feature flag, your only options are to push a hotfix, manually revert a config file, or call someone at 3am to log into a dashboard and change a value by hand. None of these are acceptable for customer-facing AI features.
This section covers two distinct problems that flag-based infrastructure solves: the technical problem of reverting a bad AI config instantly, and the organizational problem of ensuring that model and prompt changes go through appropriate review before they ever reach production.
Kill switch mechanics and revert speed
The technical case for flag-based rollback is straightforward once you understand what happens when a flag is disabled. With SDK-based flag evaluation, when your application calls something like growthbook.getFeatureValue('llm-config', defaultConfig), disabling the flag causes the platform to stop serving that flag's rules and return nothing — so the SDK falls back to the defaultConfig value you provided. All targeting rules are ignored. No redeploy. No pipeline. No waiting.
For an AI system where the model ID, prompt template, and temperature are all stored as a JSON payload in a feature flag, this means a single toggle in a dashboard immediately routes every subsequent request back to the previous known-good configuration. The contrast with deploy-based rollback is stark: a traditional rollback requires a code push, CI/CD pipeline execution, and deployment time — all while the degraded config continues serving users. For AI configurations, which live in the flag rather than in code, the flag is the only rollback mechanism that operates at the speed the problem demands.
Automated rollback with guardrail metrics
Manual kill switches require someone to notice the problem first. Automated rollback closes that gap by monitoring guardrail metrics during a staged rollout and triggering a revert when a regression is detected — without human intervention.
GrowthBook's Safe Rollouts include an Auto Rollback toggle that automatically disables the rollout rule if any guardrail metric fails significantly, using one-sided sequential testing to detect regressions while minimizing false positives. The ramp-up schedule — 10%, 25%, 50%, 75%, then 100% — is designed so that a regression surfaces early, when only a fraction of users are exposed. If guardrails fail at 10%, the rollout stops and reverts before the majority of traffic is ever affected.
The metric selection question matters especially for AI. A degraded model response may not spike your error rate at all — it might show up in engagement drop-off, task completion rate, or user satisfaction scores. That means your guardrail metrics need to include user-facing quality signals, not just infrastructure health indicators.
LaunchDarkly's Guarded Releases, which reached general availability in 2025, offers a comparable automated remediation capability with application performance thresholds and telemetry integrations. Notably, not every flag platform provides this natively — Flagsmith, for instance, requires an external Datadog Workflows integration to approximate automated rollback, which introduces meaningful operational complexity.
Approval workflows for model and prompt changes
A prompt change that alters how an AI responds to users is functionally a code change. It modifies production behavior, it can introduce regressions, and it can cause harm if it reaches users without review. Treating it as a casual config edit — something anyone on the team can push directly to production — is a governance gap that will eventually cause an incident.
The correct pattern is to route all model config changes through the same approval workflow that governs code changes. GrowthBook supports drafts, revisions, merge conflict handling, and requiring approvals as part of its Publishing and Approval Flows. Unleash offers change requests with four-eyes approvals at the enterprise tier. LaunchDarkly provides approval workflows with custom roles. The specific platform matters less than the principle: no prompt template, model ID swap, or hyperparameter change should reach production without a documented review step.
For teams with compliance requirements, this isn't optional. An audit trail that shows who approved a prompt change, when, and what the previous value was is the same artifact your security and compliance teams expect for code deployments. If your AI configuration lives in feature flags, it inherits that audit trail automatically — the same infrastructure that governs your code changes now governs your model configs. That's the architectural argument for why flags are the right home for AI configuration, not ad-hoc config files or direct API calls that leave no paper trail and offer no rollback path.
Implementing feature flags for AI models: a practical starting point
The through-line of this article is simple: AI configuration changes — model IDs, prompt templates, hyperparameters — are production changes, and they deserve the same controls as code. The gap isn't a tooling gap so much as a framing gap.
Once you treat your LLM config as a runtime variable rather than a deployment artifact, the right infrastructure becomes obvious. Feature flags aren't bolted on top of your AI release process. They are the release process.
JSON payloads vs. simple toggles: why the architecture decision matters first
Start with a single multivariate JSON flag that bundles your full LLM configuration — model ID, prompt template, and inference parameters together. Splitting these across separate flags creates interaction effects you can't reason about cleanly. If you're evaluating platforms, GrowthBook's JSON flag type with schema validation is worth examining specifically for this use case, since it lets you enforce required fields before a config reaches production.
The rollout sequence is only as safe as the discipline around it
The ramp sequence matters less than the discipline around it. What separates a controlled rollout from an optimistic deployment is the habit of defining your rollback threshold before you advance — not after a metric starts moving. Internal employees first, then a beta cohort, then a broader tier. The flag is your kill switch at every stage.
Governance feels like overhead until the first incident
Governance feels like overhead until the first incident. A prompt change that skips review is a code change that skipped review — same risk, same consequences. Build the approval workflow before your team is large enough to make skipping it tempting. The audit trail you get for free when config lives in flags is the same artifact your security team will ask for anyway.
There's a real tension worth naming: the faster your team iterates on models and prompts, the more you need this infrastructure — but building it takes time that feels like it's slowing you down. The honest answer is that the investment front-loads the friction. Teams that skip it don't move faster; they just absorb the cost later, usually during an incident.
This article was written to give you a complete picture of how this infrastructure fits together, not just the theory. If it helps you avoid one uncontrolled rollout or make one model promotion decision with actual evidence behind it, it's done its job.
What to do next: where you start depends on where you are
The right first move depends on your current state. No flag infrastructure at all means the highest-leverage action is putting your model ID and system prompt behind a single JSON feature flag — even before you build any rollout logic. That one change gives you runtime control and a rollback path, which is the most valuable property to have first.
Already have flags but they're simple booleans? Audit which AI configs are still living in environment variables or hardcoded strings and migrate the highest-risk ones to JSON flags. JSON flags already in place means the next investment is defining guardrail metrics before your next rollout — latency, error rate, and at least one user-facing quality signal — so that your next ramp has a defined stop condition rather than a hopeful one.
Related insights
- Feature flags: what they are, how they work
- 7 Best Practices to Implement Feature Flags at Scale in 2026
- Why Choose GrowthBook for Feature Flagging
Related Articles
CLI vs MCP: TLDR, it doesn’t matter as much as you think
Your agent should know how to do the job whichever interface it uses.
Your coding agent can already run terminal commands. Your product manager's assistant can connect to remote tools. Both need to inspect the same experiment, understand the same metric, and prepare changes to the same feature flag.
That creates a familiar question: should you give them a CLI or an MCP server?
The usual answers mix several concerns. A CLI is easy to compose with scripts. MCP gives compatible clients a standard way to discover and invoke tools. Skills can supply the instructions that make either route useful. None of those statements tells you whether the agent knows your release process or can distinguish a draft from a live change.
For GrowthBook, the choice has become smaller. We've built the agent layer around a REST API and reusable workflow skills, then made those available through different interfaces. Your choice of CLI or MCP no longer needs to determine which product workflow your agent can understand.
It still determines how the agent connects, executes requests, and handles credentials. This article explains that distinction, shows the same disabled-flag operation through both routes, and gives you a way to evaluate the integration you actually intend to use.
The debate combines three different questions
Start with a specific job: prepare a release control for a new checkout flow. The agent needs to know which project owns checkout, which environments exist, what the default behavior should be, and whether the proposed flag key already exists.
Choosing an interface answers only part of that job.
How can the agent reach the product?
A command-line interface lets an agent invoke a program in an environment where it has shell access. That program can make authenticated API requests and return data. You can also run the same commands yourself, put them in scripts, and preserve output for inspection.
The Model Context Protocol architecture connects a host application to servers through clients. An MCP server exposes capabilities that a compatible host can discover and call. The host might be a coding environment, a desktop assistant, or another agent application.
These categories overlap. A terminal-based agent can also be an MCP client. A local MCP server can be a process launched by that client. A remote MCP connection can work without the host giving the model a general-purpose shell.
So “terminal agent” doesn't automatically mean “use the CLI,” and “MCP” doesn't automatically mean “remote service.” Ask what access the runtime provides before choosing the adapter.
What procedure should the agent follow?
An API can expose a create operation without teaching an agent when to use it. A syntactically valid feature flag might still have the wrong project, an ambiguous key, or a default value that doesn't match the application.
That's the job of a skill: reusable instructions for a recognizable task. The Agent Skills format gives those instructions a portable structure, including a skill entry point and supporting references. A workflow can describe the context to inspect, the sequence to follow, and the evidence to return.
For the checkout example, useful guidance says to resolve environments, check the permanent key, choose a value type, and create the flag disabled. Merely exposing a create command leaves those decisions to the agent's general knowledge.
Skills are also useful when the right answer is to take no action yet. “Create a flag” and “design an experiment” are related requests, but they can require different objects and sequencing. An agent needs enough product guidance to make that distinction.
What context makes the procedure relevant?
Even a good playbook needs the current product state. The agent should inspect the objects in your GrowthBook organization and, when it is changing application code, the repository's SDK initialization and tests.
Those are different sources of context. GrowthBook supplies flags, experiments, metric definitions, and other authorized product resources. Your coding runtime supplies the application files it can access. An MCP connection doesn't by itself grant repository access, and a CLI doesn't by itself explain your metric definitions.
Evaluate the combination: access to the product, a maintained procedure, and enough context to apply that procedure. A comparison that changes all three at once cannot isolate the effect of CLI versus MCP.
GrowthBook puts the durable parts below the interface
We separated the agent layer into responsibilities that can be reused. The product API exposes operations. Skills explain workflows. The interface connects the agent to those two layers.
This is the architectural shift behind GrowthBook's agent experience. Instead of making every client integration carry its own interpretation of experimentation, we can share the interpretation and change how it is delivered.
The API provides the operations
GrowthBook's REST API makes product objects addressable. An agent can inspect a project, retrieve a flag, or perform an authorized operation against those objects.
The rebuilt API described in GrowthBook 4.4 uses shared schemas to support types, validation, and API documentation. The CLI is generated from the API definition. That reduces the amount of separately maintained command logic needed to expose product operations.
The OpenAPI specification explains the underlying principle: a machine-readable description can represent an HTTP API's paths, operations, inputs, and responses. Generation is useful because it derives another interface from that contract.
Generation doesn't mean any installed binary knows every future endpoint. The generated client still has a version, and your GrowthBook server still has a version. A self-hosted instance can lag behind the public documentation. Treat API compatibility as a deployment fact you verify.
Skills provide the workflow guidance
GrowthBook's feature-flags skill routes requests to more specific workflows. Creating a flag, inspecting dependencies, changing targeting, requesting review, and publishing a revision are separate procedures.
That structure lets the agent load the guidance relevant to the current job. The skill authoring guidance on progressive disclosure supports this approach: keep the entry point focused and place detail in references that are loaded as needed.
The benefit is maintainability as well as context management. A correction to a flag workflow can be made in the maintained skill rather than repeated across a series of tool descriptions and client-specific prompts.
There are two shell concepts to keep distinct. The generated growthbook CLI is a terminal interface to the product API. The skills repository also includes a small gb-call helper for making REST requests. Installing a skill doesn't make that helper a globally available command; its location depends on the installation. Both can reach the API, but their configuration and invocation are different.
MCP provides a thin bridge
The current GrowthBook MCP implementation exposes four tools: growthbook_list_skills, growthbook_read_skill, growthbook_api_read, and growthbook_api_write.
An agent discovers a domain, reads the relevant workflow, then makes API calls. It doesn't need a dedicated MCP tool for every product endpoint. Calls described as gb-call GET in a skill map to the read tool; mutating methods map to the write tool with the same path and body.
The bridge does not run the shell helper. It forwards authenticated requests. Product-side permissions and validation remain important because a small tool surface can still reach many operations.
Shared guidance also has a delivery boundary: the MCP server bundles skills when it is built, using a pinned source revision. A newly edited skill in the repository needs to be included in a subsequent server build before that server supplies it. “Shared source” is a maintainability claim, not a promise of instant synchronization.
Explore the shared flag workflows
Open the feature-flags skill to see the procedures that guide creation, targeting, reviews, and publishing. Bring the same workflow knowledge to the agent interface your team uses.
View the Flag Skill
The same disabled flag can use either route
The practical question is whether the interface changes the job. Let's create a Boolean flag called checkout-redesign, with a false default and every environment disabled.
This is an API example for a sandbox project, not an application integration tutorial. It creates a control object; your application still needs a correctly configured GrowthBook SDK to evaluate it. The feature flag documentation covers the product concepts behind that integration.
Connect and verify before preparing the write
For the terminal route, install the current GrowthBook CLI, configure a Personal Access Token through its interactive setup, and inspect the available command syntax:
Check that the active server and identity are the ones you intend to use. Prefer a tested, pinned CLI version in repeatable automation. Don't include a live token in a command you plan to share.
For an MCP route, the GrowthBook Cloud setup uses the hosted endpoint and browser OAuth. In a client using the documented mcpServers configuration shape, such as Cursor, add:
Complete the browser authorization and confirm tool discovery. Other clients can use a different configuration shape. Self-hosted GrowthBook users should follow the local stdio or self-hosted HTTP instructions rather than point a private instance at the Cloud service.
Next, let the connected agent discover the flag-creation workflow. These are tool-call parameters, shown separately in the order to invoke them:
The returned guidance should describe the prerequisites for creating the flag. If the workflow isn't present, resolve the server's bundled skill version before substituting instructions from an older integration.
Resolve real IDs and construct one payload
The flag-creation workflow checks existing keys, lists projects, and resolves environments. In the MCP client, invoke the read tool with each of these argument objects:
For the CLI route, perform that discovery with the corresponding command groups documented in the CLI command catalog:
Inspect each command's help if your installed version differs. The returned keys, project IDs, and environment IDs are the inputs to the payload below. In MCP, the equivalent read-tool arguments are:
Verify that checkout-redesign isn't taken, identify the sandbox project's ID, and enumerate every actual environment. These read responses determine the payload; don't treat example environment names as a complete inventory of your organization.
Save the following as flag.json, replacing <PROJECT_ID> and the environment keys with the discovered values. This example assumes the only environments are staging and production:
The API represents defaultValue as a string, even for a Boolean flag. Explicitly disabling each environment avoids relying on the organization's default state for newly created flags. Use a PAT for this example so the caller has a user identity for attribution.
Review the permanent key, project, value, and disabled states before submitting. If the intended job is an experiment, route to experiment design first instead of creating an unrelated flag and hoping to connect it later.
Submit once, then read the object back
With the CLI, the create command accepts a JSON body. In a Bash-compatible shell, inspect the request with dry run before submitting:
The first command previews a request; it doesn't prove the server will accept it. Run the second only after checking the preview and approving this creation. In PowerShell, use that shell's file-reading syntax rather than paste the Bash substitution unchanged.
The PowerShell equivalent, using the same reviewed file, is:
The same distinction applies: preview first, approve the specific creation, then submit. Neither command changes the application's deployed code.
Through MCP, call growthbook_api_write with the same payload serialized into its body string. Substitute the same resolved project and environment names:
Choose one creation route. Don't run both against the same key; they target the same product object, and the second creation should encounter an existing key.
Verify through the CLI get command:
Or invoke growthbook_api_read with:
Check the returned key, project, value type, default value, disabled state of every environment, and absence of unintended rules. Report those facts rather than “success.” No targeting or exposure change is included in this example.
Choose the route that fits the runtime
Once the procedure and API operation are shared, the remaining choice is operational. Where does the agent run? What can it execute? How does it authenticate? What does your team need to inspect afterward?
Shell access favors terminal composition
A shell-capable agent can run the CLI alongside repository tools, save structured output, and combine product inspection with local checks. That is useful when the job includes code changes, generated files, or a reproducible script.
For example, a flag audit can produce an inventory file while an engineer separately checks code references. The agent should preserve the association between the GrowthBook key and each repository reference. The final artifact can include both the product state and the proposed code diff.
The shell makes composition available; it doesn't make a workflow safe by default. A script can select the wrong profile or swallow an error. Decide which failures stop the workflow, where outputs are stored, and which steps need a reviewable artifact.
Remote tool access favors MCP clients
MCP is useful when the host already provides a tool connection but doesn't expose a terminal. A PM can inspect an experiment through a compatible assistant without installing a product CLI inside that assistant's runtime.
The MCP authorization specification defines an authorization framework for HTTP transports. GrowthBook Cloud uses browser OAuth, while local stdio configurations use credentials supplied to the process. Those are distinct deployment paths.
MCP also gives hosts a structured description of tools. The tool specification includes annotations such as read-only and destructive hints. Hosts can use those signals in their interaction design, but the signals are not an access-control boundary. The product must still enforce the caller's actual permissions.
Use a small decision table
| Your immediate requirement | Route to consider | What to verify |
|---|---|---|
| Combine product operations with local files and scripts | CLI plus maintained skills | Shell access, profile, installed version, structured output |
| Connect a compatible assistant without a shell | Hosted MCP | HTTP/OAuth support, identity, tools, bundled workflow |
| Keep requests inside a self-hosted deployment | CLI or self-hosted MCP | Network reachability, server version, credentials, trusted app origin |
| Reuse a procedure across different team clients | Shared skills through either route | Workflow revision, API compatibility, expected review artifact |
You can support more than one row. An engineer's CLI and a PM's MCP connection can operate on the same objects with different identities and permissions. Standardizing the workflow is often more useful than standardizing everyone's client.
Compare actual outcomes, not tool counts
Four MCP tools can address a large API. A large CLI command catalog can expose the same operations. Counting entries tells you little about whether an agent completed the intended job correctly.
Likewise, there is no universal token or speed result implied by the architecture. The client, model, instructions, response size, pagination, and number of requests all affect a real run. Measure the work you need instead of importing a result from an unrelated integration.
Keep the evaluation task constant
Use one sandbox project and one bounded task. For a read test, ask each route to find a named flag and identify its enabled environments, current revision, and relevant dependencies. For a write test, use separate temporary keys with the same reviewed disabled-flag requirements.
Give both runs the same expected output. Record whether each identified the correct objects, loaded the relevant workflow, handled missing data, and preserved the boundary between creation and exposure. Measure elapsed time and context consumption only alongside those correctness checks.
If one run gets rich workflow guidance and the other gets only a command name, you're testing the guidance as well as the interface. That can still be a useful product evaluation, but describe what changed.
Test failure behavior deliberately
Try a nonexistent object, an existing key, and a credential without the required permission. The agent should identify the failure and stop the dependent action. It should not invent an ID, silently pick another environment, or reinterpret a permission error as permission to retry with a broader credential.
The revision workflow adds another important test: multiple open drafts. When a chain of changes belongs in one revision, the agent must preserve its version rather than repeatedly ask for whichever draft is latest.
A creation receipt and a publication receipt are different artifacts. If your evaluation only checks that a request returned without error, it can miss the point where a proposed change became live or where a draft failed to include the intended edits.
Record the versions behind the result
Capture the CLI version, server version when available, client configuration, and the skill text or revision used. With MCP, inspect the guidance actually returned by the connected server. With installed skills, inspect the local instructions actually loaded.
That record makes a later regression diagnosable. A changed result may come from new guidance, a client update, a changed API, or different product state. “MCP became worse” is too broad to help you fix it.
Build workflows that survive the next client
The most useful consequence of GrowthBook's design is organizational. Engineers, PMs, and analysts can choose the agent surface that fits their work while referring to the same flags, experiments, and metrics.
An engineer can prepare a disabled flag from a terminal. A reviewer can inspect its draft in the app. An assistant connected through MCP can retrieve the same object and explain its state. The handoff should carry an ID, a revision where applicable, the proposed behavior, and the decision still required.
This is where the CLI-versus-MCP question loses much of its importance. Your release procedure doesn't need to be rewritten around each client. The product holds state; the API exposes operations; skills explain the work; the chosen interface delivers access.
Start with one inspection or sandbox creation. Require the agent to return a concrete object and its verified state. Then add the next workflow only when the current one is reliable enough for your team to review.
GrowthBook's feature flag platform gives that work a common place to live. The durable choice is how you connect release control and measurement to your team's process. CLI and MCP are ways to carry that process into the tools people already use.
Start with a common foundation
Set up GrowthBook for the flags, experiments, and metrics your agents will share. Then choose the CLI or MCP connection that fits each team member's work.
Get Started With GrowthBook
Standard deviation vs variance: Key differences for A/B testers
A metric can have wide user-to-user variation and still yield a precise experiment estimate.
That is why confusing standard deviation with standard error creates bad A/B testing decisions. A dashboard may show that users differ substantially while the average effect is estimated closely enough to guide a release. Those statements can both be true.
Variance and standard deviation describe spread. Variance works on a squared scale; standard deviation takes its square root to return to the metric's original units. Neither is automatically the uncertainty around an experiment result.
For A/B testers, the useful distinction is practical: use standard deviation to communicate outcome variability, use variance in the calculations that combine uncertainty, and inspect the standard error or interval for the treatment effect. Start by making sure all three describe the correct unit of analysis.
Understand the difference in units
Variance uses squared distances
Outcomes above and below a mean have deviations with opposite signs. Simply averaging those signed deviations gives zero. Variance avoids that cancellation by squaring each deviation before combining them.
The sample variance formula is s² = Σ(xᵢ − x̄)² / (n − 1), where x̄ is the sample mean and n is the number of observations. The symbol Σ means to add the terms across observations.
If the outcome is measured in seconds, variance is measured in seconds squared. That is mathematically useful but less intuitive for explaining a product experience. “Variance is 4 seconds squared” does not mean users differ by 4 seconds.
Standard deviation returns to the original scale
Standard deviation is s = √s². A variance of 4 seconds squared corresponds to a standard deviation of 2 seconds. Both describe the same dispersion, expressed on different scales.
Do not define standard deviation as the average absolute distance from the mean. That is a different measure. Squaring deviations gives large deviations more influence, and taking the square root afterward does not undo that weighting.
| Quantity | What it describes | Units for task time |
|---|---|---|
| Variance | Squared dispersion in outcomes | Seconds squared |
| Standard deviation | Outcome dispersion on the original scale | Seconds |
| Standard error of the mean | Sampling variability of an estimated mean | Seconds |
Neither variance nor standard deviation requires a normally distributed population merely to be calculated. Normality matters for particular interpretations and inferential procedures, not for the existence of these summaries when the required moments are finite.
Calculate both from the same observations
Work through a small dataset
Use four synthetic task times: 2, 4, 4, and 6 seconds. This tiny dataset is for arithmetic, not evidence about a real product or a basis for confident inference.
The mean is (2 + 4 + 4 + 6) / 4 = 4 seconds. Deviations from the mean are −2, 0, 0, and 2 seconds. Their squares are 4, 0, 0, and 4 seconds squared, totaling 8.
Treating these observations as a sample, the usual sample variance is 8 / (4 − 1) = 8/3, or about 2.667 seconds squared. The sample standard deviation is √(8/3), or about 1.633 seconds.
This is a useful check on a report. If someone describes the same sample's variance as 1.633, they have probably substituted the standard deviation. If the units are missing, the mistake is much harder to spot.
Choose sample or population formulas deliberately
If those four observations constitute the entire finite population you want to describe, population variance divides by 4: 8/4 = 2 seconds squared. Population standard deviation is √2, or about 1.414 seconds.
The population and sample distinction depends on what you are estimating. Having every recorded event from an experiment does not necessarily make those events the full population of future users or future conditions you care about.
The n − 1 denominator makes the usual sample variance an unbiased estimator of population variance under independent, identically distributed sampling with finite variance. It does not make the square root exactly unbiased for population standard deviation. Nor does it correct biased sampling or dependence between observations.
Verify the calculation with explicit function names
Python's statistics functions distinguish sample calculations from population calculations. With Python 3 installed, this standalone example requires no external packages or data:
The first line of output is 2.667 1.633; the second is 2 1.414. The different denominators explain the difference. Before comparing statistics from two tools, check their sample/population defaults and handling of missing values.
Find the noise you can reduce
Explore practical variance reduction methods, including metric design and CUPED, and the conditions that make each approach useful.
Read the Variance Reduction Guide
Use standard errors for experiment uncertainty
Distinguish users from estimated means
Standard deviation describes how users' outcomes vary. Standard error describes how an estimator would vary across repeated samples. For an independent sample, the estimated standard error of its mean is s / √n.
Suppose a user-level task-time metric has a sample standard deviation of 20 seconds among 10,000 independent users. The estimated standard error of the mean is 20/√10,000 = 0.2 seconds. Large differences between users coexist with a much smaller uncertainty around their average.
Collecting four times as many independent users would halve this standard error if the outcome variance stayed similar. It would not mechanically halve user-level standard deviation. The individual experience remains variable even as the estimated mean becomes more precise.
Combine arm variances before taking the square root
For independent treatment and control groups, a simple unadjusted difference in means has estimated variance s²ₜ/nₜ + s²꜀/n꜀. Its standard error is the square root of that sum. You do not add the two arm standard deviations and call the result the experiment's uncertainty.
Take a separate synthetic summary with 10,000 independent users in each arm and a 20-second standard deviation in both. The variance of the estimated difference is 400/10,000 + 400/10,000 = 0.08 seconds squared. Its standard error is √0.08, about 0.283 seconds.
If the observed treatment-minus-control difference is −1 second, an illustrative large-sample, unadjusted 95% normal interval is −1 ± 1.96 × 0.283, approximately −1.554 to −0.446 seconds. Negative values mean faster completion under this metric definition.
This example assumes independent groups and an appropriate large-sample approximation. It is not the exact output of GrowthBook's relative-lift, sequential, covariate-adjusted, or ratio-metric methods. GrowthBook's statistical details describe how the platform estimates effects and their uncertainty.
Avoid interpreting intervals as outcome ranges
That interval concerns the mean treatment effect. It does not say that 95% of individual users improved by between 0.446 and 1.554 seconds. An individual user's causal effect is a different quantity, and the experiment does not observe both potential outcomes for the same person.
Likewise, mean ± one standard deviation is not a universal statement about where 68% of users fall. That familiar interpretation depends on an approximately normal distribution. Skewed task times or revenue can look very different.
The frequentist confidence level describes the repeated-sampling coverage of the interval procedure under its assumptions. It is not a posterior probability assigned to a fixed realized interval.
Check what your metric actually measures
Match the variance to the randomization unit
If users are randomized and each contributes many requests, request rows are not automatically independent experimental units. Treating 100 requests from one person as 100 unrelated users can make uncertainty look smaller than it should.
Define the outcome at the intended unit or use a method that accounts for clustering. GrowthBook's aggregate data requirements are relevant when supplying precomputed summaries: the sample size, sum, and dispersion must correspond to the statistic and analysis method.
Two teams can report different standard deviations without either making an arithmetic mistake. One may summarize per-request latency while the other summarizes each user's average latency. They are describing different outcome distributions. Clarify the grain before comparing their numbers.
Paired measurements need another distinction. If the same users provide before-and-after outcomes, the measurements within each pair can be correlated. Analyze each user's difference or use an appropriate paired method; do not apply the independent-arm formula automatically. The paired-means framework treats those differences as the observations of interest. Pairing also does not remove time trends or establish causality by itself; the design still determines which comparison is defensible.
Inspect tails instead of deleting them casually
Large observations influence both variance and standard deviation. A few unusually long tasks can dominate a time metric; a few large purchasers can dominate revenue. Determine whether those values reflect valid behavior, instrumentation defects, retries, or unit errors.
A valid expensive user is not automatically an outlier to discard. Removing high values after seeing an experiment result can change the estimand and introduce selective analysis. Decide justified exclusions or transformations before interpreting the test.
Report complementary summaries when they answer useful questions: a histogram, median, or a relevant tail percentile can show what a single spread measure hides. Those summaries do not replace the mean effect if the mean remains the decision metric.
Keep units consistent across pipelines
Converting seconds to milliseconds multiplies every value by 1,000. Standard deviation also multiplies by 1,000, but variance multiplies by 1,000,000. A dramatic numerical variance increase can therefore be a unit conversion rather than a product regression.
For the four-value example, sample standard deviation becomes approximately 1,633 milliseconds and variance becomes approximately 2,666,667 milliseconds squared. The underlying task times have not changed. Store units with metric definitions and inspect them when comparing dashboards.
A lower variance caused solely by rescaling a metric does not create more statistical information. The effect size rescales too. Separate a numerical change in units from an actual improvement in the precision of your estimate.
Also avoid averaging standard deviations from daily reports and labeling the result a weekly standard deviation. The complete-period spread depends on both variation within days and differences between daily means. Keep the underlying observations or sufficient summaries that support the intended calculation. A simple average of displayed daily values generally loses information needed to reconstruct the full distribution.
Turn spread into a better testing plan
Define the outcome before planning the sample
Start with the metric definition: population, identity, aggregation, window, units, exclusions, and desired effect. Estimate variability from representative historical data at that same grain.
Historical variance is an input, not a guarantee. New traffic, a different user mix, or a changed measurement window can alter it. Explore plausible variance and effect-size assumptions rather than treating one historical number as exact.
Keep the two arm variances separate when appropriate. A treatment can change variability as well as its mean. The simple equal-standard-deviation illustration above is a teaching assumption, not a requirement that production groups have identical spread.
Reduce estimation noise without hiding behavior
CUPED and post-stratification use suitable pre-experiment information and covariates to improve effect estimation. CUPED stands for Controlled-experiment Using Pre-Experiment Data. Its value depends on useful predictive information and a valid implementation.
Adjustment can reduce uncertainty in the treatment effect without making real users' outcomes less variable. That distinction prevents a reporting error: a narrower adjusted interval is not evidence that customers now experience a more consistent product.
Also separate missing data from genuine zeros. A user with no purchase may correctly contribute zero revenue; a missing instrumentation event might represent an unknown outcome. Treating both identically can distort the mean and its variance before any statistical engine sees the data.
Report the right quantity for the decision
| Reader's question | Useful quantity |
|---|---|
| How much do individual outcomes differ? | Standard deviation, with distribution context |
| What enters this uncertainty calculation? | Variance at the correct analysis unit |
| How precisely is the effect estimated? | Standard error and an appropriate interval |
| Is the measured difference worth acting on? | Effect size, uncertainty, guardrails, and decision context |
For a clear experiment readout, show the metric's unit and analysis grain, the estimated effect, and its appropriate uncertainty interval. Add outcome spread when it helps readers understand the population or diagnose a noisy metric.
GrowthBook's experimentation platform brings defined metrics and statistical analysis together. A trustworthy interpretation still starts with your measurement choices: describe spread with standard deviation, calculate with variance where required, and judge precision using the estimator's uncertainty.
Build a clearer experiment readout
Connect your metrics to GrowthBook and evaluate effects with the analysis method and measurement definitions your team has chosen.
Get Started With Experimentation
Catching 80 defects means little until you know how many defects existed.
Suppose a detector flags 125 suspicious software changes. Engineers inspect them and confirm 80 real defects. That tells you something about alert quality, but it does not tell you how many defects the detector missed. To calculate true positive rate, you need the actual positives outside the alert queue too.
True positive rate, or TPR, measures the share of actual positive cases that a system correctly identifies. It is also called recall or sensitivity. The formula is TP / (TP + FN), where TP means true positives and FN means false negatives.
The arithmetic is simple. Choosing the right population, labels, threshold, and denominator is the difficult part. This guide works through an engineering example, distinguishes neighboring metrics, and explains what changes when the “test” is an A/B experiment rather than an item classifier.
Start with actual positives, not alerts
Define the positive class
“Positive” means the condition you want to detect. It does not necessarily mean a good outcome. In a defect detector, positive means a real defect. In a spam filter, it means spam. Write that definition before calculating anything.
A true positive is an item that is actually positive and predicted positive. A false negative is actually positive but predicted negative. Those two groups together form the denominator: all actual positives in the evaluation sample.
The recall definition is therefore conditional on actual class membership. TPR asks, “Among the real defects, what proportion did we catch?” It does not ask, “Among the alerts, what proportion were real?”
Fix the unit and observation window
Choose a consistent evaluation unit. A code change, an individual finding, and a repository are different units. If one change generates 12 duplicate alerts, counting alerts in the numerator and changes in the denominator produces a meaningless rate.
For the example, each unit is one code change, evaluated once by one fixed detector version at one fixed threshold. An independent review process eventually labels each change as defective or not defective. The observation window is long enough for those labels to mature.
This resembles the discipline needed when defining experiment metrics: the event, identity, time window, and denominator must describe the same question. A correct formula applied to incompatible counts still gives the wrong answer.
Calculate TPR from the confusion matrix
Put every evaluated item in one cell
Consider an original synthetic evaluation of 1,000 code changes. Reviewers establish that 100 contain a defect and 900 do not. The detector catches 80 defects, misses 20, and incorrectly flags 45 nondefective changes.
| Actual label | Predicted positive | Predicted negative | Total |
|---|---|---|---|
| Defective | 80 true positives | 20 false negatives | 100 |
| Not defective | 45 false positives | 855 true negatives | 900 |
| Total | 125 | 875 | 1,000 |
Read the labels rather than relying on cell position. Libraries and reports can arrange axes differently. The scikit-learn confusion matrix convention uses actual classes on rows and predicted classes on columns, with class order also affecting where each value appears.
Check the totals before calculating rates. The four cells sum to 1,000. True positives plus false negatives sum to 100 actual defects. Predicted positives total 125, which is a different denominator.
Divide detected positives by all actual positives
The calculation is:
TPR = TP / (TP + FN) = 80 / (80 + 20) = 80 / 100 = 0.80 = 80%.
Interpret it in words: the detector caught 80% of the actual defects in this evaluation sample. It missed 20%, so the false negative rate is 20%. For a binary task with complete labels, TPR and false negative rate sum to 100%.
Do not divide 80 by 1,000. That would measure true positives as a share of all evaluated changes, not recall. Do not divide 80 by 125 either. That is precision, which answers the alert-quality question.
The result describes this sample and threshold. It does not establish that every future workload will have the same recall, or that all defect categories are equally detectable. Keep the counts beside the percentage so readers can see the support behind it.
Handle zero positive support explicitly
If the sample contains no actual defects, TP + FN equals zero. Sample TPR is undefined. Returning 0% would imply the detector missed known defects; returning 100% would imply it successfully found them. Neither interpretation is supported.
Software libraries need a practical zero-division convention. That convention can return a placeholder or warning, but it does not create evidence. Report “TPR unavailable: zero actual positives” and preserve the support count. A pipeline can then distinguish an unevaluable slice from measured poor performance.
Check the experiment error risks
See how power, repeated checks, and multiple comparisons affect the reliability of experimental decisions beyond a single classifier score.
Read the False Positives GuideCompare TPR with precision and false positives
Keep the denominators visible
For the same 1,000 changes, the other common metrics tell different stories:
| Metric | Calculation | Result |
|---|---|---|
| True positive rate / recall | 80 / 100 actual positives | 80% |
| Precision | 80 / 125 predicted positives | 64% |
| False positive rate | 45 / 900 actual negatives | 5% |
| Specificity | 855 / 900 actual negatives | 95% |
| Accuracy | (80 + 855) / 1,000 changes | 93.5% |
The classification metric definitions differ because they condition on different populations. Here, 36% of alerts are false, but the false positive rate is 5%. Calling both numbers “the false positive rate” hides the operational distinction.
Accuracy can also look reassuring in an imbalanced population. A detector that labels all 1,000 changes nondefective would be 90% accurate while detecting none of the 100 defects. TPR exposes that failure, though it still needs the false-positive side of the picture.
Choose a threshold for the decision
Suppose a stricter threshold, evaluated on the same labeled changes, catches 60 defects and incorrectly flags 10 nondefective changes. Its TPR is 60%, its precision is about 85.7%, and its false positive rate is about 1.11%.
This threshold reduces the review queue from 125 changes to 70. It also misses 40 defects instead of 20. Whether that tradeoff is acceptable depends on the consequences of a missed defect, the review team's capacity, and what action follows an alert.
A ROC curve plots TPR against false positive rate across thresholds. It helps compare operating points. It does not pick the operational threshold for you, and an area-under-curve summary does not specify the queue size at the threshold you intend to deploy.
If you tune thresholds using the same labels you later report as a final evaluation, the score can be optimistic. Use separate validation and held-out evaluation data, or another defensible evaluation design, and freeze the decision rule before the final assessment.
Specify the action at each threshold too. A low-confidence finding might create a suggestion, while a high-confidence finding might block a merge. Those are different interventions with different error costs. One aggregate recall figure should not imply that every detected item received the same treatment.
Make sure the labels can support the claim
Audit predicted negatives too
If engineers review only alerts, you can estimate precision within the reviewed alert population. You cannot directly count missed defects among unreviewed negatives. Assuming those changes are clean makes FN artificially small and TPR artificially high.
An evaluation may need independent review of a representative sample of predicted negatives, delayed incident labels, or a curated benchmark with known defects. Each approach has limits. Incident labels miss undetected failures; curated examples can differ from ordinary production work.
If the evaluation oversamples suspicious changes or certain repositories, report that design. Use appropriate weighting when estimating performance for the real workload. A simple unweighted confusion matrix describes the sampled dataset, which may not represent the deployed population.
Preserve uncertainty and segment support
An 80% estimate based on 4 detected defects out of 5 is much less precise than one based on 800 out of 1,000. The important sample size for recall is the count of actual positives, not just the total number of changes evaluated.
Report positive support by important slices, such as language, defect category, or repository type. A high pooled TPR can hide weak performance on a rare but expensive failure mode. At the same time, avoid declaring tiny slices definitively better or worse from a few examples.
Counts alone do not make observations independent. Many similar changes from one repository can share a common pattern. Use uncertainty methods that fit the evaluation design rather than assuming every row is an unrelated trial.
Separate label delay from label absence
A recently deployed change might appear nondefective because nobody has had time to find the bug. Treating it as a confirmed negative gives the newest cohort a different labeling standard from older cohorts.
Define when an item becomes eligible for evaluation and track unresolved labels separately. Keep model version, threshold, label version, and observation dates with the result. That makes a later score change interpretable: you can tell whether the detector changed, the workload changed, or the labels matured.
For a warehouse-native measurement workflow, these definitions belong in the underlying data model. Joining model decisions to review outcomes still requires identity and time alignment; a queryable warehouse does not automatically resolve ambiguous ground truth.
Keep comparable populations together
When comparing detector versions, evaluate them on the same held-out cases where possible. Otherwise, an apparent recall improvement could come from an easier mix of defects. Preserve a breakdown of new, removed, and unchanged cases so reviewers can see what changed between evaluations.
When combining reports, add the underlying TP and FN counts for nonoverlapping, consistently defined populations, then calculate the pooled rate. An unweighted average of repository percentages answers a different question because it gives a tiny repository the same influence as a large one. Either summary can be useful, but label the weighting explicitly.
If the same change appears in several reports, deduplicate or model that repeated observation before pooling. Repeated scans do not create additional independent defects. Keep an audit trail from the final counts back to the evaluated item identifiers so a suspicious rate can be investigated.
Connect detection quality to product decisions
Distinguish classifier recall from statistical power
In hypothesis testing, statistical power is the probability that a testing procedure rejects the null when a specified alternative is true. It is a repeated-test detection probability, depending on assumptions such as effect size, variance, sample size, and decision rule.
That is related to the idea of detecting a real signal, but it is not the same dataset as a labeled-item confusion matrix. In an ordinary A/B program, you do not observe the true causal effect of every experiment with certainty. Dividing declared winners by all experiments does not reveal a ground-truth TPR.
Power planning asks how often your procedure would detect a particular real effect under its assumptions. A classifier evaluation asks how many labeled positives were detected in the evaluated sample. Keep those questions separate when discussing GrowthBook's statistical methods.
Evaluate the intervention as well as the detector
An improved offline detector can still create a worse product workflow. More alerts might overwhelm engineers, delay releases, or catch defects too late to matter. Conversely, a modest recall change might be valuable if it catches the failures with the highest operational cost.
When appropriate, use randomized experiments to evaluate the downstream intervention: which review experience helps teams resolve defects, with what delay and cost? Keep detector recall, review workload, and product outcomes as separate measures.
GrowthBook's experimentation platform can analyze defined outcomes from such tests. Your team supplies reliable assignment, exposure, labels, and metric definitions. Neither a high TPR nor a significant product result eliminates the need to inspect the other evidence.
Use a repeatable reporting checklist
Before sharing a TPR number, state the positive class, evaluation unit, detector version, threshold, and label process. Show TP and FN beside the rate. Include precision or false positive rate so readers can assess the workload created by detection.
Then ask whether predicted negatives were evaluated, whether labels had time to mature, and whether the sample reflects the population where you will act. Mark zero-support slices unavailable and explain material gaps in coverage.
The useful result is a statement someone can audit: “At this threshold, the detector found 80 of 100 labeled defects in these 1,000 evaluated changes, while producing 45 false alerts.” That is a clearer basis for a decision than “our model is 80% accurate.”
Measure what the change delivers
Connect your data and defined metrics to GrowthBook to evaluate how a product or workflow change affects the outcomes your team cares about.
Start Measuring With GrowthBookReady to ship faster?
No credit card required. Start with feature flags, experimentation, and product analytics—free.
