Experiments

Data Science

Experimenting with different LLM providers (OpenAI vs Anthropic vs OSS)

Jun 5, 2026

min read

A graphic of a bar chart with an arrow pointing upward.

Picking an LLM provider based on leaderboard rankings is like hiring a chef because they scored well on a written exam.

The benchmarks are real, the scores are carefully measured, and they tell you almost nothing about whether that model will perform well on your specific product with your actual users. The only reliable way to make this decision is to run controlled experiments on your own data — the same way you'd test any product change that affects user outcomes.

This guide is for engineers and product teams who are actively building LLM-powered features and need a defensible process for choosing between OpenAI, Anthropic, and open-source alternatives. Here's what you'll learn:

Why static benchmarks fail to predict production behavior — and what to use instead
The five dimensions every LLM provider comparison needs to measure: quality, latency, cost, reliability, and safety
How to structure a controlled experiment with proper randomization, confound controls, and metric instrumentation
How to navigate the cost-quality trade-off between frontier models and cheaper open-source options
How to turn experiment results into a production strategy with gradual rollouts, fallback logic, and multi-provider routing

The article moves in order from measurement framework to experiment design to deployment strategy. Each section builds on the last, so by the end you'll have a complete process — not just a checklist of things to consider.

Why static benchmarks don't tell you which LLM provider is right for your product

If you've spent any time evaluating LLM providers, you've almost certainly started with the leaderboards. Artificial Analysis, the Hugging Face Open LLM Leaderboard, Vellum — they're authoritative-looking, regularly updated, and full of the kind of quantitative rigor that makes a decision feel grounded.

And then you pick the top-ranked model, integrate it into your product, and something feels off. The responses are technically correct but tonally wrong. Latency is fine in testing but unpredictable under load. Users aren't engaging the way you expected. The benchmark said this model was best. What happened?

What happened is that benchmarks were never measuring what you actually needed to know.

Benchmarks measure narrow correctness, not production fit

Public leaderboards work by running models against standardized inputs, scoring outputs against predefined criteria, and aggregating the results into a ranking. The benchmarks themselves — GPQA Diamond for graduate-level reasoning, MMLU for broad knowledge coverage, AIME for mathematical problem-solving, SWE-Bench for software engineering tasks — are carefully constructed and genuinely useful for what they measure. The problem is what they measure.

As the evaluation team at HoneyHive puts it directly: "A benchmark with general scope but uncertain validity is fundamentally useless." Benchmarks deliver real signal only for narrow, objectively measurable tasks — the canonical example is SQL query generation against a predefined schema, where correctness is binary and the task definition is precise.

The moment you move to open-ended, context-dependent, user-facing tasks, the apples-to-apples comparison that makes benchmarks valid starts to break down. Your production prompt distribution doesn't look like the benchmark inputs. Your definition of a good response doesn't map to the scoring rubric. And aggregated scores hide variance on the specific subtasks that actually matter to your product.

Even the leaderboard providers acknowledge this. Prompts.ai notes explicitly that LLM comparison tools "lack task-specific insights" — a concession worth sitting with, given that task-specific performance is precisely what you're trying to evaluate.

The gap between rankings and production behavior

Here's the pattern that should make you skeptical: model providers report consistently climbing benchmark scores across every major leaderboard, yet practitioners building production systems report that real-world utility improvements feel much slower. The benchmarks say the models are getting dramatically better. The engineers shipping those models into products have a more complicated story to tell.

The structural reason for this gap is that benchmark evaluation criteria don't reflect user satisfaction. A model can score well on coherence metrics while producing responses that feel robotic to your users. It can ace factual recall benchmarks while failing at the specific domain knowledge your use case requires.

Artificial Analysis offers coherence and tone metrics alongside its performance data, which is more useful than raw accuracy scores — but still comes with "limited customization" for specific workflows. No leaderboard can capture whether a model's response style fits your brand voice, whether its refusal behavior is calibrated to your risk tolerance, or whether it handles the edge cases your users actually encounter.

Why task-specific experiments are the only reliable signal

The practitioner community has largely converged on this conclusion from experience. The team behind promptfoo, a benchmark harness built specifically because general benchmarks were insufficient, puts it plainly: test models on your own data and examples rather than extrapolating from general benchmarks. That recommendation comes from building evaluation infrastructure, not from theory.

The implication is methodological. If you want to know which LLM provider performs best on your customer support chatbot, your code review tool, or your document summarization pipeline, you need to run controlled experiments on your own data, with your own prompts, measuring outcomes that reflect what your users actually care about.

Benchmark rankings are a reasonable way to narrow the field — they can tell you which models are worth including in your evaluation. But the decision itself requires a different methodology entirely: structured experimentation with statistical rigor, the same kind you'd apply to any product change with measurable user impact.

This guide is for engineers, PMs, and data scientists who are building LLM-powered features and need a defensible process for choosing between providers — not a benchmark lookup, but a controlled experiment with statistical rigor. By the end, you'll have a complete framework for designing provider comparisons, instrumenting them correctly, and translating results into a production routing strategy.

Here's what the guide covers:

The five dimensions to measure — quality, latency, cost, reliability, and safety — and what to instrument for each
How to structure a controlled experiment — randomization, confound prevention, and the inference log fields that make downstream analysis possible
The cost-quality trade-off — a 300x price gap that requires experimental validation, not assumptions
Gradual rollouts, fallbacks, and multi-provider routing — how to move from experiment results to a production provider strategy
A practical starting roadmap — what to do first, and the three infrastructure prerequisites that determine whether your data is trustworthy

The five dimensions you need to measure in any LLM provider comparison: quality, latency, cost, reliability, and safety

Before you run a single experiment comparing OpenAI, Anthropic, or an open-source alternative, you need to decide what you're actually measuring. This sounds obvious, but most teams skip it. They run a few prompts, read the outputs, and form an impression.

AWS's ML team calls this "vibes-based evaluation," and they're blunt about why it fails: subjective bias causes evaluators to favor confident-sounding outputs over accurate ones, limited sample coverage misses edge cases entirely, and inconsistency means two engineers reviewing the same outputs will reach different conclusions based on whether they weight brevity or factual detail. The result is experiment data you can't act on.

The five dimensions below give you a measurement vocabulary before you instrument anything. Each one requires a specific metric, not a general impression.

Quality: relevance, faithfulness, and task accuracy

Quality is the dimension teams think they're measuring when they're actually just pattern-matching on style. The three metrics that matter are output relevance (does the response address what was asked?), faithfulness (are factual claims accurate and grounded in the provided context?), and bias (does the model systematically skew outputs in ways that affect your use case?).

These need to be defined against your specific task — a customer support bot and a code generation tool require entirely different quality rubrics. Throughput without quality degradation is also part of this picture: a model that produces excellent outputs at low volume but degrades under load has a quality problem, not just a performance problem.

What to instrument: a task-specific scoring rubric applied to a representative sample of outputs, ideally with both automated scoring (LLM-as-judge or rule-based checks) and a human review set for calibration.

Latency: average response time and variance

Average latency is the metric everyone tracks. Latency variance is the one that actually breaks user experiences. Cloud-based LLMs have throttling limits that can't be removed even at the highest account tier — this creates unpredictable spikes that averages obscure entirely.

A provider with a 400ms median response time but a 95th-percentile of 4 seconds will feel broken to users even though the average looks acceptable. Artificial Analysis publishes latency data showing that the same underlying model can behave significantly differently across API providers — the model version alone doesn't determine your latency profile.

Metrics to track: p50, p95, and p99 latency per request, measured at the API call level, segmented by prompt length and time of day.

Cost: token economics and per-task spend

Token pricing is the headline number, but per-task cost is what your finance team will eventually ask about. Input and output tokens are priced separately, and prompt engineering choices that improve quality often increase token count — meaning cost and quality are coupled in ways that per-token pricing hides.

For on-premise or self-hosted open-source deployments, the cost model shifts entirely to infrastructure: GPU hours, memory, and serving overhead replace token fees.

What to log: total tokens consumed per task type (not per request), cost per successful completion, and cost variance across prompt template versions.

Reliability: uptime, error rates, and throttling behavior

Reliability under load is where provider differences become operationally significant. The relevant metrics are error rate (5xx responses, timeouts, and malformed outputs), throttling frequency (how often requests are rate-limited and at what volume thresholds), and quality consistency under sustained load.

A provider that performs well in a 100-request test may degrade measurably at production scale. Without systematic tracking, these degradations are invisible until they become incidents.

What to track: error rate by error type, rate-limit events per hour, and a quality sample drawn specifically from high-traffic periods rather than off-peak testing windows.

Safety: unsafe output rates, bias, and compliance behavior

Safety is the dimension most consistently missed by informal evaluation. Vibes-based testing rarely surfaces unsafe outputs because reviewers aren't specifically probing for them. The measurable signals here are unsafe output rate (responses that violate your content policy, measured against a defined test set of adversarial prompts), bias rates on demographic or sensitive topics, and compliance behavior — providers differ meaningfully in how they handle privacy-sensitive inputs and geopolitically contested content.

These differences matter for regulated industries and global products in ways that benchmark scores don't capture.

What to define: a predefined safety test suite run against each provider variant, with pass/fail criteria established before the experiment begins — not after you've seen the results.

Defining these metrics upfront does more than make your experiment interpretable. It forces the question of which dimensions actually matter for your use case, which is the prerequisite for any cost-quality trade-off decision.

A warehouse-native experiment platform makes it practical to track all five dimensions within a single experiment framework — so quality scores, latency percentiles, cost-per-request, error rates, and safety flag rates are all queryable against the same assignment data without routing inference logs through a third party.

Structuring a controlled experiment that produces defensible provider comparisons

The statistical principles behind A/B testing UI changes apply directly to provider comparisons — and most of the feature flag and analytics infrastructure you already have for product experimentation can be repurposed for this. The differences are specific: LLM experiments have a non-obvious randomization choice (per-request versus per-user assignment) that doesn't come up in UI testing, prompt templates need to be locked across arms in a way that CSS changes don't, and model version aliases can silently update underneath a running experiment in a way that a button color never will. The rest of this section explains each of those differences and how to control for them.

Per-request vs. per-user assignment: why the choice contaminates your metrics

The first decision is your unit of randomization: per-request, per-session, or per-user. Per-request randomization is technically simpler — each API call independently rolls a provider assignment — but it exposes the same user to inconsistent experiences within a single session, which contaminates any downstream metric that depends on user perception (satisfaction ratings, follow-up queries, task completion). Per-user assignment is cleaner for measuring outcomes that accumulate over time, and it's what you want if retention or engagement is in your metric set.

Consistent hashing is what makes per-user assignment stable without storing anything. Your flag SDK takes the user's ID and a fixed experiment identifier, runs them through a hash function, and always produces the same output — so the same user always routes to the same provider, on every request, without a database lookup or session cookie.

If you stored assignments in a cookie instead, you'd lose them on browser clears and get mixed exposure data. If model versions or experiment parameters change mid-test, sticky bucketing preserves the original assignment so you're not inadvertently mixing treatment conditions for users who are already mid-experiment.

Before you introduce a real provider difference, run an A/A test — both arms routing to the same provider — to confirm that your traffic splitting and metric collection are working correctly. It's a low-cost validation step that catches instrumentation bugs before they corrupt your actual comparison data.

On sample size: a practical floor is 200 conversion events per variation, which means if your LLM feature has a 10% task-completion rate, you need at least 4,000 users exposed before results are interpretable. Run the experiment for at least one full weekly cycle to avoid skewing results toward atypical traffic segments.

Keep in mind that LLM model versions can change during a long experiment window — pin to specific version strings, not aliases like gpt-4 that may silently update underneath you.

Prompt consistency and confound prevention

The most common confound in LLM provider experiments isn't traffic splitting — it's prompt variation. If your system prompt, few-shot examples, or temperature settings differ across arms, you're measuring prompt quality, not provider quality. All arms must use identical prompt templates, evaluated against the same input distribution.

Beyond prompts, three other confounds deserve explicit controls. Response caching needs to be disabled or applied consistently across arms; a cache hit on one provider and a cache miss on another will produce latency differences that have nothing to do with the provider itself.

Rate-limit differences are subtler: a provider hitting rate limits under load will show artificially high latency and elevated error rates, which looks like a provider quality problem but is actually a capacity problem at your traffic volume. Model version pinning matters because Artificial Analysis tracks over 500 AI model endpoints and shows that the same underlying model can vary significantly in latency and price across different API providers — your experiment variable needs to be defined as model + API endpoint + configuration, not just the model family name.

The inference log fields that make downstream analysis possible

Log the following per inference call: provider ID, model version string, time to first token, total completion time, input token count, output token count, and a request or session ID that joins to downstream user behavior events. Cost can be calculated directly from token counts multiplied by published rates, which means your billing exposure is instrumentable from the same log rather than requiring a separate data pull.

The join between inference logs and user outcome events is where most teams underinvest. If those two data streams live in different systems, you can measure latency and cost easily but can't connect them to whether the user actually completed their task or came back the next day.

A warehouse-native approach — where your experimentation platform connects directly to your own Snowflake, BigQuery, or Redshift instance rather than routing events through a third-party SaaS — keeps inference logs and behavioral data in the same place for analysis without moving PII outside your infrastructure.

One non-obvious design decision: if users are assigned to a provider at session start but only encounter the LLM feature partway through the session, your analysis should filter to users who actually received an LLM response. Including users who were assigned but never exposed inflates noise and reduces your ability to detect real differences.

This is the activation metric pattern — define the activation event (e.g., "user submitted a query that triggered an LLM call") and use it as the filter boundary for your outcome metrics.

Average metrics can also mask differential performance across user segments. A provider that performs well on English queries may degrade significantly for non-English speakers or domain-specific inputs. Segment your results before declaring a winner.

Feature flags as zero-latency traffic routing for provider experiments

Feature flags are the cleanest mechanism for routing requests to different providers without code deploys. The flag value — something like llm_provider: "openai" | "anthropic" | "llama-3-70b" — is evaluated locally in the application using a locally cached payload, which means the routing decision adds zero latency overhead to your inference pipeline.

That matters when latency is one of your primary measurement dimensions. The application reads the flag, routes the API call to the corresponding provider, and fires a tracking event that writes the assignment to your data warehouse.

SDK-based flag evaluation adds no network call per evaluation and fires a tracking callback on each assignment exposure — meaning every time a user is assigned to a provider variant, your code logs that event to your data warehouse. That log entry is what connects "this user was assigned to Anthropic" with "this user completed their task three minutes later" when you run your analysis. Without it, you have inference logs and you have user behavior logs, but no reliable way to link them.

Sequential testing, which lets you call a winner early if one provider is clearly dominant on cost or latency, is particularly useful for LLM experiments where model versions may change if the experiment runs too long. Prerequisite flags let you gate the provider experiment behind other conditions — for example, running the comparison only for users on a paid tier where the cost difference between providers has meaningful business impact.

The result of this architecture is an experiment that produces the same kind of statistically defensible signal you'd expect from a product A/B test — not a benchmark lookup, not a manual evaluation, but a controlled comparison against the actual user population and tasks that define your product.

The cost-quality trade-off: a 300x price gap that requires experimental validation, not assumptions

The pricing gap between frontier and budget LLMs is not subtle. Claude Opus 4 costs $75 per million output tokens. Llama 4 Scout, available through inference providers, costs $0.25 per million output tokens. That's a 300x difference. Whether that gap is justified depends entirely on what you're asking the model to do — and the only reliable way to know is to run an experiment, not to assume.

The pricing landscape: a 300x gap that keeps widening

The full spectrum of LLM pricing today runs from roughly $0.07 per million input tokens at the budget floor to $75 per million output tokens at the frontier ceiling. Within that range, there are meaningful tiers. Claude Sonnet 4 sits at $3/$15 per million tokens. GPT-4.1 comes in at $2/$8. Gemini 2.5 Flash at $0.15/$0.60 represents strong mid-tier value. GPT-4.1 nano, explicitly positioned for high-volume latency-sensitive workloads, matches Llama 4 Scout at $0.10/$0.40.

The token economics described earlier become even more complex at the endpoint level. As one engineer at OpenRouter noted in a practitioner discussion, "the right lens is actually the price per token by endpoint, not by model" — because thinking modes, caching discounts, fast versus slow endpoints, and prompt-length-dependent pricing all affect what you actually pay.

A team that benchmarks Claude Opus 4 against Llama 4 Scout on headline price but ignores caching behavior may be comparing the wrong numbers.

Frontier models like Claude Opus 4 score around 95 out of 100 on composite quality evaluations. Budget models cluster in the 89–92 range.

That 6-point gap sounds modest until you consider what it represents at the tail of the distribution — the hard cases, the edge inputs, the prompts where a wrong answer has real consequences.

A task-stakes framework for model selection

The right way to think about model selection is across two axes: the stakes of a wrong answer, and the volume of requests you're running.

High-stakes, lower-volume tasks — legal contract review, medical triage, complex multi-step code generation, financial analysis — are where frontier models earn their premium. If a model error produces a materially wrong legal interpretation or a missed diagnosis, the $75/M output cost is a rounding error compared to the downstream risk. This is the quadrant where Claude Opus 4 or GPT-4.1 is the rational choice.

High-volume, low-stakes tasks — webpage summarization, content tagging, classification, FAQ responses — are where frontier pricing becomes economically unsustainable at scale. Running a million summarization requests per day through Claude Opus 4 costs $75,000 in output tokens alone. The same workload on Llama 4 Scout costs $250. If the quality difference doesn't produce a measurable difference in user outcomes, you're paying 300x for nothing.

The middle tier — GPT-4.1 mini, Gemini 2.5 Flash, Claude Haiku 3.5 — handles the large category of medium-stakes tasks where you need reliable performance but can't justify frontier pricing at volume. GPT-4.1 mini cuts cost by roughly 80% compared to GPT-4.1 with surprisingly small quality tradeoffs on structured tasks, making it a strong default for most product-facing workloads.

Volume discounts, committed-use agreements, and batch processing can reduce costs by 25–50% depending on the provider, which shifts the math further toward mid-tier and budget models for predictable workloads.

Validating the trade-off with downstream business metrics

The framework above gives you a starting hypothesis. Experiment data is what confirms or overturns it.

The right validation question isn't "which model scores higher on our internal eval?" It's "does the quality difference between models produce a measurable difference in the business metric we actually care about?" For a summarization feature, if users engage equally with Llama 4 Scout output and Claude Opus 4 output, the 300x cost premium is unjustified — full stop. For a legal review workflow, if error rates or user correction rates differ meaningfully between models, the premium is justified.

This is where the experiment infrastructure described earlier closes the loop. A warehouse-native experiment platform with read-only data access lets you join inference cost data with downstream user outcome metrics in a single pipeline — without routing sensitive data through a third party.

Metric correlation analysis is particularly useful here: it surfaces whether an experiment that reduces cost also moves retention, task completion, or purchase rate, making the trade-off empirically visible rather than assumed.

Pricing data changes frequently — the figures cited here reflect mid-2026 rates, and inference costs have been falling. That trajectory reinforces the case for running experiments now rather than waiting for the market to stabilize. The model that's 10x cheaper than the frontier today may be 2x cheaper in six months, which changes the math on every cost-quality decision you've already made.

From experiment results to provider strategy: gradual rollouts, fallbacks, and multi-provider routing

Running a controlled experiment to compare OpenAI, Anthropic, and open-source alternatives gets you to a decision. But the decision itself — "Anthropic wins on quality, let's ship it" — is only the beginning of the work.

How you move traffic to the winning provider, what happens when that provider has an outage at 2am, and whether you eventually route different task types to different providers: these are the infrastructure questions that determine whether your experiment results translate into durable production value or a one-time configuration change that nobody touches again.

Gradual traffic shifting with guardrail metrics

The instinct after a clean experiment result is to cut over completely. Resist it. A controlled experiment runs on a sample of your traffic under specific conditions — it won't surface every edge case that emerges at full scale. A gradual rollout (5% → 25% → 50% → 100%) gives you the observability window to catch problems before they affect your entire user base.

What makes a gradual rollout safe is guardrail metrics — specific thresholds that trigger a rollback automatically if breached during ramp-up. For an LLM provider migration, your guardrails should cover at minimum: error rate, p95 latency, cost per request, and a user satisfaction signal.

If your new Anthropic integration starts returning 5xx errors at 3% of requests during a 10% rollout, a guardrail on error rate should halt the ramp and route traffic back to OpenAI before the problem scales. Statistical guardrails formalize this pattern with dedicated guardrail metrics in the analysis settings, separate from primary and secondary metrics, with configurable risk thresholds — which means the rollback logic is part of the experiment definition, not an afterthought in a runbook.

Token counts can also spike unexpectedly with certain prompt patterns or user inputs at scale — a cost-per-request guardrail will surface this before it becomes a budget problem.

Handling provider outages and fallback logic

Single-provider dependency is a production risk that's easy to underestimate until it materializes. In October 2025, a major infrastructure outage affected roughly 99% of server-side SDKs globally for approximately 24 hours — a concrete illustration of what cascading failure looks like when infrastructure depends on a single external service without fallback logic.

The same failure mode applies to LLM providers: if your application routes all inference through one provider and that provider goes down, you need a path that doesn't require a code deploy to recover.

The fallback architecture is straightforward in principle: primary provider → secondary provider → cached response or graceful degradation. The operational challenge is making the switch fast enough to matter.

Feature flag kill switches solve this — the ability to instantly deactivate a provider variant and route to a fallback without touching application code. When the flag evaluation happens locally from a cached payload rather than via a network call to an external flag service, the fallback mechanism doesn't inherit the same availability risk as the provider it's protecting against.

Multi-provider routing as the mature end state

The most sophisticated teams don't pick a single winning provider and stop there. They route different task types to different providers within a single workflow — a cheap open-source model handling intent classification or content moderation, a frontier model handling final response generation where quality directly affects user experience.

This architecture reflects a straightforward economic reality: paying frontier model prices for every inference step in a pipeline is unnecessary when many of those steps are low-stakes and high-volume.

The routing logic lives in feature flags rather than application code, which means you can adjust which provider handles which task type without a deployment. Feature flags support attribute-based targeting rules, so you can route specific user segments or request types to specific providers, and kill switches allow instant cutover if a provider degrades. This also provides natural redundancy: if one provider degrades, you can shift its traffic to another via a flag update.

Character.AI uses GrowthBook to "compare different modeling techniques from the perspective of our users," which is exactly this kind of ongoing, production-integrated routing optimization rather than a one-time benchmark exercise. That kind of continuous routing optimization is only sustainable when the flag system, the experiment analysis, and the metrics all live in the same platform — which is the unified architecture the warehouse-native approach described earlier is designed to support.

The feature flag system that ran your experiment should also run production routing

The feature flag system that managed your initial experiment should also manage the production routing that follows from it. When the routing decisions and the experiment results that informed them live in the same platform, you eliminate the data pipeline complexity of reconciling a flag management tool with a separate analytics layer.

Practically, this means using the same targeting and rollout controls for provider routing that you used for the experiment itself: environment-scoped flags for staged rollouts, kill switches for emergency cutover, and attribute-based targeting to route specific user segments or request types to specific providers. The experiment was a temporary measurement instrument; the feature flag is the permanent control plane. Treating them as the same infrastructure — rather than separate tools that need to be kept in sync — is what makes provider strategy manageable as your LLM usage scales.

Turning experiment results into a provider strategy you can actually maintain

The through-line of this article is simple: LLM provider decisions made from benchmarks are guesses dressed up as analysis. The only way to know which provider performs better on your product is to measure it on your users, with your prompts, against outcomes that actually matter to your business. Everything else — the leaderboard rankings, the pricing comparisons, the vibes-based evaluations — is useful for narrowing the field, not for making the call.

Pick one feature, two providers, and a primary metric before you write any instrumentation code

The teams that get stuck are usually the ones trying to compare everything at once. Pick one LLM-powered feature where you have enough traffic to reach statistical significance in a reasonable timeframe, and define two or three metrics before you write a line of instrumentation code.

Quality and latency are almost always on that list; cost becomes the third metric once you have a quality baseline you're satisfied with. A focused first experiment — one feature, two providers, a clear primary metric — teaches you more about your measurement infrastructure than any amount of planning does.

Multi-provider routing is where you end up after experiments, not where you start

Multi-provider routing isn't a starting point — it's where you end up after you've run enough experiments to know which task types actually benefit from frontier model quality and which don't. The task-stakes framework from the cost section gives you the hypothesis; your experiment results give you the evidence.

Start routing different task types to different providers only after you have data showing that the quality difference between models produces a measurable difference in user outcomes for that specific task. Otherwise you're adding operational complexity without a business case.

Three infrastructure prerequisites that determine whether your experiment data is trustworthy

Three things need to be in place before your first experiment produces trustworthy results: a way to route traffic without code deploys, a place to join inference logs with user outcome events, and guardrail thresholds defined before the experiment runs — not after you've seen the data.

A warehouse-native experiment approach handles the second and third together, letting you define guardrail metrics in the same framework as your primary metrics and query everything against your own data warehouse. The feature flag system that routes your experiment traffic should be the same system that manages production routing after you've called a winner.

This article is meant to give you a complete, usable process — not just a framework to think about. If it saves you from a vibes-based evaluation or a benchmark lookup that leads you to the wrong provider, it's done its job.

What to do next: Identify one LLM-powered feature in your product that has measurable user outcomes — task completion, satisfaction rating, return visits, anything quantifiable. Define your primary metric and one guardrail metric. Then run an A/A test to confirm your instrumentation is working before you introduce a real provider difference. That's the whole first step. Everything else in this article follows from having clean data to work with.

Related insights

Sign up for free

Take Growthbook for a spin, no credit card required.

Create my account

Example H2

See All Articles

Experiments

Feature Flags

CLI vs MCP: TLDR, it doesn’t matter as much as you think

Oct 21, 2026

min read

Your agent should know how to do the job whichever interface it uses.

Your coding agent can already run terminal commands. Your product manager's assistant can connect to remote tools. Both need to inspect the same experiment, understand the same metric, and prepare changes to the same feature flag.

That creates a familiar question: should you give them a CLI or an MCP server?

The usual answers mix several concerns. A CLI is easy to compose with scripts. MCP gives compatible clients a standard way to discover and invoke tools. Skills can supply the instructions that make either route useful. None of those statements tells you whether the agent knows your release process or can distinguish a draft from a live change.

For GrowthBook, the choice has become smaller. We've built the agent layer around a REST API and reusable workflow skills, then made those available through different interfaces. Your choice of CLI or MCP no longer needs to determine which product workflow your agent can understand.

It still determines how the agent connects, executes requests, and handles credentials. This article explains that distinction, shows the same disabled-flag operation through both routes, and gives you a way to evaluate the integration you actually intend to use.

The debate combines three different questions

Start with a specific job: prepare a release control for a new checkout flow. The agent needs to know which project owns checkout, which environments exist, what the default behavior should be, and whether the proposed flag key already exists.

Choosing an interface answers only part of that job.

How can the agent reach the product?

A command-line interface lets an agent invoke a program in an environment where it has shell access. That program can make authenticated API requests and return data. You can also run the same commands yourself, put them in scripts, and preserve output for inspection.

The Model Context Protocol architecture connects a host application to servers through clients. An MCP server exposes capabilities that a compatible host can discover and call. The host might be a coding environment, a desktop assistant, or another agent application.

These categories overlap. A terminal-based agent can also be an MCP client. A local MCP server can be a process launched by that client. A remote MCP connection can work without the host giving the model a general-purpose shell.

So “terminal agent” doesn't automatically mean “use the CLI,” and “MCP” doesn't automatically mean “remote service.” Ask what access the runtime provides before choosing the adapter.

What procedure should the agent follow?

An API can expose a create operation without teaching an agent when to use it. A syntactically valid feature flag might still have the wrong project, an ambiguous key, or a default value that doesn't match the application.

That's the job of a skill: reusable instructions for a recognizable task. The Agent Skills format gives those instructions a portable structure, including a skill entry point and supporting references. A workflow can describe the context to inspect, the sequence to follow, and the evidence to return.

For the checkout example, useful guidance says to resolve environments, check the permanent key, choose a value type, and create the flag disabled. Merely exposing a create command leaves those decisions to the agent's general knowledge.

Skills are also useful when the right answer is to take no action yet. “Create a flag” and “design an experiment” are related requests, but they can require different objects and sequencing. An agent needs enough product guidance to make that distinction.

What context makes the procedure relevant?

Even a good playbook needs the current product state. The agent should inspect the objects in your GrowthBook organization and, when it is changing application code, the repository's SDK initialization and tests.

Those are different sources of context. GrowthBook supplies flags, experiments, metric definitions, and other authorized product resources. Your coding runtime supplies the application files it can access. An MCP connection doesn't by itself grant repository access, and a CLI doesn't by itself explain your metric definitions.

Evaluate the combination: access to the product, a maintained procedure, and enough context to apply that procedure. A comparison that changes all three at once cannot isolate the effect of CLI versus MCP.

GrowthBook puts the durable parts below the interface

We separated the agent layer into responsibilities that can be reused. The product API exposes operations. Skills explain workflows. The interface connects the agent to those two layers.

This is the architectural shift behind GrowthBook's agent experience. Instead of making every client integration carry its own interpretation of experimentation, we can share the interpretation and change how it is delivered.

The API provides the operations

GrowthBook's REST API makes product objects addressable. An agent can inspect a project, retrieve a flag, or perform an authorized operation against those objects.

The rebuilt API described in GrowthBook 4.4 uses shared schemas to support types, validation, and API documentation. The CLI is generated from the API definition. That reduces the amount of separately maintained command logic needed to expose product operations.

The OpenAPI specification explains the underlying principle: a machine-readable description can represent an HTTP API's paths, operations, inputs, and responses. Generation is useful because it derives another interface from that contract.

Generation doesn't mean any installed binary knows every future endpoint. The generated client still has a version, and your GrowthBook server still has a version. A self-hosted instance can lag behind the public documentation. Treat API compatibility as a deployment fact you verify.

Skills provide the workflow guidance

GrowthBook's feature-flags skill routes requests to more specific workflows. Creating a flag, inspecting dependencies, changing targeting, requesting review, and publishing a revision are separate procedures.

That structure lets the agent load the guidance relevant to the current job. The skill authoring guidance on progressive disclosure supports this approach: keep the entry point focused and place detail in references that are loaded as needed.

The benefit is maintainability as well as context management. A correction to a flag workflow can be made in the maintained skill rather than repeated across a series of tool descriptions and client-specific prompts.

There are two shell concepts to keep distinct. The generated growthbook CLI is a terminal interface to the product API. The skills repository also includes a small gb-call helper for making REST requests. Installing a skill doesn't make that helper a globally available command; its location depends on the installation. Both can reach the API, but their configuration and invocation are different.

MCP provides a thin bridge

The current GrowthBook MCP implementation exposes four tools: growthbook_list_skills, growthbook_read_skill, growthbook_api_read, and growthbook_api_write.

An agent discovers a domain, reads the relevant workflow, then makes API calls. It doesn't need a dedicated MCP tool for every product endpoint. Calls described as gb-call GET in a skill map to the read tool; mutating methods map to the write tool with the same path and body.

The bridge does not run the shell helper. It forwards authenticated requests. Product-side permissions and validation remain important because a small tool surface can still reach many operations.

Shared guidance also has a delivery boundary: the MCP server bundles skills when it is built, using a pinned source revision. A newly edited skill in the repository needs to be included in a subsequent server build before that server supplies it. “Shared source” is a maintainability claim, not a promise of instant synchronization.

Explore the shared flag workflows

Open the feature-flags skill to see the procedures that guide creation, targeting, reviews, and publishing. Bring the same workflow knowledge to the agent interface your team uses.

View the Flag Skill

The same disabled flag can use either route

The practical question is whether the interface changes the job. Let's create a Boolean flag called checkout-redesign, with a false default and every environment disabled.

This is an API example for a sandbox project, not an application integration tutorial. It creates a control object; your application still needs a correctly configured GrowthBook SDK to evaluate it. The feature flag documentation covers the product concepts behind that integration.

Connect and verify before preparing the write

For the terminal route, install the current GrowthBook CLI, configure a Personal Access Token through its interactive setup, and inspect the available command syntax:

npm install -g growthbook
growthbook configure
growthbook whoami
growthbook features create --help
growthbook features get --help

Check that the active server and identity are the ones you intend to use. Prefer a tested, pinned CLI version in repeatable automation. Don't include a live token in a command you plan to share.

For an MCP route, the GrowthBook Cloud setup uses the hosted endpoint and browser OAuth. In a client using the documented mcpServers configuration shape, such as Cursor, add:

{
  "mcpServers": {
    "growthbook": {
      "url": "https://mcp.growthbook.io/mcp"
    }
  }
}

Complete the browser authorization and confirm tool discovery. Other clients can use a different configuration shape. Self-hosted GrowthBook users should follow the local stdio or self-hosted HTTP instructions rather than point a private instance at the Cloud service.

Next, let the connected agent discover the flag-creation workflow. These are tool-call parameters, shown separately in the order to invoke them:

{"name":"growthbook_list_skills","arguments":{}}

{"name":"growthbook_read_skill","arguments":{"name":"feature-flags"}}

{"name":"growthbook_read_skill","arguments":{"name":"feature-flags/references/flag-create"}}

The returned guidance should describe the prerequisites for creating the flag. If the workflow isn't present, resolve the server's bundled skill version before substituting instructions from an older integration.

Resolve real IDs and construct one payload

The flag-creation workflow checks existing keys, lists projects, and resolves environments. In the MCP client, invoke the read tool with each of these argument objects:

For the CLI route, perform that discovery with the corresponding command groups documented in the CLI command catalog:

growthbook features list-keys --output-format json
growthbook projects list --output-format json
growthbook environments list --output-format json

Inspect each command's help if your installed version differs. The returned keys, project IDs, and environment IDs are the inputs to the payload below. In MCP, the equivalent read-tool arguments are:

{"path":"/api/v2/feature-keys"}

{"path":"/api/v1/projects"}

{"path":"/api/v1/environments"}

Verify that checkout-redesign isn't taken, identify the sandbox project's ID, and enumerate every actual environment. These read responses determine the payload; don't treat example environment names as a complete inventory of your organization.

Save the following as flag.json, replacing <PROJECT_ID> and the environment keys with the discovered values. This example assumes the only environments are staging and production:

{
  "id": "checkout-redesign",
  "valueType": "boolean",
  "defaultValue": "false",
  "description": "Control exposure to the redesigned checkout",
  "project": "<PROJECT_ID>",
  "environments": {
    "staging": { "enabled": false },
    "production": { "enabled": false }
  }
}

The API represents defaultValue as a string, even for a Boolean flag. Explicitly disabling each environment avoids relying on the organization's default state for newly created flags. Use a PAT for this example so the caller has a user identity for attribution.

Review the permanent key, project, value, and disabled states before submitting. If the intended job is an experiment, route to experiment design first instead of creating an unrelated flag and hoping to connect it later.

Submit once, then read the object back

With the CLI, the create command accepts a JSON body. In a Bash-compatible shell, inspect the request with dry run before submitting:

growthbook features create --body "$(cat flag.json)" --dry-run
growthbook features create --body "$(cat flag.json)" --output-format json

The first command previews a request; it doesn't prove the server will accept it. Run the second only after checking the preview and approving this creation. In PowerShell, use that shell's file-reading syntax rather than paste the Bash substitution unchanged.

The PowerShell equivalent, using the same reviewed file, is:

$flagPayload = Get-Content -LiteralPath .\flag.json -Raw
growthbook features create --body $flagPayload --dry-run
growthbook features create --body $flagPayload --output-format json

The same distinction applies: preview first, approve the specific creation, then submit. Neither command changes the application's deployed code.

Through MCP, call growthbook_api_write with the same payload serialized into its body string. Substitute the same resolved project and environment names:

{
  "method": "POST",
  "path": "/api/v2/features",
  "body": "{\"id\":\"checkout-redesign\",\"valueType\":\"boolean\",\"defaultValue\":\"false\",\"description\":\"Control exposure to the redesigned checkout\",\"project\":\"<PROJECT_ID>\",\"environments\":{\"staging\":{\"enabled\":false},\"production\":{\"enabled\":false}}}"
}

Choose one creation route. Don't run both against the same key; they target the same product object, and the second creation should encounter an existing key.

Verify through the CLI get command:

growthbook features get --id checkout-redesign --output-format json

Or invoke growthbook_api_read with:

{"path":"/api/v2/features/checkout-redesign"}

Check the returned key, project, value type, default value, disabled state of every environment, and absence of unintended rules. Report those facts rather than “success.” No targeting or exposure change is included in this example.

Choose the route that fits the runtime

Once the procedure and API operation are shared, the remaining choice is operational. Where does the agent run? What can it execute? How does it authenticate? What does your team need to inspect afterward?

Shell access favors terminal composition

A shell-capable agent can run the CLI alongside repository tools, save structured output, and combine product inspection with local checks. That is useful when the job includes code changes, generated files, or a reproducible script.

For example, a flag audit can produce an inventory file while an engineer separately checks code references. The agent should preserve the association between the GrowthBook key and each repository reference. The final artifact can include both the product state and the proposed code diff.

The shell makes composition available; it doesn't make a workflow safe by default. A script can select the wrong profile or swallow an error. Decide which failures stop the workflow, where outputs are stored, and which steps need a reviewable artifact.

Remote tool access favors MCP clients

MCP is useful when the host already provides a tool connection but doesn't expose a terminal. A PM can inspect an experiment through a compatible assistant without installing a product CLI inside that assistant's runtime.

The MCP authorization specification defines an authorization framework for HTTP transports. GrowthBook Cloud uses browser OAuth, while local stdio configurations use credentials supplied to the process. Those are distinct deployment paths.

MCP also gives hosts a structured description of tools. The tool specification includes annotations such as read-only and destructive hints. Hosts can use those signals in their interaction design, but the signals are not an access-control boundary. The product must still enforce the caller's actual permissions.

Use a small decision table

Your immediate requirement	Route to consider	What to verify
Combine product operations with local files and scripts	CLI plus maintained skills	Shell access, profile, installed version, structured output
Connect a compatible assistant without a shell	Hosted MCP	HTTP/OAuth support, identity, tools, bundled workflow
Keep requests inside a self-hosted deployment	CLI or self-hosted MCP	Network reachability, server version, credentials, trusted app origin
Reuse a procedure across different team clients	Shared skills through either route	Workflow revision, API compatibility, expected review artifact

You can support more than one row. An engineer's CLI and a PM's MCP connection can operate on the same objects with different identities and permissions. Standardizing the workflow is often more useful than standardizing everyone's client.

Compare actual outcomes, not tool counts

Four MCP tools can address a large API. A large CLI command catalog can expose the same operations. Counting entries tells you little about whether an agent completed the intended job correctly.

Likewise, there is no universal token or speed result implied by the architecture. The client, model, instructions, response size, pagination, and number of requests all affect a real run. Measure the work you need instead of importing a result from an unrelated integration.

Keep the evaluation task constant

Use one sandbox project and one bounded task. For a read test, ask each route to find a named flag and identify its enabled environments, current revision, and relevant dependencies. For a write test, use separate temporary keys with the same reviewed disabled-flag requirements.

Give both runs the same expected output. Record whether each identified the correct objects, loaded the relevant workflow, handled missing data, and preserved the boundary between creation and exposure. Measure elapsed time and context consumption only alongside those correctness checks.

If one run gets rich workflow guidance and the other gets only a command name, you're testing the guidance as well as the interface. That can still be a useful product evaluation, but describe what changed.

Test failure behavior deliberately

Try a nonexistent object, an existing key, and a credential without the required permission. The agent should identify the failure and stop the dependent action. It should not invent an ID, silently pick another environment, or reinterpret a permission error as permission to retry with a broader credential.

The revision workflow adds another important test: multiple open drafts. When a chain of changes belongs in one revision, the agent must preserve its version rather than repeatedly ask for whichever draft is latest.

A creation receipt and a publication receipt are different artifacts. If your evaluation only checks that a request returned without error, it can miss the point where a proposed change became live or where a draft failed to include the intended edits.

Record the versions behind the result

Capture the CLI version, server version when available, client configuration, and the skill text or revision used. With MCP, inspect the guidance actually returned by the connected server. With installed skills, inspect the local instructions actually loaded.

That record makes a later regression diagnosable. A changed result may come from new guidance, a client update, a changed API, or different product state. “MCP became worse” is too broad to help you fix it.

Build workflows that survive the next client

The most useful consequence of GrowthBook's design is organizational. Engineers, PMs, and analysts can choose the agent surface that fits their work while referring to the same flags, experiments, and metrics.

An engineer can prepare a disabled flag from a terminal. A reviewer can inspect its draft in the app. An assistant connected through MCP can retrieve the same object and explain its state. The handoff should carry an ID, a revision where applicable, the proposed behavior, and the decision still required.

This is where the CLI-versus-MCP question loses much of its importance. Your release procedure doesn't need to be rewritten around each client. The product holds state; the API exposes operations; skills explain the work; the chosen interface delivers access.

Start with one inspection or sandbox creation. Require the agent to return a concrete object and its verified state. Then add the next workflow only when the current one is reliable enough for your team to review.

GrowthBook's feature flag platform gives that work a common place to live. The durable choice is how you connect release control and measurement to your team's process. CLI and MCP are ways to carry that process into the tools people already use.

Start with a common foundation

Set up GrowthBook for the flags, experiments, and metrics your agents will share. Then choose the CLI or MCP connection that fits each team member's work.

Get Started With GrowthBook

Experiments

Analytics

Standard deviation vs variance: Key differences for A/B testers

Oct 21, 2026

min read

A metric can have wide user-to-user variation and still yield a precise experiment estimate.

That is why confusing standard deviation with standard error creates bad A/B testing decisions. A dashboard may show that users differ substantially while the average effect is estimated closely enough to guide a release. Those statements can both be true.

Variance and standard deviation describe spread. Variance works on a squared scale; standard deviation takes its square root to return to the metric's original units. Neither is automatically the uncertainty around an experiment result.

For A/B testers, the useful distinction is practical: use standard deviation to communicate outcome variability, use variance in the calculations that combine uncertainty, and inspect the standard error or interval for the treatment effect. Start by making sure all three describe the correct unit of analysis.

Understand the difference in units

Variance uses squared distances

Outcomes above and below a mean have deviations with opposite signs. Simply averaging those signed deviations gives zero. Variance avoids that cancellation by squaring each deviation before combining them.

The sample variance formula is s² = Σ(xᵢ − x̄)² / (n − 1), where x̄ is the sample mean and n is the number of observations. The symbol Σ means to add the terms across observations.

If the outcome is measured in seconds, variance is measured in seconds squared. That is mathematically useful but less intuitive for explaining a product experience. “Variance is 4 seconds squared” does not mean users differ by 4 seconds.

Standard deviation returns to the original scale

Standard deviation is s = √s². A variance of 4 seconds squared corresponds to a standard deviation of 2 seconds. Both describe the same dispersion, expressed on different scales.

Do not define standard deviation as the average absolute distance from the mean. That is a different measure. Squaring deviations gives large deviations more influence, and taking the square root afterward does not undo that weighting.

Quantity	What it describes	Units for task time
Variance	Squared dispersion in outcomes	Seconds squared
Standard deviation	Outcome dispersion on the original scale	Seconds
Standard error of the mean	Sampling variability of an estimated mean	Seconds

Neither variance nor standard deviation requires a normally distributed population merely to be calculated. Normality matters for particular interpretations and inferential procedures, not for the existence of these summaries when the required moments are finite.

Calculate both from the same observations

Work through a small dataset

Use four synthetic task times: 2, 4, 4, and 6 seconds. This tiny dataset is for arithmetic, not evidence about a real product or a basis for confident inference.

The mean is (2 + 4 + 4 + 6) / 4 = 4 seconds. Deviations from the mean are −2, 0, 0, and 2 seconds. Their squares are 4, 0, 0, and 4 seconds squared, totaling 8.

Treating these observations as a sample, the usual sample variance is 8 / (4 − 1) = 8/3, or about 2.667 seconds squared. The sample standard deviation is √(8/3), or about 1.633 seconds.

This is a useful check on a report. If someone describes the same sample's variance as 1.633, they have probably substituted the standard deviation. If the units are missing, the mistake is much harder to spot.

Choose sample or population formulas deliberately

If those four observations constitute the entire finite population you want to describe, population variance divides by 4: 8/4 = 2 seconds squared. Population standard deviation is √2, or about 1.414 seconds.

The population and sample distinction depends on what you are estimating. Having every recorded event from an experiment does not necessarily make those events the full population of future users or future conditions you care about.

The n − 1 denominator makes the usual sample variance an unbiased estimator of population variance under independent, identically distributed sampling with finite variance. It does not make the square root exactly unbiased for population standard deviation. Nor does it correct biased sampling or dependence between observations.

Verify the calculation with explicit function names

Python's statistics functions distinguish sample calculations from population calculations. With Python 3 installed, this standalone example requires no external packages or data:

from statistics import variance, stdev, pvariance, pstdev

times = [2, 4, 4, 6]
print(round(variance(times), 3), round(stdev(times), 3))
print(round(pvariance(times), 3), round(pstdev(times), 3))

The first line of output is 2.667 1.633; the second is 2 1.414. The different denominators explain the difference. Before comparing statistics from two tools, check their sample/population defaults and handling of missing values.

Find the noise you can reduce

Explore practical variance reduction methods, including metric design and CUPED, and the conditions that make each approach useful.

Read the Variance Reduction Guide

Use standard errors for experiment uncertainty

Distinguish users from estimated means

Standard deviation describes how users' outcomes vary. Standard error describes how an estimator would vary across repeated samples. For an independent sample, the estimated standard error of its mean is s / √n.

Suppose a user-level task-time metric has a sample standard deviation of 20 seconds among 10,000 independent users. The estimated standard error of the mean is 20/√10,000 = 0.2 seconds. Large differences between users coexist with a much smaller uncertainty around their average.

Collecting four times as many independent users would halve this standard error if the outcome variance stayed similar. It would not mechanically halve user-level standard deviation. The individual experience remains variable even as the estimated mean becomes more precise.

Combine arm variances before taking the square root

For independent treatment and control groups, a simple unadjusted difference in means has estimated variance s²ₜ/nₜ + s²꜀/n꜀. Its standard error is the square root of that sum. You do not add the two arm standard deviations and call the result the experiment's uncertainty.

Take a separate synthetic summary with 10,000 independent users in each arm and a 20-second standard deviation in both. The variance of the estimated difference is 400/10,000 + 400/10,000 = 0.08 seconds squared. Its standard error is √0.08, about 0.283 seconds.

If the observed treatment-minus-control difference is −1 second, an illustrative large-sample, unadjusted 95% normal interval is −1 ± 1.96 × 0.283, approximately −1.554 to −0.446 seconds. Negative values mean faster completion under this metric definition.

This example assumes independent groups and an appropriate large-sample approximation. It is not the exact output of GrowthBook's relative-lift, sequential, covariate-adjusted, or ratio-metric methods. GrowthBook's statistical details describe how the platform estimates effects and their uncertainty.

Avoid interpreting intervals as outcome ranges

That interval concerns the mean treatment effect. It does not say that 95% of individual users improved by between 0.446 and 1.554 seconds. An individual user's causal effect is a different quantity, and the experiment does not observe both potential outcomes for the same person.

Likewise, mean ± one standard deviation is not a universal statement about where 68% of users fall. That familiar interpretation depends on an approximately normal distribution. Skewed task times or revenue can look very different.

The frequentist confidence level describes the repeated-sampling coverage of the interval procedure under its assumptions. It is not a posterior probability assigned to a fixed realized interval.

Check what your metric actually measures

Match the variance to the randomization unit

If users are randomized and each contributes many requests, request rows are not automatically independent experimental units. Treating 100 requests from one person as 100 unrelated users can make uncertainty look smaller than it should.

Define the outcome at the intended unit or use a method that accounts for clustering. GrowthBook's aggregate data requirements are relevant when supplying precomputed summaries: the sample size, sum, and dispersion must correspond to the statistic and analysis method.

Two teams can report different standard deviations without either making an arithmetic mistake. One may summarize per-request latency while the other summarizes each user's average latency. They are describing different outcome distributions. Clarify the grain before comparing their numbers.

Paired measurements need another distinction. If the same users provide before-and-after outcomes, the measurements within each pair can be correlated. Analyze each user's difference or use an appropriate paired method; do not apply the independent-arm formula automatically. The paired-means framework treats those differences as the observations of interest. Pairing also does not remove time trends or establish causality by itself; the design still determines which comparison is defensible.

Inspect tails instead of deleting them casually

Large observations influence both variance and standard deviation. A few unusually long tasks can dominate a time metric; a few large purchasers can dominate revenue. Determine whether those values reflect valid behavior, instrumentation defects, retries, or unit errors.

A valid expensive user is not automatically an outlier to discard. Removing high values after seeing an experiment result can change the estimand and introduce selective analysis. Decide justified exclusions or transformations before interpreting the test.

Report complementary summaries when they answer useful questions: a histogram, median, or a relevant tail percentile can show what a single spread measure hides. Those summaries do not replace the mean effect if the mean remains the decision metric.

Keep units consistent across pipelines

Converting seconds to milliseconds multiplies every value by 1,000. Standard deviation also multiplies by 1,000, but variance multiplies by 1,000,000. A dramatic numerical variance increase can therefore be a unit conversion rather than a product regression.

For the four-value example, sample standard deviation becomes approximately 1,633 milliseconds and variance becomes approximately 2,666,667 milliseconds squared. The underlying task times have not changed. Store units with metric definitions and inspect them when comparing dashboards.

A lower variance caused solely by rescaling a metric does not create more statistical information. The effect size rescales too. Separate a numerical change in units from an actual improvement in the precision of your estimate.

Also avoid averaging standard deviations from daily reports and labeling the result a weekly standard deviation. The complete-period spread depends on both variation within days and differences between daily means. Keep the underlying observations or sufficient summaries that support the intended calculation. A simple average of displayed daily values generally loses information needed to reconstruct the full distribution.

Turn spread into a better testing plan

Define the outcome before planning the sample

Start with the metric definition: population, identity, aggregation, window, units, exclusions, and desired effect. Estimate variability from representative historical data at that same grain.

Historical variance is an input, not a guarantee. New traffic, a different user mix, or a changed measurement window can alter it. Explore plausible variance and effect-size assumptions rather than treating one historical number as exact.

Keep the two arm variances separate when appropriate. A treatment can change variability as well as its mean. The simple equal-standard-deviation illustration above is a teaching assumption, not a requirement that production groups have identical spread.

Reduce estimation noise without hiding behavior

CUPED and post-stratification use suitable pre-experiment information and covariates to improve effect estimation. CUPED stands for Controlled-experiment Using Pre-Experiment Data. Its value depends on useful predictive information and a valid implementation.

Adjustment can reduce uncertainty in the treatment effect without making real users' outcomes less variable. That distinction prevents a reporting error: a narrower adjusted interval is not evidence that customers now experience a more consistent product.

Also separate missing data from genuine zeros. A user with no purchase may correctly contribute zero revenue; a missing instrumentation event might represent an unknown outcome. Treating both identically can distort the mean and its variance before any statistical engine sees the data.

Report the right quantity for the decision

Reader's question	Useful quantity
How much do individual outcomes differ?	Standard deviation, with distribution context
What enters this uncertainty calculation?	Variance at the correct analysis unit
How precisely is the effect estimated?	Standard error and an appropriate interval
Is the measured difference worth acting on?	Effect size, uncertainty, guardrails, and decision context

For a clear experiment readout, show the metric's unit and analysis grain, the estimated effect, and its appropriate uncertainty interval. Add outcome spread when it helps readers understand the population or diagnose a noisy metric.

GrowthBook's experimentation platform brings defined metrics and statistical analysis together. A trustworthy interpretation still starts with your measurement choices: describe spread with standard deviation, calculate with variance where required, and judge precision using the estimator's uncertainty.

Build a clearer experiment readout

Connect your metrics to GrowthBook and evaluate effects with the analysis method and measurement definitions your team has chosen.

Get Started With Experimentation

Experiments

Analytics

How to calculate true positive rate (TPR)

Oct 21, 2026

min read

Catching 80 defects means little until you know how many defects existed.

Suppose a detector flags 125 suspicious software changes. Engineers inspect them and confirm 80 real defects. That tells you something about alert quality, but it does not tell you how many defects the detector missed. To calculate true positive rate, you need the actual positives outside the alert queue too.

True positive rate, or TPR, measures the share of actual positive cases that a system correctly identifies. It is also called recall or sensitivity. The formula is TP / (TP + FN), where TP means true positives and FN means false negatives.

The arithmetic is simple. Choosing the right population, labels, threshold, and denominator is the difficult part. This guide works through an engineering example, distinguishes neighboring metrics, and explains what changes when the “test” is an A/B experiment rather than an item classifier.

Start with actual positives, not alerts

Define the positive class

“Positive” means the condition you want to detect. It does not necessarily mean a good outcome. In a defect detector, positive means a real defect. In a spam filter, it means spam. Write that definition before calculating anything.

A true positive is an item that is actually positive and predicted positive. A false negative is actually positive but predicted negative. Those two groups together form the denominator: all actual positives in the evaluation sample.

The recall definition is therefore conditional on actual class membership. TPR asks, “Among the real defects, what proportion did we catch?” It does not ask, “Among the alerts, what proportion were real?”

Fix the unit and observation window

Choose a consistent evaluation unit. A code change, an individual finding, and a repository are different units. If one change generates 12 duplicate alerts, counting alerts in the numerator and changes in the denominator produces a meaningless rate.

For the example, each unit is one code change, evaluated once by one fixed detector version at one fixed threshold. An independent review process eventually labels each change as defective or not defective. The observation window is long enough for those labels to mature.

This resembles the discipline needed when defining experiment metrics: the event, identity, time window, and denominator must describe the same question. A correct formula applied to incompatible counts still gives the wrong answer.

Calculate TPR from the confusion matrix

Put every evaluated item in one cell

Consider an original synthetic evaluation of 1,000 code changes. Reviewers establish that 100 contain a defect and 900 do not. The detector catches 80 defects, misses 20, and incorrectly flags 45 nondefective changes.

Actual label	Predicted positive	Predicted negative	Total
Defective	80 true positives	20 false negatives	100
Not defective	45 false positives	855 true negatives	900
Total	125	875	1,000

Read the labels rather than relying on cell position. Libraries and reports can arrange axes differently. The scikit-learn confusion matrix convention uses actual classes on rows and predicted classes on columns, with class order also affecting where each value appears.

Check the totals before calculating rates. The four cells sum to 1,000. True positives plus false negatives sum to 100 actual defects. Predicted positives total 125, which is a different denominator.

Divide detected positives by all actual positives

The calculation is:

TPR = TP / (TP + FN) = 80 / (80 + 20) = 80 / 100 = 0.80 = 80%.

Interpret it in words: the detector caught 80% of the actual defects in this evaluation sample. It missed 20%, so the false negative rate is 20%. For a binary task with complete labels, TPR and false negative rate sum to 100%.

Do not divide 80 by 1,000. That would measure true positives as a share of all evaluated changes, not recall. Do not divide 80 by 125 either. That is precision, which answers the alert-quality question.

The result describes this sample and threshold. It does not establish that every future workload will have the same recall, or that all defect categories are equally detectable. Keep the counts beside the percentage so readers can see the support behind it.

Handle zero positive support explicitly

If the sample contains no actual defects, TP + FN equals zero. Sample TPR is undefined. Returning 0% would imply the detector missed known defects; returning 100% would imply it successfully found them. Neither interpretation is supported.

Software libraries need a practical zero-division convention. That convention can return a placeholder or warning, but it does not create evidence. Report “TPR unavailable: zero actual positives” and preserve the support count. A pipeline can then distinguish an unevaluable slice from measured poor performance.

Check the experiment error risks

See how power, repeated checks, and multiple comparisons affect the reliability of experimental decisions beyond a single classifier score.

Read the False Positives Guide

Compare TPR with precision and false positives

Keep the denominators visible

For the same 1,000 changes, the other common metrics tell different stories:

Metric	Calculation	Result
True positive rate / recall	80 / 100 actual positives	80%
Precision	80 / 125 predicted positives	64%
False positive rate	45 / 900 actual negatives	5%
Specificity	855 / 900 actual negatives	95%
Accuracy	(80 + 855) / 1,000 changes	93.5%

The classification metric definitions differ because they condition on different populations. Here, 36% of alerts are false, but the false positive rate is 5%. Calling both numbers “the false positive rate” hides the operational distinction.

Accuracy can also look reassuring in an imbalanced population. A detector that labels all 1,000 changes nondefective would be 90% accurate while detecting none of the 100 defects. TPR exposes that failure, though it still needs the false-positive side of the picture.

Choose a threshold for the decision

Suppose a stricter threshold, evaluated on the same labeled changes, catches 60 defects and incorrectly flags 10 nondefective changes. Its TPR is 60%, its precision is about 85.7%, and its false positive rate is about 1.11%.

This threshold reduces the review queue from 125 changes to 70. It also misses 40 defects instead of 20. Whether that tradeoff is acceptable depends on the consequences of a missed defect, the review team's capacity, and what action follows an alert.

A ROC curve plots TPR against false positive rate across thresholds. It helps compare operating points. It does not pick the operational threshold for you, and an area-under-curve summary does not specify the queue size at the threshold you intend to deploy.

If you tune thresholds using the same labels you later report as a final evaluation, the score can be optimistic. Use separate validation and held-out evaluation data, or another defensible evaluation design, and freeze the decision rule before the final assessment.

Specify the action at each threshold too. A low-confidence finding might create a suggestion, while a high-confidence finding might block a merge. Those are different interventions with different error costs. One aggregate recall figure should not imply that every detected item received the same treatment.

Make sure the labels can support the claim

Audit predicted negatives too

If engineers review only alerts, you can estimate precision within the reviewed alert population. You cannot directly count missed defects among unreviewed negatives. Assuming those changes are clean makes FN artificially small and TPR artificially high.

An evaluation may need independent review of a representative sample of predicted negatives, delayed incident labels, or a curated benchmark with known defects. Each approach has limits. Incident labels miss undetected failures; curated examples can differ from ordinary production work.

If the evaluation oversamples suspicious changes or certain repositories, report that design. Use appropriate weighting when estimating performance for the real workload. A simple unweighted confusion matrix describes the sampled dataset, which may not represent the deployed population.

Preserve uncertainty and segment support

An 80% estimate based on 4 detected defects out of 5 is much less precise than one based on 800 out of 1,000. The important sample size for recall is the count of actual positives, not just the total number of changes evaluated.

Report positive support by important slices, such as language, defect category, or repository type. A high pooled TPR can hide weak performance on a rare but expensive failure mode. At the same time, avoid declaring tiny slices definitively better or worse from a few examples.

Counts alone do not make observations independent. Many similar changes from one repository can share a common pattern. Use uncertainty methods that fit the evaluation design rather than assuming every row is an unrelated trial.

Separate label delay from label absence

A recently deployed change might appear nondefective because nobody has had time to find the bug. Treating it as a confirmed negative gives the newest cohort a different labeling standard from older cohorts.

Define when an item becomes eligible for evaluation and track unresolved labels separately. Keep model version, threshold, label version, and observation dates with the result. That makes a later score change interpretable: you can tell whether the detector changed, the workload changed, or the labels matured.

For a warehouse-native measurement workflow, these definitions belong in the underlying data model. Joining model decisions to review outcomes still requires identity and time alignment; a queryable warehouse does not automatically resolve ambiguous ground truth.

Keep comparable populations together

When comparing detector versions, evaluate them on the same held-out cases where possible. Otherwise, an apparent recall improvement could come from an easier mix of defects. Preserve a breakdown of new, removed, and unchanged cases so reviewers can see what changed between evaluations.

When combining reports, add the underlying TP and FN counts for nonoverlapping, consistently defined populations, then calculate the pooled rate. An unweighted average of repository percentages answers a different question because it gives a tiny repository the same influence as a large one. Either summary can be useful, but label the weighting explicitly.

If the same change appears in several reports, deduplicate or model that repeated observation before pooling. Repeated scans do not create additional independent defects. Keep an audit trail from the final counts back to the evaluated item identifiers so a suspicious rate can be investigated.

Connect detection quality to product decisions

Distinguish classifier recall from statistical power

In hypothesis testing, statistical power is the probability that a testing procedure rejects the null when a specified alternative is true. It is a repeated-test detection probability, depending on assumptions such as effect size, variance, sample size, and decision rule.

That is related to the idea of detecting a real signal, but it is not the same dataset as a labeled-item confusion matrix. In an ordinary A/B program, you do not observe the true causal effect of every experiment with certainty. Dividing declared winners by all experiments does not reveal a ground-truth TPR.

Power planning asks how often your procedure would detect a particular real effect under its assumptions. A classifier evaluation asks how many labeled positives were detected in the evaluated sample. Keep those questions separate when discussing GrowthBook's statistical methods.

Evaluate the intervention as well as the detector

An improved offline detector can still create a worse product workflow. More alerts might overwhelm engineers, delay releases, or catch defects too late to matter. Conversely, a modest recall change might be valuable if it catches the failures with the highest operational cost.

When appropriate, use randomized experiments to evaluate the downstream intervention: which review experience helps teams resolve defects, with what delay and cost? Keep detector recall, review workload, and product outcomes as separate measures.

GrowthBook's experimentation platform can analyze defined outcomes from such tests. Your team supplies reliable assignment, exposure, labels, and metric definitions. Neither a high TPR nor a significant product result eliminates the need to inspect the other evidence.

Use a repeatable reporting checklist

Before sharing a TPR number, state the positive class, evaluation unit, detector version, threshold, and label process. Show TP and FN beside the rate. Include precision or false positive rate so readers can assess the workload created by detection.

Then ask whether predicted negatives were evaluated, whether labels had time to mature, and whether the sample reflects the population where you will act. Mark zero-support slices unavailable and explain material gaps in coverage.

The useful result is a statement someone can audit: “At this threshold, the detector found 80 of 100 labeled defects in these 1,000 evaluated changes, while producing 45 false alerts.” That is a clearer basis for a decision than “our model is 80% accurate.”

Measure what the change delivers

Connect your data and defined metrics to GrowthBook to evaluate how a product or workflow change affects the outcomes your team cares about.

Start Measuring With GrowthBook

Ready to ship faster?

No credit card required. Start with feature flags, experimentation, and product analytics—free.

Get Started

Book a Demo

Simplified white illustration of a right angle ruler or carpenter's square tool.

White checkmark symbol with a scattered pixelated effect around its edges on a transparent background.

Picking an LLM provider based on leaderboard rankings is like hiring a chef because they scored well on a written exam.

Why static benchmarks don't tell you which LLM provider is right for your product

Benchmarks measure narrow correctness, not production fit

The gap between rankings and production behavior

Why task-specific experiments are the only reliable signal

The five dimensions you need to measure in any LLM provider comparison: quality, latency, cost, reliability, and safety

Quality: relevance, faithfulness, and task accuracy

Latency: average response time and variance

Cost: token economics and per-task spend

Reliability: uptime, error rates, and throttling behavior

Safety: unsafe output rates, bias, and compliance behavior

Structuring a controlled experiment that produces defensible provider comparisons

Per-request vs. per-user assignment: why the choice contaminates your metrics

Prompt consistency and confound prevention

The inference log fields that make downstream analysis possible

Feature flags as zero-latency traffic routing for provider experiments

The cost-quality trade-off: a 300x price gap that requires experimental validation, not assumptions

The pricing landscape: a 300x gap that keeps widening

A task-stakes framework for model selection

Validating the trade-off with downstream business metrics

From experiment results to provider strategy: gradual rollouts, fallbacks, and multi-provider routing

Gradual traffic shifting with guardrail metrics

Handling provider outages and fallback logic

Multi-provider routing as the mature end state

The feature flag system that ran your experiment should also run production routing

Turning experiment results into a provider strategy you can actually maintain

Pick one feature, two providers, and a primary metric before you write any instrumentation code

Multi-provider routing is where you end up after experiments, not where you start

Three infrastructure prerequisites that determine whether your experiment data is trustworthy

Related insights

Sign up for free

Table of Contents

Related Articles

CLI vs MCP: TLDR, it doesn’t matter as much as you think

Your agent should know how to do the job whichever interface it uses.

The debate combines three different questions

How can the agent reach the product?

What procedure should the agent follow?

What context makes the procedure relevant?

GrowthBook puts the durable parts below the interface

The API provides the operations

Skills provide the workflow guidance

MCP provides a thin bridge

Explore the shared flag workflows

The same disabled flag can use either route

Connect and verify before preparing the write

Resolve real IDs and construct one payload

Submit once, then read the object back

Choose the route that fits the runtime

Shell access favors terminal composition

Remote tool access favors MCP clients

Use a small decision table

Compare actual outcomes, not tool counts

Keep the evaluation task constant

Test failure behavior deliberately

Record the versions behind the result

Build workflows that survive the next client

Start with a common foundation

Standard deviation vs variance: Key differences for A/B testers

A metric can have wide user-to-user variation and still yield a precise experiment estimate.

Understand the difference in units

Variance uses squared distances

Standard deviation returns to the original scale

Calculate both from the same observations

Work through a small dataset

Choose sample or population formulas deliberately

Verify the calculation with explicit function names

Find the noise you can reduce

Use standard errors for experiment uncertainty

Distinguish users from estimated means

Combine arm variances before taking the square root

Avoid interpreting intervals as outcome ranges

Check what your metric actually measures

Match the variance to the randomization unit

Inspect tails instead of deleting them casually

Keep units consistent across pipelines

Turn spread into a better testing plan

Define the outcome before planning the sample

Reduce estimation noise without hiding behavior

Report the right quantity for the decision