How do you A/B test an LLM when results aren't deterministic?
A/B testing an LLM is not the same as A/B testing a button color.
The core assumption that makes standard A/B testing work — that your treatment is a stable, fixed thing — breaks the moment you introduce a model that produces different outputs every time it runs, even with the same prompt and the same settings. Research testing five LLMs under nominally deterministic conditions found accuracy variations of up to 15% were observed across runs. That's not noise you can ignore. That's a problem you have to design around.
This guide is for engineers, PMs, and data teams who are shipping AI features and need to know whether their changes are actually working. If you've tried applying standard experiment logic to LLM outputs and gotten results that felt unreliable — or if you're about to run your first LLM experiment and want to avoid the common traps — this is written for you. Here's what you'll learn:
- Why LLM non-determinism breaks traditional A/B testing assumptions, and what that means for your experiment design
- How to define what you're actually comparing — prompts, models, hyperparameters, and configurations — so your results are causally interpretable
- Which metrics can detect real signal through output variability, and which ones produce noise dressed up as data
- When to use offline prompt evaluation versus live A/B tests, and how to sequence them correctly
- How to structure the statistics — power analysis, randomization unit, test duration, and canary rollouts — so your results hold up
The article moves in that order: from why the problem exists, to how to isolate what you're testing, to how to measure it, to how to run the experiment correctly. None of these steps are optional. Each one addresses a specific failure mode that produces confident-looking results that are actually wrong.
Why LLM non-determinism makes traditional A/B testing assumptions break down
Standard A/B testing is built on a quiet assumption that almost never gets stated explicitly: the treatment is stable. When you assign a user to variant B — the green button, the shorter headline, the reordered checkout flow — variant B is the same thing every time that user encounters it.
The output doesn't drift between assignment and measurement. This stability is so fundamental to how experiments are designed and analyzed that most testing frameworks don't bother to document it. It's just assumed.
LLMs violate this assumption at the architectural level. And the violation isn't subtle.
LLM outputs are samples, not artifacts
A UI element is a fixed artifact — it looks the same every time. An LLM response is more like a roll of a weighted die: the model has learned which types of responses are most likely for a given prompt, but it doesn't produce the same response every time. Which specific response you get is decided fresh at the moment the model runs.
This breaks the core requirement for treatment stability. When you assign a user to "variant B — the new prompt," you're not assigning them to a fixed experience. You're assigning them to a distribution of possible experiences, and which specific experience they get is determined at the moment of inference.
The treatment isn't a thing; it's a process. That distinction matters enormously for how you design and interpret experiments.
Temperature is not the fix it appears to be
The instinct to set temperature to zero is understandable. Greedy sampling — always selecting the highest-probability token — is theoretically deterministic. If you remove the randomness, the argument goes, you get a stable output you can test against.
The problem is that this theoretical determinism doesn't survive contact with production infrastructure. Research from Penn State and Comcast AI Technologies (Atil et al., arXiv:2408.04667) tested five LLMs configured to be deterministic across eight common tasks over ten runs. The results are striking: accuracy variations of up to 15% were observed across runs, with a gap between best and worst possible performance of up to 70%. None of the five models consistently delivered repeatable accuracy across all tasks.
The reason is a quirk of how modern GPUs do math. In short: the order in which a GPU performs millions of parallel calculations can vary slightly between runs, and those tiny differences in rounding accumulate into meaningfully different outputs — even when the model settings are identical.
The root cause is structural. Floating-point arithmetic on GPUs is non-associative — (a + b) + c does not necessarily equal a + (b + c) due to finite precision and rounding. When attention scores and logits are computed in parallel across GPU threads, the execution order can vary, and those rounding differences compound through the computation.
Add to this the co-mingling of data in input buffers across concurrent requests — a property that researchers note may be "essential to the efficient use of compute resources" — and you have non-determinism that is architectural, not incidental. Even self-hosted inference with open-source libraries like vLLM or SGLang doesn't eliminate it.
The Thinking Machines Lab analysis puts it plainly: the concurrency-plus-floating-point explanation is "not entirely wrong" but incomplete. The full picture is messier, and no major hosted LLM currently guarantees deterministic output even at temperature zero with a fixed seed.
Why comparing raw outputs between variants doesn't work
Even if you accept the variance, you might think you can still compare outputs statistically — run enough samples, measure quality scores, see which variant wins. The problem is that there's no stable ground truth for either variant. Variant A doesn't have "an output" — it has a distribution of outputs. Variant B has a different distribution.
Comparing a sample from one distribution to a sample from another, without accounting for within-variant variance, produces results that are easy to misread.
A practitioner observation from the Hacker News discussion on this topic captures the failure mode precisely: "You can't use correct unit tests or evaluation sets to prove anything about inputs you haven't tested." This generalizes directly to A/B tests. A result on a sampled set of outputs doesn't generalize to the full input distribution, especially when context sensitivity means the same user query can produce different outputs depending on what preceded it in the session.
Ignoring output variance compounds every other experimental error
The downstream consequences are predictable once you understand the mechanics. Teams that apply traditional A/B testing logic to LLM outputs without accounting for output variance will tend to attribute sampling noise to the treatment, under-power experiments because they've underestimated variance, and produce false positives from comparing unstable distributions.
A single-run evaluation of any LLM variant — the kind of quick check that's perfectly reasonable for a UI change — is an unreliable basis for a shipping decision when that variant's output can swing 15% in accuracy across runs under nominally identical conditions.
The standard pitfalls of A/B testing — Twyman's Law, measurement error, inflated false positive rates from running multiple metrics — don't disappear in LLM experiments. They compound. Higher structural variance means the signal-to-noise ratio is worse to begin with, which means every other source of experimental error matters more, not less. The solution isn't to abandon statistical rigor; it's to adapt the experiment design to account for what LLMs actually are.
Defining what you're comparing when A/B testing LLM outputs: prompts, models, and configurations
When engineers say they want to "A/B test their LLM," they're usually describing a goal, not an experiment. The goal is reasonable — figure out which version of the AI system works better. But "the AI system" is not a single variable.
It's a stack of independently configurable layers, and treating them as one undifferentiated thing to test is how you end up with results you can't act on.
The discipline that makes LLM experimentation valid is the same one that makes any experiment valid: change one thing at a time. In an LLM context, that requires first being clear about what the distinct "things" actually are.
Five independently testable layers in any LLM system
An LLM system has at least five layers that can be independently varied and therefore independently tested. Prompt wording and structure is the most obvious — the specific instructions, examples, and framing you pass to the model. Entirely separate from that is model version or provider: comparing GPT-4o against Claude against Gemini is a different experiment from comparing two prompt variants on the same model.
Braintrust explicitly treats these as parallel test surfaces in their evaluation tooling, run different prompt versions, models, or parameters simultaneously — not as a single combined test, but as distinct comparisons.
Hyperparameters — temperature, top-p, max tokens — constitute a third layer, controlling the sampling behavior of the model in ways that can meaningfully shift output characteristics independent of any prompt change. A fourth layer worth isolating is system prompt configuration and fine-tuning, particularly relevant for teams that have customized model behavior at the training or instruction level. Rounding out the five is the end-to-end user experience — how the AI output is surfaced, formatted, and integrated into the product — which often gets conflated with the others when it deserves its own test.
Latitude's framework for LLM evaluation maps these layers to different measurement concerns: technical performance metrics, user experience metrics, and business impact metrics. The implication is that different test surfaces tend to move different downstream indicators, which is another reason to keep them separate.
The isolation principle is not optional
Conflating these layers doesn't just make your results messy — it makes them causally uninterpretable. Braintrust puts the problem concretely: "The fundamental challenge with prompt development is that changes have unpredictable effects. Adding an example might improve one scenario while breaking another." If you simultaneously change the prompt and swap the model, you cannot know which one drove the change in outcomes.
You've run two experiments at once and produced evidence for neither.
GrowthBook's experimentation documentation states this as a general law: "The fewer variables that are involved in the experiment, the more causality can be implied in the results." In traditional A/B testing, violating this principle is a mistake you can sometimes recover from with more data. In LLM A/B testing, where outputs are already variable by design, it's closer to fatal — the background variation is already high enough that you cannot afford to add more confusion by changing two things at once. You need every methodological advantage you can get, and variable isolation is the cheapest one available.
The Hacker News community has felt this failure mode from the user side. When Anthropic ran undisclosed A/B tests on their API, one practitioner observed: "Any negative result can now be thrown straight into the trash because of the chance Anthropic put you on the wrong side of an A/B test." That's not just a complaint about transparency — it's a precise description of what happens when variable isolation breaks down. The signal becomes untrustworthy for everyone downstream.
From output quality to user behavior
The most important reframe in LLM experimentation is shifting the question from "which prompt produces better text?" to "which configuration produces better user outcomes?" These are not the same question, and optimizing for the first doesn't reliably produce improvements in the second.
Character.AI's use of GrowthBook for model-level comparison illustrates the distinction. Landon Smith, Head of Post-Training at Character.AI, describes the approach as comparing "different modeling techniques from the perspective of our users" — not evaluating output quality in isolation, but measuring how modeling choices translate into user experience. The test surface is the model layer; the measurement layer is user behavior.
This reframe also clarifies how to choose which layer to test first. If your hypothesis is about engagement, you're probably testing UX or prompt presentation. If your hypothesis is about output accuracy or cost, you're probably testing model version or hyperparameters.
The test surface should be determined by which downstream behavior you're trying to move — and that determination has to happen before you design the experiment, not after you look at the results.
Choosing metrics that can actually detect signal through LLM output variability
The measurement problem in LLM experiments isn't just technical — it's conceptual. Most teams approach LLM evaluation the same way they'd approach any software quality check: read some outputs, form an impression, maybe run a few spot-checks. That approach collapses under experimental conditions.
When you're trying to detect whether prompt variant A outperforms prompt variant B across thousands of interactions, impressions don't aggregate. You need metrics that do.
Why output text is a poor primary metric
Because of the output variance established in the first section, comparing text strings directly across variants doesn't produce signal — it produces noise dressed up as data.
Traditional text similarity metrics like BLEU and ROUGE make this worse, not better. They measure surface-level token overlap, which fails to capture semantic equivalence. A response that says "the account balance is insufficient" and one that says "you don't have enough funds" score poorly against each other despite being functionally identical.
Confident AI's evaluation guidance is explicit on this point: semantic nuance in LLM outputs is not captured by statistical scorers, which is precisely why they're unreliable as primary metrics in experiments.
The practical consequence is that if you anchor your experiment to output text quality — even with automated scoring — you're measuring the wrong thing. What you actually want to know is whether one variant produces better downstream outcomes for users. That requires different instrumentation entirely.
Quantitative behavioral and operational metrics worth tracking
The metrics that aggregate cleanly in LLM experiments are the ones that describe what users do in response to outputs, and what the system costs to produce them. Latency, token usage, cost per query, task completion rate, user retention, and conversion events are all measurable, stable, and statistically tractable in ways that output text is not.
The trade-off between these dimensions is real and often non-obvious. Adding a Chain-of-Thought instruction to a prompt might improve factual accuracy while doubling latency and token cost. If you're only tracking accuracy, you'll ship a change that degrades user experience and inflates your inference bill. Tracking both dimensions simultaneously is what makes the trade-off visible before it becomes a production problem.
For teams whose behavioral metrics — retention, conversion, session depth — live in a data warehouse rather than a real-time event stream, warehouse-native experimentation removes the need to re-instrument data collection. GrowthBook connects directly to your existing data warehouse — Snowflake, BigQuery, Redshift, ClickHouse, Databricks, Athena, Postgres, and more — which matters when LLM experiment outcomes are measured in days or weeks, not seconds.
Latitude's framework for mapping metrics to measurement methods is useful here: technical performance metrics like response time and F1 score are best captured through automated evaluation; user experience metrics like satisfaction ratings come from feedback mechanisms and system logs; business impact metrics like retention come from analytics pipelines. Each category requires different instrumentation, and all three are necessary for a complete picture.
One important constraint: tracking too many metrics simultaneously inflates your false positive rate in ways that can make experiments misleading. GrowthBook's A/A test analysis quantifies this directly — with five unrelated metrics in an experiment, the probability of at least one false positive reaches 41%. With two metrics, it's 19%.
The implication for LLM experiments is that metric selectivity isn't just good hygiene; it's a statistical necessity. Pick the metrics most causally connected to your experiment goal, and resist the temptation to track everything because you can.
What quantitative metrics miss that users notice immediately
Quantitative metrics tell you what happened; qualitative feedback tells you why. Thumbs up/down signals, satisfaction ratings, session logs, and targeted surveys are harder to aggregate statistically, but they provide ground truth that automated metrics routinely miss.
Users notice when LLM quality degrades — in tone, in relevance, in the subtle ways a response fails to address what they actually asked — even when task completion rates hold steady.
The Hacker News community's reaction to undisclosed LLM A/B testing makes this concrete: practitioners are acutely aware of quality variation and attribute negative experiences to experimentation when they notice inconsistency. That perception matters, and it won't show up in your latency dashboard.
LLM-as-judge and automated scoring approaches
For teams that need scalable quality assessment without manual review at every step, LLM-as-judge has emerged as the most practically useful approach. The pattern is straightforward: use a separate LLM to evaluate outputs against natural language rubrics — criteria like faithfulness to source material, relevance to the user's query, or absence of hallucination — and produce structured scores that can be aggregated across experiment arms.
Named approaches in this space include G-Eval, Prometheus (the evaluator model, distinct from the infrastructure monitoring tool of the same name), GPTScore, SelfCheckGPT, and QAG Score. These model-based scorers are more semantically aware than statistical alternatives, though they carry higher inference cost and introduce their own consistency questions.
The right metrics also depend on what you're building. For RAG pipelines, the relevant dimensions are faithfulness, answer relevancy, and contextual precision and recall. For agent systems, task completion rate, tool correctness, and step efficiency matter more than fluency. Applying generic quality metrics to specialized systems produces generic, unhelpful results.
The practical synthesis: a measurement stack for LLM experiments should combine operational metrics (latency, cost, token usage) with behavioral outcomes (retention, task completion, conversion), qualitative feedback signals, and automated scoring on the dimensions most relevant to your specific system. Experiment platforms that surface metric correlations are worth noting here — the ability to visualize how experiments jointly affect two metrics simultaneously is particularly useful for surfacing the trade-offs that LLM experiments routinely produce.
Landon Smith, Head of Post-Training at Character.AI, describes the value directly: "GrowthBook has been an invaluable tool for Character.AI, helping us develop our models into a great consumer experience. We can compare different modeling techniques from the perspective of our users — guiding our research in the direction that best serves our product." That framing — comparing modeling techniques from the perspective of users — is the right orientation for LLM measurement generally. The text of the output is an intermediate artifact. What users experience, and what they do next, is the actual signal.
When to use offline prompt evaluation versus live A/B tests in production
Offline evaluations and live A/B tests are not two names for the same thing. They answer different questions, and conflating them is one of the more reliable ways to ship a prompt change that looks good in your notebook and quietly degrades user experience in production.
The practical question most teams face isn't which method to use — it's understanding what each one actually tells you, and in what order to use them.
What offline evaluation is good for — and where it breaks down
Offline evaluation means running your prompt or model against a fixed, labeled dataset and scoring outputs against defined criteria. The appeal is real: it's fast, repeatable, and makes regression testing tractable. When you have a stable evaluation set, you can quickly see whether a new prompt version improves or degrades performance on known cases before any user sees the change.
The limitation is equally real. As Label Studio puts it directly: "A model can look improved offline and still cause new issues in real usage." Your evaluation dataset is always an approximation of production. It underrepresents edge cases, misses novel user behaviors, and reflects historical patterns that may have already shifted.
Offline evals also struggle with subjective quality dimensions — tone, helpfulness, policy adherence — unless you've built a rubric that reliably captures those qualities, which is harder than it sounds. A model that clears every benchmark you've defined can still fail the moment it encounters the actual distribution of things your users ask.
What live A/B testing adds that offline evals cannot
Live tests expose your LLM to production inputs — the full, unpredictable distribution of real user queries, including the long tail of phrasings and edge cases that no curated dataset fully represents. More importantly, they measure downstream user behavior: engagement, task completion, satisfaction signals. Those are the outcomes that actually matter to the product, and they're invisible to offline scoring.
Statsig frames this plainly: "A model can look great in a notebook, then fall apart once it meets production traffic and real users." Offline gains are hypotheses. Live tests are where you confirm whether those hypotheses hold.
The two-stage workflow: validate offline, then route live traffic
Both Label Studio and Statsig independently converge on the same sequencing logic, which makes it worth treating as consensus rather than one team's opinion: use offline evaluation to iterate quickly, and use online testing to confirm that improvements are real.
In practice, the trigger for starting offline evals is any meaningful change — a new model version, a major prompt rewrite, or a change that puts cost, latency, or safety targets at risk. Lock a dataset, define a grader, anchor your decisions on that fixed setup. When offline gains look steady, that's the signal to move — not when they look perfect.
Statsig's guidance here is worth internalizing: "Do not wait for perfection. Move from sandbox to production when gaps narrow, not vanish."
The transition to live testing doesn't mean full exposure. Start with shadow scoring — run both variants against live traffic, score them silently, and compare grades before any user sees a difference. Then open a small traffic flight. Scale as wins persist.
Why controlled rollout requires traffic routing, not just deployment
The two-stage workflow is only practical if you have a reliable mechanism for routing a fraction of live traffic to a new variant without a full deployment. Feature flags are that mechanism. They let you expose a new prompt or model to, say, five percent of users, monitor behavioral metrics in real time, and roll back instantly if guardrail metrics degrade — without touching your deployment pipeline.
This is where the architecture of your experimentation platform matters. GrowthBook SDKs download flag rules as a locally cached payload and evaluate every flag check in-process with zero network latency, which means flag evaluation itself doesn't add latency to LLM inference — a meaningful consideration when you're already managing token generation time.
Consistent user bucketing (the same user always receives the same variant) prevents within-user contamination that would otherwise make behavioral metrics uninterpretable. Platforms that connect guardrail metric monitoring directly to traffic controls close the loop between observation and action — GrowthBook's Safe Rollouts monitor guardrail metrics like error rates and latency, and surface warnings directly in the rollout dashboard, within the same interface used to manage the experiment itself.
The offline-to-live sequence isn't a theoretical best practice. It's the operational response to the non-determinism problem: because you can't fully anticipate what production inputs will look like, no amount of offline testing substitutes for real signal — but shipping without offline validation first is how you expose users to regressions you could have caught in an afternoon.
Five design decisions that determine whether your LLM experiment statistics are valid
Standard statistical principles don't stop working when you're testing LLM outputs — but they do require deliberate adaptation. The core challenge isn't that rigor becomes impossible; it's that the assumptions baked into standard experiment design (stable variance, clean randomization, interpretable output comparisons) quietly break in ways that produce confident-looking results that are actually wrong.
One team ran a two-week A/B test on an improved prompt, saw p=0.03 at +1.2% improvement, called it a win, and rolled out fully — only to discover six months later, through a customer audit, that the prompt had been producing subtly incorrect summaries the entire time. Click-through rates and session lengths looked fine. The semantic quality had drifted in ways the metrics couldn't see.
The fix isn't to abandon statistical testing. It's to make five specific design decisions correctly.
Power analysis when variance isn't constant
Standard power analysis calculations assume roughly constant variance across your input distribution. LLM output variance doesn't work that way — it's heteroskedastic, scaling with task difficulty. Simple lookup queries may show only 5–10% output divergence across runs; complex reasoning tasks can show 40–60% divergence.
When you run a single aggregate power calculation across a mixed workload, your sample size estimate will be accurate for the easy queries and wildly wrong for the hard ones.
The practical implication: treat any standard power calculation as a floor, not a target. GrowthBook's general rule of thumb — at least 200 conversion events per variation — is a reasonable minimum starting point, but in high-variance LLM contexts it's almost certainly insufficient for detecting meaningful differences in complex task performance. Where possible, segment your power analysis by task complexity and size for the hardest segment. If you can't segment, apply a conservative multiplier and accept that you'll need more data than a standard calculator suggests.
Randomize on users, not requests
This is the most common structural mistake in LLM experiment design. If you randomize on individual requests, the same user can receive variant A in one session and variant B in another. That contaminates the experiment and makes your results uninterpretable — you're no longer measuring the effect of a treatment on a person, you're measuring a noisy mixture of both treatments on the same people.
The correct randomization unit is the user. Latitude explicitly recommends this, and the implementation pattern matters: assignment needs to be stable across sessions without requiring stored state. GrowthBook handles this through a consistent hashing algorithm (MurmurHash3) that maps user identifiers to variants deterministically — the same user always gets the same variant as long as experiment settings don't change, with no additional cookies required.
The Hacker News community has flagged exactly this concern from the other side: when LLM providers run silent A/B tests with unstable assignment, any longitudinal research on tool effectiveness becomes uninterpretable. Stable, transparent user-level assignment is what makes results defensible.
Test duration and novelty effects
The minimum duration principle exists in standard A/B testing because traffic patterns vary across days and weeks. In LLM experiments, there's an additional reason to enforce it: novelty effects. Users often behave differently when they first encounter a new AI response style — they may engage more, explore more, or rate responses more generously simply because the experience is new. That early signal doesn't reflect steady-state behavior.
GrowthBook's documentation recommends 1–2 weeks as a typical minimum, with some tests requiring a month or more. For LLM experiments, treat two weeks as the floor and be especially skeptical of results that look strong in the first few days. If your experiment shows a strong positive signal in days one through three and then flattens, you're likely looking at novelty, not a real effect.
Statistical methods for aggregated behavioral outcomes
Because you're measuring user-level behavioral outcomes rather than output-level quality scores, the statistical methods that apply to LLM A/B testing are the same ones that apply to any behavioral experiment — with one important adaptation. Variance reduction techniques matter more here than in typical product experiments, because LLM output variance inflates the noise in your downstream metrics.
GrowthBook supports CUPED (Controlled-experiment Using Pre-Experiment Data) and post-stratification, both of which reduce variance by controlling for pre-experiment user characteristics. In practice, CUPED can meaningfully reduce the sample size required to reach significance — which matters when you're running experiments that need weeks of data to accumulate.
GrowthBook's activation metric feature handles the filtering explicitly: users who were assigned to an experiment but never actually triggered the LLM feature should be excluded from analysis, or they'll dilute your effect estimate.
GrowthBook offers both frequentist and Bayesian engines, with Bayesian as the default. For LLM experiments where you expect small effect sizes and high variance, Bayesian methods have a practical advantage: they let you reason about the probability that one variant is better, rather than forcing a binary significant/not-significant decision on noisy data. Consistent user bucketing and activation metric filtering are worth enforcing here, specifically because they handle the structural mistakes that contaminate LLM experiment results most often.
Canary deployments as a risk management layer
The fifth design decision isn't statistical — it's operational. Before you commit to a full experiment, a canary deployment lets you expose a small slice of traffic to the new variant and monitor guardrail metrics before the experiment proper begins. This is particularly valuable for LLM changes because the failure modes are often qualitative and slow to surface: a prompt that produces subtly worse responses won't show up in latency metrics, but it will eventually show up in retention and satisfaction signals.
Feature flag rollout controls support gradual exposure with percentage-based rollouts, targeted rollout by user segment, and instant kill switches — which maps directly to what a canary pattern requires in practice. Start at one to five percent of traffic, watch your guardrail metrics for 24–48 hours, and only open the full experiment if nothing alarming appears. This doesn't replace the experiment — it de-risks the experiment by catching catastrophic failures before they affect your full user base.
Putting it together: the sequence that makes LLM experimentation defensible
The five sections above each address a distinct failure mode. The non-determinism section explains why standard assumptions break. The variable isolation section explains what you're actually testing. The metrics section explains what to measure. The offline-versus-live section explains the sequencing. The statistical design section explains how to make the numbers trustworthy. None of these is optional, but they're also not independent — they form a sequence, and the sequence matters.
Start with offline evals before you touch production traffic
Before any live traffic is involved, you need a stable evaluation setup: a locked dataset, a defined grader, and a baseline score for your current configuration. This doesn't need to be elaborate — it needs to be stable. The purpose isn't to prove your new variant is better; it's to establish a reference point that makes regression detectable.
The metric selectivity principle discussed earlier — where five simultaneous metrics produce a 41% false positive rate — applies with even more force when you're scaling to multiple sequential experiments. Lock your primary metric before you start, and treat secondary metrics as diagnostic rather than decisive.
When offline gains look steady across multiple evaluation runs, that's the signal to move to live traffic. Not when they look perfect — when they look consistent.
Build a measurement stack that survives output variability
The measurement infrastructure needs to be in place before the experiment starts, not assembled during it. That means instrumentation for behavioral outcomes (task completion, retention, conversion), operational metrics (latency, token usage, cost), and at least one qualitative feedback signal.
For teams running warehouse-native experimentation, this means confirming that your behavioral events are flowing into your data warehouse and that your experiment platform can query them directly — avoiding the need to re-instrument or duplicate data pipelines.
The LLM-as-judge layer, if you're using one, should be validated offline before it's used to score live traffic. An evaluator that produces inconsistent scores is worse than no evaluator, because it adds noise to a signal that's already noisy.
Scale to live experiments with proper statistical controls
The live experiment phase is where the statistical design decisions from the previous section become operational. User-level randomization needs to be confirmed before traffic opens. The activation metric filter needs to be set so that unexposed users don't dilute the analysis. The minimum duration needs to be enforced even if early results look strong.
Canary first, then full experiment. Watch guardrail metrics during the canary phase. If nothing alarming surfaces, open the full experiment and let it run to the pre-specified duration. Resist the temptation to call it early — the novelty effect is real, and results that look strong in the first week often flatten by week two.
Where to begin depending on where you are
The right entry point depends on where your team currently sits.
Teams that have no existing evaluation infrastructure should start with offline evals, not live experiments. Build a dataset of 50–100 representative inputs, define a grader (even a simple rubric scored manually), and establish a baseline score for your current prompt or model. That baseline is what makes future changes interpretable. Without it, you have no reference point, and every experiment result is floating in space.
Teams that have offline evals but have never run a live LLM experiment should focus on the randomization and duration decisions before anything else. Confirm that your experiment platform assigns users — not requests — to variants, and that assignment is stable across sessions. Set a minimum duration before you start, and commit to it. These two decisions prevent the most common structural errors that produce false positives in LLM experiments.
For teams already running live experiments but getting results that feel unreliable, the most likely culprits are metric selection and sample size. Check whether you're tracking more than two or three metrics simultaneously — if so, your false positive rate is almost certainly elevated. Check whether your power calculation accounted for the variance in your specific task distribution, or whether it used a generic formula. Both of these are fixable without changing your experiment infrastructure.
Related Articles
In healthcare, “Can we randomize it?” is the wrong first question. Start with “Could either experience change care, rights, privacy, or access?”
A/B testing can improve digital intake, appointment access, patient education, clinician workflows, and administrative operations. It can also create unacceptable risk when teams treat a clinical or consent decision like an ordinary conversion funnel.
The difference is not the label on the method. A/B tests are randomized experiments. What matters is the treatment, purpose, affected population, data flow, and oversight required in the organization and jurisdiction. This guide provides a practical product framework, not a substitute for legal, clinical, privacy, security, or institutional review.
Draw the boundary before designing variants
Create an intake step that classifies the proposed change before anyone builds a treatment. At minimum, ask:
- Can the change alter diagnosis, treatment, triage, dosage, or clinical recommendations?
- Can it delay or discourage access to care, accommodations, or urgent help?
- Does it change informed consent, privacy choice, required disclosure, or patient cost?
- Does it use protected or sensitive health information for assignment or measurement?
- Does it include children, people in crisis, or another population requiring added protection?
- Is the purpose internal quality improvement, or is it designed to contribute to generalizable knowledge?
- Could the software function fall within medical-device or clinical decision-support oversight?
The HHS quality-improvement guidance says many activities limited to improving patient care and collecting operational data are not research under the cited human-subjects regulations. It also states that some quality-improvement activities can have a research purpose, in which case human-subject protections may apply. A product team should not make that determination informally; route it to the organization’s authorized office.
Likewise, software that influences clinical decisions is not automatically an ordinary product surface. The FDA’s January 2026 clinical decision-support guidance explains that some software functions are excluded from the device definition while other patient- or caregiver-facing functions can remain subject to digital-health policy. Clinical and regulatory owners need to classify the function before experimentation.
Start with lower-risk operational questions
The safest early program tests reversible changes where both variants meet the same clinical, accessibility, privacy, and disclosure requirements.
Appointment reminder timing
Compare 2 approved reminder schedules or message structures to reduce missed appointments. Keep required details, opt-out behavior, language support, and urgent-contact instructions constant.
Use completed appointments or timely rescheduling as the primary outcome. Track cancellations, patient contacts, message delivery, opt-outs, wrong-recipient risk, and differences across language, age, disability, or access groups. A higher click rate is not enough if no-show rates or trust worsen.
Patient portal navigation
Test whether a clearer information architecture helps people complete a high-value administrative task, such as finding results, updating insurance, or sending a non-urgent message. Preserve emergency guidance and clinical escalation paths in both variants.
Measure successful task completion and time to completion. Guard against repeated navigation, abandonment, accessibility failures, mistaken message routing, and increased call-center burden. Use usability testing before the A/B test to catch failures randomization should never expose.
Administrative form sequence
Compare a long form with a staged flow, or test the order of non-clinical fields. Do not omit information needed for safe care, billing transparency, consent, or legal compliance.
Measure accurate completion, not just submission. Track validation errors, correction rates, staff rework, abandonment, and time to appointment. If the treatment collects sensitive data, confirm necessity and access controls before launch.
Educational content layout
Test 2 ways to present the same clinician-approved information: summary-first versus stepwise, text plus illustration versus text alone, or a clear action checklist versus a dense paragraph. Keep the medical meaning, risks, contraindications, and escalation advice equivalent.
Use a comprehension or appropriate next-action metric when feasible. Page time and clicks can be misleading. Accessibility, language quality, and comprehension across health-literacy levels belong in the guardrail plan.
Review the design before launch
Use a trustworthy experiment-design session to pressure-test metrics, safety checks, and decision rules before exposing patients or clinicians.
Watch the Experiment Design Session
Use stronger controls for care-adjacent products
Some product changes are not clinical interventions but can still influence care. They need clinical ownership, narrower eligibility, conservative ramps, and explicit stopping criteria.
Clinician workflow support
A test might compare how a work queue prioritizes administrative follow-up, how a note template reduces documentation work, or how a non-diagnostic alert is presented. The treatment should not silently alter the clinical standard of care.
Randomize at the unit that prevents contamination. Individual clinician assignment may fail when teams share queues and handoffs; clinic- or unit-level clusters may better match the workflow. Measure task completion and time saved, with guardrails for missed work, overrides, escalations, documentation quality, and staff workload.
Preventive-care outreach
Compare approved outreach content or channels for people already eligible under the same clinical rule. Do not experiment with whether one group receives necessary care or required notice.
Use completed appropriate follow-up as the primary outcome. Track opt-outs, unreachable patients, scheduling capacity, disparities, complaints, and downstream cancellations. If the treatment drives demand beyond operational capacity, a messaging lift can make access worse.
Digital adherence support
Test the presentation or timing of an approved reminder, checklist, or educational cue. Avoid treatment changes that could be interpreted as personalized medical advice without the corresponding validation and oversight.
Measure the intended behavior with caution. Self-reported completion or app engagement is not a clinical outcome. Include adverse-event reporting, escalation pathways, disengagement, and privacy events where relevant.
Feature rollout in health software
Use feature flags to separate deployment from release, start with internal or trained cohorts, and expand only when technical and clinical guardrails remain healthy. GrowthBook’s feature flag platform supports targeted rollouts and kill switches, while the experiment layer measures impact.
The rollback plan must describe more than turning off a flag. Determine whether the old experience remains clinically and operationally safe, how queued work is reconciled, what happens to partial workflows, and who is authorized to stop exposure.
Protect data by design
Do not send a broad event stream to an experimentation vendor and decide later which fields were unnecessary. Inventory the data before implementation:
| Data question | Required decision |
|---|---|
| Assignment | What is the least identifiable stable unit that works? |
| Eligibility | Which sensitive attributes are truly needed? |
| Exposure | What event proves the treatment was delivered? |
| Outcomes | Can metrics be computed inside the governed data environment? |
| Access | Which roles can view assignments, segments, and results? |
| Retention | When are raw records, logs, and exports removed? |
The HHS minimum-necessary guidance describes limiting uses, disclosures, and requests for protected health information to what is needed for the intended purpose, with policies based on roles and recurring versus non-routine access. Apply that principle to experiment attributes, debugging logs, dashboards, and downloaded readouts.
Pseudonymous identifiers reduce exposure but do not automatically make a dataset non-sensitive or outside applicable rules. Review linkability, small cohorts, free-text fields, URLs, device metadata, and combinations that can reveal a condition. Never put clinical details or identifiers in feature names, variation labels, or URLs.
A warehouse-native experimentation approach can query approved metrics where the organization already governs them. Architecture does not create compliance on its own; teams still need contracts, access control, auditability, retention rules, security review, and configuration that matches the approved data flow.
Keep unsafe questions out of product experimentation
An experimentation policy should name prohibited or separately governed categories. Product teams should not discover the boundary only after a proposal reaches launch review.
Do not use an ordinary product A/B test to withhold a clinically indicated service, emergency direction, safety warning, accessibility accommodation, required disclosure, or legally protected choice. Do not reduce the visibility of risks to improve completion. Do not randomize a diagnostic or treatment recommendation without the clinical, regulatory, and research framework appropriate to that intervention.
Avoid treatments that exploit fear, urgency, shame, or uncertainty about health. A message can increase appointment conversion while undermining informed choice. Likewise, do not test whether patients tolerate a harder cancellation, more confusing privacy control, or hidden cost. Both variants must meet the organization’s baseline standard for respectful and comprehensible communication.
Clinical AI and decision-support changes need an evaluation program beyond a click-based A/B test. Validate the model offline, examine performance and failure modes across relevant populations, review human factors, and stage deployment with clinical monitoring. An online comparison may contribute evidence only after both treatments meet the safety threshold for exposure.
When an activity may be human-subjects research, follow the institution’s process before enrolling or exposing anyone. HHS research-oversight training states that covered non-exempt human-subjects research requires the applicable review and that informed consent requirements apply unless the IRB authorizes otherwise. The product team should preserve the determination, protocol version, approved treatment, and reporting obligations with the experiment record.
Finally, do not interpret lack of detected harm as proof of safety. Rare adverse events, small vulnerable groups, and outcomes that occur after the experiment window may be underpowered. Use prior evidence, incident monitoring, qualitative reports, and post-rollout surveillance alongside the randomized estimate.
Define patient-centered metrics and guardrails
Healthcare teams need more than a conversion scorecard. Build a measurement hierarchy:
- Primary outcome: the operational or patient-facing result that answers the decision.
- Process diagnostics: steps that explain why the treatment worked or failed.
- Safety guardrails: outcomes that trigger a stop or clinical review.
- Equity checks: predeclared groups where access or benefit could differ.
- Operational guardrails: staffing, wait time, rework, cost, and downstream capacity.
Define the practical threshold before launch. A statistically detectable change may be too small to justify implementation, and a neutral aggregate can hide meaningful harm in a protected or vulnerable group. At the same time, slicing results across many small subgroups increases false-positive risk and can expose sensitive attributes. Predeclare the equity questions that matter and use appropriate privacy and multiple-testing controls.
GrowthBook supports reusable fact tables and metrics so teams can keep definitions reviewable. Use a power analysis for the primary outcome and critical guardrails. If the required sample or duration is unrealistic, do not weaken the standard; use usability research, simulation, staged quality improvement, or a larger treatment contrast.
Create a healthcare experiment review packet
Before launch, the owner should provide one reviewable packet:
- purpose, hypothesis, and operational decision
- classification and required oversight determination
- affected population and exclusion criteria
- clinical, privacy, security, accessibility, and compliance approvals
- treatment screenshots or workflow diagrams
- assignment, exposure, and data-flow design
- primary outcome, diagnostics, guardrails, and equity checks
- sample plan and stopping rule
- rollout stages, monitoring owner, and rollback procedure
- patient or clinician communication plan, if applicable
- documentation and retention plan
Use an approval matrix that names accountable people. Product approval does not replace clinical approval; a privacy review does not settle human-subjects research status; and an IRB determination does not automatically approve the production security architecture.
The WHO clinical-trial best-practices guidance emphasizes ethical standards, regulatory considerations, patient-centered research, transparency, and stakeholder collaboration. Not every healthcare product experiment is a clinical trial, but high-risk work should inherit the same respect for people and evidence.
Build trust into the experimentation program
Start with reversible operational improvements where both experiences are already acceptable. Prove that the team can classify risk, minimize data, validate assignment, monitor safety, and document decisions before expanding scope.
Publish internal rules for what teams may test, what requires added review, and what is out of bounds. Maintain an experiment registry and audit trail. Record neutral and negative results so a new team does not repeat the same risky idea.
GrowthBook can support the controlled delivery and analysis layer through experimentation, feature flags, permissions, and warehouse-defined metrics. The organization remains responsible for the clinical, ethical, legal, privacy, and operational framework around every test.
In healthcare, speed is valuable only when the learning process protects the people whose behavior creates the data.
Build a governed test workflow
Connect controlled releases to reviewable metrics and decision rules while keeping healthcare data in your approved architecture.
Get Started With GrowthBook
The right statistical test is determined by the question and data-generating process, not by which function is easiest to run. Start with the outcome, groups, and dependence structure; the test name comes later.
Z-tests, t-tests, chi-square tests, and analysis of variance (ANOVA) all compare observed data with a null model. They differ in the kind of outcome they model, the uncertainty they estimate, and the number or structure of groups they can compare.
For a simple product experiment, a useful first pass is:
- continuous outcome, two independent groups: usually a Welch two-sample t-test
- binary proportion, two large independent groups: a two-proportion z-test is common
- categorical counts across groups: chi-square test, if expected counts are adequate
- continuous outcome across three or more groups: one-way ANOVA or Welch ANOVA
Those rules are a starting point. Paired observations, clusters, ratios, repeated measures, heavy tails, covariate adjustment, or sequential monitoring require a model that reflects the design.
Choose from the outcome and hypothesis
Write the estimand before choosing a test. An estimand is the quantity the experiment is trying to estimate: a difference in mean revenue, a difference in conversion probability, or an association between two categorical variables.
| Question | Outcome | Common test |
|---|---|---|
| Did average order value change between A and B? | Continuous | Welch two-sample t-test |
| Did signup probability change between A and B? | Binary | Two-proportion z-test |
| Is plan choice associated with variant? | Categorical, 3+ levels | Chi-square test of independence |
| Do mean task times differ across four variants? | Continuous | One-way ANOVA |
| Did the same users' scores change before and after? | Paired continuous | Paired t-test |
The number of groups alone is insufficient. Conversion in four variants is still categorical data; a chi-square or binomial model may fit. Revenue in two groups is continuous; a t-test or regression is more natural.
The University of Michigan's statistical-test guide uses the same sequence: identify variable types and the relationship being tested before selecting a method.
When to use a z-test
A z-test compares a standardized estimate with the standard normal distribution. The classical one-sample z-test for a mean assumes the population standard deviation is known. That condition is unusual in product analytics, where variability is estimated from the current sample.
Z-tests remain common for proportions. In a two-arm conversion experiment, the estimate is:
Under the null of equal proportions and with adequate counts, the standardized difference is approximately normal. This yields a two-proportion z-test.
Use it when:
- the outcome is a binary count summarized as successes and failures
- assignment groups are independent
- sample sizes make the normal approximation credible
- the hypothesis and one- or two-sided direction were set before analysis
Do not rely on a universal “n greater than 30” rule. For rare events, 30 observations can produce almost no successes; for balanced common events, approximation quality can be good. Inspect expected successes and failures and use an exact or model-based method when counts are sparse.
In high-volume online experiments, a normal approximation is also used for many sample means through the central limit theorem. The important question is whether the estimator's sampling distribution and variance calculation are valid for the metric, not whether the raw user values look perfectly normal.
When to use a t-test
A t-test is designed for inference about means when the variance is estimated from sample data. That extra variance uncertainty produces a t distribution with heavier tails than the standard normal, especially at small sample sizes.
For two independent groups, default to Welch's t-test unless equal variance is justified. Welch's version does not assume the two population variances are equal and handles unequal group sizes. NIST's two-sample t-test reference shows the unequal-variance standard error based on each group's sample variance and size.
Use an independent two-sample t-test when:
- the outcome is numeric and the mean is the target
- the two groups contain different experimental units
- observations are independent within the model
- the mean and standard error behave well enough for the sample size
Use a paired t-test when each value has a meaningful partner: the same user's before-and-after score, or deliberately matched units. The analysis reduces each pair to a difference and tests the mean of those differences. Treating paired data as independent discards information and computes the wrong standard error.
The t-test can be sensitive to extreme values because the sample mean and variance are sensitive to them. Product metrics such as revenue or session duration are often skewed. At scale, the mean may still have a usable sampling distribution, but inspect outliers, data quality, and the estimand. Robust inference, transformations, winsorization policies, or bootstrap methods may be more appropriate when a few observations dominate the result.
Reduce variance before launch
Learn how CUPED and covariate adjustment can sharpen experiment estimates without changing the randomized comparison.
Explore Variance Reduction
When to use a chi-square test
Pearson's chi-square statistic compares observed category counts with counts expected under a null hypothesis. Two common forms are:
- goodness of fit: does one categorical distribution match specified probabilities?
- independence or homogeneity: is a categorical outcome distributed the same way across groups?
Suppose an onboarding experiment records three outcomes: completed, skipped, and abandoned. Cross-tabulate outcome by variant. A chi-square test asks whether the outcome distribution is independent of variant.
The test statistic sums (observed - expected)^2 / expected across cells. NIST's chi-square documentation describes the same comparison of binned frequency distributions.
Use a chi-square test when observations contribute counts to mutually exclusive categories and expected cell counts are large enough for the asymptotic approximation. With sparse cells, combine categories only when substantively justified or use an exact method such as Fisher's exact test for a two-by-two table.
A chi-square result says the distributions differ somewhere. It does not provide the most decision-friendly effect estimate by itself. Report category proportions, absolute differences, uncertainty intervals, and the cells contributing to the pattern.
For a binary two-arm experiment, the Pearson chi-square test and a two-sided two-proportion z-test are closely related: under standard conditions, the chi-square statistic with one degree of freedom equals the squared z statistic. Choose the representation that matches the hypothesis and reporting needs.
When to use ANOVA
ANOVA compares variation between group means with unexplained variation within groups. A one-way ANOVA tests the null that all population means are equal across levels of one factor.
Use it for a continuous outcome across three or more independent groups when the global question is whether any mean differs. Classical ANOVA assumes independent errors, normally distributed residuals within the model, and equal variances. Welch ANOVA relaxes the equal-variance assumption; R's 0 implements that approximation.
ANOVA's F-test is an omnibus test. A significant result means at least one mean differs, but it does not identify which one. Use planned contrasts or multiplicity-aware post-hoc comparisons to answer the product question.
ANOVA is more than a rule for “three or more groups.” Multi-factor ANOVA can estimate main effects and interactions in multivariate or factorial experiments. Repeated-measures or clustered data need corresponding error structures rather than a basic one-way calculation.
Why several t-tests are not a substitute for ANOVA
With four variants there are six pairwise comparisons. Testing each at 0.05 creates multiple opportunities for a false positive. An omnibus ANOVA tests one global null first, and planned follow-ups can use Tukey, Holm, Bonferroni, or another procedure appropriate to the family of claims.
The Bonferroni correction is simple and conservative. The right procedure depends on whether the goal is all pairwise comparisons, treatments versus one control, or a small set of preplanned contrasts. Define that family before looking at the ranking.
ANOVA and regression are also two views of the same linear-model machinery. R's 0 documentation describes aov as a wrapper around linear models for experimental designs. Regression is often more flexible when the analysis includes covariates, interactions, or unbalanced data.
Assumptions that change the choice
Before running any of the four tests, verify:
Independence and assignment unit
If the experiment randomizes accounts but analyzes users as independent observations, standard errors will usually be too small. Analyze at the randomization unit or use cluster-aware inference. If users can appear in both groups, repair the assignment or use a model that represents the dependence.
Paired or repeated observations
The same user measured twice is not two independent users. Use a paired test or repeated-measures model. For experiments with many events per user, aggregate to the user level or use appropriate clustered methods.
Outcome distribution and metric construction
Check missingness, zero inflation, extreme tails, ratio denominators, and censoring. A test can be mathematically correct for the supplied numbers while the metric itself misrepresents the user outcome.
Variance assumptions
Prefer Welch's t-test or Welch ANOVA when group variances may differ. Equal sample sizes do not prove equal variance, and a preliminary variance test can introduce another decision layer.
Sample size and sparse cells
Approximate z and chi-square methods need enough information in the relevant cells. Low-frequency guardrails and small segments may need exact methods or longer collection.
A product experimentation decision tree
Use this sequence before opening a statistics package:
- What unit was randomized: user, account, device, session, or region?
- What is the primary estimand: mean, proportion, category distribution, or model coefficient?
- Are groups independent, paired, repeated, or clustered?
- Are there two groups, several groups, or multiple factors?
- Do expected counts and sample sizes support the approximation?
- Are variances, tails, or outliers likely to break the default model?
- How many confirmatory hypotheses can trigger the decision?
- Was the test direction and stopping rule declared before launch?
Then choose the simplest model that answers the exact question. A two-proportion z-test may be perfect for signup conversion, while a t-test handles mean revenue and a chi-square test handles plan mix in the same experiment. Different metrics can require different tests.
Report effects, not only test names
The test produces a statistic and p-value under a null model. The guide to interpreting a t-test p-value shows why that number needs the effect, interval, and degrees of freedom beside it. The product decision needs more:
- the effect estimate in business units
- a confidence or credible interval
- sample sizes and allocation
- baseline and treatment values
- assumption and data-quality checks
- the planned hypothesis family
- practical thresholds and guardrails
GrowthBook's statistics documentation explains the frequentist and Bayesian engines available for experiment analysis. Whichever framework is used, review effect magnitude and uncertainty together. A small p-value can accompany a trivial lift in a huge sample, while a valuable estimated lift can remain uncertain in a small one.
Choose the test by tracing the data back to the experiment design. For three or more continuous-outcome variants, the deeper ANOVA guide covers the omnibus F-test, planned contrasts, and Welch alternative. When the outcome, assignment unit, dependence, and hypothesis are explicit, the difference between z, t, chi-square, and ANOVA becomes a modeling decision rather than a memorization exercise.
Analyze tests with context
Connect experiment assignments to trusted metrics, inspect uncertainty, and keep decision rules visible to the whole team.
Get Started With GrowthBookAn experiment with control plus three variants creates more than one comparison. ANOVA gives the team one principled global test of whether the variants differ before it starts hunting for a winner.
Analysis of variance, or ANOVA, is a family of statistical models for comparing group means and decomposing sources of variation. In a one-way product experiment, the “factor” is the assigned variant and its “levels” are control, B, C, and D.
The basic ANOVA question is deliberately broad: if all variants had the same population mean, would the observed separation among their sample means be surprising relative to the noise within variants?
That question is useful, but incomplete. A significant ANOVA result does not say which variant won, whether the lift is large enough to ship, or whether assumptions and instrumentation are sound. Those conclusions require planned contrasts, uncertainty intervals, and experiment-quality checks.
How ANOVA compares means through variance
ANOVA separates total variability into components:
- between-group variation: how far each group mean is from the overall mean
- within-group variation: how far individual observations are from their group mean
Each sum of squares is divided by its degrees of freedom to produce a mean square. The F statistic is:
Under the null hypothesis that all group means are equal, both quantities estimate the same underlying error variance, so their ratio should often be near 1. When group means are separated relative to the residual noise, F grows.
NIST's one-way ANOVA explanation describes this as comparing the level mean square with the residual mean square. The p-value is the probability, under the null model and assumptions, of an F statistic at least as large as the observed one.
For k groups and N total observations, one-way ANOVA usually has:
The numerator asks how much the k means vary. The denominator pools information about variability inside the groups.
A four-variant experiment example
Suppose a SaaS team tests four onboarding flows and measures projects created per eligible account during the first week.
| Variant | Accounts | Mean projects | Standard deviation |
|---|---|---|---|
| Control | 1,000 | 2.30 | 1.80 |
| B | 1,020 | 2.42 | 1.84 |
| C | 990 | 2.61 | 1.91 |
| D | 1,010 | 2.36 | 1.79 |
The null hypothesis is:
The alternative is that not all four means are equal. Notice what it does not say: “C is best.” The global alternative includes any pattern where at least one mean differs.
If the F-test rejects the null, the team should evaluate the comparisons it planned. It might compare every treatment with control, or test one contrast between the current flow and the average of three new concepts. The comparison plan should reflect the decision, not the visual ranking in the finished dashboard.
Make multiple tests trustworthy
See how experimentation leaders plan hypotheses, guardrails, and review practices when a result surface contains many possible claims.
Watch the Trustworthy Experiments TalkWhy not run every pairwise t-test?
Four groups create six pairs. If the team runs six independent tests at alpha 0.05 and treats any significant result as proof, the probability of at least one false positive across the family can exceed 0.05.
ANOVA gives one global test of the equality of all means. It also estimates residual variation using all groups, which can be more efficient than estimating it afresh for each pair under the classical equal-variance model.
The global test does not eliminate multiplicity in follow-up comparisons. R's Tukey HSD documentation explicitly notes that ordinary t-tests inflate the probability of a false declaration across a family. Choose the follow-up procedure for the comparisons the decision actually needs:
- every pair: Tukey-style simultaneous comparisons
- every treatment versus control: Dunnett-style comparisons
- a few planned product questions: predeclared contrasts with a suitable adjustment
- a conservative small family: a Bonferroni or Holm correction
An omnibus test can also be nonsignificant while one carefully planned contrast is persuasive, because the hypotheses and power differ. Decide before launch whether the global null or a treatment-versus-control contrast is the primary decision test.
Unequal group sizes do not automatically invalidate ANOVA, but they make the variance assumption and contrast plan more consequential. If allocation is intentionally uneven, power the smallest comparison that drives the decision and preserve the assignment probabilities. When variances and sample sizes both differ, classical pooled ANOVA can behave poorly; Welch ANOVA or a regression with suitable standard errors is usually easier to defend.
Planned contrasts can also use product structure that the global test ignores. Instead of comparing every pair, a team might compare control with the average of three related treatments, or compare two low-intensity treatments with two high-intensity treatments. A small set of predeclared contrasts often answers the business question with more power and clearer multiplicity control than an exhaustive winner search.
ANOVA assumptions in experiments
The familiar one-way fixed-effects model can be written as:
Classical inference depends on the residuals and design, not on a requirement that the combined raw outcome form one bell curve. NIST's model reference assumes independent, normally distributed errors with mean zero and common variance.
Independent observations
The analysis unit must respect randomization. If accounts are assigned but every user within an account is treated as independent, the standard error ignores clustering. Aggregate at the account level or use cluster-robust or hierarchical methods.
Repeated events from one user create the same problem. Ten sessions from one user do not carry the same independent information as ten users.
Appropriate residual behavior
ANOVA is often robust to moderate non-normality with balanced, sufficiently large groups, but severe skew, outliers, censoring, or zero inflation can make the mean unstable or the F approximation unreliable. Diagnose residuals and assess whether the mean is still the business estimand.
Equal variance for classical one-way ANOVA
Classical ANOVA assumes a common population variance. This can fail when a treatment changes both the mean and spread, or when groups serve different traffic mixes. Unequal group sizes make the problem more consequential.
SciPy's 0 supports Welch ANOVA when equal_var=False. Welch's method relaxes equal population variances and adjusts the degrees of freedom.
Correct outcome model
ANOVA targets a continuous mean. Conversion is binary; event counts are discrete; time-to-churn can be censored. Large-sample mean inference can sometimes work, but logistic, Poisson or negative-binomial, survival, or other generalized models may better represent the outcome and produce interpretable effects.
One-way, two-way, and repeated-measures ANOVA
“ANOVA” names a family rather than one calculation.
One-way ANOVA
One categorical factor with multiple levels, such as four assigned onboarding variants. This is the usual A/B/n example.
Two-way or factorial ANOVA
Two controlled factors, such as headline and layout. The model estimates each main effect plus their interaction. The interaction asks whether one factor's effect changes with the other. This is central to a properly designed multivariate test.
Repeated-measures ANOVA
The same units are observed under multiple conditions or times. Dependence is part of the design and must be modeled. A basic independent one-way ANOVA is invalid for repeated measurements.
ANCOVA
Analysis of covariance adds continuous covariates to the group comparison. In randomized experiments, pre-experiment covariates can improve precision when they are chosen and measured without post-treatment contamination. GrowthBook's guide to variance reduction explains the same motivation in online experimentation.
Run one-way ANOVA in Python
At the action boundary, keep one numeric observation per independent analysis unit in each group. In SciPy:
Before running it, confirm that rows match the randomization unit and missing values have a documented policy. Afterward, inspect group summaries and residual behavior. The p-value alone cannot reveal a broken exposure join or a few enormous outliers.
In R, aov(outcome ~ variant, data = experiment) fits the classical model. R documents 1 as a linear-model interface, which helps explain why ANOVA, regression, and contrasts are closely connected.
Interpret the ANOVA table
A standard output contains:
- degrees of freedom
- sum of squares
- mean square
- F statistic
- p-value
Suppose the output reports F(3, 4016) = 6.8, p < 0.001. Under the model, the observed ratio of between-variant to within-variant variation is unlikely if all four population means are equal. It does not mean every treatment beats control or that any effect is commercially important.
Add the quantities the product decision needs:
- each mean and sample size
- differences from control in original units
- simultaneous or comparison-specific intervals
- an effect-size measure when useful
- guardrail and data-quality results
- the follow-up comparison method
Avoid ranking noisy means without uncertainty. The highest observed variant has benefited from both its true effect and sampling variation, especially when many variants were screened.
Common ANOVA mistakes
Treating events as independent users
Repeated events make the nominal sample size huge and uncertainty too narrow. Preserve the assignment unit.
Using ANOVA for every metric shape
The word “variant” does not imply ANOVA. Match the outcome distribution and estimand to a model.
Checking assumptions after selecting a winner
Write the model, outlier policy, transformation, and variance choice before the ranking is visible. Result-driven switching creates hidden researcher degrees of freedom.
Treating a significant F-test as a winner declaration
Follow with the planned contrasts. The omnibus test only rejects equality of all means.
Ignoring practical significance
A very large experiment can detect a tiny difference. Compare intervals with a minimum practical effect and account for implementation cost and guardrails.
Use ANOVA as part of an experiment plan
Before launch, specify the factor and levels, independent unit, primary continuous outcome, minimum effect, sample-size plan, variance assumption, global or contrast hypothesis, comparison family, and stopping rule.
Then verify assignment and exposure before interpreting the model. A sample ratio mismatch can signal that observed group counts no longer reflect the planned randomization. No F-test can repair biased exposure data.
ANOVA is valuable because it turns a field of variant means into a structured model of signal and noise. The broader z-test, t-test, chi-square, and ANOVA guide shows when the outcome and hypothesis call for another member of that family. Use the omnibus test for the global question, planned contrasts for the decision, and effect estimates for practical judgment. That sequence makes a multiple-variant test easier to defend than a dashboard full of uncoordinated p-values.
Compare variants with discipline
Run controlled experiments, connect trusted metrics, and review treatment effects and uncertainty in one shared workflow.
Start With GrowthBookReady to ship faster?
No credit card required. Start with feature flags, experimentation, and product analytics—free.
