How to experiment with personalization in AI systems
Standard A/B testing breaks when the treatment is never the same twice.
That's the core problem with running AI personalization experiments — your recommendation engine, dynamic content system, or LLM-based feature is generating a unique experience for every user, which means the foundational assumption behind classical A/B testing (one treatment, applied the same way to everyone) doesn't hold.
The result is noisy data, lift numbers that don't replicate, and false confidence in model improvements that are actually just reflections of pre-existing user quality differences.
This guide is for engineers, PMs, and data teams who are building or iterating on AI-driven personalization systems and need a reliable way to measure whether those systems are actually working. Here's what you'll learn:
- Why standard A/B testing produces structurally misleading results for AI personalization — and what breaks specifically
- How to separate user-level effects from model-level effects so you're measuring model quality, not user quality
- How to measure lift accurately when every user receives a different AI-generated experience
- How to detect bias and fairness problems that aggregate metrics will hide
- What infrastructure primitives you need to run continuous AI personalization experiments at scale
The article moves in order from diagnosis to execution. It starts with why the standard testing method fails for this class of system, then works through experiment design, lift measurement, bias detection, and finally the infrastructure layer that makes all of it work in production.
Each section builds on the one before it, so the techniques in the later sections only make sense once the structural problems in the earlier ones are clear.
Why traditional A/B testing breaks down for AI personalization
Classical A/B testing earned its reputation as the gold standard for causal inference because it does one thing exceptionally well: it isolates the effect of a single, stable change applied uniformly to a randomly assigned group.
That structural simplicity is also its limitation. When the "treatment" is an AI system generating unique outputs for every user in real time, the foundational assumptions that make A/B testing valid start to crack — not metaphorically, but in ways that produce structurally misleading results.
The fixed-treatment assumption standard A/B testing actually requires
When Professor Alexandre Belloni at Duke Fuqua describes A/B testing as the gold standard for determining causal effect, the implicit qualifier is that the treatment is fixed and uniform. Think of the Obama 2008 campaign's headline and button-color tests: every user in the treatment group saw the same headline, the same button. The comparison was clean because the intervention was identical across the group.
That condition — one treatment, applied the same way to everyone in the variant — is what allows you to attribute outcome differences to the treatment rather than to noise or pre-existing user differences.
Standard A/B testing infrastructure is built around this assumption. Randomization creates comparable groups. A single variable changes. You measure whether the outcome difference exceeds what chance would produce.
How AI personalization violates that assumption by design
A recommendation engine doesn't serve the same ranked list to every user in the treatment group. A dynamic content system doesn't generate the same copy for every session. An LLM-based feature doesn't produce the same response twice. That's the point — personalization is valuable precisely because it adapts to the individual.
But this creates a fundamental problem for standard A/B testing: there is no single "treatment" to evaluate. Two users assigned to the same variant are receiving materially different interventions, driven by the model's real-time decisions about their context, history, and predicted preferences.
As GrowthBook's AI testing playbook notes, the goal of tuning "AI responses across thousands of use cases" is structurally incompatible with one-variable-at-a-time testing. Interaction effects compound this further — AI personalization systems often have multiple model components active simultaneously, and standard A/B frameworks have no native way to account for how those components interact across users.
What this means for experiment validity
The validity consequences are specific and serious. When an AI system routes users based on model decisions, naive treatment/control comparisons don't measure model quality — they measure a mixture of model quality and pre-existing user differences.
The mechanism behind this is covered in the next section; the point here is that the contamination is structural, not incidental.
Even when randomization is technically in place, the heterogeneity of AI outputs creates a second problem. GrowthBook's documentation puts it plainly: "Experimentation results work on averages, and this can hide a lot of systemic biases that may exist."
An average treatment effect calculated over a population where every user received a different AI-generated experience is an average of fundamentally different interventions. It tells you something, but not what you think it tells you — and it can easily mask divergent effects across user segments that would be visible if you were measuring a uniform treatment.
The symptoms practitioners experience — noisy results, inconclusive tests, lift estimates that don't replicate — often trace back to this structural mismatch. The problem isn't that the team ran the test wrong. It's that the testing method was designed for a different class of system.
This is diagnosable, and it's solvable. But the solution requires rethinking randomization design, how lift is measured, and what infrastructure is needed to run valid experiments on systems where the treatment is never the same twice.
Separating user-level effects from model-level effects in AI personalization experiments
The most dangerous failure mode in AI personalization experiments isn't a bug in your statistical test — it's a structural flaw in how you've set up the comparison.
When a recommendation engine routes higher-value users toward certain experiences, a naive treatment/control comparison doesn't measure whether your model is better. It measures whether your model happened to find better users. That distinction is the difference between a genuine insight and a false positive you ship to production.
What selection bias actually looks like in AI systems
Spotify's engineering team describes their personalization systems as learning "relationships between user characteristics — age, past behavior, product preferences — and preferred experiences". That's precisely what makes personalization valuable. It's also precisely what makes naive A/B comparisons invalid.
When a model's job is to identify the best experience for each user based on their characteristics, it is by design routing different users to different treatments. If your experiment assigns users to treatment and control groups after — or alongside — that routing logic, the groups are no longer exchangeable.
Treatment users may be systematically higher-value, more engaged, or more likely to convert, not because your model is better, but because your model already learned to find them. GrowthBook's own documentation puts it plainly: "Experimentation results work on averages, and this can hide a lot of systemic biases that may exist." In AI systems, those biases are often baked into the routing logic itself.
Randomization unit design — why user-level hashing is the foundation
The structural fix is to move randomization upstream of the model's decision-making. Randomization must happen at the user level, using deterministic user-level hashing, before the personalization layer makes any routing decisions.
This ensures that the same user consistently lands in the same experimental bucket regardless of session, device, or model state — and that bucket assignment is independent of any signal the model might use to route them.
Spotify's architectural approach captures the principle cleanly: "We build personalization systems using our ML/AI stack and improve them through our experimentation stack." Those are treated as distinct technical domains, not interleaved ones. The experiment layer sits above the personalization layer in the decision stack. When they're entangled, the model's routing decisions contaminate the randomization.
GrowthBook supports randomization across flexible units — user, location, postal code, URL path — which allows teams to lock assignment at the user level before any model inference occurs. The choice of randomization unit isn't a configuration detail; it's the primary mechanism for ensuring your treatment and control groups are actually comparable.
Holdout groups for isolating model-level effects
Even with clean user-level randomization, measuring the model's aggregate contribution requires a holdout group — a segment of users who receive no personalization, or a fixed baseline experience, while the rest of the population is exposed to the model. The holdout gives you a clean counterfactual: what would have happened without the personalization layer at all?
Without a holdout, you can compare model versions against each other, but you can't measure whether the personalization system as a whole is generating lift over a non-personalized baseline.
That's a meaningful gap, especially early in a system's lifecycle when the fundamental question is whether the model is adding value, not just which variant of it performs best.
GrowthBook supports holdout experiments, and this capability matters specifically because it separates the question of model quality from the question of user quality. A model that looks good in an A/B test against another model variant might still be underperforming a simple non-personalized baseline — or it might be outperforming it only for a specific user cohort, which would itself be a signal worth investigating.
Auditing for residual confounding
Even with correct randomization and holdout groups, residual selection bias can persist — particularly if the model was trained on data that already reflects historical routing decisions.
GrowthBook's warehouse-native architecture runs statistical computations directly against your existing data warehouse — Snowflake, BigQuery, Redshift, or Postgres — which means every calculation is reproducible and auditable without pulling data into a proprietary engine. That auditability is what lets you verify that your randomization is actually clean, not just assumed to be.
Segment-level analysis is the practical tool for detecting residual confounding: if observed lift is concentrated in high-value user cohorts rather than distributed across the population, that's a signal that user quality, not model quality, is driving the result.
Targeted rollout capabilities support this kind of cohort-level inspection. And for cases where pre-experiment user characteristics are imbalanced despite randomization, CUPED and post-stratification — both supported in GrowthBook — provide regression adjustment that can partially correct for those differences.
Landon Smith, Head of Post-Training at Character.AI, describes the practical value of this architecture: "GrowthBook has been an invaluable tool for Character.AI, helping us develop our models into a great consumer experience. We can compare different modeling techniques from the perspective of our users — guiding our research in the direction that best serves our product." That framing — comparing modeling techniques from the perspective of users, not just metrics — is exactly the orientation that user/model separation makes possible.
Lift numbers lie when every user receives a different AI-generated treatment
Getting randomization right is necessary but not sufficient. Once you've structured your experiment correctly, you still face a harder problem: computing a lift number that actually reflects model performance rather than measurement artifacts.
In AI personalization systems, the standard approach — calculate average treatment effect, run a t-test, ship if p < 0.05 — produces numbers that look credible but often aren't.
Why average treatment effects fall short for heterogeneous AI outputs
In a classical A/B test, every user in the treatment group receives the same stimulus. The average treatment effect is meaningful because you're averaging over a single, consistent intervention. In an AI personalization system, User A gets a recommendation driven by purchase history, User B gets one driven by browsing recency, and User C gets one based on predicted lifetime value. The "treatment" is a distribution of outputs, not a fixed stimulus.
When you compute ATE across this distribution, you're averaging across fundamentally different interventions. That average might look positive while the model is actively hurting engagement for a large segment of mid-tier users — if a small cohort of high-value users is responding exceptionally well, they'll pull the aggregate number upward and mask the damage elsewhere.
The standard lift formula — (treatment metric − control metric) / control metric — assumes you're comparing stable, consistent means on both sides. That assumption holds when everyone in the treatment group received the same thing. When treatment content varies continuously across users, the "treatment mean" is an average of fundamentally different interventions, and the lift number you compute is averaging across experiences that aren't comparable.
SUTVA violations and the evolving treatment problem
There's a deeper statistical issue that most teams don't catch until their results stop replicating. The Stable Unit Treatment Value Assumption (SUTVA), foundational to causal inference, requires two conditions: no interference between units, and a single version of the treatment. AI personalization systems routinely violate the second condition.
AI models analyze behavior in real time, which means the experience a user receives at day one of your experiment is not the same experience they receive at day thirty. If the model retrains or adapts during the experiment window — which is the entire point of a continuously learning system — then your early-period data and late-period data are measuring different interventions.
Pooling them produces a blended estimate that accurately describes neither. You can end up with a "significant" result that reflects the model's improvement trajectory rather than a stable treatment effect, which means the result won't hold when you try to reproduce it against a different baseline.
The practical mitigation is to lock model weights during the experiment window, or to explicitly segment your analysis by time cohort and check for consistency across periods before pooling. If lift is only appearing in the second half of your experiment, that's a signal worth investigating before you ship.
CUPED and regression adjustment for pre-experiment normalization
Even with a locked model and clean randomization, pre-existing user variance will inflate your standard errors and force you to run longer experiments than necessary. Heavy users, power purchasers, and highly engaged users all have metric trajectories that started before your experiment did.
If those users happen to skew toward your treatment group, you'll see lift that reflects user quality, not model quality.
CUPED — Controlled-experiment Using Pre-Experiment Data — addresses this by using each user's pre-experiment behavior as a baseline correction. If a user was already a heavy purchaser before your experiment started, CUPED accounts for that, so your lift estimate reflects the model's effect rather than the user's pre-existing tendencies.
The result is tighter confidence intervals — which means you can detect smaller real effects with the same sample size, or reach statistical confidence faster.
Platforms like GrowthBook support CUPED and post-stratification natively as part of their statistical framework, meaning this isn't a custom analysis you need to build outside your experiment tooling. GrowthBook's Bayesian engine extends this further by allowing informative priors from historical experiments — if past model comparisons have averaged 1% lift, you can encode that knowledge to tighten credible intervals on new experiments rather than starting from scratch each time.
Activation metrics and guardrail metrics
Not every user assigned to your treatment group actually receives a meaningfully different experience. Some users never trigger the recommendation engine. A second group visits the surface but doesn't interact with the personalization layer at all. For a third group, behavioral history is too sparse for the model to generate a confident output.
Including these users in your lift calculation dilutes the signal — you're averaging a real effect over a large population of users who experienced nothing different.
Activation metrics solve this by filtering the analysis to users who were genuinely exposed to a differentiated experience. The key discipline is defining activation criteria before the experiment runs, based on what it means to have actually received the personalized treatment, not based on which users showed positive outcomes.
Guardrail metrics are the other half of the discipline. Lift in your primary metric is only meaningful if it isn't purchased at the cost of degraded experience elsewhere — slower page loads, higher error rates, increased session abandonment. These need to be pre-specified, not added after you see the results.
GrowthBook's A/A test documentation makes the stakes concrete: with five unrelated metrics in a single experiment, the probability of at least one false positive reaches 41%. With two metrics, it's 19%. Post-hoc guardrail selection is just p-hacking with extra steps. Specify your primary metric, specify your guardrails, and hold to that list before you look at results.
The goal of this entire measurement stack — CUPED normalization, activation filtering, pre-specified guardrails — is to answer a single question cleanly: did the model-treated group actually behave differently, or are you measuring something else?
For AI personalization, where the causal chain from model output to user behavior is complex and partially opaque, lift measurement's focus on actual change against a baseline is almost always the right tradeoff.
Avoiding bias and fairness pitfalls in AI personalization experiments
A personalization model that shows +3% lift in aggregate can still be quietly degrading the experience for a meaningful slice of your user base. If your experiment reporting stops at overall averages, you will never see it.
This isn't a statistical edge case — it's a structural property of how AI systems learn, and it requires deliberate design choices to surface.
How AI personalization systems encode and amplify historical bias
The problem starts in training. Personalization models learn from historical behavioral data, which means they inherit whatever disparities already existed in user outcomes. GrowthBook's own documentation states this plainly: algorithmic systems can "learn or otherwise encode real-world biases in their operation, and then further amplify/reinforce those biases."
The amplification mechanism is what makes this particularly difficult to contain. When a biased model generates biased recommendations, those recommendations produce biased behavioral signals, which then feed back into the next training cycle with even more concentrated bias. The model doesn't drift toward fairness over time — it drifts away from it.
Sparse data compounds the problem. GrowthBook's documentation specifically flags that "sparse or poor data quality leads to objective-setting errors and system designs that lead to suboptimal outcomes for many groups of end users."
Underrepresented users generate less behavioral signal, so the model has less to work with for those groups — and its outputs are correspondingly less reliable for them, even when aggregate performance looks strong.
This isn't hypothetical. Research into LLM-based hiring tools found that models were 15% more likely to select whichever candidate appeared first in the prompt, all else being equal — a systematic positional bias that was completely invisible in aggregate accuracy metrics but measurable once researchers structured their analysis to look for it.
The same dynamic applies to recommendation engines and dynamic content systems: the bias exists in the decision pattern, not in the headline number.
Why aggregate lift metrics hide differential impact
GrowthBook's documentation is direct on this point: "Experimentation results work on averages, and this can hide a lot of systemic biases that may exist." The formal statistical mechanism here is Simpson's Paradox — a phenomenon where a trend visible in aggregate data reverses or disappears when the data is disaggregated by subgroup.
The Berkeley admissions case is the canonical illustration: overall admission rates appeared to favor male applicants (44% vs. 35%), but when broken down by department, women had higher admission rates in most departments. The aggregate number was real but misleading.
The same structure appears in AI personalization experiments when a model over-serves high-engagement or data-rich segments while underperforming for others — the overall lift is genuine, but it's being driven by a subset of users, and the rest are getting a worse experience than baseline.
Goodhart's Law adds another layer of risk. When a single aggregate metric becomes the optimization target, the model learns to maximize that metric for the users where it's easiest to do so. That tends to be your highest-signal, most-engaged segments — which means the model is systematically deprioritizing everyone else.
Metric slicing and cohort-level guardrails
The practical response is to build differential impact analysis into the experiment design from the start, not as a post-hoc audit. GrowthBook's documentation is explicit that "it is possible to measure this effect and ensure that results account for these groups" — but only if you instrument for it.
Metric slicing means breaking experiment results down by the user dimensions that matter for your product: demographic segments, behavioral cohorts, geographic regions, account tenure, or whatever groupings are relevant to your user base.
GrowthBook's product analytics capabilities support pivot tables and dimension-based filtering, which allows teams to surface differential treatment effects across cohorts rather than relying solely on aggregate experiment readouts. The guidance from GrowthBook's Simpson's Paradox documentation applies directly: "analyze the data by considering all relevant variables and subgroups" and "ensure that the experimental groups are similar in terms of demographics and behavior."
Guardrail metrics need to follow the same logic. A guardrail defined only at the aggregate level will pass a model that harms a specific cohort while lifting overall numbers.
Guardrail metrics should be defined per segment for any experiment where differential impact is a plausible risk — and for AI personalization experiments, it almost always is. When a guardrail fires, GrowthBook's kill-switch capability allows teams to immediately deactivate the feature without a full deployment cycle, which matters when the harm being detected is happening to real users in production.
Targeted rollouts serve a related function: validating model performance on specific cohorts before broad release, rather than discovering differential impact after the fact at full traffic. Shipping to a segment first and measuring cohort-level outcomes is a more honest test of whether your personalization model is actually working for all of your users — not just the ones it was implicitly optimized for.
The infrastructure gap that makes continuous AI personalization testing fail
The measurement techniques covered earlier in this article — regression adjustment, activation filtering, cohort-level slicing — only work in production if the infrastructure underneath them is built for the job.
Most experimentation stacks were designed for periodic A/B tests: launch a variant, wait two weeks, read a result, ship or kill. AI personalization doesn't work that way. Hightouch describes the gap plainly: rather than waiting weeks for a single test to conclude, AI-driven systems can test thousands of variations in parallel, adapting content, offers, and timing in real time. That scale demands infrastructure that treats experimentation as a continuous operational process, not a scheduled event.
Linking feature flags to model versions
The first primitive you need is a clean linkage between your feature flag system and your model versioning. Without it, you cannot reliably attribute outcome changes to a specific model version versus any other concurrent change in your system.
Feature flags serve as the deployment primitive here — they control which users are exposed to which model variant, enable gradual rollouts to limit blast radius, and provide an instant kill switch if a new model degrades a guardrail metric before you've caught it in your analysis. GrowthBook SDKs evaluate flags from a locally cached payload, so flag checks resolve in sub-millisecond time and your application continues to function correctly even if GrowthBook's servers are temporarily unavailable.
Feature flag rollout systems that expose this as a first-class capability allow experiments to be directly tied to flag IDs. The practical value is that your model deployment and your experiment assignment stay in sync — you're not managing two separate systems that can drift out of alignment.
Bandit-style traffic allocation for adaptive learning
Static 50/50 splits make sense when your treatment is fixed. When you're testing multiple model variants and want to minimize exposure to underperforming ones, multi-arm bandits are a better fit.
Spotify's engineering team describes bandits as dynamically reallocating traffic from poorly performing treatments to successful ones — which compresses the time between launching a variant and making a confident rollout decision.
There's an important distinction worth keeping in mind: multi-arm bandits operate at the model level, shifting traffic between discrete variants. Contextual bandits operate at the user level, selecting a variant based on user features — which Spotify explicitly characterizes as "essentially personalization."
Both are valid tools, but they answer different questions. Multi-arm bandits are the right primitive for model comparison experiments; contextual bandits belong in the personalization layer itself. Platforms that support native bandit configuration — burn-in period, schedule interval, and conversion window — treat this as a production-ready capability rather than a theoretical one.
Warehouse-native analysis as the source of truth
Analysis that runs inside a vendor's proprietary engine creates a reproducibility problem. You can't audit the calculation, you can't add metrics retroactively, and you can't slice results against user attributes that live in your own warehouse.
For AI personalization experiments — where you need cohort-level breakdowns to catch differential impact and regression adjustment to normalize pre-experiment variance — that opacity is a structural liability.
Warehouse-native analysis solves this by running statistical computations directly against your existing data infrastructure, whether that's Snowflake, BigQuery, Redshift, or Postgres. GrowthBook's architecture is built on this model: metrics are defined in SQL, calculations are reproducible, and you can add metrics to a completed experiment after the fact.
That last capability matters more than it sounds — in AI personalization experiments, you often discover the right guardrail metric after you've already launched.
Capturing institutional knowledge to compound personalization learnings
High-volume experimentation creates a compounding knowledge problem. Without program-level memory, teams repeat tests they've already run because nothing connects past learnings to new ideas.
Individual experiments produce results; a program produces compounding understanding of what works for which users under which conditions. Those are different things.
The infrastructure requirement here is a searchable, structured record of experiment history — not just results, but hypotheses, metric choices, and rollout decisions. GrowthBook addresses this through a learning library that surfaces past experiments, cumulative impact tracking across the full experiment program, and insights dashboards that make win rates and metric performance visible at the program level.
The goal is to make the tenth experiment in a personalization domain meaningfully informed by the first nine, rather than starting from scratch each time.
Taken together, these four primitives — flag-linked model versioning, adaptive traffic allocation, warehouse-native analysis, and institutional knowledge capture — form the infrastructure layer that makes continuous AI personalization testing viable. None of them is sufficient alone.
A bandit without reproducible analysis produces fast results you can't trust. Warehouse-native analysis without institutional capture produces auditable results you'll forget. The stack only works as a system.
What valid AI personalization experiments actually require
The through-line of this article is a single, uncomfortable truth: the standard experimentation playbook was built for a class of system that AI personalization is not.
When the treatment is a distribution of outputs rather than a fixed stimulus, when the model is routing users based on signals that correlate with user quality, when the system is learning during the experiment window (the SUTVA violation covered in the measurement section above) — the numbers you get from a naive A/B test are not wrong in an obvious way. They're wrong in a way that looks credible, which is worse.
The non-negotiable foundations: randomization, measurement, and bias detection
Three things have to be right before anything else matters. Randomization must happen upstream of the model's routing logic — not alongside it, not after it.
Lift measurement requires filtering to users who actually received a differentiated experience, applying pre-experiment variance reduction, and evaluating against pre-specified guardrails. Bias detection cannot be retrofitted — it has to be built into the experiment design from the start, because aggregate lift numbers will never surface differential harm on their own.
Infrastructure decisions that determine whether your experiment results are trustworthy
If you're early in building a personalization system, the most important infrastructure decision is keeping your experiment layer structurally separate from your personalization layer — and making sure feature flags are the mechanism that links model versions to experiment assignments. GrowthBook feature flags control which users are exposed to which model variant, enable gradual rollouts with guardrail metrics, and provide an instant kill switch without requiring a deploy.
If you're further along and running multiple model variants in parallel, warehouse-native analysis becomes the non-negotiable: you need to be able to audit your calculations, slice by cohort, and add guardrail metrics after launch without rebuilding your analysis pipeline. GrowthBook's warehouse-native architecture supports all of this without pulling your experiment data into a black box.
Start with the holdout: the one experiment that surfaces every structural problem
The best first experiment is usually the simplest one: a holdout test that answers whether your personalization system is generating any lift at all over a non-personalized baseline. Not a model comparison. Not a multi-variant test. Just: does the model add value?
That question is harder to answer than it sounds, and getting a clean answer to it will surface most of the structural issues — selection bias, SUTVA violations, missing activation criteria — before you're running at full scale.
This article was written to give you a complete picture of what valid AI personalization experiments actually require, and to make the path from diagnosis to execution as clear as possible. The problems are real, but they're solvable with the right design choices.
What to do next: Start by auditing your current experiment setup against the randomization question: does your bucket assignment happen before or after the model makes routing decisions? If you can't answer that with certainty, that's your first fix — everything downstream of a contaminated randomization is unreliable. If your randomization is clean, move to measurement: do you have activation criteria defined, and are your guardrail metrics pre-specified? If both are in place, run a holdout experiment against a non-personalized baseline before you run any model comparison. That sequence — randomization, measurement hygiene, holdout baseline — will give you a foundation you can actually build on.
Related insights
- GrowthBook 4.4: AI-Native Experimentation & Feature Flags
- UPS Delivered $500M in Revenue Using Experimentation
- What Is A/B Testing? A Complete Guide
Related Articles
In healthcare, “Can we randomize it?” is the wrong first question. Start with “Could either experience change care, rights, privacy, or access?”
A/B testing can improve digital intake, appointment access, patient education, clinician workflows, and administrative operations. It can also create unacceptable risk when teams treat a clinical or consent decision like an ordinary conversion funnel.
The difference is not the label on the method. A/B tests are randomized experiments. What matters is the treatment, purpose, affected population, data flow, and oversight required in the organization and jurisdiction. This guide provides a practical product framework, not a substitute for legal, clinical, privacy, security, or institutional review.
Draw the boundary before designing variants
Create an intake step that classifies the proposed change before anyone builds a treatment. At minimum, ask:
- Can the change alter diagnosis, treatment, triage, dosage, or clinical recommendations?
- Can it delay or discourage access to care, accommodations, or urgent help?
- Does it change informed consent, privacy choice, required disclosure, or patient cost?
- Does it use protected or sensitive health information for assignment or measurement?
- Does it include children, people in crisis, or another population requiring added protection?
- Is the purpose internal quality improvement, or is it designed to contribute to generalizable knowledge?
- Could the software function fall within medical-device or clinical decision-support oversight?
The HHS quality-improvement guidance says many activities limited to improving patient care and collecting operational data are not research under the cited human-subjects regulations. It also states that some quality-improvement activities can have a research purpose, in which case human-subject protections may apply. A product team should not make that determination informally; route it to the organization’s authorized office.
Likewise, software that influences clinical decisions is not automatically an ordinary product surface. The FDA’s January 2026 clinical decision-support guidance explains that some software functions are excluded from the device definition while other patient- or caregiver-facing functions can remain subject to digital-health policy. Clinical and regulatory owners need to classify the function before experimentation.
Start with lower-risk operational questions
The safest early program tests reversible changes where both variants meet the same clinical, accessibility, privacy, and disclosure requirements.
Appointment reminder timing
Compare 2 approved reminder schedules or message structures to reduce missed appointments. Keep required details, opt-out behavior, language support, and urgent-contact instructions constant.
Use completed appointments or timely rescheduling as the primary outcome. Track cancellations, patient contacts, message delivery, opt-outs, wrong-recipient risk, and differences across language, age, disability, or access groups. A higher click rate is not enough if no-show rates or trust worsen.
Patient portal navigation
Test whether a clearer information architecture helps people complete a high-value administrative task, such as finding results, updating insurance, or sending a non-urgent message. Preserve emergency guidance and clinical escalation paths in both variants.
Measure successful task completion and time to completion. Guard against repeated navigation, abandonment, accessibility failures, mistaken message routing, and increased call-center burden. Use usability testing before the A/B test to catch failures randomization should never expose.
Administrative form sequence
Compare a long form with a staged flow, or test the order of non-clinical fields. Do not omit information needed for safe care, billing transparency, consent, or legal compliance.
Measure accurate completion, not just submission. Track validation errors, correction rates, staff rework, abandonment, and time to appointment. If the treatment collects sensitive data, confirm necessity and access controls before launch.
Educational content layout
Test 2 ways to present the same clinician-approved information: summary-first versus stepwise, text plus illustration versus text alone, or a clear action checklist versus a dense paragraph. Keep the medical meaning, risks, contraindications, and escalation advice equivalent.
Use a comprehension or appropriate next-action metric when feasible. Page time and clicks can be misleading. Accessibility, language quality, and comprehension across health-literacy levels belong in the guardrail plan.
Review the design before launch
Use a trustworthy experiment-design session to pressure-test metrics, safety checks, and decision rules before exposing patients or clinicians.
Watch the Experiment Design Session
Use stronger controls for care-adjacent products
Some product changes are not clinical interventions but can still influence care. They need clinical ownership, narrower eligibility, conservative ramps, and explicit stopping criteria.
Clinician workflow support
A test might compare how a work queue prioritizes administrative follow-up, how a note template reduces documentation work, or how a non-diagnostic alert is presented. The treatment should not silently alter the clinical standard of care.
Randomize at the unit that prevents contamination. Individual clinician assignment may fail when teams share queues and handoffs; clinic- or unit-level clusters may better match the workflow. Measure task completion and time saved, with guardrails for missed work, overrides, escalations, documentation quality, and staff workload.
Preventive-care outreach
Compare approved outreach content or channels for people already eligible under the same clinical rule. Do not experiment with whether one group receives necessary care or required notice.
Use completed appropriate follow-up as the primary outcome. Track opt-outs, unreachable patients, scheduling capacity, disparities, complaints, and downstream cancellations. If the treatment drives demand beyond operational capacity, a messaging lift can make access worse.
Digital adherence support
Test the presentation or timing of an approved reminder, checklist, or educational cue. Avoid treatment changes that could be interpreted as personalized medical advice without the corresponding validation and oversight.
Measure the intended behavior with caution. Self-reported completion or app engagement is not a clinical outcome. Include adverse-event reporting, escalation pathways, disengagement, and privacy events where relevant.
Feature rollout in health software
Use feature flags to separate deployment from release, start with internal or trained cohorts, and expand only when technical and clinical guardrails remain healthy. GrowthBook’s feature flag platform supports targeted rollouts and kill switches, while the experiment layer measures impact.
The rollback plan must describe more than turning off a flag. Determine whether the old experience remains clinically and operationally safe, how queued work is reconciled, what happens to partial workflows, and who is authorized to stop exposure.
Protect data by design
Do not send a broad event stream to an experimentation vendor and decide later which fields were unnecessary. Inventory the data before implementation:
| Data question | Required decision |
|---|---|
| Assignment | What is the least identifiable stable unit that works? |
| Eligibility | Which sensitive attributes are truly needed? |
| Exposure | What event proves the treatment was delivered? |
| Outcomes | Can metrics be computed inside the governed data environment? |
| Access | Which roles can view assignments, segments, and results? |
| Retention | When are raw records, logs, and exports removed? |
The HHS minimum-necessary guidance describes limiting uses, disclosures, and requests for protected health information to what is needed for the intended purpose, with policies based on roles and recurring versus non-routine access. Apply that principle to experiment attributes, debugging logs, dashboards, and downloaded readouts.
Pseudonymous identifiers reduce exposure but do not automatically make a dataset non-sensitive or outside applicable rules. Review linkability, small cohorts, free-text fields, URLs, device metadata, and combinations that can reveal a condition. Never put clinical details or identifiers in feature names, variation labels, or URLs.
A warehouse-native experimentation approach can query approved metrics where the organization already governs them. Architecture does not create compliance on its own; teams still need contracts, access control, auditability, retention rules, security review, and configuration that matches the approved data flow.
Keep unsafe questions out of product experimentation
An experimentation policy should name prohibited or separately governed categories. Product teams should not discover the boundary only after a proposal reaches launch review.
Do not use an ordinary product A/B test to withhold a clinically indicated service, emergency direction, safety warning, accessibility accommodation, required disclosure, or legally protected choice. Do not reduce the visibility of risks to improve completion. Do not randomize a diagnostic or treatment recommendation without the clinical, regulatory, and research framework appropriate to that intervention.
Avoid treatments that exploit fear, urgency, shame, or uncertainty about health. A message can increase appointment conversion while undermining informed choice. Likewise, do not test whether patients tolerate a harder cancellation, more confusing privacy control, or hidden cost. Both variants must meet the organization’s baseline standard for respectful and comprehensible communication.
Clinical AI and decision-support changes need an evaluation program beyond a click-based A/B test. Validate the model offline, examine performance and failure modes across relevant populations, review human factors, and stage deployment with clinical monitoring. An online comparison may contribute evidence only after both treatments meet the safety threshold for exposure.
When an activity may be human-subjects research, follow the institution’s process before enrolling or exposing anyone. HHS research-oversight training states that covered non-exempt human-subjects research requires the applicable review and that informed consent requirements apply unless the IRB authorizes otherwise. The product team should preserve the determination, protocol version, approved treatment, and reporting obligations with the experiment record.
Finally, do not interpret lack of detected harm as proof of safety. Rare adverse events, small vulnerable groups, and outcomes that occur after the experiment window may be underpowered. Use prior evidence, incident monitoring, qualitative reports, and post-rollout surveillance alongside the randomized estimate.
Define patient-centered metrics and guardrails
Healthcare teams need more than a conversion scorecard. Build a measurement hierarchy:
- Primary outcome: the operational or patient-facing result that answers the decision.
- Process diagnostics: steps that explain why the treatment worked or failed.
- Safety guardrails: outcomes that trigger a stop or clinical review.
- Equity checks: predeclared groups where access or benefit could differ.
- Operational guardrails: staffing, wait time, rework, cost, and downstream capacity.
Define the practical threshold before launch. A statistically detectable change may be too small to justify implementation, and a neutral aggregate can hide meaningful harm in a protected or vulnerable group. At the same time, slicing results across many small subgroups increases false-positive risk and can expose sensitive attributes. Predeclare the equity questions that matter and use appropriate privacy and multiple-testing controls.
GrowthBook supports reusable fact tables and metrics so teams can keep definitions reviewable. Use a power analysis for the primary outcome and critical guardrails. If the required sample or duration is unrealistic, do not weaken the standard; use usability research, simulation, staged quality improvement, or a larger treatment contrast.
Create a healthcare experiment review packet
Before launch, the owner should provide one reviewable packet:
- purpose, hypothesis, and operational decision
- classification and required oversight determination
- affected population and exclusion criteria
- clinical, privacy, security, accessibility, and compliance approvals
- treatment screenshots or workflow diagrams
- assignment, exposure, and data-flow design
- primary outcome, diagnostics, guardrails, and equity checks
- sample plan and stopping rule
- rollout stages, monitoring owner, and rollback procedure
- patient or clinician communication plan, if applicable
- documentation and retention plan
Use an approval matrix that names accountable people. Product approval does not replace clinical approval; a privacy review does not settle human-subjects research status; and an IRB determination does not automatically approve the production security architecture.
The WHO clinical-trial best-practices guidance emphasizes ethical standards, regulatory considerations, patient-centered research, transparency, and stakeholder collaboration. Not every healthcare product experiment is a clinical trial, but high-risk work should inherit the same respect for people and evidence.
Build trust into the experimentation program
Start with reversible operational improvements where both experiences are already acceptable. Prove that the team can classify risk, minimize data, validate assignment, monitor safety, and document decisions before expanding scope.
Publish internal rules for what teams may test, what requires added review, and what is out of bounds. Maintain an experiment registry and audit trail. Record neutral and negative results so a new team does not repeat the same risky idea.
GrowthBook can support the controlled delivery and analysis layer through experimentation, feature flags, permissions, and warehouse-defined metrics. The organization remains responsible for the clinical, ethical, legal, privacy, and operational framework around every test.
In healthcare, speed is valuable only when the learning process protects the people whose behavior creates the data.
Build a governed test workflow
Connect controlled releases to reviewable metrics and decision rules while keeping healthcare data in your approved architecture.
Get Started With GrowthBook
The right statistical test is determined by the question and data-generating process, not by which function is easiest to run. Start with the outcome, groups, and dependence structure; the test name comes later.
Z-tests, t-tests, chi-square tests, and analysis of variance (ANOVA) all compare observed data with a null model. They differ in the kind of outcome they model, the uncertainty they estimate, and the number or structure of groups they can compare.
For a simple product experiment, a useful first pass is:
- continuous outcome, two independent groups: usually a Welch two-sample t-test
- binary proportion, two large independent groups: a two-proportion z-test is common
- categorical counts across groups: chi-square test, if expected counts are adequate
- continuous outcome across three or more groups: one-way ANOVA or Welch ANOVA
Those rules are a starting point. Paired observations, clusters, ratios, repeated measures, heavy tails, covariate adjustment, or sequential monitoring require a model that reflects the design.
Choose from the outcome and hypothesis
Write the estimand before choosing a test. An estimand is the quantity the experiment is trying to estimate: a difference in mean revenue, a difference in conversion probability, or an association between two categorical variables.
| Question | Outcome | Common test |
|---|---|---|
| Did average order value change between A and B? | Continuous | Welch two-sample t-test |
| Did signup probability change between A and B? | Binary | Two-proportion z-test |
| Is plan choice associated with variant? | Categorical, 3+ levels | Chi-square test of independence |
| Do mean task times differ across four variants? | Continuous | One-way ANOVA |
| Did the same users' scores change before and after? | Paired continuous | Paired t-test |
The number of groups alone is insufficient. Conversion in four variants is still categorical data; a chi-square or binomial model may fit. Revenue in two groups is continuous; a t-test or regression is more natural.
The University of Michigan's statistical-test guide uses the same sequence: identify variable types and the relationship being tested before selecting a method.
When to use a z-test
A z-test compares a standardized estimate with the standard normal distribution. The classical one-sample z-test for a mean assumes the population standard deviation is known. That condition is unusual in product analytics, where variability is estimated from the current sample.
Z-tests remain common for proportions. In a two-arm conversion experiment, the estimate is:
Under the null of equal proportions and with adequate counts, the standardized difference is approximately normal. This yields a two-proportion z-test.
Use it when:
- the outcome is a binary count summarized as successes and failures
- assignment groups are independent
- sample sizes make the normal approximation credible
- the hypothesis and one- or two-sided direction were set before analysis
Do not rely on a universal “n greater than 30” rule. For rare events, 30 observations can produce almost no successes; for balanced common events, approximation quality can be good. Inspect expected successes and failures and use an exact or model-based method when counts are sparse.
In high-volume online experiments, a normal approximation is also used for many sample means through the central limit theorem. The important question is whether the estimator's sampling distribution and variance calculation are valid for the metric, not whether the raw user values look perfectly normal.
When to use a t-test
A t-test is designed for inference about means when the variance is estimated from sample data. That extra variance uncertainty produces a t distribution with heavier tails than the standard normal, especially at small sample sizes.
For two independent groups, default to Welch's t-test unless equal variance is justified. Welch's version does not assume the two population variances are equal and handles unequal group sizes. NIST's two-sample t-test reference shows the unequal-variance standard error based on each group's sample variance and size.
Use an independent two-sample t-test when:
- the outcome is numeric and the mean is the target
- the two groups contain different experimental units
- observations are independent within the model
- the mean and standard error behave well enough for the sample size
Use a paired t-test when each value has a meaningful partner: the same user's before-and-after score, or deliberately matched units. The analysis reduces each pair to a difference and tests the mean of those differences. Treating paired data as independent discards information and computes the wrong standard error.
The t-test can be sensitive to extreme values because the sample mean and variance are sensitive to them. Product metrics such as revenue or session duration are often skewed. At scale, the mean may still have a usable sampling distribution, but inspect outliers, data quality, and the estimand. Robust inference, transformations, winsorization policies, or bootstrap methods may be more appropriate when a few observations dominate the result.
Reduce variance before launch
Learn how CUPED and covariate adjustment can sharpen experiment estimates without changing the randomized comparison.
Explore Variance Reduction
When to use a chi-square test
Pearson's chi-square statistic compares observed category counts with counts expected under a null hypothesis. Two common forms are:
- goodness of fit: does one categorical distribution match specified probabilities?
- independence or homogeneity: is a categorical outcome distributed the same way across groups?
Suppose an onboarding experiment records three outcomes: completed, skipped, and abandoned. Cross-tabulate outcome by variant. A chi-square test asks whether the outcome distribution is independent of variant.
The test statistic sums (observed - expected)^2 / expected across cells. NIST's chi-square documentation describes the same comparison of binned frequency distributions.
Use a chi-square test when observations contribute counts to mutually exclusive categories and expected cell counts are large enough for the asymptotic approximation. With sparse cells, combine categories only when substantively justified or use an exact method such as Fisher's exact test for a two-by-two table.
A chi-square result says the distributions differ somewhere. It does not provide the most decision-friendly effect estimate by itself. Report category proportions, absolute differences, uncertainty intervals, and the cells contributing to the pattern.
For a binary two-arm experiment, the Pearson chi-square test and a two-sided two-proportion z-test are closely related: under standard conditions, the chi-square statistic with one degree of freedom equals the squared z statistic. Choose the representation that matches the hypothesis and reporting needs.
When to use ANOVA
ANOVA compares variation between group means with unexplained variation within groups. A one-way ANOVA tests the null that all population means are equal across levels of one factor.
Use it for a continuous outcome across three or more independent groups when the global question is whether any mean differs. Classical ANOVA assumes independent errors, normally distributed residuals within the model, and equal variances. Welch ANOVA relaxes the equal-variance assumption; R's 0 implements that approximation.
ANOVA's F-test is an omnibus test. A significant result means at least one mean differs, but it does not identify which one. Use planned contrasts or multiplicity-aware post-hoc comparisons to answer the product question.
ANOVA is more than a rule for “three or more groups.” Multi-factor ANOVA can estimate main effects and interactions in multivariate or factorial experiments. Repeated-measures or clustered data need corresponding error structures rather than a basic one-way calculation.
Why several t-tests are not a substitute for ANOVA
With four variants there are six pairwise comparisons. Testing each at 0.05 creates multiple opportunities for a false positive. An omnibus ANOVA tests one global null first, and planned follow-ups can use Tukey, Holm, Bonferroni, or another procedure appropriate to the family of claims.
The Bonferroni correction is simple and conservative. The right procedure depends on whether the goal is all pairwise comparisons, treatments versus one control, or a small set of preplanned contrasts. Define that family before looking at the ranking.
ANOVA and regression are also two views of the same linear-model machinery. R's 0 documentation describes aov as a wrapper around linear models for experimental designs. Regression is often more flexible when the analysis includes covariates, interactions, or unbalanced data.
Assumptions that change the choice
Before running any of the four tests, verify:
Independence and assignment unit
If the experiment randomizes accounts but analyzes users as independent observations, standard errors will usually be too small. Analyze at the randomization unit or use cluster-aware inference. If users can appear in both groups, repair the assignment or use a model that represents the dependence.
Paired or repeated observations
The same user measured twice is not two independent users. Use a paired test or repeated-measures model. For experiments with many events per user, aggregate to the user level or use appropriate clustered methods.
Outcome distribution and metric construction
Check missingness, zero inflation, extreme tails, ratio denominators, and censoring. A test can be mathematically correct for the supplied numbers while the metric itself misrepresents the user outcome.
Variance assumptions
Prefer Welch's t-test or Welch ANOVA when group variances may differ. Equal sample sizes do not prove equal variance, and a preliminary variance test can introduce another decision layer.
Sample size and sparse cells
Approximate z and chi-square methods need enough information in the relevant cells. Low-frequency guardrails and small segments may need exact methods or longer collection.
A product experimentation decision tree
Use this sequence before opening a statistics package:
- What unit was randomized: user, account, device, session, or region?
- What is the primary estimand: mean, proportion, category distribution, or model coefficient?
- Are groups independent, paired, repeated, or clustered?
- Are there two groups, several groups, or multiple factors?
- Do expected counts and sample sizes support the approximation?
- Are variances, tails, or outliers likely to break the default model?
- How many confirmatory hypotheses can trigger the decision?
- Was the test direction and stopping rule declared before launch?
Then choose the simplest model that answers the exact question. A two-proportion z-test may be perfect for signup conversion, while a t-test handles mean revenue and a chi-square test handles plan mix in the same experiment. Different metrics can require different tests.
Report effects, not only test names
The test produces a statistic and p-value under a null model. The guide to interpreting a t-test p-value shows why that number needs the effect, interval, and degrees of freedom beside it. The product decision needs more:
- the effect estimate in business units
- a confidence or credible interval
- sample sizes and allocation
- baseline and treatment values
- assumption and data-quality checks
- the planned hypothesis family
- practical thresholds and guardrails
GrowthBook's statistics documentation explains the frequentist and Bayesian engines available for experiment analysis. Whichever framework is used, review effect magnitude and uncertainty together. A small p-value can accompany a trivial lift in a huge sample, while a valuable estimated lift can remain uncertain in a small one.
Choose the test by tracing the data back to the experiment design. For three or more continuous-outcome variants, the deeper ANOVA guide covers the omnibus F-test, planned contrasts, and Welch alternative. When the outcome, assignment unit, dependence, and hypothesis are explicit, the difference between z, t, chi-square, and ANOVA becomes a modeling decision rather than a memorization exercise.
Analyze tests with context
Connect experiment assignments to trusted metrics, inspect uncertainty, and keep decision rules visible to the whole team.
Get Started With GrowthBookAn experiment with control plus three variants creates more than one comparison. ANOVA gives the team one principled global test of whether the variants differ before it starts hunting for a winner.
Analysis of variance, or ANOVA, is a family of statistical models for comparing group means and decomposing sources of variation. In a one-way product experiment, the “factor” is the assigned variant and its “levels” are control, B, C, and D.
The basic ANOVA question is deliberately broad: if all variants had the same population mean, would the observed separation among their sample means be surprising relative to the noise within variants?
That question is useful, but incomplete. A significant ANOVA result does not say which variant won, whether the lift is large enough to ship, or whether assumptions and instrumentation are sound. Those conclusions require planned contrasts, uncertainty intervals, and experiment-quality checks.
How ANOVA compares means through variance
ANOVA separates total variability into components:
- between-group variation: how far each group mean is from the overall mean
- within-group variation: how far individual observations are from their group mean
Each sum of squares is divided by its degrees of freedom to produce a mean square. The F statistic is:
Under the null hypothesis that all group means are equal, both quantities estimate the same underlying error variance, so their ratio should often be near 1. When group means are separated relative to the residual noise, F grows.
NIST's one-way ANOVA explanation describes this as comparing the level mean square with the residual mean square. The p-value is the probability, under the null model and assumptions, of an F statistic at least as large as the observed one.
For k groups and N total observations, one-way ANOVA usually has:
The numerator asks how much the k means vary. The denominator pools information about variability inside the groups.
A four-variant experiment example
Suppose a SaaS team tests four onboarding flows and measures projects created per eligible account during the first week.
| Variant | Accounts | Mean projects | Standard deviation |
|---|---|---|---|
| Control | 1,000 | 2.30 | 1.80 |
| B | 1,020 | 2.42 | 1.84 |
| C | 990 | 2.61 | 1.91 |
| D | 1,010 | 2.36 | 1.79 |
The null hypothesis is:
The alternative is that not all four means are equal. Notice what it does not say: “C is best.” The global alternative includes any pattern where at least one mean differs.
If the F-test rejects the null, the team should evaluate the comparisons it planned. It might compare every treatment with control, or test one contrast between the current flow and the average of three new concepts. The comparison plan should reflect the decision, not the visual ranking in the finished dashboard.
Make multiple tests trustworthy
See how experimentation leaders plan hypotheses, guardrails, and review practices when a result surface contains many possible claims.
Watch the Trustworthy Experiments TalkWhy not run every pairwise t-test?
Four groups create six pairs. If the team runs six independent tests at alpha 0.05 and treats any significant result as proof, the probability of at least one false positive across the family can exceed 0.05.
ANOVA gives one global test of the equality of all means. It also estimates residual variation using all groups, which can be more efficient than estimating it afresh for each pair under the classical equal-variance model.
The global test does not eliminate multiplicity in follow-up comparisons. R's Tukey HSD documentation explicitly notes that ordinary t-tests inflate the probability of a false declaration across a family. Choose the follow-up procedure for the comparisons the decision actually needs:
- every pair: Tukey-style simultaneous comparisons
- every treatment versus control: Dunnett-style comparisons
- a few planned product questions: predeclared contrasts with a suitable adjustment
- a conservative small family: a Bonferroni or Holm correction
An omnibus test can also be nonsignificant while one carefully planned contrast is persuasive, because the hypotheses and power differ. Decide before launch whether the global null or a treatment-versus-control contrast is the primary decision test.
Unequal group sizes do not automatically invalidate ANOVA, but they make the variance assumption and contrast plan more consequential. If allocation is intentionally uneven, power the smallest comparison that drives the decision and preserve the assignment probabilities. When variances and sample sizes both differ, classical pooled ANOVA can behave poorly; Welch ANOVA or a regression with suitable standard errors is usually easier to defend.
Planned contrasts can also use product structure that the global test ignores. Instead of comparing every pair, a team might compare control with the average of three related treatments, or compare two low-intensity treatments with two high-intensity treatments. A small set of predeclared contrasts often answers the business question with more power and clearer multiplicity control than an exhaustive winner search.
ANOVA assumptions in experiments
The familiar one-way fixed-effects model can be written as:
Classical inference depends on the residuals and design, not on a requirement that the combined raw outcome form one bell curve. NIST's model reference assumes independent, normally distributed errors with mean zero and common variance.
Independent observations
The analysis unit must respect randomization. If accounts are assigned but every user within an account is treated as independent, the standard error ignores clustering. Aggregate at the account level or use cluster-robust or hierarchical methods.
Repeated events from one user create the same problem. Ten sessions from one user do not carry the same independent information as ten users.
Appropriate residual behavior
ANOVA is often robust to moderate non-normality with balanced, sufficiently large groups, but severe skew, outliers, censoring, or zero inflation can make the mean unstable or the F approximation unreliable. Diagnose residuals and assess whether the mean is still the business estimand.
Equal variance for classical one-way ANOVA
Classical ANOVA assumes a common population variance. This can fail when a treatment changes both the mean and spread, or when groups serve different traffic mixes. Unequal group sizes make the problem more consequential.
SciPy's 0 supports Welch ANOVA when equal_var=False. Welch's method relaxes equal population variances and adjusts the degrees of freedom.
Correct outcome model
ANOVA targets a continuous mean. Conversion is binary; event counts are discrete; time-to-churn can be censored. Large-sample mean inference can sometimes work, but logistic, Poisson or negative-binomial, survival, or other generalized models may better represent the outcome and produce interpretable effects.
One-way, two-way, and repeated-measures ANOVA
“ANOVA” names a family rather than one calculation.
One-way ANOVA
One categorical factor with multiple levels, such as four assigned onboarding variants. This is the usual A/B/n example.
Two-way or factorial ANOVA
Two controlled factors, such as headline and layout. The model estimates each main effect plus their interaction. The interaction asks whether one factor's effect changes with the other. This is central to a properly designed multivariate test.
Repeated-measures ANOVA
The same units are observed under multiple conditions or times. Dependence is part of the design and must be modeled. A basic independent one-way ANOVA is invalid for repeated measurements.
ANCOVA
Analysis of covariance adds continuous covariates to the group comparison. In randomized experiments, pre-experiment covariates can improve precision when they are chosen and measured without post-treatment contamination. GrowthBook's guide to variance reduction explains the same motivation in online experimentation.
Run one-way ANOVA in Python
At the action boundary, keep one numeric observation per independent analysis unit in each group. In SciPy:
Before running it, confirm that rows match the randomization unit and missing values have a documented policy. Afterward, inspect group summaries and residual behavior. The p-value alone cannot reveal a broken exposure join or a few enormous outliers.
In R, aov(outcome ~ variant, data = experiment) fits the classical model. R documents 1 as a linear-model interface, which helps explain why ANOVA, regression, and contrasts are closely connected.
Interpret the ANOVA table
A standard output contains:
- degrees of freedom
- sum of squares
- mean square
- F statistic
- p-value
Suppose the output reports F(3, 4016) = 6.8, p < 0.001. Under the model, the observed ratio of between-variant to within-variant variation is unlikely if all four population means are equal. It does not mean every treatment beats control or that any effect is commercially important.
Add the quantities the product decision needs:
- each mean and sample size
- differences from control in original units
- simultaneous or comparison-specific intervals
- an effect-size measure when useful
- guardrail and data-quality results
- the follow-up comparison method
Avoid ranking noisy means without uncertainty. The highest observed variant has benefited from both its true effect and sampling variation, especially when many variants were screened.
Common ANOVA mistakes
Treating events as independent users
Repeated events make the nominal sample size huge and uncertainty too narrow. Preserve the assignment unit.
Using ANOVA for every metric shape
The word “variant” does not imply ANOVA. Match the outcome distribution and estimand to a model.
Checking assumptions after selecting a winner
Write the model, outlier policy, transformation, and variance choice before the ranking is visible. Result-driven switching creates hidden researcher degrees of freedom.
Treating a significant F-test as a winner declaration
Follow with the planned contrasts. The omnibus test only rejects equality of all means.
Ignoring practical significance
A very large experiment can detect a tiny difference. Compare intervals with a minimum practical effect and account for implementation cost and guardrails.
Use ANOVA as part of an experiment plan
Before launch, specify the factor and levels, independent unit, primary continuous outcome, minimum effect, sample-size plan, variance assumption, global or contrast hypothesis, comparison family, and stopping rule.
Then verify assignment and exposure before interpreting the model. A sample ratio mismatch can signal that observed group counts no longer reflect the planned randomization. No F-test can repair biased exposure data.
ANOVA is valuable because it turns a field of variant means into a structured model of signal and noise. The broader z-test, t-test, chi-square, and ANOVA guide shows when the outcome and hypothesis call for another member of that family. Use the omnibus test for the global question, planned contrasts for the decision, and effect estimates for practical judgment. That sequence makes a multiple-variant test easier to defend than a dashboard full of uncoordinated p-values.
Compare variants with discipline
Run controlled experiments, connect trusted metrics, and review treatment effects and uncertainty in one shared workflow.
Start With GrowthBookReady to ship faster?
No credit card required. Start with feature flags, experimentation, and product analytics—free.
