Understanding false causality and examples
Your dashboard shows a metric moving in the right direction right after a feature launch.
The story writes itself — and that's exactly the problem. The causal narrative feels earned because the data is real, but data showing that two things happened together says nothing about whether one caused the other. Teams that skip that distinction don't just get wrong answers — they ship the wrong features, fund the wrong campaigns, and optimize for metrics that have no real connection to the outcomes they care about.
This article is for engineers, product managers, and data analysts who work with experiment results and analytics data and want to reason about it more carefully. Whether you're new to statistical thinking or just looking to sharpen how you evaluate causal claims, here's what you'll learn:
- What false causality is and why human cognition makes it the default reasoning error, not the exception
- The specific subtypes — post hoc, cum hoc, the Texas Sharpshooter fallacy, regression to the mean, and more — and how each one produces a different flavor of wrong conclusion
- Real examples, from ice cream and drowning rates to product analytics and the UC Berkeley admissions case, that show how the same error scales from obvious to invisible
- How confounding variables create spurious correlations and why Simpson's Paradox can make aggregate data point in the exact wrong direction
- Practical safeguards — A/B testing, pre-registration, multiple testing corrections, and anti-peeking discipline — that help teams build systems that catch these errors before they become decisions
The article moves from concept to taxonomy to examples to methodology. By the end, you'll have a working vocabulary for spotting false causal reasoning in the wild and a concrete set of practices for avoiding it in your own work.
False causality is an informal fallacy with expensive consequences
Every data team has been there: a metric moves, someone finds a correlated variable, and within minutes a causal story has taken shape. The feature launch caused retention to spike. The email campaign caused the revenue jump. The new onboarding flow caused the drop in churn.
These conclusions feel earned — they're backed by data, after all. But the data shows association, not causation, and conflating the two is one of the most consequential reasoning errors in analytical work. That error has a name: false causality.
This article is designed to be genuinely useful to practitioners who work with data every day. Here's what we'll cover:
- What false causality is and why it's formally classified as a logical fallacy
- The main subtypes: post hoc, cum hoc, Texas Sharpshooter, Simpson's Paradox, and more
- Real-world examples from business and product analytics
- How confounding variables produce spurious correlations
- Why controlled experiments work — and how they can still fail
- The organizational habits that make causal discipline stick
Non causa pro causa: the formal definition
False causality — also called the false cause fallacy — is formally classified in logic as non causa pro causa, Latin for "not the cause for the cause." It belongs to the category of informal fallacies, specifically fallacies of presumption: arguments that presume something to be true that has not actually been established.
The logical pattern is straightforward: A is regularly associated with B; therefore, A causes B. The inference fails because correlation establishes association — it tells you that two variables move together — but it says nothing about mechanism, direction, or exclusivity.
Something else entirely might be driving both A and B. A might actually be caused by B, not the other way around. Or the relationship might be coincidental, a statistical artifact with no meaningful structure at all.
False causality is also an umbrella term. Post hoc ergo propter hoc, cum hoc ergo propter hoc, the third-cause fallacy, and several other named variants are all subtypes of the same underlying error. What they share is the unwarranted leap from "these things are associated" to "one of them explains the other."
Why the human brain defaults to this error
The reason false causality is so pervasive isn't carelessness or low analytical skill. It's cognition working exactly as designed.
Humans are pattern-seeking by nature. When we observe two events occurring together — especially in sequence — we instinctively reach for a causal explanation. As one framing puts it: "We naturally look for explanations when we notice patterns. This tendency can lead us straight into faulty causal reasoning."
The same cognitive shortcut that helped early humans survive (if predators appeared near the river, avoid the river) becomes a systematic liability when applied to dashboards and experiment readouts.
This matters because it means false causality isn't an outlier mistake made by unsophisticated analysts. It is the default output of human cognition applied to correlated data without rigorous controls. Recognizing the error requires deliberate effort precisely because the incorrect inference feels natural and even obvious.
Why it costs teams real money and strategic direction
In product, marketing, and data work, false causality doesn't just produce wrong answers in a vacuum — it produces wrong decisions that compound over time. Teams ship features that didn't actually drive the outcomes attributed to them. Marketing budgets get reallocated toward campaigns that happened to coincide with seasonal trends. Optimization efforts get directed at proxy metrics that have no genuine causal link to the goals they're supposed to represent.
That last failure mode has a specific name: Goodhart's Law. When a proxy metric is not strongly causally linked to the target metric, pushing hard on the proxy may have no effect on the actual goal — or may actively break the correlation that made the proxy seem useful in the first place.
The causal assumption embedded in metric selection turns out to be false, and the entire optimization strategy built on top of it collapses.
False causality also surfaces in subtler ways. Simpson's Paradox — where an apparent trend in aggregate data reverses entirely once a confounding variable is accounted for — is a documented, real-world example of how false causal conclusions can emerge from legitimate data analyzed without sufficient rigor.
The UC Berkeley 1973 admissions case is the canonical illustration, and it's worth noting that the data wasn't fabricated or cherry-picked; the false causal conclusion arose from a failure to account for a hidden variable.
Understanding false causality starts here, with the definition: an informal logical fallacy in which association is mistaken for causation. Everything else — the subtypes, the examples, the experimental safeguards — is built on this foundation.
The main types of false causality fallacies: post hoc, cum hoc, and more
False causality isn't a single mistake — it's a family of related errors, each with its own mechanism for producing a faulty causal conclusion. What they share is "the illogical assumption that a specific factor caused a specific effect."
But the way each subtype arrives at that assumption differs, and recognizing those differences is what allows you to catch the error in a dashboard, a business review, or a team meeting. Here's a working taxonomy.
Post hoc ergo propter hoc: mistaking sequence for cause
The Latin phrase translates roughly to "after this, therefore because of this." The mechanism is simple: A happens before B, so A must have caused B. The error is treating temporal sequence as causal evidence.
The classic illustration is superstitious reasoning: "Every time I wear this jersey, my team loses — it must be unlucky." The jersey preceded the loss, so it gets blamed. In business contexts, this looks like: "We launched a new homepage last quarter, and revenue went up — the redesign drove growth." Maybe it did. But the fact that one thing preceded another tells you nothing about whether a causal relationship exists.
Cum hoc ergo propter hoc: simultaneous correlation
This variant drops the temporal element entirely. Two things move together — they rise and fall in tandem — so one must be causing the other. The canonical example is ice cream sales and drowning incidents. Both spike in summer and drop in winter.
The actual driver is warm weather, which independently increases both swimming activity and ice cream consumption. Neither causes the other.
The distinction from post hoc matters: in cum hoc reasoning, there's no claim that one event preceded the other. The error is purely about co-occurrence being mistaken for causation.
The third-cause fallacy: the hidden variable
Closely related to cum hoc, the third-cause fallacy occurs when two correlated variables are treated as causally linked while an unmeasured third variable — the actual driver — is ignored. Warm weather is the third cause in the ice cream example. This error becomes particularly dangerous at scale, because the correlation can be statistically robust and the hidden variable genuinely difficult to identify without deliberate investigation.
The Texas Sharpshooter fallacy: retrofitting patterns to data
The name comes from an image of a Texan firing at a barn wall and then painting a target around the bullet holes. In data analysis, the equivalent is examining results without a pre-set hypothesis, finding a cluster that looks meaningful, and treating it as a discovery. GrowthBook's experimentation documentation defines it precisely as "cherry-picking data clusters to suit a particular argument, hypothesis, or bias."
This fallacy is directly tied to the multiple testing problem. If you test 20 metrics at a 5% significance threshold, you have roughly a 64% probability of finding at least one false positive by chance alone — even if nothing real is happening.
The Texas Sharpshooter fallacy is what happens when analysts don't account for that and report the significant result as if it were a genuine finding. GrowthBook's docs flag this as a specific risk when teams "analyze the data in multiple ways or look at various subgroups without adjusting for multiple comparisons."
Wrong direction: reversing cause and effect
Sometimes the causal relationship is real, but the direction is inverted. A study might find that hospitals are associated with higher mortality rates and conclude that hospitals cause death — when in reality, people go to hospitals because they are already sick or injured. The association is genuine; the interpretation is backwards.
In product analytics, this shows up when teams observe that heavy users engage more with a new feature and conclude the feature is driving engagement, when the actual pattern is that highly engaged users are simply more likely to try new features.
The regression fallacy: mistaking natural variation for intervention
Extreme values tend to move back toward average over time — this is regression to the mean, and it happens regardless of any intervention. The regression fallacy occurs when that natural movement is credited to something that happened in between.
A sales team has its worst month on record, leadership introduces a new process, and the next month performance rebounds. The process gets the credit. But some portion of that rebound would have happened anyway, simply because extreme low performance rarely persists. Without a control group, there's no way to separate the intervention's effect from the natural correction.
Each of these subtypes produces the same surface-level error — a causal claim that isn't warranted — but through meaningfully different paths. Knowing which one you're looking at shapes how you'd go about disproving it.
False causality examples: from ice cream and drowning rates to flawed business decisions
The easiest way to understand false causality is to start somewhere almost embarrassingly obvious — and then notice how the same mistake, dressed in more sophisticated clothing, shows up in your team's quarterly review.
The classic examples that make the pattern click
Ice cream sales and drowning incidents both spike in summer. If you plotted them on a chart, you'd see a near-perfect correlation. A naive reading of that chart might suggest that ice cream consumption somehow increases drowning risk — or, absurdly, that drowning incidents drive ice cream sales.
Neither is true. Both are driven by a third variable: hot weather draws people to pools and beaches while simultaneously driving ice cream consumption. The correlation is real; the causal relationship is entirely fabricated.
The same structure appears in simpler superstitions. "Every time I wear my lucky socks, we win the game" is a post hoc ergo propter hoc error — the socks came before the win, so the socks must have caused it. Or consider the political version: "After the new mayor took office, crime went up."
The temporal sequence feels like an explanation, but it isn't one. Crime trends are shaped by economic conditions, policing policy, demographic shifts, and dozens of other variables that have nothing to do with who won the last election.
These examples feel obvious in isolation. The problem is that the underlying reasoning pattern — "these two things happened together, so one must have caused the other" — is exactly the same pattern your team uses when it looks at a dashboard.
Where false causality causes real business harm
The business version of the ice cream problem is subtler but structurally identical. Imagine a marketing team that launches a campaign in early Q4, and sales spike two weeks later. The campaign gets the credit.
But Q4 also brings seasonal buying patterns, a competitor's product recall, and a PR moment from an unrelated news story. Without isolating those variables, attributing the spike to the campaign is the same logical error as blaming the mayor for the crime rate.
A more insidious version involves proxy metrics. Product teams routinely use metrics like "items added to cart" as a stand-in for purchases, assuming a causal link between the two. But as GrowthBook's documentation on experimentation problems notes, "if the proxy metric is not strongly causally linked to the target metric, pressing hard on the proxy may have no effect on the goal metric, or might actually cause the correlation to break."
Optimizing aggressively for a proxy that isn't causally connected to the outcome you care about is false causality operationalized into your product roadmap.
The cost of acting on these false causal assumptions is real. Merritt Aho, Digital Analytics Lead at Breeze Airways, put it plainly: "People only see the wins, but there's actually greater value in avoiding losses. We've stopped changes that could have cost millions." That's the business case for taking false causality seriously — not as an academic concern, but as a source of expensive, avoidable decisions.
When the numbers lie in product analytics
The most dangerous false causality in analytics contexts is the kind that hides inside aggregate data. The UC Berkeley 1973 admissions case is the canonical example. Looking at overall admission rates, men appeared to be admitted at a significantly higher rate (44%) than women (35%) — a pattern that seemed to implicate gender bias.
But when researchers broke the data down by department, women actually had higher admission rates than men in many departments. In the Department of Education, for instance, women were admitted at a 77% rate compared to 62% for men.
The confounding variable was department choice. Women disproportionately applied to more competitive departments with lower overall acceptance rates. Once that variable was accounted for, the apparent pattern of discrimination reversed entirely. The aggregate number wasn't lying exactly — it was just answering a different question than the one people thought they were asking.
Product analytics teams run into this constantly. An aggregate metric improves, but when you segment by acquisition channel or device type, the improvement disappears — or exists only in one cohort that happened to grow. Twyman's Law offers a useful heuristic here: "Any data or figure that looks interesting or different is usually wrong."
When a result looks surprisingly clean, the more likely explanation is a data or implementation problem, not a genuine causal effect.
Industry-wide A/B test success rates sit around 33%, meaning teams' intuitions about what will move a metric are wrong roughly two-thirds of the time. That's not a reason for paralysis — it's a reason to be skeptical of causal stories that haven't been tested.
Correlation vs. causation: how confounding variables drive false causality
Most false causality errors don't happen because analysts are careless. They happen because a hidden third variable is quietly driving both sides of an observed relationship, making two unrelated things look like cause and effect. Understanding this mechanism — the confounding variable — is essential for anyone who makes decisions based on data.
What is a confounding variable?
A confounding variable is a third factor that independently influences both the apparent cause and the apparent effect, producing a spurious association between them. The two observed variables aren't causally linked at all; they're both downstream of something else.
A clean example from product analytics: users who have activated more in-app notifications tend to spend more time in the app. The tempting interpretation is that notifications drive engagement. But the actual driver may be that power users — people who already love the product — are both more likely to turn on notifications and more likely to spend hours in the app.
User engagement level is the confounder. It explains both behaviors independently, and the correlation between notifications and time-in-app is entirely spurious.
What makes confounders particularly dangerous is that they're often unmeasured or unrecognized. If you never think to look for the third variable, the spurious correlation looks like solid evidence.
Why correlation is an unreliable proxy for causation
Correlation measures the degree to which two variables move together. It says nothing about whether one causes the other, or whether both are being driven by something else entirely. This is the formal basis of the cum hoc ergo propter hoc fallacy — mistaking simultaneous correlation for causation.
In the presence of a confounder, two completely unrelated variables can show strong, consistent correlation. The pattern looks compelling. It replicates across time periods. It shows up in your dashboards. And it's still entirely misleading.
For product and marketing teams, the practical cost is real: optimizing based on correlated metrics without testing for causation means investing in features, campaigns, or interventions that have no actual effect on outcomes. The metric moves, but not because of anything you did.
Simpson's Paradox — when the aggregate pattern lies
The most dramatic illustration of confounding-driven false causality is Simpson's Paradox: a statistical phenomenon where a trend appears in aggregate data but disappears or reverses entirely when the data is broken down by subgroup.
The UC Berkeley 1973 admissions case — covered in detail in the previous section — is the canonical illustration: the aggregate data pointed in the exact wrong direction once department choice was accounted for as a confounding variable.
This is confounding at its most extreme: the observed pattern didn't just understate the true relationship, it pointed in the wrong direction entirely. Any decision made on the basis of the aggregate data would have been not just imprecise but actively wrong.
Controlled experiments as the methodological response
The reason controlled experiments — particularly randomized A/B tests — are the gold standard for establishing causation is precisely because they neutralize confounders. By randomly assigning subjects to treatment and control conditions, randomization distributes confounding variables roughly equally across groups.
Whatever third factors exist, they're present in both groups, so they can't explain away a difference in outcomes.
Observational analysis can partially address confounding through stratification and statistical controls, but these approaches require you to identify and measure the confounders in advance — which is exactly what you often can't do. Randomization handles confounders you haven't thought of yet.
For teams running experiments, this means paying close attention to whether experimental groups are actually comparable in demographics and behavior before drawing conclusions. Platforms built for rigorous experimentation — GrowthBook among them — incorporate techniques like CUPED and Sample Ratio Mismatch detection specifically to catch the kinds of group imbalances that can reintroduce confounding even within a structured experiment. The mechanics of randomization are necessary but not always sufficient; the analysis has to hold up too.
Controlled experiments neutralize confounders — but only when the analysis holds up
Understanding false causality is one thing. Building systems that reliably avoid it is another. The honest answer to "how do I stop drawing false causal conclusions?" is: run controlled experiments. But that answer is incomplete without a serious accounting of the ways experiments themselves can go wrong.
A/B testing as the gold standard — and why it works
The reason controlled experiments are so effective at establishing causation comes down to a single mechanism: random assignment. When you randomly split users into a control group and a treatment group, you distribute confounding variables roughly equally across both groups.
The hidden factors that corrupt observational data — seasonality, user demographics, concurrent product changes — don't disappear, but they stop being a problem because they affect both groups equally. What's left is the isolated effect of the variable you're actually testing.
This is precisely what observational analysis cannot do. You can control for confounders you know about, but you can't control for the ones you haven't thought of. Random assignment handles both categories at once.
That said, even well-designed experiments fail when they're poorly planned. GrowthBook's pre-experiment guide frames the problem directly: poorly planned experiments waste time and lead to bad decisions. Defining your hypothesis, primary metric, and success criteria before you start isn't bureaucratic overhead — it's the thing that makes your results interpretable.
Statistical pitfalls inside experiments
Here's the uncomfortable truth: you can run a properly randomized A/B test and still draw a false causal conclusion. The mechanism is usually one of three things.
P-hacking happens when analysts — often unconsciously — explore different metrics, time periods, or user segments until they find a statistically significant result. The problem isn't malice; it's that statistical significance at p < 0.05 means you'll see a false positive 5% of the time by chance alone. If you're testing enough slices of your data, false positives become nearly inevitable.
The multiple testing problem makes this concrete. If you test the same hypothesis across 20 different metrics at a 5% significance level, the probability of finding at least one statistically significant result purely by chance is around 64% — assuming those metrics are independent, which they often aren't in digital products.
These corrections work by raising the bar for what counts as statistically significant when you're testing many things at once — the more tests you run, the stricter the threshold needs to be. GrowthBook's documentation on experimentation problems names the standard approaches: Bonferroni correction, False Discovery Rate correction, and the Benjamini-Hochberg procedure, each offering a different trade-off between sensitivity and specificity. The practical takeaway is simpler: if you're tracking a large number of metrics, treat any single significant result as a signal to run a follow-up experiment, not as a conclusion.
Peeking — stopping an experiment early because the results look good — is a subtler failure mode. Every time you look at interim results and consider stopping, you inflate your false positive rate. The fix is straightforward in principle: set a predetermined sample size or duration before the experiment starts and commit to it. For teams that need more flexibility, sequential testing methods are specifically designed to allow early stopping without inflating error rates.
Pre-registration and the Texas Sharpshooter problem
The Texas Sharpshooter fallacy — shooting at a barn, then painting a target around the bullet holes — has a direct experimental equivalent: analyzing your data first, then constructing a hypothesis to fit what you found. It's easy to do accidentally. You run an experiment, dig into the results looking for something interesting, find an unexpected segment that shows a strong effect, and report that as your finding.
But that's not a hypothesis you tested; it's a pattern you noticed after the fact.
The defense is pre-registration in practice: write down your hypothesis, your primary metric, and your definition of success before you look at any results. This isn't just a statistical formality — it's what separates a finding from a story you told yourself about your data. GrowthBook's documentation notes that unusually large or surprising results should trigger skepticism rather than celebration until you've ruled out implementation errors. If a result looks too good, it probably is.
Building habits that outlast any single experiment
Avoiding false causality at scale isn't purely a statistical problem — it's an organizational one. When a measure becomes a target, it ceases to be a good measure. Teams that optimize for proxy metrics (items added to cart, say, rather than completed purchases) can produce results that look causal but aren't, because the proxy may not be strongly linked to the outcome that actually matters.
The teams that consistently avoid false causality share a few habits: they define hypotheses before running experiments, they apply correction methods when testing multiple metrics, they treat surprising results as a reason to investigate rather than celebrate, and they run follow-up experiments to validate findings before acting on them. Methodology gets you most of the way there. Discipline in execution gets you the rest.
The causal story feels true because pattern recognition is what brains do
The through-line of this article is simple: the causal story your brain constructs from correlated data feels true because pattern recognition is what brains do. That's not a flaw to fix — it's a feature to compensate for. The compensation is methodology: controlled experiments, pre-registered hypotheses, multiple testing corrections, and the discipline to treat surprising results as a reason to investigate rather than ship.
A quick-reference summary: the most common false causality patterns to watch for
Post hoc errors show up whenever a metric moves after a launch and the launch gets the credit. In dashboards where two lines move together and no one asks what's driving both, that's cum hoc reasoning. Any analysis that started with the data rather than a hypothesis is a candidate for the Texas Sharpshooter fallacy.
And Simpson's Paradox is waiting in every aggregate metric that hasn't been segmented — the UC Berkeley case is a reminder that the aggregate number can point in the exact wrong direction while every subgroup tells the opposite story.
Practical questions to ask before drawing any causal conclusion
Before you attribute a metric movement to a cause, ask three things: Was there a pre-specified hypothesis before the data was collected? Is there a plausible third variable that could explain both sides of the correlation? And does the pattern hold when you break it down by meaningful subgroups? These aren't bureaucratic checkboxes — they're the questions that separate a finding from a story you told yourself about your data.
What to do next
Look at the last causal claim your team acted on — a feature that "drove" a metric, a campaign that "caused" a lift. Ask whether it was tested with a control group, whether the hypothesis was written before the data was collected, and whether the result held across subgroups. If the answer to any of those is no, you've found your starting point. Run the follow-up experiment before the next decision gets made on the same assumption.
The teams that get this right are more disciplined organizationally than statistically
The teams that get this right aren't necessarily more sophisticated statistically — they're more disciplined organizationally. They write down what they expect to see before they run an experiment. They apply corrections when they're tracking multiple metrics. They treat Twyman's Law as a real heuristic: if a result looks surprisingly clean, the first assumption is that something is wrong, not that something is working.
Statistical guardrails built into experimentation platforms — sample ratio mismatch detection and variance reduction techniques — catch the kinds of group imbalances and noise that quietly corrupt even well-randomized experiments. But the tooling only helps if the habits are already there.
This article was written to be genuinely useful to practitioners who are tired of making expensive decisions on the basis of correlations that felt like causes. If it gave you a sharper vocabulary for one meeting or one experiment review, it did its job.
Related insights
Related Articles
In healthcare, “Can we randomize it?” is the wrong first question. Start with “Could either experience change care, rights, privacy, or access?”
A/B testing can improve digital intake, appointment access, patient education, clinician workflows, and administrative operations. It can also create unacceptable risk when teams treat a clinical or consent decision like an ordinary conversion funnel.
The difference is not the label on the method. A/B tests are randomized experiments. What matters is the treatment, purpose, affected population, data flow, and oversight required in the organization and jurisdiction. This guide provides a practical product framework, not a substitute for legal, clinical, privacy, security, or institutional review.
Draw the boundary before designing variants
Create an intake step that classifies the proposed change before anyone builds a treatment. At minimum, ask:
- Can the change alter diagnosis, treatment, triage, dosage, or clinical recommendations?
- Can it delay or discourage access to care, accommodations, or urgent help?
- Does it change informed consent, privacy choice, required disclosure, or patient cost?
- Does it use protected or sensitive health information for assignment or measurement?
- Does it include children, people in crisis, or another population requiring added protection?
- Is the purpose internal quality improvement, or is it designed to contribute to generalizable knowledge?
- Could the software function fall within medical-device or clinical decision-support oversight?
The HHS quality-improvement guidance says many activities limited to improving patient care and collecting operational data are not research under the cited human-subjects regulations. It also states that some quality-improvement activities can have a research purpose, in which case human-subject protections may apply. A product team should not make that determination informally; route it to the organization’s authorized office.
Likewise, software that influences clinical decisions is not automatically an ordinary product surface. The FDA’s January 2026 clinical decision-support guidance explains that some software functions are excluded from the device definition while other patient- or caregiver-facing functions can remain subject to digital-health policy. Clinical and regulatory owners need to classify the function before experimentation.
Start with lower-risk operational questions
The safest early program tests reversible changes where both variants meet the same clinical, accessibility, privacy, and disclosure requirements.
Appointment reminder timing
Compare 2 approved reminder schedules or message structures to reduce missed appointments. Keep required details, opt-out behavior, language support, and urgent-contact instructions constant.
Use completed appointments or timely rescheduling as the primary outcome. Track cancellations, patient contacts, message delivery, opt-outs, wrong-recipient risk, and differences across language, age, disability, or access groups. A higher click rate is not enough if no-show rates or trust worsen.
Patient portal navigation
Test whether a clearer information architecture helps people complete a high-value administrative task, such as finding results, updating insurance, or sending a non-urgent message. Preserve emergency guidance and clinical escalation paths in both variants.
Measure successful task completion and time to completion. Guard against repeated navigation, abandonment, accessibility failures, mistaken message routing, and increased call-center burden. Use usability testing before the A/B test to catch failures randomization should never expose.
Administrative form sequence
Compare a long form with a staged flow, or test the order of non-clinical fields. Do not omit information needed for safe care, billing transparency, consent, or legal compliance.
Measure accurate completion, not just submission. Track validation errors, correction rates, staff rework, abandonment, and time to appointment. If the treatment collects sensitive data, confirm necessity and access controls before launch.
Educational content layout
Test 2 ways to present the same clinician-approved information: summary-first versus stepwise, text plus illustration versus text alone, or a clear action checklist versus a dense paragraph. Keep the medical meaning, risks, contraindications, and escalation advice equivalent.
Use a comprehension or appropriate next-action metric when feasible. Page time and clicks can be misleading. Accessibility, language quality, and comprehension across health-literacy levels belong in the guardrail plan.
Review the design before launch
Use a trustworthy experiment-design session to pressure-test metrics, safety checks, and decision rules before exposing patients or clinicians.
Watch the Experiment Design Session
Use stronger controls for care-adjacent products
Some product changes are not clinical interventions but can still influence care. They need clinical ownership, narrower eligibility, conservative ramps, and explicit stopping criteria.
Clinician workflow support
A test might compare how a work queue prioritizes administrative follow-up, how a note template reduces documentation work, or how a non-diagnostic alert is presented. The treatment should not silently alter the clinical standard of care.
Randomize at the unit that prevents contamination. Individual clinician assignment may fail when teams share queues and handoffs; clinic- or unit-level clusters may better match the workflow. Measure task completion and time saved, with guardrails for missed work, overrides, escalations, documentation quality, and staff workload.
Preventive-care outreach
Compare approved outreach content or channels for people already eligible under the same clinical rule. Do not experiment with whether one group receives necessary care or required notice.
Use completed appropriate follow-up as the primary outcome. Track opt-outs, unreachable patients, scheduling capacity, disparities, complaints, and downstream cancellations. If the treatment drives demand beyond operational capacity, a messaging lift can make access worse.
Digital adherence support
Test the presentation or timing of an approved reminder, checklist, or educational cue. Avoid treatment changes that could be interpreted as personalized medical advice without the corresponding validation and oversight.
Measure the intended behavior with caution. Self-reported completion or app engagement is not a clinical outcome. Include adverse-event reporting, escalation pathways, disengagement, and privacy events where relevant.
Feature rollout in health software
Use feature flags to separate deployment from release, start with internal or trained cohorts, and expand only when technical and clinical guardrails remain healthy. GrowthBook’s feature flag platform supports targeted rollouts and kill switches, while the experiment layer measures impact.
The rollback plan must describe more than turning off a flag. Determine whether the old experience remains clinically and operationally safe, how queued work is reconciled, what happens to partial workflows, and who is authorized to stop exposure.
Protect data by design
Do not send a broad event stream to an experimentation vendor and decide later which fields were unnecessary. Inventory the data before implementation:
| Data question | Required decision |
|---|---|
| Assignment | What is the least identifiable stable unit that works? |
| Eligibility | Which sensitive attributes are truly needed? |
| Exposure | What event proves the treatment was delivered? |
| Outcomes | Can metrics be computed inside the governed data environment? |
| Access | Which roles can view assignments, segments, and results? |
| Retention | When are raw records, logs, and exports removed? |
The HHS minimum-necessary guidance describes limiting uses, disclosures, and requests for protected health information to what is needed for the intended purpose, with policies based on roles and recurring versus non-routine access. Apply that principle to experiment attributes, debugging logs, dashboards, and downloaded readouts.
Pseudonymous identifiers reduce exposure but do not automatically make a dataset non-sensitive or outside applicable rules. Review linkability, small cohorts, free-text fields, URLs, device metadata, and combinations that can reveal a condition. Never put clinical details or identifiers in feature names, variation labels, or URLs.
A warehouse-native experimentation approach can query approved metrics where the organization already governs them. Architecture does not create compliance on its own; teams still need contracts, access control, auditability, retention rules, security review, and configuration that matches the approved data flow.
Keep unsafe questions out of product experimentation
An experimentation policy should name prohibited or separately governed categories. Product teams should not discover the boundary only after a proposal reaches launch review.
Do not use an ordinary product A/B test to withhold a clinically indicated service, emergency direction, safety warning, accessibility accommodation, required disclosure, or legally protected choice. Do not reduce the visibility of risks to improve completion. Do not randomize a diagnostic or treatment recommendation without the clinical, regulatory, and research framework appropriate to that intervention.
Avoid treatments that exploit fear, urgency, shame, or uncertainty about health. A message can increase appointment conversion while undermining informed choice. Likewise, do not test whether patients tolerate a harder cancellation, more confusing privacy control, or hidden cost. Both variants must meet the organization’s baseline standard for respectful and comprehensible communication.
Clinical AI and decision-support changes need an evaluation program beyond a click-based A/B test. Validate the model offline, examine performance and failure modes across relevant populations, review human factors, and stage deployment with clinical monitoring. An online comparison may contribute evidence only after both treatments meet the safety threshold for exposure.
When an activity may be human-subjects research, follow the institution’s process before enrolling or exposing anyone. HHS research-oversight training states that covered non-exempt human-subjects research requires the applicable review and that informed consent requirements apply unless the IRB authorizes otherwise. The product team should preserve the determination, protocol version, approved treatment, and reporting obligations with the experiment record.
Finally, do not interpret lack of detected harm as proof of safety. Rare adverse events, small vulnerable groups, and outcomes that occur after the experiment window may be underpowered. Use prior evidence, incident monitoring, qualitative reports, and post-rollout surveillance alongside the randomized estimate.
Define patient-centered metrics and guardrails
Healthcare teams need more than a conversion scorecard. Build a measurement hierarchy:
- Primary outcome: the operational or patient-facing result that answers the decision.
- Process diagnostics: steps that explain why the treatment worked or failed.
- Safety guardrails: outcomes that trigger a stop or clinical review.
- Equity checks: predeclared groups where access or benefit could differ.
- Operational guardrails: staffing, wait time, rework, cost, and downstream capacity.
Define the practical threshold before launch. A statistically detectable change may be too small to justify implementation, and a neutral aggregate can hide meaningful harm in a protected or vulnerable group. At the same time, slicing results across many small subgroups increases false-positive risk and can expose sensitive attributes. Predeclare the equity questions that matter and use appropriate privacy and multiple-testing controls.
GrowthBook supports reusable fact tables and metrics so teams can keep definitions reviewable. Use a power analysis for the primary outcome and critical guardrails. If the required sample or duration is unrealistic, do not weaken the standard; use usability research, simulation, staged quality improvement, or a larger treatment contrast.
Create a healthcare experiment review packet
Before launch, the owner should provide one reviewable packet:
- purpose, hypothesis, and operational decision
- classification and required oversight determination
- affected population and exclusion criteria
- clinical, privacy, security, accessibility, and compliance approvals
- treatment screenshots or workflow diagrams
- assignment, exposure, and data-flow design
- primary outcome, diagnostics, guardrails, and equity checks
- sample plan and stopping rule
- rollout stages, monitoring owner, and rollback procedure
- patient or clinician communication plan, if applicable
- documentation and retention plan
Use an approval matrix that names accountable people. Product approval does not replace clinical approval; a privacy review does not settle human-subjects research status; and an IRB determination does not automatically approve the production security architecture.
The WHO clinical-trial best-practices guidance emphasizes ethical standards, regulatory considerations, patient-centered research, transparency, and stakeholder collaboration. Not every healthcare product experiment is a clinical trial, but high-risk work should inherit the same respect for people and evidence.
Build trust into the experimentation program
Start with reversible operational improvements where both experiences are already acceptable. Prove that the team can classify risk, minimize data, validate assignment, monitor safety, and document decisions before expanding scope.
Publish internal rules for what teams may test, what requires added review, and what is out of bounds. Maintain an experiment registry and audit trail. Record neutral and negative results so a new team does not repeat the same risky idea.
GrowthBook can support the controlled delivery and analysis layer through experimentation, feature flags, permissions, and warehouse-defined metrics. The organization remains responsible for the clinical, ethical, legal, privacy, and operational framework around every test.
In healthcare, speed is valuable only when the learning process protects the people whose behavior creates the data.
Build a governed test workflow
Connect controlled releases to reviewable metrics and decision rules while keeping healthcare data in your approved architecture.
Get Started With GrowthBook
The right statistical test is determined by the question and data-generating process, not by which function is easiest to run. Start with the outcome, groups, and dependence structure; the test name comes later.
Z-tests, t-tests, chi-square tests, and analysis of variance (ANOVA) all compare observed data with a null model. They differ in the kind of outcome they model, the uncertainty they estimate, and the number or structure of groups they can compare.
For a simple product experiment, a useful first pass is:
- continuous outcome, two independent groups: usually a Welch two-sample t-test
- binary proportion, two large independent groups: a two-proportion z-test is common
- categorical counts across groups: chi-square test, if expected counts are adequate
- continuous outcome across three or more groups: one-way ANOVA or Welch ANOVA
Those rules are a starting point. Paired observations, clusters, ratios, repeated measures, heavy tails, covariate adjustment, or sequential monitoring require a model that reflects the design.
Choose from the outcome and hypothesis
Write the estimand before choosing a test. An estimand is the quantity the experiment is trying to estimate: a difference in mean revenue, a difference in conversion probability, or an association between two categorical variables.
| Question | Outcome | Common test |
|---|---|---|
| Did average order value change between A and B? | Continuous | Welch two-sample t-test |
| Did signup probability change between A and B? | Binary | Two-proportion z-test |
| Is plan choice associated with variant? | Categorical, 3+ levels | Chi-square test of independence |
| Do mean task times differ across four variants? | Continuous | One-way ANOVA |
| Did the same users' scores change before and after? | Paired continuous | Paired t-test |
The number of groups alone is insufficient. Conversion in four variants is still categorical data; a chi-square or binomial model may fit. Revenue in two groups is continuous; a t-test or regression is more natural.
The University of Michigan's statistical-test guide uses the same sequence: identify variable types and the relationship being tested before selecting a method.
When to use a z-test
A z-test compares a standardized estimate with the standard normal distribution. The classical one-sample z-test for a mean assumes the population standard deviation is known. That condition is unusual in product analytics, where variability is estimated from the current sample.
Z-tests remain common for proportions. In a two-arm conversion experiment, the estimate is:
Under the null of equal proportions and with adequate counts, the standardized difference is approximately normal. This yields a two-proportion z-test.
Use it when:
- the outcome is a binary count summarized as successes and failures
- assignment groups are independent
- sample sizes make the normal approximation credible
- the hypothesis and one- or two-sided direction were set before analysis
Do not rely on a universal “n greater than 30” rule. For rare events, 30 observations can produce almost no successes; for balanced common events, approximation quality can be good. Inspect expected successes and failures and use an exact or model-based method when counts are sparse.
In high-volume online experiments, a normal approximation is also used for many sample means through the central limit theorem. The important question is whether the estimator's sampling distribution and variance calculation are valid for the metric, not whether the raw user values look perfectly normal.
When to use a t-test
A t-test is designed for inference about means when the variance is estimated from sample data. That extra variance uncertainty produces a t distribution with heavier tails than the standard normal, especially at small sample sizes.
For two independent groups, default to Welch's t-test unless equal variance is justified. Welch's version does not assume the two population variances are equal and handles unequal group sizes. NIST's two-sample t-test reference shows the unequal-variance standard error based on each group's sample variance and size.
Use an independent two-sample t-test when:
- the outcome is numeric and the mean is the target
- the two groups contain different experimental units
- observations are independent within the model
- the mean and standard error behave well enough for the sample size
Use a paired t-test when each value has a meaningful partner: the same user's before-and-after score, or deliberately matched units. The analysis reduces each pair to a difference and tests the mean of those differences. Treating paired data as independent discards information and computes the wrong standard error.
The t-test can be sensitive to extreme values because the sample mean and variance are sensitive to them. Product metrics such as revenue or session duration are often skewed. At scale, the mean may still have a usable sampling distribution, but inspect outliers, data quality, and the estimand. Robust inference, transformations, winsorization policies, or bootstrap methods may be more appropriate when a few observations dominate the result.
Reduce variance before launch
Learn how CUPED and covariate adjustment can sharpen experiment estimates without changing the randomized comparison.
Explore Variance Reduction
When to use a chi-square test
Pearson's chi-square statistic compares observed category counts with counts expected under a null hypothesis. Two common forms are:
- goodness of fit: does one categorical distribution match specified probabilities?
- independence or homogeneity: is a categorical outcome distributed the same way across groups?
Suppose an onboarding experiment records three outcomes: completed, skipped, and abandoned. Cross-tabulate outcome by variant. A chi-square test asks whether the outcome distribution is independent of variant.
The test statistic sums (observed - expected)^2 / expected across cells. NIST's chi-square documentation describes the same comparison of binned frequency distributions.
Use a chi-square test when observations contribute counts to mutually exclusive categories and expected cell counts are large enough for the asymptotic approximation. With sparse cells, combine categories only when substantively justified or use an exact method such as Fisher's exact test for a two-by-two table.
A chi-square result says the distributions differ somewhere. It does not provide the most decision-friendly effect estimate by itself. Report category proportions, absolute differences, uncertainty intervals, and the cells contributing to the pattern.
For a binary two-arm experiment, the Pearson chi-square test and a two-sided two-proportion z-test are closely related: under standard conditions, the chi-square statistic with one degree of freedom equals the squared z statistic. Choose the representation that matches the hypothesis and reporting needs.
When to use ANOVA
ANOVA compares variation between group means with unexplained variation within groups. A one-way ANOVA tests the null that all population means are equal across levels of one factor.
Use it for a continuous outcome across three or more independent groups when the global question is whether any mean differs. Classical ANOVA assumes independent errors, normally distributed residuals within the model, and equal variances. Welch ANOVA relaxes the equal-variance assumption; R's 0 implements that approximation.
ANOVA's F-test is an omnibus test. A significant result means at least one mean differs, but it does not identify which one. Use planned contrasts or multiplicity-aware post-hoc comparisons to answer the product question.
ANOVA is more than a rule for “three or more groups.” Multi-factor ANOVA can estimate main effects and interactions in multivariate or factorial experiments. Repeated-measures or clustered data need corresponding error structures rather than a basic one-way calculation.
Why several t-tests are not a substitute for ANOVA
With four variants there are six pairwise comparisons. Testing each at 0.05 creates multiple opportunities for a false positive. An omnibus ANOVA tests one global null first, and planned follow-ups can use Tukey, Holm, Bonferroni, or another procedure appropriate to the family of claims.
The Bonferroni correction is simple and conservative. The right procedure depends on whether the goal is all pairwise comparisons, treatments versus one control, or a small set of preplanned contrasts. Define that family before looking at the ranking.
ANOVA and regression are also two views of the same linear-model machinery. R's 0 documentation describes aov as a wrapper around linear models for experimental designs. Regression is often more flexible when the analysis includes covariates, interactions, or unbalanced data.
Assumptions that change the choice
Before running any of the four tests, verify:
Independence and assignment unit
If the experiment randomizes accounts but analyzes users as independent observations, standard errors will usually be too small. Analyze at the randomization unit or use cluster-aware inference. If users can appear in both groups, repair the assignment or use a model that represents the dependence.
Paired or repeated observations
The same user measured twice is not two independent users. Use a paired test or repeated-measures model. For experiments with many events per user, aggregate to the user level or use appropriate clustered methods.
Outcome distribution and metric construction
Check missingness, zero inflation, extreme tails, ratio denominators, and censoring. A test can be mathematically correct for the supplied numbers while the metric itself misrepresents the user outcome.
Variance assumptions
Prefer Welch's t-test or Welch ANOVA when group variances may differ. Equal sample sizes do not prove equal variance, and a preliminary variance test can introduce another decision layer.
Sample size and sparse cells
Approximate z and chi-square methods need enough information in the relevant cells. Low-frequency guardrails and small segments may need exact methods or longer collection.
A product experimentation decision tree
Use this sequence before opening a statistics package:
- What unit was randomized: user, account, device, session, or region?
- What is the primary estimand: mean, proportion, category distribution, or model coefficient?
- Are groups independent, paired, repeated, or clustered?
- Are there two groups, several groups, or multiple factors?
- Do expected counts and sample sizes support the approximation?
- Are variances, tails, or outliers likely to break the default model?
- How many confirmatory hypotheses can trigger the decision?
- Was the test direction and stopping rule declared before launch?
Then choose the simplest model that answers the exact question. A two-proportion z-test may be perfect for signup conversion, while a t-test handles mean revenue and a chi-square test handles plan mix in the same experiment. Different metrics can require different tests.
Report effects, not only test names
The test produces a statistic and p-value under a null model. The guide to interpreting a t-test p-value shows why that number needs the effect, interval, and degrees of freedom beside it. The product decision needs more:
- the effect estimate in business units
- a confidence or credible interval
- sample sizes and allocation
- baseline and treatment values
- assumption and data-quality checks
- the planned hypothesis family
- practical thresholds and guardrails
GrowthBook's statistics documentation explains the frequentist and Bayesian engines available for experiment analysis. Whichever framework is used, review effect magnitude and uncertainty together. A small p-value can accompany a trivial lift in a huge sample, while a valuable estimated lift can remain uncertain in a small one.
Choose the test by tracing the data back to the experiment design. For three or more continuous-outcome variants, the deeper ANOVA guide covers the omnibus F-test, planned contrasts, and Welch alternative. When the outcome, assignment unit, dependence, and hypothesis are explicit, the difference between z, t, chi-square, and ANOVA becomes a modeling decision rather than a memorization exercise.
Analyze tests with context
Connect experiment assignments to trusted metrics, inspect uncertainty, and keep decision rules visible to the whole team.
Get Started With GrowthBookAn experiment with control plus three variants creates more than one comparison. ANOVA gives the team one principled global test of whether the variants differ before it starts hunting for a winner.
Analysis of variance, or ANOVA, is a family of statistical models for comparing group means and decomposing sources of variation. In a one-way product experiment, the “factor” is the assigned variant and its “levels” are control, B, C, and D.
The basic ANOVA question is deliberately broad: if all variants had the same population mean, would the observed separation among their sample means be surprising relative to the noise within variants?
That question is useful, but incomplete. A significant ANOVA result does not say which variant won, whether the lift is large enough to ship, or whether assumptions and instrumentation are sound. Those conclusions require planned contrasts, uncertainty intervals, and experiment-quality checks.
How ANOVA compares means through variance
ANOVA separates total variability into components:
- between-group variation: how far each group mean is from the overall mean
- within-group variation: how far individual observations are from their group mean
Each sum of squares is divided by its degrees of freedom to produce a mean square. The F statistic is:
Under the null hypothesis that all group means are equal, both quantities estimate the same underlying error variance, so their ratio should often be near 1. When group means are separated relative to the residual noise, F grows.
NIST's one-way ANOVA explanation describes this as comparing the level mean square with the residual mean square. The p-value is the probability, under the null model and assumptions, of an F statistic at least as large as the observed one.
For k groups and N total observations, one-way ANOVA usually has:
The numerator asks how much the k means vary. The denominator pools information about variability inside the groups.
A four-variant experiment example
Suppose a SaaS team tests four onboarding flows and measures projects created per eligible account during the first week.
| Variant | Accounts | Mean projects | Standard deviation |
|---|---|---|---|
| Control | 1,000 | 2.30 | 1.80 |
| B | 1,020 | 2.42 | 1.84 |
| C | 990 | 2.61 | 1.91 |
| D | 1,010 | 2.36 | 1.79 |
The null hypothesis is:
The alternative is that not all four means are equal. Notice what it does not say: “C is best.” The global alternative includes any pattern where at least one mean differs.
If the F-test rejects the null, the team should evaluate the comparisons it planned. It might compare every treatment with control, or test one contrast between the current flow and the average of three new concepts. The comparison plan should reflect the decision, not the visual ranking in the finished dashboard.
Make multiple tests trustworthy
See how experimentation leaders plan hypotheses, guardrails, and review practices when a result surface contains many possible claims.
Watch the Trustworthy Experiments Talk
Why not run every pairwise t-test?
Four groups create six pairs. If the team runs six independent tests at alpha 0.05 and treats any significant result as proof, the probability of at least one false positive across the family can exceed 0.05.
ANOVA gives one global test of the equality of all means. It also estimates residual variation using all groups, which can be more efficient than estimating it afresh for each pair under the classical equal-variance model.
The global test does not eliminate multiplicity in follow-up comparisons. R's Tukey HSD documentation explicitly notes that ordinary t-tests inflate the probability of a false declaration across a family. Choose the follow-up procedure for the comparisons the decision actually needs:
- every pair: Tukey-style simultaneous comparisons
- every treatment versus control: Dunnett-style comparisons
- a few planned product questions: predeclared contrasts with a suitable adjustment
- a conservative small family: a Bonferroni or Holm correction
An omnibus test can also be nonsignificant while one carefully planned contrast is persuasive, because the hypotheses and power differ. Decide before launch whether the global null or a treatment-versus-control contrast is the primary decision test.
Unequal group sizes do not automatically invalidate ANOVA, but they make the variance assumption and contrast plan more consequential. If allocation is intentionally uneven, power the smallest comparison that drives the decision and preserve the assignment probabilities. When variances and sample sizes both differ, classical pooled ANOVA can behave poorly; Welch ANOVA or a regression with suitable standard errors is usually easier to defend.
Planned contrasts can also use product structure that the global test ignores. Instead of comparing every pair, a team might compare control with the average of three related treatments, or compare two low-intensity treatments with two high-intensity treatments. A small set of predeclared contrasts often answers the business question with more power and clearer multiplicity control than an exhaustive winner search.
ANOVA assumptions in experiments
The familiar one-way fixed-effects model can be written as:
Classical inference depends on the residuals and design, not on a requirement that the combined raw outcome form one bell curve. NIST's model reference assumes independent, normally distributed errors with mean zero and common variance.
Independent observations
The analysis unit must respect randomization. If accounts are assigned but every user within an account is treated as independent, the standard error ignores clustering. Aggregate at the account level or use cluster-robust or hierarchical methods.
Repeated events from one user create the same problem. Ten sessions from one user do not carry the same independent information as ten users.
Appropriate residual behavior
ANOVA is often robust to moderate non-normality with balanced, sufficiently large groups, but severe skew, outliers, censoring, or zero inflation can make the mean unstable or the F approximation unreliable. Diagnose residuals and assess whether the mean is still the business estimand.
Equal variance for classical one-way ANOVA
Classical ANOVA assumes a common population variance. This can fail when a treatment changes both the mean and spread, or when groups serve different traffic mixes. Unequal group sizes make the problem more consequential.
SciPy's 0 supports Welch ANOVA when equal_var=False. Welch's method relaxes equal population variances and adjusts the degrees of freedom.
Correct outcome model
ANOVA targets a continuous mean. Conversion is binary; event counts are discrete; time-to-churn can be censored. Large-sample mean inference can sometimes work, but logistic, Poisson or negative-binomial, survival, or other generalized models may better represent the outcome and produce interpretable effects.
One-way, two-way, and repeated-measures ANOVA
“ANOVA” names a family rather than one calculation.
One-way ANOVA
One categorical factor with multiple levels, such as four assigned onboarding variants. This is the usual A/B/n example.
Two-way or factorial ANOVA
Two controlled factors, such as headline and layout. The model estimates each main effect plus their interaction. The interaction asks whether one factor's effect changes with the other. This is central to a properly designed multivariate test.
Repeated-measures ANOVA
The same units are observed under multiple conditions or times. Dependence is part of the design and must be modeled. A basic independent one-way ANOVA is invalid for repeated measurements.
ANCOVA
Analysis of covariance adds continuous covariates to the group comparison. In randomized experiments, pre-experiment covariates can improve precision when they are chosen and measured without post-treatment contamination. GrowthBook's guide to variance reduction explains the same motivation in online experimentation.
Run one-way ANOVA in Python
At the action boundary, keep one numeric observation per independent analysis unit in each group. In SciPy:
Before running it, confirm that rows match the randomization unit and missing values have a documented policy. Afterward, inspect group summaries and residual behavior. The p-value alone cannot reveal a broken exposure join or a few enormous outliers.
In R, aov(outcome ~ variant, data = experiment) fits the classical model. R documents 1 as a linear-model interface, which helps explain why ANOVA, regression, and contrasts are closely connected.
Interpret the ANOVA table
A standard output contains:
- degrees of freedom
- sum of squares
- mean square
- F statistic
- p-value
Suppose the output reports F(3, 4016) = 6.8, p < 0.001. Under the model, the observed ratio of between-variant to within-variant variation is unlikely if all four population means are equal. It does not mean every treatment beats control or that any effect is commercially important.
Add the quantities the product decision needs:
- each mean and sample size
- differences from control in original units
- simultaneous or comparison-specific intervals
- an effect-size measure when useful
- guardrail and data-quality results
- the follow-up comparison method
Avoid ranking noisy means without uncertainty. The highest observed variant has benefited from both its true effect and sampling variation, especially when many variants were screened.
Common ANOVA mistakes
Treating events as independent users
Repeated events make the nominal sample size huge and uncertainty too narrow. Preserve the assignment unit.
Using ANOVA for every metric shape
The word “variant” does not imply ANOVA. Match the outcome distribution and estimand to a model.
Checking assumptions after selecting a winner
Write the model, outlier policy, transformation, and variance choice before the ranking is visible. Result-driven switching creates hidden researcher degrees of freedom.
Treating a significant F-test as a winner declaration
Follow with the planned contrasts. The omnibus test only rejects equality of all means.
Ignoring practical significance
A very large experiment can detect a tiny difference. Compare intervals with a minimum practical effect and account for implementation cost and guardrails.
Use ANOVA as part of an experiment plan
Before launch, specify the factor and levels, independent unit, primary continuous outcome, minimum effect, sample-size plan, variance assumption, global or contrast hypothesis, comparison family, and stopping rule.
Then verify assignment and exposure before interpreting the model. A sample ratio mismatch can signal that observed group counts no longer reflect the planned randomization. No F-test can repair biased exposure data.
ANOVA is valuable because it turns a field of variant means into a structured model of signal and noise. The broader z-test, t-test, chi-square, and ANOVA guide shows when the outcome and hypothesis call for another member of that family. Use the omnibus test for the global question, planned contrasts for the decision, and effect estimates for practical judgment. That sequence makes a multiple-variant test easier to defend than a dashboard full of uncoordinated p-values.
Compare variants with discipline
Run controlled experiments, connect trusted metrics, and review treatment effects and uncertainty in one shared workflow.
Start With GrowthBookReady to ship faster?
No credit card required. Start with feature flags, experimentation, and product analytics—free.
