True Positive: Definition and Examples in Testing
Most teams tracking model performance or running A/B experiments focus on whether their system is catching real positives — but that's only one piece of a four-part picture.
A true positive only means something useful when you understand it alongside false positives, false negatives, and true negatives. Miss that context, and you end up optimizing the wrong thing, sometimes badly.
This article is for engineers, product managers, and data teams who work with classifiers, ML models, or experimentation platforms and want a clear, practical grip on how true positives fit into real evaluation work. Here's what you'll learn:
- What a true positive is and how it fits into the four-outcome binary classification framework
- How to read a confusion matrix and why accuracy alone will mislead you on imbalanced data
- How to calculate the true positive rate (TPR), what it actually measures, and where it goes by other names like sensitivity and recall
- Why maximizing true positives isn't always the right goal — and how to think about the sensitivity-specificity trade-off
- How true positives work specifically in A/B testing, including the common practices that inflate false discoveries and suppress real ones
Each section builds on the last. Start with the definition, work through the math, and finish with the practical implications for experimentation — including how GrowthBook handles peeking, multiple testing, and statistical power in ways that directly affect whether your test results are real.
A true positive has a two-part definition — and getting either part wrong breaks your analysis
Precision matters when you're evaluating whether a test, model, or experiment is actually working. The term "true positive" gets used loosely in practice — often as a shorthand for "a correct result" — but that framing is incomplete in a way that causes real problems when you're trying to reason about model performance or test quality.
A true positive has a specific, two-part definition, and understanding it exactly is the foundation for everything else in this space.
Why "a correct result" is not specific enough
A true positive occurs when a test predicts a positive outcome and the underlying condition is genuinely present. Both conditions must hold simultaneously: the test says positive, and the ground truth is positive.
As Wikipedia frames it in the context of diagnostic testing, sensitivity (the true positive rate) is "the probability of a positive test result, conditioned on the individual truly being positive". That word conditioned is doing important work — it means the ground truth is already established as positive, and the question is whether the test correctly reflects that reality.
This distinction matters because "a correct result" is not specific enough. A test that correctly identifies a negative case — someone who doesn't have a disease, a transaction that isn't fraudulent — is also producing a correct result. That outcome is a true negative, which is a different category entirely. A true positive is specifically a correct positive identification.
The four-outcome binary classification framework
No test outcome exists in isolation. A true positive only has meaning when you understand it as one of four possible outcomes in any binary classification system. When a test is applied to a case where the condition either exists or doesn't, and the test either flags it or doesn't, you get exactly four combinations:
The four cells of this table represent the complete universe of outcomes for any binary classifier. A test that labels everything as positive would accumulate a high count of true positives, but it would also generate false positives on every negative case. That's not a useful test. The true positive count only becomes meaningful when you can see it in context alongside the other three outcomes.
The definition holds across domains; what changes is the cost of getting it wrong
The definition is domain-agnostic. The underlying logic — test says positive, condition is real — applies identically whether you're working in healthcare, financial services, or software systems.
In medical screening, a cancer test that correctly flags a patient who actually has cancer is producing a true positive. The clinical stakes here are high: a test with a high true positive rate catches real cases early, enabling timely treatment. Missing those cases — producing false negatives instead — carries serious consequences for patient outcomes.
In fraud detection, a true positive occurs when a model flags a transaction as fraudulent and that transaction is, in fact, fraudulent. Compliance teams rely on these correct identifications to prevent financial losses and protect customer accounts. The system is doing exactly what it's supposed to do.
Software and model evaluation follow the same logic: a classification model produces a true positive when it correctly identifies a positive instance — a defective component on a production line, a spam email in a filtering system, a bug flagged by a static analysis tool that is a genuine defect. The condition exists, and the model found it.
Across all three domains, the definition holds. What changes is the cost structure around errors — how damaging it is to miss a real positive versus how damaging it is to flag a false one. But that's a question of trade-offs, not of the definition itself.
True positives only have meaning inside the full four-outcome confusion matrix
A true positive doesn't exist in isolation. Its meaning only becomes clear when you place it alongside the three other outcomes that any classifier or test can produce: true negatives, false positives, and false negatives.
The confusion matrix is the minimum unit of analysis for evaluating test quality — and if you're only tracking how often your model catches real positives, you're missing most of the picture.
How the four cells map every possible prediction against reality
A confusion matrix is a 2×2 table that maps every prediction a model makes against what actually occurred. One axis represents the actual class; the other represents the predicted class. The diagonal — top-left to bottom-right — captures correct predictions. The off-diagonal cells capture errors.
A concrete example makes this tangible. Consider a cancer screening classifier evaluated on 12 individuals — 8 who actually have cancer and 4 who don't. If the model makes 9 correct predictions but misclassifies 3 — calling 2 cancer patients cancer-free and flagging 1 healthy person as having cancer — you have 2 false negatives and 1 false positive.
The model looks reasonably accurate at first glance, but the error breakdown reveals two very different failure modes with very different consequences.
One note on convention: some sources place actual classes on rows and predicted classes on columns; others reverse this. Both are valid. The table above follows the row-as-actual convention used by Wikipedia and most ML tooling, but you'll encounter both in practice.
False positives (Type I errors): the cost of crying wolf
A false positive occurs when a model predicts positive but the actual outcome is negative. In machine learning, this is formally called a Type I error. In spam detection, it's flagging a legitimate email as spam. In fraud detection, it's blocking a valid transaction.
In A/B testing, the framing is slightly different but structurally identical. GrowthBook's documentation defines a Type I error as a situation where "your metrics all appear to be winners, but in reality the experiment has no effect." The test fires a signal; the signal is wrong. Acting on that signal — shipping a feature, changing a product flow — means making a real decision based on noise.
False positives carry costs that depend entirely on context. In some domains, they're annoying but recoverable. In others, they trigger irreversible actions: unnecessary medical treatment, a blocked customer, a shipped feature that degrades the product.
False negatives (Type II errors): the cost of missing what's real
A false negative is the mirror image: the model predicts negative, but the actual outcome was positive. This is a Type II error. In medical screening, it's a missed diagnosis. In fraud detection, it's a fraudulent transaction that goes through undetected.
In A/B testing, a Type II error occurs when "the data aren't showing a clear winner or loser when actually a variation is much better or worse." The consequence is that teams either collect more data — extending an experiment that already has an answer — or make a blind decision without the signal they needed. Real improvements go undetected; real regressions go unaddressed.
The cost asymmetry between Type I and Type II errors is domain-specific and worth making explicit in any system you're evaluating. Missing a cancer diagnosis is not the same kind of failure as flagging a healthy patient. Missing a winning A/B test variant is not the same kind of failure as shipping a losing one.
Why no single cell in the confusion matrix tells you whether your model is good
No single cell in the confusion matrix tells you whether your model is good. The matrix has to be read as a whole, and the derived metrics — accuracy, precision, recall — each weight the four cells differently.
Accuracy is the most intuitive: (TP + TN) / all predictions. But it's also the most misleading on imbalanced datasets. A model that predicts "negative" for every single input on a dataset where positives appear only 1% of the time achieves 99% accuracy while being completely useless. It has a perfect true negative rate and a true positive rate of zero. As one practitioner put it, such a classifier "could be losslessly replaced by a rock."
Precision (TP / TP + FP) tells you how often a positive prediction is correct — critical when false positives are expensive. Recall, also called sensitivity (TP / TP + FN), tells you how much of the actual positive class you're capturing — critical when false negatives are expensive. These metrics pull in opposite directions, and optimizing one typically costs you the other.
All of these metrics are also threshold-dependent. Change the classification threshold and every cell in the matrix shifts. The confusion matrix you compute at one threshold is not the confusion matrix you'd compute at another, which is why evaluating a classifier at a single operating point is rarely sufficient for understanding its real-world behavior.
True positive rate: the metric that tells you how much of reality your system is actually catching
Understanding what a true positive is gets you halfway there. The more useful question for practitioners is: how do you measure how many real positives your system is actually catching? That's what True Positive Rate (TPR) answers — and it's a metric precise enough to calculate, compare, and optimize.
The TPR formula and what it actually measures
TPR is calculated as:
TPR = TP / (TP + FN)
The numerator is the count of true positives — cases where your model or test correctly identified a real positive. The denominator is the total universe of actual positive cases: every real positive that existed, whether your system caught it (TP) or missed it (FN). The result is a value between 0 and 1, where 1.0 means every real positive was detected and nothing slipped through.
You'll encounter this metric under different names depending on the field. In medicine and statistics, it's called sensitivity. In machine learning, it's called recall. Wikipedia's formal definition captures it cleanly: sensitivity is "the probability of a positive test result, conditioned on the individual truly being positive." All three terms refer to the same calculation.
One practical note: a high TPR tells you the test is sensitive — it catches most real positives. But it doesn't tell you that any specific positive result is correct. Here's why that matters: imagine a disease that affects 1 in 10,000 people. Even a test with 99% TPR will produce more false positives than true positives in that population, simply because there are so few actual cases to find.
The probability that a positive result is real depends on how common the condition is — a separate calculation called positive predictive value (PPV). Conflating TPR with PPV is a common mistake with real consequences.
What high and low TPR signals about your system
A high TPR means your model is catching most of the real positive cases and producing few false negatives. A low TPR means real positives are routinely slipping through — your system is missing what it's supposed to find.
The stakes of a low TPR vary dramatically by domain. In healthcare, a missed diagnosis is a false negative with potentially irreversible consequences. In fraud detection, a missed fraudulent transaction carries direct financial and reputational costs. These are the domains where TPR is typically the primary metric to optimize, because the cost of a false negative far outweighs the cost of a false positive.
In A/B testing, statistical power functions as the domain-specific analog to TPR — it represents the probability that a real effect will be detected, which is precisely what TPR measures in classification contexts.
That said, a high TPR is not universally the right target. Pushing TPR toward 1.0 typically requires lowering the classification threshold, which increases false positives. Whether that trade-off is acceptable depends entirely on the cost structure of your specific problem — a point the next section addresses directly.
TPR and the ROC curve
The ROC (Receiver Operating Characteristic) curve is the standard tool for visualizing how TPR behaves across different classification thresholds. It plots TPR on the y-axis against the false positive rate (FPR) on the x-axis. Each point on the curve represents a different threshold setting.
As you lower the classification threshold, more cases get flagged as positive. TPR rises — you catch more real positives — but FPR rises too, because more negatives get incorrectly flagged. Raise the threshold and the reverse happens: fewer false positives, but more real positives missed. The ROC curve makes this trade-off visible across the full range of possible thresholds rather than at a single fixed point.
A classifier with a curve that hugs the top-left corner of the plot is performing well: it achieves high TPR at low FPR. A curve that runs diagonally from bottom-left to top-right is no better than random guessing. The shape of the curve tells you how much flexibility you have in setting a threshold before sensitivity degrades meaningfully.
The sensitivity-specificity trade-off: why maximizing true positives isn't always the goal
There's an intuitive appeal to the idea that a good classifier should catch as many true positives as possible. In practice, that instinct leads teams astray. Maximizing sensitivity — your true positive rate — always comes at a cost, and understanding that cost is what separates a well-calibrated system from one that creates as many problems as it solves.
The fundamental trade-off: why you can't maximize both
Sensitivity and specificity move in opposite directions. As Wikipedia's treatment of the topic states directly: "higher sensitivities will mean lower specificities and vice versa".
The mechanism is straightforward. When you lower a classification threshold to catch more true positives, you inevitably sweep in more negatives along with them. More true positives means more false positives — which means lower specificity. There's no configuration that escapes this relationship. The question is never whether to accept this trade-off, but where to set it.
When high sensitivity is the right call
The case for prioritizing sensitivity is strongest when a missed positive carries severe consequences. Wikipedia's criterion is precise: prioritize sensitivity "when the consequence of failing to treat the condition is serious and/or the treatment is very effective and has minimal side effects."
Cancer screening is the canonical example. A false negative — a test that misses a malignancy — means a patient goes untreated while the disease progresses. The downstream cost of that miss dwarfs the cost of a false positive, which typically means an additional confirmatory test. The asymmetry is clear: one error is inconvenient, the other can be fatal.
Fraud detection follows similar logic. Missing a fraudulent transaction carries real financial and reputational consequences for a business, while a false positive — a legitimate transaction flagged incorrectly — creates customer friction but is recoverable. In domains like these, false negatives are the more expensive error, and systems should be tuned accordingly.
When specificity deserves priority
The calculus flips when false positives carry their own serious costs. Wikipedia identifies the relevant condition: specificity matters most "when people who are identified as having a condition may be subjected to more testing, expense, stigma, anxiety, etc."
Confirmatory diagnostic testing is the clearest case. A false positive diagnosis can trigger unnecessary treatment, psychological harm, or lasting stigma — costs that are real and sometimes irreversible. The initial screening test can afford to be sensitive; the confirmatory test needs to be specific.
The same principle applies in A/B testing. A false positive in an experiment means declaring a winning variant when no real effect exists, then shipping a change that doesn't actually improve the product. The cost here isn't just a wasted engineering cycle — it's the compounding effect of making product decisions on noise.
Threshold selection is a business decision, not a statistical one
The classification threshold determines where on the sensitivity-specificity trade-off curve your system operates. There is no universally correct setting. The right threshold depends entirely on the relative cost of each error type in your specific context, and that cost calculation belongs to the domain, not the algorithm.
This becomes especially concrete in A/B testing when multiple metrics are evaluated simultaneously. GrowthBook's documentation on experimentation pitfalls notes that testing the same hypothesis across 20 metrics at a 5% significance level produces roughly a 64% probability of finding at least one statistically significant result by chance alone.
That's what happens when sensitivity is implicitly maximized without any mechanism to control specificity. Correction methods like Benjamini-Hochberg and Bonferroni deliberately sacrifice some sensitivity — accepting that a few real effects may go undetected — in order to reduce the rate of false discoveries.
That trade-off isn't a flaw in the methodology. It's the methodology working as intended, calibrated to the cost structure of the problem. The teams that get this right aren't the ones chasing the highest possible true positive rate — they're the ones who have thought clearly about what a false positive actually costs them.
Related reading: Statistical Validity: What It Means in Research
True positives in A/B testing: correctly detecting real effects and avoiding false discoveries
In A/B testing, a true positive has a specific meaning: your experiment correctly identifies a variation that genuinely improves a metric, and you ship it. The test said it won. It actually won. That alignment between the statistical signal and the underlying reality is exactly what experimentation is designed to produce — and it's rarer than most teams assume.
According to GrowthBook's documentation, only about one-third of experiments successfully improve the metrics they're designed to move. Another third have no effect, and the remaining third actually hurt performance. That distribution means the majority of experiments you run will not produce true positives. In that environment, correctly identifying the real winners matters enormously — and so does avoiding the false ones.
Where true positives sit in the A/B testing decision space
The full A/B testing decision space maps directly onto the confusion matrix framework. When you decide to ship a variation and it actually won, that's a correct inference — a true positive. Every other combination where "ship" and "actual outcome" don't align is a Type I error. Shipping a variation that had no real effect, or shutting down one that actually would have won, are both failures of the classification system.
The default significance threshold in most experimentation platforms is 95% confidence. That means even in a perfectly designed experiment with no methodological problems, you'll incorrectly flag a result as significant 5% of the time. As GrowthBook's documentation puts it plainly: "5% of the time, it isn't actually better." That baseline false positive risk is unavoidable — but several common practices make it dramatically worse.
The peeking problem and multiple testing
The peeking problem occurs when teams monitor experiment results continuously and stop a test the moment results look promising. This practice inflates the false positive rate substantially. Because statistical significance fluctuates throughout a test's runtime, repeatedly checking results and stopping early when p < 0.05 appears means you're much more likely to catch a random fluctuation than a real effect.
Sequential testing addresses this by allowing continuous monitoring and early stopping without inflating false positive rates — the statistical method adjusts for the repeated looks so the error rate stays controlled.
Multiple testing compounds the problem in a different way. When you test many metrics simultaneously, the probability that at least one will appear significant by chance increases with each metric added. Google's famous "41 shades of blue" experiment illustrates how cascade testing — running A vs. B, then B vs. C, and so on — can produce mathematically invalid conclusions when not handled correctly.
GrowthBook's documentation notes that adding many metrics to any test increases false positive risk, even in a correctly configured system. The solution is pre-registering your primary metric before the experiment runs, not selecting the most flattering result afterward.
P-hacking is the logical extension of this: continuing to slice data, add metrics, or adjust segments until statistical significance appears, then reporting only the significant finding. It's not always intentional, but the effect is the same — the result looks like a true positive and isn't.
Underpowered tests don't inflate false positives — they suppress true positives
Underpowered tests don't inflate false positives — they suppress true positives. If your experiment doesn't have enough users to detect the effect size you're looking for, real improvements will fail to reach significance. GrowthBook's documentation defines this through the concept of Minimal Detectable Effect (MDE): if the actual effect is smaller than the MDE given your sample size, the test cannot detect it even when it genuinely exists.
The result is a false negative — a true positive that never gets identified.
Variance reduction techniques directly address this. CUPED reduces variance by accounting for pre-experiment user behavior, which means experiments can reach statistical significance with fewer users. That speed matters because it reduces the temptation to peek early.
A/A testing eliminates infrastructure failures before they corrupt your true positive rate
Before you can trust that your A/B test results represent true positives, you need to confirm that your experimentation infrastructure isn't generating spurious signals on its own. A/A testing — running an experiment where both variations are identical — is the standard method for this validation. If an A/A test returns statistically significant results across multiple metrics, the system itself may be broken: traffic isn't splitting correctly, metrics are misconfigured, or the SDK integration has an error.
GrowthBook recommends running A/A tests after setting up a new SDK connection and after any significant changes to your integration, data warehouse, or tracking libraries. One important calibration note: even a correctly configured A/A test may show one or two marginally significant metrics due to random chance at the 5% threshold. That's expected. What's alarming is three or four metrics all showing significance above 99% — that's a signal the system is broken, not just unlucky.
A/A testing doesn't guarantee that your subsequent A/B results are true positives. But it eliminates a category of infrastructure-level failures that would make true positive detection impossible from the start.
Putting it together: calibrating your system to the actual cost of each error
The core insight running through this entire article is simple but easy to miss: a true positive only tells you something useful when you understand it in context. The count of things your system correctly identified means nothing without knowing how many real positives it missed, how many false alarms it generated, and what each of those errors actually costs you.
That four-part frame — not just the single number — is what separates a well-calibrated system from one that looks good on the surface and fails in practice.
Match your sensitivity-specificity balance to the cost of each error type
Before you tune a threshold or evaluate a metric, make the cost asymmetry explicit. In fraud detection, a missed fraudulent transaction is typically more expensive than a blocked legitimate one. In A/B testing, shipping a feature that had no real effect compounds quietly over time in ways a missed winner usually doesn't.
Neither of those cost structures is universal — they're specific to your domain, your users, and your business. The threshold decision follows from that analysis, not the other way around.
The confusion matrix and TPR reveal what accuracy alone conceals
If you're currently evaluating model performance with accuracy alone, the confusion matrix is the right place to start. Pull the full 2×2 breakdown, compute TPR and precision separately, and look at where your errors are concentrated. A model with 95% accuracy on an imbalanced dataset may have a TPR near zero — catching nothing that actually matters. The matrix makes that visible in a way a single aggregate metric never will.
Protect true positive rates in experimentation with rigorous statistical practices
In A/B testing, the practices that inflate false positives — peeking, running too many metrics, post-hoc segmentation — are the same ones that make your true positives harder to trust. Pre-register your primary metric, run an A/A test to validate your infrastructure, and size your experiments to detect the effect you actually care about. Sequential testing and variance reduction techniques are specifically designed to address these failure modes without forcing you to choose between speed and statistical integrity.
What to do next:
- If you're evaluating a classifier: pull the full confusion matrix, compute TPR and precision separately, and compare performance at multiple thresholds before picking an operating point.
- If you're running A/B tests: pre-register your primary metric, run an A/A test to validate your infrastructure, and calculate your MDE before launching.
- If you're setting a classification threshold: write down the cost of a false positive and the cost of a false negative in your specific context before touching the threshold value.
One tension worth keeping in mind: when you add controls to reduce false positives — stricter significance thresholds, multiple testing corrections, higher sample size requirements — you will sometimes fail to detect real effects. That's not a flaw in the methodology. It's the trade-off working as intended. The goal isn't to catch every true positive at any cost. It's to build a system where the errors you make are the ones you've consciously decided are cheaper than the alternative.
Related reading
Related Articles
In healthcare, “Can we randomize it?” is the wrong first question. Start with “Could either experience change care, rights, privacy, or access?”
A/B testing can improve digital intake, appointment access, patient education, clinician workflows, and administrative operations. It can also create unacceptable risk when teams treat a clinical or consent decision like an ordinary conversion funnel.
The difference is not the label on the method. A/B tests are randomized experiments. What matters is the treatment, purpose, affected population, data flow, and oversight required in the organization and jurisdiction. This guide provides a practical product framework, not a substitute for legal, clinical, privacy, security, or institutional review.
Draw the boundary before designing variants
Create an intake step that classifies the proposed change before anyone builds a treatment. At minimum, ask:
- Can the change alter diagnosis, treatment, triage, dosage, or clinical recommendations?
- Can it delay or discourage access to care, accommodations, or urgent help?
- Does it change informed consent, privacy choice, required disclosure, or patient cost?
- Does it use protected or sensitive health information for assignment or measurement?
- Does it include children, people in crisis, or another population requiring added protection?
- Is the purpose internal quality improvement, or is it designed to contribute to generalizable knowledge?
- Could the software function fall within medical-device or clinical decision-support oversight?
The HHS quality-improvement guidance says many activities limited to improving patient care and collecting operational data are not research under the cited human-subjects regulations. It also states that some quality-improvement activities can have a research purpose, in which case human-subject protections may apply. A product team should not make that determination informally; route it to the organization’s authorized office.
Likewise, software that influences clinical decisions is not automatically an ordinary product surface. The FDA’s January 2026 clinical decision-support guidance explains that some software functions are excluded from the device definition while other patient- or caregiver-facing functions can remain subject to digital-health policy. Clinical and regulatory owners need to classify the function before experimentation.
Start with lower-risk operational questions
The safest early program tests reversible changes where both variants meet the same clinical, accessibility, privacy, and disclosure requirements.
Appointment reminder timing
Compare 2 approved reminder schedules or message structures to reduce missed appointments. Keep required details, opt-out behavior, language support, and urgent-contact instructions constant.
Use completed appointments or timely rescheduling as the primary outcome. Track cancellations, patient contacts, message delivery, opt-outs, wrong-recipient risk, and differences across language, age, disability, or access groups. A higher click rate is not enough if no-show rates or trust worsen.
Patient portal navigation
Test whether a clearer information architecture helps people complete a high-value administrative task, such as finding results, updating insurance, or sending a non-urgent message. Preserve emergency guidance and clinical escalation paths in both variants.
Measure successful task completion and time to completion. Guard against repeated navigation, abandonment, accessibility failures, mistaken message routing, and increased call-center burden. Use usability testing before the A/B test to catch failures randomization should never expose.
Administrative form sequence
Compare a long form with a staged flow, or test the order of non-clinical fields. Do not omit information needed for safe care, billing transparency, consent, or legal compliance.
Measure accurate completion, not just submission. Track validation errors, correction rates, staff rework, abandonment, and time to appointment. If the treatment collects sensitive data, confirm necessity and access controls before launch.
Educational content layout
Test 2 ways to present the same clinician-approved information: summary-first versus stepwise, text plus illustration versus text alone, or a clear action checklist versus a dense paragraph. Keep the medical meaning, risks, contraindications, and escalation advice equivalent.
Use a comprehension or appropriate next-action metric when feasible. Page time and clicks can be misleading. Accessibility, language quality, and comprehension across health-literacy levels belong in the guardrail plan.
Review the design before launch
Use a trustworthy experiment-design session to pressure-test metrics, safety checks, and decision rules before exposing patients or clinicians.
Watch the Experiment Design Session
Use stronger controls for care-adjacent products
Some product changes are not clinical interventions but can still influence care. They need clinical ownership, narrower eligibility, conservative ramps, and explicit stopping criteria.
Clinician workflow support
A test might compare how a work queue prioritizes administrative follow-up, how a note template reduces documentation work, or how a non-diagnostic alert is presented. The treatment should not silently alter the clinical standard of care.
Randomize at the unit that prevents contamination. Individual clinician assignment may fail when teams share queues and handoffs; clinic- or unit-level clusters may better match the workflow. Measure task completion and time saved, with guardrails for missed work, overrides, escalations, documentation quality, and staff workload.
Preventive-care outreach
Compare approved outreach content or channels for people already eligible under the same clinical rule. Do not experiment with whether one group receives necessary care or required notice.
Use completed appropriate follow-up as the primary outcome. Track opt-outs, unreachable patients, scheduling capacity, disparities, complaints, and downstream cancellations. If the treatment drives demand beyond operational capacity, a messaging lift can make access worse.
Digital adherence support
Test the presentation or timing of an approved reminder, checklist, or educational cue. Avoid treatment changes that could be interpreted as personalized medical advice without the corresponding validation and oversight.
Measure the intended behavior with caution. Self-reported completion or app engagement is not a clinical outcome. Include adverse-event reporting, escalation pathways, disengagement, and privacy events where relevant.
Feature rollout in health software
Use feature flags to separate deployment from release, start with internal or trained cohorts, and expand only when technical and clinical guardrails remain healthy. GrowthBook’s feature flag platform supports targeted rollouts and kill switches, while the experiment layer measures impact.
The rollback plan must describe more than turning off a flag. Determine whether the old experience remains clinically and operationally safe, how queued work is reconciled, what happens to partial workflows, and who is authorized to stop exposure.
Protect data by design
Do not send a broad event stream to an experimentation vendor and decide later which fields were unnecessary. Inventory the data before implementation:
| Data question | Required decision |
|---|---|
| Assignment | What is the least identifiable stable unit that works? |
| Eligibility | Which sensitive attributes are truly needed? |
| Exposure | What event proves the treatment was delivered? |
| Outcomes | Can metrics be computed inside the governed data environment? |
| Access | Which roles can view assignments, segments, and results? |
| Retention | When are raw records, logs, and exports removed? |
The HHS minimum-necessary guidance describes limiting uses, disclosures, and requests for protected health information to what is needed for the intended purpose, with policies based on roles and recurring versus non-routine access. Apply that principle to experiment attributes, debugging logs, dashboards, and downloaded readouts.
Pseudonymous identifiers reduce exposure but do not automatically make a dataset non-sensitive or outside applicable rules. Review linkability, small cohorts, free-text fields, URLs, device metadata, and combinations that can reveal a condition. Never put clinical details or identifiers in feature names, variation labels, or URLs.
A warehouse-native experimentation approach can query approved metrics where the organization already governs them. Architecture does not create compliance on its own; teams still need contracts, access control, auditability, retention rules, security review, and configuration that matches the approved data flow.
Keep unsafe questions out of product experimentation
An experimentation policy should name prohibited or separately governed categories. Product teams should not discover the boundary only after a proposal reaches launch review.
Do not use an ordinary product A/B test to withhold a clinically indicated service, emergency direction, safety warning, accessibility accommodation, required disclosure, or legally protected choice. Do not reduce the visibility of risks to improve completion. Do not randomize a diagnostic or treatment recommendation without the clinical, regulatory, and research framework appropriate to that intervention.
Avoid treatments that exploit fear, urgency, shame, or uncertainty about health. A message can increase appointment conversion while undermining informed choice. Likewise, do not test whether patients tolerate a harder cancellation, more confusing privacy control, or hidden cost. Both variants must meet the organization’s baseline standard for respectful and comprehensible communication.
Clinical AI and decision-support changes need an evaluation program beyond a click-based A/B test. Validate the model offline, examine performance and failure modes across relevant populations, review human factors, and stage deployment with clinical monitoring. An online comparison may contribute evidence only after both treatments meet the safety threshold for exposure.
When an activity may be human-subjects research, follow the institution’s process before enrolling or exposing anyone. HHS research-oversight training states that covered non-exempt human-subjects research requires the applicable review and that informed consent requirements apply unless the IRB authorizes otherwise. The product team should preserve the determination, protocol version, approved treatment, and reporting obligations with the experiment record.
Finally, do not interpret lack of detected harm as proof of safety. Rare adverse events, small vulnerable groups, and outcomes that occur after the experiment window may be underpowered. Use prior evidence, incident monitoring, qualitative reports, and post-rollout surveillance alongside the randomized estimate.
Define patient-centered metrics and guardrails
Healthcare teams need more than a conversion scorecard. Build a measurement hierarchy:
- Primary outcome: the operational or patient-facing result that answers the decision.
- Process diagnostics: steps that explain why the treatment worked or failed.
- Safety guardrails: outcomes that trigger a stop or clinical review.
- Equity checks: predeclared groups where access or benefit could differ.
- Operational guardrails: staffing, wait time, rework, cost, and downstream capacity.
Define the practical threshold before launch. A statistically detectable change may be too small to justify implementation, and a neutral aggregate can hide meaningful harm in a protected or vulnerable group. At the same time, slicing results across many small subgroups increases false-positive risk and can expose sensitive attributes. Predeclare the equity questions that matter and use appropriate privacy and multiple-testing controls.
GrowthBook supports reusable fact tables and metrics so teams can keep definitions reviewable. Use a power analysis for the primary outcome and critical guardrails. If the required sample or duration is unrealistic, do not weaken the standard; use usability research, simulation, staged quality improvement, or a larger treatment contrast.
Create a healthcare experiment review packet
Before launch, the owner should provide one reviewable packet:
- purpose, hypothesis, and operational decision
- classification and required oversight determination
- affected population and exclusion criteria
- clinical, privacy, security, accessibility, and compliance approvals
- treatment screenshots or workflow diagrams
- assignment, exposure, and data-flow design
- primary outcome, diagnostics, guardrails, and equity checks
- sample plan and stopping rule
- rollout stages, monitoring owner, and rollback procedure
- patient or clinician communication plan, if applicable
- documentation and retention plan
Use an approval matrix that names accountable people. Product approval does not replace clinical approval; a privacy review does not settle human-subjects research status; and an IRB determination does not automatically approve the production security architecture.
The WHO clinical-trial best-practices guidance emphasizes ethical standards, regulatory considerations, patient-centered research, transparency, and stakeholder collaboration. Not every healthcare product experiment is a clinical trial, but high-risk work should inherit the same respect for people and evidence.
Build trust into the experimentation program
Start with reversible operational improvements where both experiences are already acceptable. Prove that the team can classify risk, minimize data, validate assignment, monitor safety, and document decisions before expanding scope.
Publish internal rules for what teams may test, what requires added review, and what is out of bounds. Maintain an experiment registry and audit trail. Record neutral and negative results so a new team does not repeat the same risky idea.
GrowthBook can support the controlled delivery and analysis layer through experimentation, feature flags, permissions, and warehouse-defined metrics. The organization remains responsible for the clinical, ethical, legal, privacy, and operational framework around every test.
In healthcare, speed is valuable only when the learning process protects the people whose behavior creates the data.
Build a governed test workflow
Connect controlled releases to reviewable metrics and decision rules while keeping healthcare data in your approved architecture.
Get Started With GrowthBook
The right statistical test is determined by the question and data-generating process, not by which function is easiest to run. Start with the outcome, groups, and dependence structure; the test name comes later.
Z-tests, t-tests, chi-square tests, and analysis of variance (ANOVA) all compare observed data with a null model. They differ in the kind of outcome they model, the uncertainty they estimate, and the number or structure of groups they can compare.
For a simple product experiment, a useful first pass is:
- continuous outcome, two independent groups: usually a Welch two-sample t-test
- binary proportion, two large independent groups: a two-proportion z-test is common
- categorical counts across groups: chi-square test, if expected counts are adequate
- continuous outcome across three or more groups: one-way ANOVA or Welch ANOVA
Those rules are a starting point. Paired observations, clusters, ratios, repeated measures, heavy tails, covariate adjustment, or sequential monitoring require a model that reflects the design.
Choose from the outcome and hypothesis
Write the estimand before choosing a test. An estimand is the quantity the experiment is trying to estimate: a difference in mean revenue, a difference in conversion probability, or an association between two categorical variables.
| Question | Outcome | Common test |
|---|---|---|
| Did average order value change between A and B? | Continuous | Welch two-sample t-test |
| Did signup probability change between A and B? | Binary | Two-proportion z-test |
| Is plan choice associated with variant? | Categorical, 3+ levels | Chi-square test of independence |
| Do mean task times differ across four variants? | Continuous | One-way ANOVA |
| Did the same users' scores change before and after? | Paired continuous | Paired t-test |
The number of groups alone is insufficient. Conversion in four variants is still categorical data; a chi-square or binomial model may fit. Revenue in two groups is continuous; a t-test or regression is more natural.
The University of Michigan's statistical-test guide uses the same sequence: identify variable types and the relationship being tested before selecting a method.
When to use a z-test
A z-test compares a standardized estimate with the standard normal distribution. The classical one-sample z-test for a mean assumes the population standard deviation is known. That condition is unusual in product analytics, where variability is estimated from the current sample.
Z-tests remain common for proportions. In a two-arm conversion experiment, the estimate is:
Under the null of equal proportions and with adequate counts, the standardized difference is approximately normal. This yields a two-proportion z-test.
Use it when:
- the outcome is a binary count summarized as successes and failures
- assignment groups are independent
- sample sizes make the normal approximation credible
- the hypothesis and one- or two-sided direction were set before analysis
Do not rely on a universal “n greater than 30” rule. For rare events, 30 observations can produce almost no successes; for balanced common events, approximation quality can be good. Inspect expected successes and failures and use an exact or model-based method when counts are sparse.
In high-volume online experiments, a normal approximation is also used for many sample means through the central limit theorem. The important question is whether the estimator's sampling distribution and variance calculation are valid for the metric, not whether the raw user values look perfectly normal.
When to use a t-test
A t-test is designed for inference about means when the variance is estimated from sample data. That extra variance uncertainty produces a t distribution with heavier tails than the standard normal, especially at small sample sizes.
For two independent groups, default to Welch's t-test unless equal variance is justified. Welch's version does not assume the two population variances are equal and handles unequal group sizes. NIST's two-sample t-test reference shows the unequal-variance standard error based on each group's sample variance and size.
Use an independent two-sample t-test when:
- the outcome is numeric and the mean is the target
- the two groups contain different experimental units
- observations are independent within the model
- the mean and standard error behave well enough for the sample size
Use a paired t-test when each value has a meaningful partner: the same user's before-and-after score, or deliberately matched units. The analysis reduces each pair to a difference and tests the mean of those differences. Treating paired data as independent discards information and computes the wrong standard error.
The t-test can be sensitive to extreme values because the sample mean and variance are sensitive to them. Product metrics such as revenue or session duration are often skewed. At scale, the mean may still have a usable sampling distribution, but inspect outliers, data quality, and the estimand. Robust inference, transformations, winsorization policies, or bootstrap methods may be more appropriate when a few observations dominate the result.
Reduce variance before launch
Learn how CUPED and covariate adjustment can sharpen experiment estimates without changing the randomized comparison.
Explore Variance Reduction
When to use a chi-square test
Pearson's chi-square statistic compares observed category counts with counts expected under a null hypothesis. Two common forms are:
- goodness of fit: does one categorical distribution match specified probabilities?
- independence or homogeneity: is a categorical outcome distributed the same way across groups?
Suppose an onboarding experiment records three outcomes: completed, skipped, and abandoned. Cross-tabulate outcome by variant. A chi-square test asks whether the outcome distribution is independent of variant.
The test statistic sums (observed - expected)^2 / expected across cells. NIST's chi-square documentation describes the same comparison of binned frequency distributions.
Use a chi-square test when observations contribute counts to mutually exclusive categories and expected cell counts are large enough for the asymptotic approximation. With sparse cells, combine categories only when substantively justified or use an exact method such as Fisher's exact test for a two-by-two table.
A chi-square result says the distributions differ somewhere. It does not provide the most decision-friendly effect estimate by itself. Report category proportions, absolute differences, uncertainty intervals, and the cells contributing to the pattern.
For a binary two-arm experiment, the Pearson chi-square test and a two-sided two-proportion z-test are closely related: under standard conditions, the chi-square statistic with one degree of freedom equals the squared z statistic. Choose the representation that matches the hypothesis and reporting needs.
When to use ANOVA
ANOVA compares variation between group means with unexplained variation within groups. A one-way ANOVA tests the null that all population means are equal across levels of one factor.
Use it for a continuous outcome across three or more independent groups when the global question is whether any mean differs. Classical ANOVA assumes independent errors, normally distributed residuals within the model, and equal variances. Welch ANOVA relaxes the equal-variance assumption; R's 0 implements that approximation.
ANOVA's F-test is an omnibus test. A significant result means at least one mean differs, but it does not identify which one. Use planned contrasts or multiplicity-aware post-hoc comparisons to answer the product question.
ANOVA is more than a rule for “three or more groups.” Multi-factor ANOVA can estimate main effects and interactions in multivariate or factorial experiments. Repeated-measures or clustered data need corresponding error structures rather than a basic one-way calculation.
Why several t-tests are not a substitute for ANOVA
With four variants there are six pairwise comparisons. Testing each at 0.05 creates multiple opportunities for a false positive. An omnibus ANOVA tests one global null first, and planned follow-ups can use Tukey, Holm, Bonferroni, or another procedure appropriate to the family of claims.
The Bonferroni correction is simple and conservative. The right procedure depends on whether the goal is all pairwise comparisons, treatments versus one control, or a small set of preplanned contrasts. Define that family before looking at the ranking.
ANOVA and regression are also two views of the same linear-model machinery. R's 0 documentation describes aov as a wrapper around linear models for experimental designs. Regression is often more flexible when the analysis includes covariates, interactions, or unbalanced data.
Assumptions that change the choice
Before running any of the four tests, verify:
Independence and assignment unit
If the experiment randomizes accounts but analyzes users as independent observations, standard errors will usually be too small. Analyze at the randomization unit or use cluster-aware inference. If users can appear in both groups, repair the assignment or use a model that represents the dependence.
Paired or repeated observations
The same user measured twice is not two independent users. Use a paired test or repeated-measures model. For experiments with many events per user, aggregate to the user level or use appropriate clustered methods.
Outcome distribution and metric construction
Check missingness, zero inflation, extreme tails, ratio denominators, and censoring. A test can be mathematically correct for the supplied numbers while the metric itself misrepresents the user outcome.
Variance assumptions
Prefer Welch's t-test or Welch ANOVA when group variances may differ. Equal sample sizes do not prove equal variance, and a preliminary variance test can introduce another decision layer.
Sample size and sparse cells
Approximate z and chi-square methods need enough information in the relevant cells. Low-frequency guardrails and small segments may need exact methods or longer collection.
A product experimentation decision tree
Use this sequence before opening a statistics package:
- What unit was randomized: user, account, device, session, or region?
- What is the primary estimand: mean, proportion, category distribution, or model coefficient?
- Are groups independent, paired, repeated, or clustered?
- Are there two groups, several groups, or multiple factors?
- Do expected counts and sample sizes support the approximation?
- Are variances, tails, or outliers likely to break the default model?
- How many confirmatory hypotheses can trigger the decision?
- Was the test direction and stopping rule declared before launch?
Then choose the simplest model that answers the exact question. A two-proportion z-test may be perfect for signup conversion, while a t-test handles mean revenue and a chi-square test handles plan mix in the same experiment. Different metrics can require different tests.
Report effects, not only test names
The test produces a statistic and p-value under a null model. The guide to interpreting a t-test p-value shows why that number needs the effect, interval, and degrees of freedom beside it. The product decision needs more:
- the effect estimate in business units
- a confidence or credible interval
- sample sizes and allocation
- baseline and treatment values
- assumption and data-quality checks
- the planned hypothesis family
- practical thresholds and guardrails
GrowthBook's statistics documentation explains the frequentist and Bayesian engines available for experiment analysis. Whichever framework is used, review effect magnitude and uncertainty together. A small p-value can accompany a trivial lift in a huge sample, while a valuable estimated lift can remain uncertain in a small one.
Choose the test by tracing the data back to the experiment design. For three or more continuous-outcome variants, the deeper ANOVA guide covers the omnibus F-test, planned contrasts, and Welch alternative. When the outcome, assignment unit, dependence, and hypothesis are explicit, the difference between z, t, chi-square, and ANOVA becomes a modeling decision rather than a memorization exercise.
Analyze tests with context
Connect experiment assignments to trusted metrics, inspect uncertainty, and keep decision rules visible to the whole team.
Get Started With GrowthBookAn experiment with control plus three variants creates more than one comparison. ANOVA gives the team one principled global test of whether the variants differ before it starts hunting for a winner.
Analysis of variance, or ANOVA, is a family of statistical models for comparing group means and decomposing sources of variation. In a one-way product experiment, the “factor” is the assigned variant and its “levels” are control, B, C, and D.
The basic ANOVA question is deliberately broad: if all variants had the same population mean, would the observed separation among their sample means be surprising relative to the noise within variants?
That question is useful, but incomplete. A significant ANOVA result does not say which variant won, whether the lift is large enough to ship, or whether assumptions and instrumentation are sound. Those conclusions require planned contrasts, uncertainty intervals, and experiment-quality checks.
How ANOVA compares means through variance
ANOVA separates total variability into components:
- between-group variation: how far each group mean is from the overall mean
- within-group variation: how far individual observations are from their group mean
Each sum of squares is divided by its degrees of freedom to produce a mean square. The F statistic is:
Under the null hypothesis that all group means are equal, both quantities estimate the same underlying error variance, so their ratio should often be near 1. When group means are separated relative to the residual noise, F grows.
NIST's one-way ANOVA explanation describes this as comparing the level mean square with the residual mean square. The p-value is the probability, under the null model and assumptions, of an F statistic at least as large as the observed one.
For k groups and N total observations, one-way ANOVA usually has:
The numerator asks how much the k means vary. The denominator pools information about variability inside the groups.
A four-variant experiment example
Suppose a SaaS team tests four onboarding flows and measures projects created per eligible account during the first week.
| Variant | Accounts | Mean projects | Standard deviation |
|---|---|---|---|
| Control | 1,000 | 2.30 | 1.80 |
| B | 1,020 | 2.42 | 1.84 |
| C | 990 | 2.61 | 1.91 |
| D | 1,010 | 2.36 | 1.79 |
The null hypothesis is:
The alternative is that not all four means are equal. Notice what it does not say: “C is best.” The global alternative includes any pattern where at least one mean differs.
If the F-test rejects the null, the team should evaluate the comparisons it planned. It might compare every treatment with control, or test one contrast between the current flow and the average of three new concepts. The comparison plan should reflect the decision, not the visual ranking in the finished dashboard.
Make multiple tests trustworthy
See how experimentation leaders plan hypotheses, guardrails, and review practices when a result surface contains many possible claims.
Watch the Trustworthy Experiments Talk
Why not run every pairwise t-test?
Four groups create six pairs. If the team runs six independent tests at alpha 0.05 and treats any significant result as proof, the probability of at least one false positive across the family can exceed 0.05.
ANOVA gives one global test of the equality of all means. It also estimates residual variation using all groups, which can be more efficient than estimating it afresh for each pair under the classical equal-variance model.
The global test does not eliminate multiplicity in follow-up comparisons. R's Tukey HSD documentation explicitly notes that ordinary t-tests inflate the probability of a false declaration across a family. Choose the follow-up procedure for the comparisons the decision actually needs:
- every pair: Tukey-style simultaneous comparisons
- every treatment versus control: Dunnett-style comparisons
- a few planned product questions: predeclared contrasts with a suitable adjustment
- a conservative small family: a Bonferroni or Holm correction
An omnibus test can also be nonsignificant while one carefully planned contrast is persuasive, because the hypotheses and power differ. Decide before launch whether the global null or a treatment-versus-control contrast is the primary decision test.
Unequal group sizes do not automatically invalidate ANOVA, but they make the variance assumption and contrast plan more consequential. If allocation is intentionally uneven, power the smallest comparison that drives the decision and preserve the assignment probabilities. When variances and sample sizes both differ, classical pooled ANOVA can behave poorly; Welch ANOVA or a regression with suitable standard errors is usually easier to defend.
Planned contrasts can also use product structure that the global test ignores. Instead of comparing every pair, a team might compare control with the average of three related treatments, or compare two low-intensity treatments with two high-intensity treatments. A small set of predeclared contrasts often answers the business question with more power and clearer multiplicity control than an exhaustive winner search.
ANOVA assumptions in experiments
The familiar one-way fixed-effects model can be written as:
Classical inference depends on the residuals and design, not on a requirement that the combined raw outcome form one bell curve. NIST's model reference assumes independent, normally distributed errors with mean zero and common variance.
Independent observations
The analysis unit must respect randomization. If accounts are assigned but every user within an account is treated as independent, the standard error ignores clustering. Aggregate at the account level or use cluster-robust or hierarchical methods.
Repeated events from one user create the same problem. Ten sessions from one user do not carry the same independent information as ten users.
Appropriate residual behavior
ANOVA is often robust to moderate non-normality with balanced, sufficiently large groups, but severe skew, outliers, censoring, or zero inflation can make the mean unstable or the F approximation unreliable. Diagnose residuals and assess whether the mean is still the business estimand.
Equal variance for classical one-way ANOVA
Classical ANOVA assumes a common population variance. This can fail when a treatment changes both the mean and spread, or when groups serve different traffic mixes. Unequal group sizes make the problem more consequential.
SciPy's 0 supports Welch ANOVA when equal_var=False. Welch's method relaxes equal population variances and adjusts the degrees of freedom.
Correct outcome model
ANOVA targets a continuous mean. Conversion is binary; event counts are discrete; time-to-churn can be censored. Large-sample mean inference can sometimes work, but logistic, Poisson or negative-binomial, survival, or other generalized models may better represent the outcome and produce interpretable effects.
One-way, two-way, and repeated-measures ANOVA
“ANOVA” names a family rather than one calculation.
One-way ANOVA
One categorical factor with multiple levels, such as four assigned onboarding variants. This is the usual A/B/n example.
Two-way or factorial ANOVA
Two controlled factors, such as headline and layout. The model estimates each main effect plus their interaction. The interaction asks whether one factor's effect changes with the other. This is central to a properly designed multivariate test.
Repeated-measures ANOVA
The same units are observed under multiple conditions or times. Dependence is part of the design and must be modeled. A basic independent one-way ANOVA is invalid for repeated measurements.
ANCOVA
Analysis of covariance adds continuous covariates to the group comparison. In randomized experiments, pre-experiment covariates can improve precision when they are chosen and measured without post-treatment contamination. GrowthBook's guide to variance reduction explains the same motivation in online experimentation.
Run one-way ANOVA in Python
At the action boundary, keep one numeric observation per independent analysis unit in each group. In SciPy:
Before running it, confirm that rows match the randomization unit and missing values have a documented policy. Afterward, inspect group summaries and residual behavior. The p-value alone cannot reveal a broken exposure join or a few enormous outliers.
In R, aov(outcome ~ variant, data = experiment) fits the classical model. R documents 1 as a linear-model interface, which helps explain why ANOVA, regression, and contrasts are closely connected.
Interpret the ANOVA table
A standard output contains:
- degrees of freedom
- sum of squares
- mean square
- F statistic
- p-value
Suppose the output reports F(3, 4016) = 6.8, p < 0.001. Under the model, the observed ratio of between-variant to within-variant variation is unlikely if all four population means are equal. It does not mean every treatment beats control or that any effect is commercially important.
Add the quantities the product decision needs:
- each mean and sample size
- differences from control in original units
- simultaneous or comparison-specific intervals
- an effect-size measure when useful
- guardrail and data-quality results
- the follow-up comparison method
Avoid ranking noisy means without uncertainty. The highest observed variant has benefited from both its true effect and sampling variation, especially when many variants were screened.
Common ANOVA mistakes
Treating events as independent users
Repeated events make the nominal sample size huge and uncertainty too narrow. Preserve the assignment unit.
Using ANOVA for every metric shape
The word “variant” does not imply ANOVA. Match the outcome distribution and estimand to a model.
Checking assumptions after selecting a winner
Write the model, outlier policy, transformation, and variance choice before the ranking is visible. Result-driven switching creates hidden researcher degrees of freedom.
Treating a significant F-test as a winner declaration
Follow with the planned contrasts. The omnibus test only rejects equality of all means.
Ignoring practical significance
A very large experiment can detect a tiny difference. Compare intervals with a minimum practical effect and account for implementation cost and guardrails.
Use ANOVA as part of an experiment plan
Before launch, specify the factor and levels, independent unit, primary continuous outcome, minimum effect, sample-size plan, variance assumption, global or contrast hypothesis, comparison family, and stopping rule.
Then verify assignment and exposure before interpreting the model. A sample ratio mismatch can signal that observed group counts no longer reflect the planned randomization. No F-test can repair biased exposure data.
ANOVA is valuable because it turns a field of variant means into a structured model of signal and noise. The broader z-test, t-test, chi-square, and ANOVA guide shows when the outcome and hypothesis call for another member of that family. Use the omnibus test for the global question, planned contrasts for the decision, and effect estimates for practical judgment. That sequence makes a multiple-variant test easier to defend than a dashboard full of uncoordinated p-values.
Compare variants with discipline
Run controlled experiments, connect trusted metrics, and review treatment effects and uncertainty in one shared workflow.
Start With GrowthBookReady to ship faster?
No credit card required. Start with feature flags, experimentation, and product analytics—free.
