What are experimental units in research?
Pick the wrong entity to randomize in your experiment, and your p-values, sample size, and conclusions all break — even if the rest of your analysis is flawless.
This is the core problem with misidentifying experimental units, and it happens constantly: in academic labs counting cells instead of mice, in product teams counting pageviews instead of users, in any experiment where the thing being measured gets confused with the thing that actually received the treatment.
This article is for engineers, PMs, and data practitioners who run experiments — whether that means lab studies, field tests, or product A/B tests — and want to understand how experimental units work well enough to get them right. Here's what you'll learn:
- What experimental units are, including their dual role as both the recipient of a treatment and the basis for statistical inference
- How to correctly identify the experimental unit in your own study using a practical diagnostic checklist
- Why experimental units and sampling units are different things, and how confusing them produces pseudo-replication
- How experimental units determine your true sample size and what under-replication actually costs you
- How these principles apply directly to A/B testing and product experimentation, including how platforms like GrowthBook make the randomization unit an explicit design decision
The article moves from the foundational concept through identification, common errors, sample size implications, and finally to digital experimentation — so whether you're designing a study from scratch or auditing one that's already running, you'll find the relevant section without having to read linearly.
The entity that receives the treatment: defining the experimental unit
Before any statistical analysis can be trusted, one question has to be answered correctly: what, exactly, is the experimental unit? It sounds like a technical formality, but getting this wrong doesn't just introduce noise into your results — it can invalidate them entirely.
Whether you're running a clinical trial, a behavioral study, or a product A/B test, the experimental unit is the foundation everything else is built on.
Independence is the criterion, not physical form
An experimental unit is the entity that receives a treatment independently of all other units in a study. The NC3Rs Experimental Design Advisory group defines it this way: "The experimental unit is the entity subjected to an intervention independently of all other units. It must be possible to assign any two experimental units to different treatment groups."
That independence criterion is doing a lot of work in the definition. It's not enough that something receives a treatment — it has to receive it in a way that is logically separable from every other entity in the study. If one unit's treatment assignment or response influences another's, they aren't truly independent experimental units.
The ARRIVE Guidelines, which govern reporting standards for animal research, add a further condition: experimental units "should not influence each other on the outcomes that are measured."
The experimental unit can take many forms. In the most straightforward case, it's an individual person or animal, each independently allocated to a treatment group. But it can also be a cage of animals receiving a shared diet, a litter of pups whose dam was treated, a specific body region of a single animal receiving a topical drug, or even the same animal across distinct time periods in a crossover design. What matters is not the physical form of the entity — it's whether the treatment was applied independently.
You'll sometimes see the experimental unit referred to as the "unit of randomisation", particularly in clinical and biomedical research contexts. The terms are functionally synonymous and worth knowing, since practitioners in different fields tend to favor one over the other.
The dual role: treatment receipt and the basis for inference
Here's where many researchers — even experienced ones — miss something important. The experimental unit doesn't just receive a treatment. It also serves as the entity about which you draw population-level conclusions.
NC3Rs states both roles explicitly: the experimental unit is "the entity you want to make inferences about (in the population) based on the sample (in your experiment)" and simultaneously "the entity subjected to an intervention independently of all other units." These two roles are inseparable, and the second one — the inference role — is what makes correct identification so consequential.
Sample size, as defined by both NC3Rs and the ARRIVE Guidelines, is the number of experimental units per group. That's not a technical footnote; it's a direct statement that your power calculations, your degrees of freedom, and your ability to generalize findings all hinge on how you've counted experimental units. If you're counting the wrong thing, your sample size estimate is wrong, and everything downstream follows.
Misidentifying the unit doesn't just add noise — it invalidates the analysis
The consequences of misidentifying the experimental unit aren't subtle. NC3Rs is direct about it: "If you do not correctly identify the experimental unit, there is a risk you overestimate your sample size which could invalidate the results of your statistical analysis and conclusions."
A concrete example from the ARRIVE Guidelines makes this tangible. Suppose a researcher takes 50 cell measurements from a single mouse. If the mouse is the experimental unit — because the treatment was applied to the mouse, not to individual cells — then those 50 measurements represent a sample size of one, not fifty. Treating them as 50 independent observations inflates the apparent sample size, distorts the statistical analysis, and produces conclusions that can't be trusted.
This is why identifying the experimental unit isn't a step you return to after designing your study. It's the first design decision, and every subsequent choice — how many subjects to recruit, how to structure your analysis, what inferences you're entitled to draw — depends on getting it right.
Treatment assignment, not measurement, determines the experimental unit
Knowing the definition of an experimental unit is one thing. Knowing which entity in your study actually qualifies is another. The distinction that unlocks correct identification is this: the experimental unit is defined by what receives the treatment, not by what gets measured.
These are often different entities, and conflating them is one of the most common errors in study design — committed by experienced researchers as often as by newcomers.
The guiding question: what received the treatment independently?
The most reliable diagnostic question you can ask is: which entity in this study received the treatment independently of all other entities?
NC3Rs EDA offers a sharper version of this test: "It must be possible to assign any two experimental units to different treatment groups." If two entities in your study always receive the same treatment because they are physically or logically grouped together, neither one is the experimental unit — the group is. This single test resolves most identification problems before they become analytical errors.
The key move is separating "what did I measure?" from "what received the treatment?" These questions have different answers more often than researchers expect, and the experimental unit is always the answer to the second question.
The aquarium example: when the container is the unit, not its contents
Consider a study testing the effect of a water additive on fish. The researcher applies the additive to the aquarium, not to individual fish. Every fish in a given aquarium experiences identical water conditions — they are not independent recipients of the treatment. The aquarium is what received the treatment independently. Therefore, the aquarium is the experimental unit, and individual fish are the measurement unit.
This distinction matters enormously for analysis. If the study contains ten fish across two aquariums, the researcher does not have ten experimental units — they have two. The fish provide multiple measurements, but those measurements are not statistically independent of one another. NC3Rs EDA makes this principle explicit: taking multiple measurements from the same entity does not multiply the number of experimental units.
The aquarium example also illustrates a pattern worth internalizing: the experimental unit is frequently larger than the entity being measured. NC3Rs EDA notes that the unit "may be bigger than the animal (e.g. a litter or a cage)" — the cage of animals, not the individual animal, is the experimental unit when animals within a cage cannot be assigned to different treatments independently.
The restaurant example: scaling the principle to real-world experiments
The same logic applies outside the lab. Imagine a restaurant chain testing a new menu layout at a subset of its locations. Customers are measured — order size, satisfaction scores, return visits — but the treatment was assigned at the restaurant level. Every customer who walks into a treated location receives the same menu. Those customers are not independent recipients of the treatment; the restaurant is. The restaurant is the experimental unit. Individual customers are the measurement unit.
This scales directly to digital product contexts. If a feature is rolled out to all users in a geographic region, the region is the experimental unit, not the individual user. Misidentifying your experimental units can lead to overestimating your sample size, skewing your statistical analysis, and producing invalid conclusions. The error is not abstract — it produces wrong numbers and wrong decisions.
Four diagnostic questions that resolve most identification problems
When you are unsure which entity in your study is the experimental unit, work through these diagnostic questions in order. Start by asking what entity actually received the treatment — not what you measured, but what the intervention was applied to. From there, apply the NC3Rs EDA independence test: could you have assigned this entity to a different treatment group without affecting any other entity?
Next, look for nesting: are there smaller entities grouped inside your candidate unit that all received the same treatment? If so, the candidate unit — not the nested entities — is the experimental unit. Finally, check whether your unit is larger than you initially assumed, since the experimental unit is often the cage, the location, the region, or the account rather than the individual subject inside it.
In practice, this identification step happens at the design stage — and in modern experimentation platforms, it is made explicit. GrowthBook, for instance, requires teams to configure a hashAttribute when setting up an experiment, which is the attribute used to assign entities to treatment groups. Selecting that attribute is the act of identifying the experimental unit. The platform supports user, location, postal code, URL path, and other randomization units precisely because the right choice depends on how the treatment is actually assigned — which is exactly the question this checklist is designed to answer.
Experimental units vs. sampling units: the structural difference that pseudo-replication exploits
Most statistical errors don't announce themselves. Pseudo-replication — the mistake of treating measurements taken on sampling units as if they were independent experimental units — is particularly insidious because the data looks fine, the model runs without errors, and the p-value comes back significant. The problem is that the significance isn't real.
Understanding the difference between experimental units and sampling units is what separates a valid analysis from a convincing-looking one.
Sampling units live inside experimental units — and cannot stand in for them
A sampling unit is a fraction of an experimental unit — not a separate entity that independently receives a treatment, but a piece of something that does. The distinction is structural: experimental units exist at the level where treatment is assigned; sampling units exist within experimental units and are measured to characterize them.
The pairing shows up across research contexts in a consistent pattern. A fish tank is an experimental unit; an individual fish pulled from that tank is a sampling unit. A cage holding five birds is an experimental unit; one bird from that cage is a sampling unit. A field plot is an experimental unit; a quadrant within that plot is a sampling unit. In every case, the sampling unit shares its treatment assignment with the experimental unit it belongs to — it didn't receive the treatment independently, so it cannot be treated as an independent observation for statistical purposes.
This matters because, as agricultural statisticians put it directly: variation of observations within an experimental unit will not give you treatment differences. Only variation between experimental units provides the basis for testing whether a treatment had an effect. When you collapse that distinction, you're no longer measuring what you think you're measuring.
What pseudo-replication actually does to your statistics
Here's the error in concrete terms. Suppose you have two fish tanks. Tank A receives Treatment 1; Tank B receives Treatment 2. Each tank holds 10 fish, and you record a measurement on every fish — giving you 20 data points. If you enter those 20 observations into a statistical model and treat each fish as an independent experimental unit, you've committed pseudo-replication.
The immediate consequence is that your statistical test is counting the wrong number of independent data points. With 20 fish observations and 2 treatment groups, the test behaves as though you have 18 pieces of independent evidence to work with. You don't — you have 2 experimental units, one per treatment. The test's internal math is dividing by a number that doesn't reflect reality, which makes the result look more statistically significant than it actually is.
The p-value comes back small, but the small p-value is an artifact of miscounting, not evidence of a real treatment effect.
The mechanism is exactly this — counting sampling units as experimental units inflates n, which inflates the apparent degrees of freedom, which makes effects appear more statistically significant than they are.
Recognizing the error before it propagates
The fish tank example is easy to diagnose in retrospect, but the same structural mistake appears in less obvious forms. Multiple measurements taken on the same patient over time, repeated observations from the same store location, or several page views from the same user session — all of these are sampling units nested within experimental units, not independent experimental units in their own right.
The diagnostic question is the same one that identifies the experimental unit in the first place: what entity received the treatment independently? Everything measured within that entity is a sampling unit. Counting those measurements as separate experimental units doesn't increase your statistical power — it creates the illusion of power that isn't there.
Modern experimentation platforms have formalized this distinction at the infrastructure level. Tools like GrowthBook expose the randomization unit — the entity that gets hashed to determine treatment assignment — as an explicit configuration parameter. That design choice reflects a real methodological constraint: if a team assigns treatments at the user level but analyzes outcomes at the session or pageview level, treating each session as an independent observation, they're committing the digital equivalent of the fish tank error. The platform makes the experimental unit explicit precisely because the consequences of getting it wrong propagate through every analysis that follows.
How experimental units determine sample size and replication
Sample size is not the number of measurements in your study. It is the number of experimental units per group. That distinction sounds simple, but getting it wrong is one of the most common ways researchers and data practitioners end up with studies that are either underpowered or deceptively over-counted — and often both at once.
Replication means independent units, not more measurements
The ARRIVE Guidelines define the experimental unit as "the biological entity subjected to an intervention independently of all other units, such that it is possible to assign any two experimental units to different treatment groups." From that definition follows a direct corollary: the sample size is the count of those independent units per group, not the total number of observations collected.
Replication, in the statistically meaningful sense, requires independent experimental units receiving each treatment. It is not satisfied by taking more measurements from the same unit. As the JABSTB textbook frames it, a statistically valid sample is comprised of independent replicates of the experimental unit, generated through some random process. Both conditions — independence and randomness — are required. Repeated measurements on a single unit satisfy neither.
This matters for repeated-measures designs specifically. A before-and-after experiment that records two scores from the same subject produces more data points than experimental units. Those two scores are intrinsically linked; they do not represent two independent replications of the treatment. The experimental unit count stays at one per subject.
How misidentification inflates your n
The inflation error has a concrete form. Suppose you measure 50 individual cells taken from a single mouse. If the mouse is the experimental unit — because the mouse, not the cell, received the treatment independently — then you have n = 1, not n = 50. The 50 cell measurements are subsamples. They estimate measurement error within that one unit; they say nothing about how different mice would respond to the treatment.
Treating those 50 measurements as 50 independent replicates makes the study appear far better-powered than it actually is. The degrees of freedom are artificially inflated, confidence intervals are falsely narrow, and any resulting p-values are not trustworthy. This is the mechanism behind pseudo-replication: the study looks adequately sized on paper while being, in practice, an experiment of one.
Hierarchical designs and the question of which n to count
Real experiments are rarely flat. Biological and technical factors are typically organized in hierarchies — cells within animals, animals within cages, cages within rooms. Each level of that hierarchy raises the same question: at which level does independent treatment assignment actually occur?
The ARRIVE Guidelines are direct about this challenge: "Such hierarchies can make determining the sample size difficult (is it the number of animals, cells, or mitochondria?)". The answer is always the level at which treatment is assigned independently. If the cage receives the treatment and animals within the cage are measured, the cage is the experimental unit regardless of how many animals it contains.
Hierarchical designs can also have multiple experimental units within a single experiment. A pregnant dam receiving one treatment and her weaned pups subsequently allocated to different diets creates two distinct experimental units operating at different levels. Each has its own relevant n for its respective treatment comparison. Collapsing them into a single count produces an analysis that is wrong at both levels.
The consequences of under-replication
An experiment with only one experimental unit per treatment group cannot estimate within-treatment variability. There are no degrees of freedom available for error, which means no valid statistical test can be performed. Results from such a design cannot be generalized beyond the specific units tested, because there is no empirical basis for estimating how much those units represent a broader population.
Under-replication is not just a power problem — it is a validity problem. No amount of additional measurements per unit compensates for the absence of independent replication across units. This is why identifying the experimental unit correctly before designing a study is not a formality. It determines whether your sample size calculation reflects reality or a number that will mislead you from the start.
Platforms like GrowthBook operationalize this principle in digital experimentation by enforcing minimum metric thresholds before surfacing results — a practical guard against drawing conclusions from experiments that have not yet accumulated enough experimental units to support valid inference.
Experimental units in A/B testing and product experimentation
The same principle that determines the experimental unit in a laboratory study applies without modification to a digital A/B test: the experimental unit is whatever entity independently receives the treatment. The terminology shifts — "randomization unit" instead of "experimental unit," "traffic split" instead of "treatment assignment" — but the underlying logic is identical. Getting it wrong produces the same class of errors in a product dashboard that it produces in a published paper.
Users, sessions, regions: which entity actually received the treatment?
In most product experiments, the user is the experimental unit. A feature flag is toggled on or off for a given user, that user consistently sees one variant throughout the experiment, and the analysis counts users — not events, not pageviews — as the unit of observation. But the user is not the only valid choice.
Depending on what receives the treatment independently, the experimental unit might be a session, a device, an account, a geographic region, a store location, or a server-side entity like an API endpoint. The NC3Rs diagnostic question applies cleanly here: can any two of these entities be assigned to different treatment groups independently of each other? If the answer is yes, you have a candidate experimental unit. GrowthBook's platform reflects this variety explicitly, supporting randomization by user, location, postal code, URL path, and other attributes — a recognition that the right unit depends on the experiment, not on a platform default.
When the analysis unit doesn't match the assignment unit, independence breaks down
The choice of experimental unit determines whether the independence assumption underlying your statistical test actually holds. If users are the true experimental unit but your analysis treats sessions as independent observations, you are counting the same user's repeated sessions as if they were separate, unrelated data points. They are not. The same user's sessions are correlated by definition — same preferences, same context, same exposure history.
This is precisely what NC3Rs identifies as the core requirement: it must be possible to assign any two experimental units to different treatment groups. A session cannot be independently assigned when the user behind it has already been assigned. Misidentifying the unit in this way leads directly to overestimated sample sizes and invalid statistical conclusions — the confidence intervals look tighter than they are because the apparent sample size is inflated.
Consistent assignment is the practical safeguard against this error. Experimentation platforms address it by ensuring the same entity always receives the same variant across the experiment's duration, rather than being re-randomized on each visit. The experimental unit must receive a consistent treatment throughout — not a randomly re-assigned one on each interaction.
The pseudo-replication risk in product experimentation
Counting pageviews or events as independent experimental units when the actual unit is the user is the digital equivalent of pseudo-replication. The apparent sample size grows quickly — millions of events can accumulate in days — but the number of independent experimental units grows much more slowly. Statistical tests that treat event counts as the sample size are operating on inflated degrees of freedom, producing p-values and confidence intervals that cannot be trusted.
GrowthBook's own guidance on exposure timing addresses this directly: expose users as close to the actual treatment exposure as possible, and avoid including users who never encountered the treatment. Including unexposed users in the analysis increases noise and reduces the ability to detect real differences — the same logic that makes pseudo-replication damaging in a lab study. For cases where assignment and exposure are unavoidably separated, some platforms support an activation metric to filter the analysis down to users who actually received the treatment.
Making the experimental unit an explicit configuration decision, not a hidden assumption
The industry has responded to this problem by making experimental unit selection an explicit, configurable platform decision rather than a hidden assumption. GrowthBook exposes the randomization unit as a named configuration parameter within a unified platform that connects feature flags, experiment configuration, and metrics analysis — ensuring the entity used for assignment is the same entity reflected in downstream statistical reporting.
The platform also offers sticky bucketing for experiments where consistent assignment must be maintained even if experiment settings change. Statsig similarly surfaces flexible targeting and randomization units as a deliberate capability.
This is not a convenience feature. It is a recognition that the choice of experimental unit is a design decision with direct consequences for statistical validity, and that practitioners need to make it consciously rather than inherit a default that may not match their experiment's structure.
One question determines everything: what entity received the treatment independently?
The through-line of everything in this article is a single question: what entity received the treatment independently? That question determines your experimental unit, which determines your true sample size, which determines whether your statistical conclusions are valid or just plausible-looking. The cell isn't the unit if the mouse got the treatment. The session isn't the unit if the user was assigned. The customer isn't the unit if the restaurant was randomized. Everything else follows from getting that one thing right.
Three questions that surface the right unit before the study runs
Before you finalize any study design — or before you audit one that's already running — ask three things in order. First: what entity actually received the treatment, not what you measured? Second: could any two of those entities have been assigned to different groups without affecting each other? Third: is there a level of nesting above your candidate unit where treatment was really applied?
If you work through those questions honestly, you'll land on the right unit most of the time. The mistake almost always comes from skipping straight to "what did I measure?" and working backward.
Pseudo-replication doesn't look like an error until you check the denominator
The hardest part about pseudo-replication is that it doesn't look like an error. The data is real, the model runs, and the p-value comes back significant. What's broken is the denominator — and catching it requires checking at the design stage, not after the results are in. If you've worked through the three questions above honestly, you've already done the work. The analysis just has to reflect the same unit you identified.
The terminology changes by field; the logic doesn't
The framing changes by field — "unit of randomisation" in clinical research, "randomization unit" or hashAttribute in product experimentation, "experimental unit" in biology — but the underlying logic doesn't. Whatever entity was independently assigned to a treatment group is your experimental unit, and your analysis has to be built around that entity, not around the measurements you took from inside it. This is one of those rare methodological principles that transfers cleanly across contexts, which means getting it right once pays dividends everywhere you run experiments.
What to do next:
- If you're designing a new study, answer the three questions in this article before writing your analysis plan — not after.
- If you're in a product experimentation context, check whether your platform's randomization unit matches the entity your treatment was actually applied to.
- If you're auditing an existing study, locate the denominator in your statistical test and verify it reflects independent experimental units, not subsamples.
- If you're working in a hierarchical design (cells within animals, users within accounts, sessions within users), identify every level where treatment assignment occurs and ensure your analysis reflects the correct level.
Related insights
Related Articles
In healthcare, “Can we randomize it?” is the wrong first question. Start with “Could either experience change care, rights, privacy, or access?”
A/B testing can improve digital intake, appointment access, patient education, clinician workflows, and administrative operations. It can also create unacceptable risk when teams treat a clinical or consent decision like an ordinary conversion funnel.
The difference is not the label on the method. A/B tests are randomized experiments. What matters is the treatment, purpose, affected population, data flow, and oversight required in the organization and jurisdiction. This guide provides a practical product framework, not a substitute for legal, clinical, privacy, security, or institutional review.
Draw the boundary before designing variants
Create an intake step that classifies the proposed change before anyone builds a treatment. At minimum, ask:
- Can the change alter diagnosis, treatment, triage, dosage, or clinical recommendations?
- Can it delay or discourage access to care, accommodations, or urgent help?
- Does it change informed consent, privacy choice, required disclosure, or patient cost?
- Does it use protected or sensitive health information for assignment or measurement?
- Does it include children, people in crisis, or another population requiring added protection?
- Is the purpose internal quality improvement, or is it designed to contribute to generalizable knowledge?
- Could the software function fall within medical-device or clinical decision-support oversight?
The HHS quality-improvement guidance says many activities limited to improving patient care and collecting operational data are not research under the cited human-subjects regulations. It also states that some quality-improvement activities can have a research purpose, in which case human-subject protections may apply. A product team should not make that determination informally; route it to the organization’s authorized office.
Likewise, software that influences clinical decisions is not automatically an ordinary product surface. The FDA’s January 2026 clinical decision-support guidance explains that some software functions are excluded from the device definition while other patient- or caregiver-facing functions can remain subject to digital-health policy. Clinical and regulatory owners need to classify the function before experimentation.
Start with lower-risk operational questions
The safest early program tests reversible changes where both variants meet the same clinical, accessibility, privacy, and disclosure requirements.
Appointment reminder timing
Compare 2 approved reminder schedules or message structures to reduce missed appointments. Keep required details, opt-out behavior, language support, and urgent-contact instructions constant.
Use completed appointments or timely rescheduling as the primary outcome. Track cancellations, patient contacts, message delivery, opt-outs, wrong-recipient risk, and differences across language, age, disability, or access groups. A higher click rate is not enough if no-show rates or trust worsen.
Patient portal navigation
Test whether a clearer information architecture helps people complete a high-value administrative task, such as finding results, updating insurance, or sending a non-urgent message. Preserve emergency guidance and clinical escalation paths in both variants.
Measure successful task completion and time to completion. Guard against repeated navigation, abandonment, accessibility failures, mistaken message routing, and increased call-center burden. Use usability testing before the A/B test to catch failures randomization should never expose.
Administrative form sequence
Compare a long form with a staged flow, or test the order of non-clinical fields. Do not omit information needed for safe care, billing transparency, consent, or legal compliance.
Measure accurate completion, not just submission. Track validation errors, correction rates, staff rework, abandonment, and time to appointment. If the treatment collects sensitive data, confirm necessity and access controls before launch.
Educational content layout
Test 2 ways to present the same clinician-approved information: summary-first versus stepwise, text plus illustration versus text alone, or a clear action checklist versus a dense paragraph. Keep the medical meaning, risks, contraindications, and escalation advice equivalent.
Use a comprehension or appropriate next-action metric when feasible. Page time and clicks can be misleading. Accessibility, language quality, and comprehension across health-literacy levels belong in the guardrail plan.
Review the design before launch
Use a trustworthy experiment-design session to pressure-test metrics, safety checks, and decision rules before exposing patients or clinicians.
Watch the Experiment Design Session
Use stronger controls for care-adjacent products
Some product changes are not clinical interventions but can still influence care. They need clinical ownership, narrower eligibility, conservative ramps, and explicit stopping criteria.
Clinician workflow support
A test might compare how a work queue prioritizes administrative follow-up, how a note template reduces documentation work, or how a non-diagnostic alert is presented. The treatment should not silently alter the clinical standard of care.
Randomize at the unit that prevents contamination. Individual clinician assignment may fail when teams share queues and handoffs; clinic- or unit-level clusters may better match the workflow. Measure task completion and time saved, with guardrails for missed work, overrides, escalations, documentation quality, and staff workload.
Preventive-care outreach
Compare approved outreach content or channels for people already eligible under the same clinical rule. Do not experiment with whether one group receives necessary care or required notice.
Use completed appropriate follow-up as the primary outcome. Track opt-outs, unreachable patients, scheduling capacity, disparities, complaints, and downstream cancellations. If the treatment drives demand beyond operational capacity, a messaging lift can make access worse.
Digital adherence support
Test the presentation or timing of an approved reminder, checklist, or educational cue. Avoid treatment changes that could be interpreted as personalized medical advice without the corresponding validation and oversight.
Measure the intended behavior with caution. Self-reported completion or app engagement is not a clinical outcome. Include adverse-event reporting, escalation pathways, disengagement, and privacy events where relevant.
Feature rollout in health software
Use feature flags to separate deployment from release, start with internal or trained cohorts, and expand only when technical and clinical guardrails remain healthy. GrowthBook’s feature flag platform supports targeted rollouts and kill switches, while the experiment layer measures impact.
The rollback plan must describe more than turning off a flag. Determine whether the old experience remains clinically and operationally safe, how queued work is reconciled, what happens to partial workflows, and who is authorized to stop exposure.
Protect data by design
Do not send a broad event stream to an experimentation vendor and decide later which fields were unnecessary. Inventory the data before implementation:
| Data question | Required decision |
|---|---|
| Assignment | What is the least identifiable stable unit that works? |
| Eligibility | Which sensitive attributes are truly needed? |
| Exposure | What event proves the treatment was delivered? |
| Outcomes | Can metrics be computed inside the governed data environment? |
| Access | Which roles can view assignments, segments, and results? |
| Retention | When are raw records, logs, and exports removed? |
The HHS minimum-necessary guidance describes limiting uses, disclosures, and requests for protected health information to what is needed for the intended purpose, with policies based on roles and recurring versus non-routine access. Apply that principle to experiment attributes, debugging logs, dashboards, and downloaded readouts.
Pseudonymous identifiers reduce exposure but do not automatically make a dataset non-sensitive or outside applicable rules. Review linkability, small cohorts, free-text fields, URLs, device metadata, and combinations that can reveal a condition. Never put clinical details or identifiers in feature names, variation labels, or URLs.
A warehouse-native experimentation approach can query approved metrics where the organization already governs them. Architecture does not create compliance on its own; teams still need contracts, access control, auditability, retention rules, security review, and configuration that matches the approved data flow.
Keep unsafe questions out of product experimentation
An experimentation policy should name prohibited or separately governed categories. Product teams should not discover the boundary only after a proposal reaches launch review.
Do not use an ordinary product A/B test to withhold a clinically indicated service, emergency direction, safety warning, accessibility accommodation, required disclosure, or legally protected choice. Do not reduce the visibility of risks to improve completion. Do not randomize a diagnostic or treatment recommendation without the clinical, regulatory, and research framework appropriate to that intervention.
Avoid treatments that exploit fear, urgency, shame, or uncertainty about health. A message can increase appointment conversion while undermining informed choice. Likewise, do not test whether patients tolerate a harder cancellation, more confusing privacy control, or hidden cost. Both variants must meet the organization’s baseline standard for respectful and comprehensible communication.
Clinical AI and decision-support changes need an evaluation program beyond a click-based A/B test. Validate the model offline, examine performance and failure modes across relevant populations, review human factors, and stage deployment with clinical monitoring. An online comparison may contribute evidence only after both treatments meet the safety threshold for exposure.
When an activity may be human-subjects research, follow the institution’s process before enrolling or exposing anyone. HHS research-oversight training states that covered non-exempt human-subjects research requires the applicable review and that informed consent requirements apply unless the IRB authorizes otherwise. The product team should preserve the determination, protocol version, approved treatment, and reporting obligations with the experiment record.
Finally, do not interpret lack of detected harm as proof of safety. Rare adverse events, small vulnerable groups, and outcomes that occur after the experiment window may be underpowered. Use prior evidence, incident monitoring, qualitative reports, and post-rollout surveillance alongside the randomized estimate.
Define patient-centered metrics and guardrails
Healthcare teams need more than a conversion scorecard. Build a measurement hierarchy:
- Primary outcome: the operational or patient-facing result that answers the decision.
- Process diagnostics: steps that explain why the treatment worked or failed.
- Safety guardrails: outcomes that trigger a stop or clinical review.
- Equity checks: predeclared groups where access or benefit could differ.
- Operational guardrails: staffing, wait time, rework, cost, and downstream capacity.
Define the practical threshold before launch. A statistically detectable change may be too small to justify implementation, and a neutral aggregate can hide meaningful harm in a protected or vulnerable group. At the same time, slicing results across many small subgroups increases false-positive risk and can expose sensitive attributes. Predeclare the equity questions that matter and use appropriate privacy and multiple-testing controls.
GrowthBook supports reusable fact tables and metrics so teams can keep definitions reviewable. Use a power analysis for the primary outcome and critical guardrails. If the required sample or duration is unrealistic, do not weaken the standard; use usability research, simulation, staged quality improvement, or a larger treatment contrast.
Create a healthcare experiment review packet
Before launch, the owner should provide one reviewable packet:
- purpose, hypothesis, and operational decision
- classification and required oversight determination
- affected population and exclusion criteria
- clinical, privacy, security, accessibility, and compliance approvals
- treatment screenshots or workflow diagrams
- assignment, exposure, and data-flow design
- primary outcome, diagnostics, guardrails, and equity checks
- sample plan and stopping rule
- rollout stages, monitoring owner, and rollback procedure
- patient or clinician communication plan, if applicable
- documentation and retention plan
Use an approval matrix that names accountable people. Product approval does not replace clinical approval; a privacy review does not settle human-subjects research status; and an IRB determination does not automatically approve the production security architecture.
The WHO clinical-trial best-practices guidance emphasizes ethical standards, regulatory considerations, patient-centered research, transparency, and stakeholder collaboration. Not every healthcare product experiment is a clinical trial, but high-risk work should inherit the same respect for people and evidence.
Build trust into the experimentation program
Start with reversible operational improvements where both experiences are already acceptable. Prove that the team can classify risk, minimize data, validate assignment, monitor safety, and document decisions before expanding scope.
Publish internal rules for what teams may test, what requires added review, and what is out of bounds. Maintain an experiment registry and audit trail. Record neutral and negative results so a new team does not repeat the same risky idea.
GrowthBook can support the controlled delivery and analysis layer through experimentation, feature flags, permissions, and warehouse-defined metrics. The organization remains responsible for the clinical, ethical, legal, privacy, and operational framework around every test.
In healthcare, speed is valuable only when the learning process protects the people whose behavior creates the data.
Build a governed test workflow
Connect controlled releases to reviewable metrics and decision rules while keeping healthcare data in your approved architecture.
Get Started With GrowthBook
The right statistical test is determined by the question and data-generating process, not by which function is easiest to run. Start with the outcome, groups, and dependence structure; the test name comes later.
Z-tests, t-tests, chi-square tests, and analysis of variance (ANOVA) all compare observed data with a null model. They differ in the kind of outcome they model, the uncertainty they estimate, and the number or structure of groups they can compare.
For a simple product experiment, a useful first pass is:
- continuous outcome, two independent groups: usually a Welch two-sample t-test
- binary proportion, two large independent groups: a two-proportion z-test is common
- categorical counts across groups: chi-square test, if expected counts are adequate
- continuous outcome across three or more groups: one-way ANOVA or Welch ANOVA
Those rules are a starting point. Paired observations, clusters, ratios, repeated measures, heavy tails, covariate adjustment, or sequential monitoring require a model that reflects the design.
Choose from the outcome and hypothesis
Write the estimand before choosing a test. An estimand is the quantity the experiment is trying to estimate: a difference in mean revenue, a difference in conversion probability, or an association between two categorical variables.
| Question | Outcome | Common test |
|---|---|---|
| Did average order value change between A and B? | Continuous | Welch two-sample t-test |
| Did signup probability change between A and B? | Binary | Two-proportion z-test |
| Is plan choice associated with variant? | Categorical, 3+ levels | Chi-square test of independence |
| Do mean task times differ across four variants? | Continuous | One-way ANOVA |
| Did the same users' scores change before and after? | Paired continuous | Paired t-test |
The number of groups alone is insufficient. Conversion in four variants is still categorical data; a chi-square or binomial model may fit. Revenue in two groups is continuous; a t-test or regression is more natural.
The University of Michigan's statistical-test guide uses the same sequence: identify variable types and the relationship being tested before selecting a method.
When to use a z-test
A z-test compares a standardized estimate with the standard normal distribution. The classical one-sample z-test for a mean assumes the population standard deviation is known. That condition is unusual in product analytics, where variability is estimated from the current sample.
Z-tests remain common for proportions. In a two-arm conversion experiment, the estimate is:
Under the null of equal proportions and with adequate counts, the standardized difference is approximately normal. This yields a two-proportion z-test.
Use it when:
- the outcome is a binary count summarized as successes and failures
- assignment groups are independent
- sample sizes make the normal approximation credible
- the hypothesis and one- or two-sided direction were set before analysis
Do not rely on a universal “n greater than 30” rule. For rare events, 30 observations can produce almost no successes; for balanced common events, approximation quality can be good. Inspect expected successes and failures and use an exact or model-based method when counts are sparse.
In high-volume online experiments, a normal approximation is also used for many sample means through the central limit theorem. The important question is whether the estimator's sampling distribution and variance calculation are valid for the metric, not whether the raw user values look perfectly normal.
When to use a t-test
A t-test is designed for inference about means when the variance is estimated from sample data. That extra variance uncertainty produces a t distribution with heavier tails than the standard normal, especially at small sample sizes.
For two independent groups, default to Welch's t-test unless equal variance is justified. Welch's version does not assume the two population variances are equal and handles unequal group sizes. NIST's two-sample t-test reference shows the unequal-variance standard error based on each group's sample variance and size.
Use an independent two-sample t-test when:
- the outcome is numeric and the mean is the target
- the two groups contain different experimental units
- observations are independent within the model
- the mean and standard error behave well enough for the sample size
Use a paired t-test when each value has a meaningful partner: the same user's before-and-after score, or deliberately matched units. The analysis reduces each pair to a difference and tests the mean of those differences. Treating paired data as independent discards information and computes the wrong standard error.
The t-test can be sensitive to extreme values because the sample mean and variance are sensitive to them. Product metrics such as revenue or session duration are often skewed. At scale, the mean may still have a usable sampling distribution, but inspect outliers, data quality, and the estimand. Robust inference, transformations, winsorization policies, or bootstrap methods may be more appropriate when a few observations dominate the result.
Reduce variance before launch
Learn how CUPED and covariate adjustment can sharpen experiment estimates without changing the randomized comparison.
Explore Variance Reduction
When to use a chi-square test
Pearson's chi-square statistic compares observed category counts with counts expected under a null hypothesis. Two common forms are:
- goodness of fit: does one categorical distribution match specified probabilities?
- independence or homogeneity: is a categorical outcome distributed the same way across groups?
Suppose an onboarding experiment records three outcomes: completed, skipped, and abandoned. Cross-tabulate outcome by variant. A chi-square test asks whether the outcome distribution is independent of variant.
The test statistic sums (observed - expected)^2 / expected across cells. NIST's chi-square documentation describes the same comparison of binned frequency distributions.
Use a chi-square test when observations contribute counts to mutually exclusive categories and expected cell counts are large enough for the asymptotic approximation. With sparse cells, combine categories only when substantively justified or use an exact method such as Fisher's exact test for a two-by-two table.
A chi-square result says the distributions differ somewhere. It does not provide the most decision-friendly effect estimate by itself. Report category proportions, absolute differences, uncertainty intervals, and the cells contributing to the pattern.
For a binary two-arm experiment, the Pearson chi-square test and a two-sided two-proportion z-test are closely related: under standard conditions, the chi-square statistic with one degree of freedom equals the squared z statistic. Choose the representation that matches the hypothesis and reporting needs.
When to use ANOVA
ANOVA compares variation between group means with unexplained variation within groups. A one-way ANOVA tests the null that all population means are equal across levels of one factor.
Use it for a continuous outcome across three or more independent groups when the global question is whether any mean differs. Classical ANOVA assumes independent errors, normally distributed residuals within the model, and equal variances. Welch ANOVA relaxes the equal-variance assumption; R's 0 implements that approximation.
ANOVA's F-test is an omnibus test. A significant result means at least one mean differs, but it does not identify which one. Use planned contrasts or multiplicity-aware post-hoc comparisons to answer the product question.
ANOVA is more than a rule for “three or more groups.” Multi-factor ANOVA can estimate main effects and interactions in multivariate or factorial experiments. Repeated-measures or clustered data need corresponding error structures rather than a basic one-way calculation.
Why several t-tests are not a substitute for ANOVA
With four variants there are six pairwise comparisons. Testing each at 0.05 creates multiple opportunities for a false positive. An omnibus ANOVA tests one global null first, and planned follow-ups can use Tukey, Holm, Bonferroni, or another procedure appropriate to the family of claims.
The Bonferroni correction is simple and conservative. The right procedure depends on whether the goal is all pairwise comparisons, treatments versus one control, or a small set of preplanned contrasts. Define that family before looking at the ranking.
ANOVA and regression are also two views of the same linear-model machinery. R's 0 documentation describes aov as a wrapper around linear models for experimental designs. Regression is often more flexible when the analysis includes covariates, interactions, or unbalanced data.
Assumptions that change the choice
Before running any of the four tests, verify:
Independence and assignment unit
If the experiment randomizes accounts but analyzes users as independent observations, standard errors will usually be too small. Analyze at the randomization unit or use cluster-aware inference. If users can appear in both groups, repair the assignment or use a model that represents the dependence.
Paired or repeated observations
The same user measured twice is not two independent users. Use a paired test or repeated-measures model. For experiments with many events per user, aggregate to the user level or use appropriate clustered methods.
Outcome distribution and metric construction
Check missingness, zero inflation, extreme tails, ratio denominators, and censoring. A test can be mathematically correct for the supplied numbers while the metric itself misrepresents the user outcome.
Variance assumptions
Prefer Welch's t-test or Welch ANOVA when group variances may differ. Equal sample sizes do not prove equal variance, and a preliminary variance test can introduce another decision layer.
Sample size and sparse cells
Approximate z and chi-square methods need enough information in the relevant cells. Low-frequency guardrails and small segments may need exact methods or longer collection.
A product experimentation decision tree
Use this sequence before opening a statistics package:
- What unit was randomized: user, account, device, session, or region?
- What is the primary estimand: mean, proportion, category distribution, or model coefficient?
- Are groups independent, paired, repeated, or clustered?
- Are there two groups, several groups, or multiple factors?
- Do expected counts and sample sizes support the approximation?
- Are variances, tails, or outliers likely to break the default model?
- How many confirmatory hypotheses can trigger the decision?
- Was the test direction and stopping rule declared before launch?
Then choose the simplest model that answers the exact question. A two-proportion z-test may be perfect for signup conversion, while a t-test handles mean revenue and a chi-square test handles plan mix in the same experiment. Different metrics can require different tests.
Report effects, not only test names
The test produces a statistic and p-value under a null model. The guide to interpreting a t-test p-value shows why that number needs the effect, interval, and degrees of freedom beside it. The product decision needs more:
- the effect estimate in business units
- a confidence or credible interval
- sample sizes and allocation
- baseline and treatment values
- assumption and data-quality checks
- the planned hypothesis family
- practical thresholds and guardrails
GrowthBook's statistics documentation explains the frequentist and Bayesian engines available for experiment analysis. Whichever framework is used, review effect magnitude and uncertainty together. A small p-value can accompany a trivial lift in a huge sample, while a valuable estimated lift can remain uncertain in a small one.
Choose the test by tracing the data back to the experiment design. For three or more continuous-outcome variants, the deeper ANOVA guide covers the omnibus F-test, planned contrasts, and Welch alternative. When the outcome, assignment unit, dependence, and hypothesis are explicit, the difference between z, t, chi-square, and ANOVA becomes a modeling decision rather than a memorization exercise.
Analyze tests with context
Connect experiment assignments to trusted metrics, inspect uncertainty, and keep decision rules visible to the whole team.
Get Started With GrowthBookAn experiment with control plus three variants creates more than one comparison. ANOVA gives the team one principled global test of whether the variants differ before it starts hunting for a winner.
Analysis of variance, or ANOVA, is a family of statistical models for comparing group means and decomposing sources of variation. In a one-way product experiment, the “factor” is the assigned variant and its “levels” are control, B, C, and D.
The basic ANOVA question is deliberately broad: if all variants had the same population mean, would the observed separation among their sample means be surprising relative to the noise within variants?
That question is useful, but incomplete. A significant ANOVA result does not say which variant won, whether the lift is large enough to ship, or whether assumptions and instrumentation are sound. Those conclusions require planned contrasts, uncertainty intervals, and experiment-quality checks.
How ANOVA compares means through variance
ANOVA separates total variability into components:
- between-group variation: how far each group mean is from the overall mean
- within-group variation: how far individual observations are from their group mean
Each sum of squares is divided by its degrees of freedom to produce a mean square. The F statistic is:
Under the null hypothesis that all group means are equal, both quantities estimate the same underlying error variance, so their ratio should often be near 1. When group means are separated relative to the residual noise, F grows.
NIST's one-way ANOVA explanation describes this as comparing the level mean square with the residual mean square. The p-value is the probability, under the null model and assumptions, of an F statistic at least as large as the observed one.
For k groups and N total observations, one-way ANOVA usually has:
The numerator asks how much the k means vary. The denominator pools information about variability inside the groups.
A four-variant experiment example
Suppose a SaaS team tests four onboarding flows and measures projects created per eligible account during the first week.
| Variant | Accounts | Mean projects | Standard deviation |
|---|---|---|---|
| Control | 1,000 | 2.30 | 1.80 |
| B | 1,020 | 2.42 | 1.84 |
| C | 990 | 2.61 | 1.91 |
| D | 1,010 | 2.36 | 1.79 |
The null hypothesis is:
The alternative is that not all four means are equal. Notice what it does not say: “C is best.” The global alternative includes any pattern where at least one mean differs.
If the F-test rejects the null, the team should evaluate the comparisons it planned. It might compare every treatment with control, or test one contrast between the current flow and the average of three new concepts. The comparison plan should reflect the decision, not the visual ranking in the finished dashboard.
Make multiple tests trustworthy
See how experimentation leaders plan hypotheses, guardrails, and review practices when a result surface contains many possible claims.
Watch the Trustworthy Experiments Talk
Why not run every pairwise t-test?
Four groups create six pairs. If the team runs six independent tests at alpha 0.05 and treats any significant result as proof, the probability of at least one false positive across the family can exceed 0.05.
ANOVA gives one global test of the equality of all means. It also estimates residual variation using all groups, which can be more efficient than estimating it afresh for each pair under the classical equal-variance model.
The global test does not eliminate multiplicity in follow-up comparisons. R's Tukey HSD documentation explicitly notes that ordinary t-tests inflate the probability of a false declaration across a family. Choose the follow-up procedure for the comparisons the decision actually needs:
- every pair: Tukey-style simultaneous comparisons
- every treatment versus control: Dunnett-style comparisons
- a few planned product questions: predeclared contrasts with a suitable adjustment
- a conservative small family: a Bonferroni or Holm correction
An omnibus test can also be nonsignificant while one carefully planned contrast is persuasive, because the hypotheses and power differ. Decide before launch whether the global null or a treatment-versus-control contrast is the primary decision test.
Unequal group sizes do not automatically invalidate ANOVA, but they make the variance assumption and contrast plan more consequential. If allocation is intentionally uneven, power the smallest comparison that drives the decision and preserve the assignment probabilities. When variances and sample sizes both differ, classical pooled ANOVA can behave poorly; Welch ANOVA or a regression with suitable standard errors is usually easier to defend.
Planned contrasts can also use product structure that the global test ignores. Instead of comparing every pair, a team might compare control with the average of three related treatments, or compare two low-intensity treatments with two high-intensity treatments. A small set of predeclared contrasts often answers the business question with more power and clearer multiplicity control than an exhaustive winner search.
ANOVA assumptions in experiments
The familiar one-way fixed-effects model can be written as:
Classical inference depends on the residuals and design, not on a requirement that the combined raw outcome form one bell curve. NIST's model reference assumes independent, normally distributed errors with mean zero and common variance.
Independent observations
The analysis unit must respect randomization. If accounts are assigned but every user within an account is treated as independent, the standard error ignores clustering. Aggregate at the account level or use cluster-robust or hierarchical methods.
Repeated events from one user create the same problem. Ten sessions from one user do not carry the same independent information as ten users.
Appropriate residual behavior
ANOVA is often robust to moderate non-normality with balanced, sufficiently large groups, but severe skew, outliers, censoring, or zero inflation can make the mean unstable or the F approximation unreliable. Diagnose residuals and assess whether the mean is still the business estimand.
Equal variance for classical one-way ANOVA
Classical ANOVA assumes a common population variance. This can fail when a treatment changes both the mean and spread, or when groups serve different traffic mixes. Unequal group sizes make the problem more consequential.
SciPy's 0 supports Welch ANOVA when equal_var=False. Welch's method relaxes equal population variances and adjusts the degrees of freedom.
Correct outcome model
ANOVA targets a continuous mean. Conversion is binary; event counts are discrete; time-to-churn can be censored. Large-sample mean inference can sometimes work, but logistic, Poisson or negative-binomial, survival, or other generalized models may better represent the outcome and produce interpretable effects.
One-way, two-way, and repeated-measures ANOVA
“ANOVA” names a family rather than one calculation.
One-way ANOVA
One categorical factor with multiple levels, such as four assigned onboarding variants. This is the usual A/B/n example.
Two-way or factorial ANOVA
Two controlled factors, such as headline and layout. The model estimates each main effect plus their interaction. The interaction asks whether one factor's effect changes with the other. This is central to a properly designed multivariate test.
Repeated-measures ANOVA
The same units are observed under multiple conditions or times. Dependence is part of the design and must be modeled. A basic independent one-way ANOVA is invalid for repeated measurements.
ANCOVA
Analysis of covariance adds continuous covariates to the group comparison. In randomized experiments, pre-experiment covariates can improve precision when they are chosen and measured without post-treatment contamination. GrowthBook's guide to variance reduction explains the same motivation in online experimentation.
Run one-way ANOVA in Python
At the action boundary, keep one numeric observation per independent analysis unit in each group. In SciPy:
Before running it, confirm that rows match the randomization unit and missing values have a documented policy. Afterward, inspect group summaries and residual behavior. The p-value alone cannot reveal a broken exposure join or a few enormous outliers.
In R, aov(outcome ~ variant, data = experiment) fits the classical model. R documents 1 as a linear-model interface, which helps explain why ANOVA, regression, and contrasts are closely connected.
Interpret the ANOVA table
A standard output contains:
- degrees of freedom
- sum of squares
- mean square
- F statistic
- p-value
Suppose the output reports F(3, 4016) = 6.8, p < 0.001. Under the model, the observed ratio of between-variant to within-variant variation is unlikely if all four population means are equal. It does not mean every treatment beats control or that any effect is commercially important.
Add the quantities the product decision needs:
- each mean and sample size
- differences from control in original units
- simultaneous or comparison-specific intervals
- an effect-size measure when useful
- guardrail and data-quality results
- the follow-up comparison method
Avoid ranking noisy means without uncertainty. The highest observed variant has benefited from both its true effect and sampling variation, especially when many variants were screened.
Common ANOVA mistakes
Treating events as independent users
Repeated events make the nominal sample size huge and uncertainty too narrow. Preserve the assignment unit.
Using ANOVA for every metric shape
The word “variant” does not imply ANOVA. Match the outcome distribution and estimand to a model.
Checking assumptions after selecting a winner
Write the model, outlier policy, transformation, and variance choice before the ranking is visible. Result-driven switching creates hidden researcher degrees of freedom.
Treating a significant F-test as a winner declaration
Follow with the planned contrasts. The omnibus test only rejects equality of all means.
Ignoring practical significance
A very large experiment can detect a tiny difference. Compare intervals with a minimum practical effect and account for implementation cost and guardrails.
Use ANOVA as part of an experiment plan
Before launch, specify the factor and levels, independent unit, primary continuous outcome, minimum effect, sample-size plan, variance assumption, global or contrast hypothesis, comparison family, and stopping rule.
Then verify assignment and exposure before interpreting the model. A sample ratio mismatch can signal that observed group counts no longer reflect the planned randomization. No F-test can repair biased exposure data.
ANOVA is valuable because it turns a field of variant means into a structured model of signal and noise. The broader z-test, t-test, chi-square, and ANOVA guide shows when the outcome and hypothesis call for another member of that family. Use the omnibus test for the global question, planned contrasts for the decision, and effect estimates for practical judgment. That sequence makes a multiple-variant test easier to defend than a dashboard full of uncoordinated p-values.
Compare variants with discipline
Run controlled experiments, connect trusted metrics, and review treatment effects and uncertainty in one shared workflow.
Start With GrowthBookReady to ship faster?
No credit card required. Start with feature flags, experimentation, and product analytics—free.
