Understanding STAR goals for effective performance
SMART goals are useful until they aren't.
They produce clear, measurable targets — but they have no mechanism for ownership, no requirement to connect a goal to a larger purpose, and no room for the kind of ambitious thinking that produces breakthrough outcomes. STAR goals were developed, in two distinct forms by two different practitioners, to fill exactly those gaps.
The problem is that the term "STAR goals" gets used inconsistently online, and most explanations conflate frameworks that work very differently and serve different purposes.
This article is for managers, team leads, and individual contributors who set performance goals and want a more effective approach than a checklist. Whether you're running quarterly planning cycles, coaching direct reports, or building your own goal practice, here's what you'll learn:
- What STAR goals actually are — including the two primary frameworks and how they differ from each other and from SMART
- Why self-created goals produce stronger ownership and engagement than top-down assigned ones
- How to connect a goal to a larger purpose so it stays motivating when progress gets hard
- How to build an action plan that turns a well-written goal into concrete, trackable steps
- How to balance realistic targets with stretch goals so your team has both a reliable baseline and an ambitious ceiling
The article moves in that order — from framework clarity, through the psychological and structural principles behind each STAR criterion, to the practical mechanics of execution. By the end, you'll have enough to choose the right framework for your context and apply it without guessing.
STAR goals come in three distinct variants — and conflating them undermines all of them
If you've searched "STAR goals" and walked away more confused than when you started, you're not alone. The term refers to at least two entirely distinct frameworks developed independently by different practitioners for different contexts — plus a third, lesser-known variant that's essentially a SMART clone with different labeling.
Before applying any STAR framework in practice, it's worth understanding exactly what you're working with, where each version came from, and what specific problem each was designed to solve.
The origin and limitations of SMART goals
SMART goals were introduced in 1981 by George T. Doran as a structured approach to writing management objectives. The acronym — Specific, Measurable, Attainable, Relevant, Time-based — gave managers and individuals a reliable checklist for turning vague intentions into concrete targets. For operational planning, SMART remains genuinely useful. It forces specificity, creates accountability through deadlines, and produces goals that can be evaluated objectively.
The problem is what SMART excludes. As personal development practitioner Signe Knutson puts it, "SMART goals are logical, but limited." SMART goals are bounded by current circumstances — they take you from A to B, but only within the range of what already seems achievable. They contain no mechanism for ownership, no requirement that a goal connect to a larger purpose, and no room for aspirational thinking that exceeds what present conditions appear to allow.
For routine operational targets, that's fine. For sustained motivation, meaningful performance, or breakthrough outcomes, those omissions matter.
The education STAR framework: self-created, tied to a greater good, action-oriented, realistic and achievable
The first STAR variant was developed by the Alliance for Catholic Education (ACE) at the University of Notre Dame, specifically for use in blended learning classrooms. The acronym stands for Self-Created, Tied to a Greater Good, Action-Oriented, and Realistic and Achievable.
Each criterion addresses a specific failure mode of conventional goal-setting. The Self-Created criterion reflects a core finding from ACE's work: "Teacher-created learning goals, personalized or otherwise, do not elicit the same sense of ownership as student-created goals." Ownership, in other words, is not a soft benefit — it's a structural requirement for goal commitment.
The Tied to a Greater Good criterion is the most distinctive departure from SMART. It requires that every goal be explicitly connected to a larger purpose — not just a metric to hit, but a reason that metric matters in the context of a longer arc. Action-Oriented means a concrete execution plan must accompany the goal, not just the target itself. Realistic and Achievable mirrors SMART's Attainable criterion, keeping goals grounded in what's genuinely possible given current conditions.
This variant is designed to replace SMART in contexts where ownership and purpose are as important as measurability — particularly anywhere goals are set collaboratively between a guide (teacher, manager) and the person doing the work.
The personal development STAR framework: stretching, tremendous, amazing, rich
The second STAR variant comes from Signe Knutson and represents a more radical departure from SMART thinking. Here, STAR stands for Stretching, Tremendous, Amazing, and Rich. These goals are explicitly not bounded by current reality. They carry no timeline requirement. They are not evaluated against attainability. The entire point is to articulate outcomes that feel impossible under present circumstances — because, as Knutson argues, "when you write it down, it enters the realm of the possible."
Knutson's practical method involves writing six goals daily in a journal: three SMART goals (practical targets across Do, Be, and Have categories) and three STAR goals (aspirational counterparts in the same categories). The STAR goals in this system are designed to invite serendipity — outcomes that conventional planning forecloses because they exceed what current conditions appear to support. This variant doesn't replace SMART; it runs alongside it.
SMART vs. STAR: a direct comparison
The table below captures the structural differences across all three frameworks:
One additional definition worth flagging: a third-party source defines "STAR" as Specific, Time-bound, Achievable, Relevant, Trackable — essentially a SMART variant with "Trackable" substituted for "Measurable." This definition circulates in some corners of the goal-setting literature but doesn't represent either of the primary frameworks covered here, and conflating it with either would muddy the distinctions that make STAR frameworks useful in the first place.
The key takeaway: if you're implementing STAR goals in a performance management or educational context, you're most likely working with the ACE framework. If you're building a personal goal practice designed to coexist with operational targets, Knutson's variant is the relevant model. Both address SMART's motivational limitations — but through fundamentally different mechanisms.
Why self-created STAR goals drive stronger ownership and engagement
There's a familiar pattern in performance management: a manager sets a goal, an employee acknowledges it, and then — despite everyone's good intentions — the goal quietly fades into the background by Q2. The problem isn't ambition or ability. It's design.
When goals are handed down rather than built up, the psychological contract between the person and the objective is compliance, not commitment. The "Self-Created" criterion in the STAR goals framework exists precisely to break that pattern.
The psychology of ownership: why agency changes everything
Ownership, in the context of goal-setting, is not a personality trait that some employees have and others don't. It's a condition that either gets engineered into the process or doesn't. Scott Burgmeyer at becomemoregp.com draws a useful distinction here: ownership is the vested interest in an outcome — the initiative, the sense of responsibility for the process itself — while accountability is accepting responsibility for results after the fact. Both matter, but ownership comes first. Without it, accountability is just blame with better vocabulary.
When someone genuinely owns a goal, the downstream effects are concrete and measurable. Intrinsic motivation replaces the need for external pressure. Because the person accepts that obstacles are theirs to navigate rather than escalate, problem-solving improves. And performance becomes more consistent because the goal isn't something being done to them — it's something they're doing for themselves.
As Kelli Binnings frames it in Brainz Magazine, "ownership gives you the confidence to move beyond intentions and into action" because you've already accepted that you're doing everything within your power to succeed. That proactive orientation also changes how failure lands: when a goal is yours, a setback becomes feedback rather than evidence that the goal was wrong to begin with.
Contrast this with top-down assigned goals, where the psychological contract is fundamentally different. The employee's job is to comply, report, and not fall short of the number. There's no mechanism for the kind of intrinsic investment that sustains effort when things get difficult.
Self-created goals outperform assigned goals — and the data explains why
The data on externally imposed goals — or goals set without genuine individual investment — is not encouraging. Research cited by Brainz Magazine from Ohio State University found that only 9% of Americans who make New Year's resolutions actually keep them, with many abandoning them within the first week.
Resolutions are, by nature, self-chosen, which makes that failure rate striking. The explanation isn't a lack of desire — it's a lack of the structural support that turns intention into execution. If self-chosen resolutions fail at that rate without a supporting framework, assigned goals with even less personal investment face steeper odds.
The workforce context compounds this. The CREW Network has noted that today's employees are no longer satisfied with simply trading time for money. Particularly in the post-COVID era, workers seek meaning, personal growth, and alignment between their work and their values. The original SMART goal model, the CREW Network argues, "fails to bridge the gap between organizational objectives and meaningful engagement that today's employees seek." Self-created goals are one structural response to that gap — not because they're softer or less rigorous, but because they activate a different kind of psychological investment.
Applying self-created goals without losing organizational alignment
The reasonable objection from managers is: if employees write their own goals, how do we maintain alignment to team and company objectives? This tension is real but resolvable, and it dissolves when you stop treating self-created goals as unconstrained goals.
The practical model looks like this: the manager establishes the context — the team's OKRs, the department's strategic priorities, the constraints that are non-negotiable. Within that defined space, the employee drafts their specific goal: what they're committing to, how they'll measure progress, and what success looks like from their vantage point. Manager and employee then review together for alignment. The organizational direction is preserved; the ownership psychology is activated.
The CREW Network describes this shift as moving toward goals that develop "both the person and the professional" — a framing that acknowledges employees aren't just execution units but people whose engagement is a performance variable. Burgmeyer illustrates the cost of skipping this step with a C-suite coaching example: an executive frustrated that her team member "seemed to be looking for directives instead of taking more initiative." That's not a character flaw in the employee. It's the predictable output of a goal-setting process that never gave them genuine agency in the first place.
Ownership is a design outcome. Engineer it into the process deliberately, and the engagement that follows is a structural consequence — not a cultural accident.
Tying performance goals to a greater purpose beyond the metric
There is a well-documented gap between goals that are technically measurable and goals that people actually care about pursuing. Atlassian's research on team goal-setting identifies one of the core reasons teams fall short: a lack of consensus on what success actually means.
That finding points to something deeper than poor metric selection. It suggests that when a goal is only defined by its number, the people responsible for hitting it have no shared story about why the number matters — and without that story, motivation erodes the moment progress stalls.
The "Tied to a Greater Good" criterion in the STAR framework exists precisely to close this gap. It requires that every goal be explicitly connected to something larger than the metric itself — a team outcome, a departmental priority, or an organizational mission. This is not inspirational language for its own sake. It is a structural requirement that forces the goal-setter to answer a question that SMART goals leave optional: why does this target matter beyond the dashboard?
Why metric-only goals fail to sustain motivation
The SMART framework's "Relevant" criterion does ask whether a goal matters. But relevance, as typically applied, is a checkbox — it confirms that a goal is not wildly off-mission without requiring the goal-setter to articulate the specific chain of value from individual action to organizational outcome. A goal like "increase conversion rate by 5% this quarter" passes the relevance test easily. It says nothing about which customers benefit, which product strategy it advances, or what happens to the team if the number moves.
This matters because human motivation under difficulty is not sustained by measurability. It is sustained by perceived significance. When progress stalls — and it will — the question a person asks is not "is this goal specific enough?" It is "is this goal worth the continued effort?" A number without narrative context cannot answer that question. A goal anchored to a clear downstream purpose can.
Self-determination theory offers a useful mechanism here: when a goal connects to something a person genuinely values — not just a metric they've been assigned — the motivation to pursue it becomes internally driven rather than externally pressured. That kind of motivation is associated with greater persistence and higher-quality effort, particularly when progress stalls. The implication for performance management is direct: a goal that explains its own importance is more resilient than one that merely tracks a metric.
The research case for purpose-connected goals
Educational research from the Alliance for Catholic Education supports this argument in a different context with the same underlying logic. Students who understand how a short-term learning target connects to a longer-term outcome demonstrate measurably greater persistence and engagement than students who are given targets without that context.
The psychological mechanism is the same whether the learner is a student or a product manager: when people can trace a near-term goal to a longer-term outcome it serves, abandoning the near-term goal carries a higher psychological cost — because doing so means abandoning the larger outcome as well.
Gallup's longitudinal engagement research reinforces this at the organizational level, consistently finding that employees who report a strong sense of purpose in their work outperform those who do not on retention, productivity, and discretionary effort. Purpose is not a soft benefit — it is a performance variable.
Translating the 'tied to a greater good' criterion into concrete goal language
The practical challenge is translating this principle into goal language that is concrete rather than aspirational. A reliable method works in three levels, moving from the metric outward to the mission.
Start by stating the metric goal in plain terms: what will be measured, by how much, and by when. Then identify the team or department outcome that this metric directly serves — not a vague value like "customer satisfaction," but a specific operational result like "reducing churn in the enterprise segment." Finally, identify the organizational mission or strategic priority that outcome advances.
Once those three levels are clear, write a single connecting sentence: "I will [metric goal] so that [team outcome], which supports [organizational priority]." Applied to an earlier example, this transforms "increase NPS by 10 points" into "increase NPS by 10 points so that the support team can demonstrate retention impact, which supports our strategy of growing revenue through existing accounts rather than new acquisition."
That sentence does not make the goal easier to hit. It makes the goal worth hitting — and that distinction is what the "Tied to a Greater Good" criterion is designed to produce.
Building the action plan: where STAR goals either become real or stay theoretical
A STAR goal without an execution roadmap is, at best, a well-articulated wish. The "Action-Oriented" criterion exists precisely because intention and outcome are not the same thing — and the gap between them is where most goals quietly die.
As Asana puts it plainly: "A goal without a plan is just a wish." The action plan is the operational bridge that answers not just what you're trying to achieve, but how you're going to get there, who is responsible for each step, and by when. For managers and individual contributors alike, this is where goal-setting becomes goal-achieving.
What makes an action plan "STAR-worthy"
Not every to-do list qualifies as an action plan in the STAR sense. A STAR-worthy action plan is a structured document that identifies specific tasks, assigns ownership, sets deadlines, and — critically — connects each step back to the goal's underlying purpose. The Alliance for Catholic Education's framework defines the action plan as the component that provides "the 'HOW' to achieve the goal," and it makes a point that's easy to overlook: the plan should identify responsibilities across multiple stakeholders, not just the person who owns the goal.
Translated into a workplace context, this means a strong action plan doesn't just list what the individual contributor needs to do. It also clarifies what the manager needs to provide (resources, approvals, unblocking), what cross-functional partners need to deliver, and when each dependency is due. This multi-stakeholder framing is what separates a STAR action plan from a personal task list — it makes the goal a shared operational commitment rather than a private aspiration.
Asana's framework for action plans identifies four concrete outcomes a well-built plan delivers: clarity about exactly what steps are needed, accountability through clear ownership and deadlines, efficiency by sequencing tasks in the right order to reduce wasted effort, and motivation by connecting daily work to a larger objective. These aren't soft benefits — they're the structural properties that make a plan executable rather than decorative.
Decomposing a STAR goal into sequenced, time-bound milestones
Once the goal is defined, the practical work is decomposition. A high-level objective like "improve API response time by 30% this quarter" means nothing operationally until it's broken into sequenced, time-bound tasks: audit current performance baselines in week one, identify the top three bottlenecks by week two, implement and test the first optimization by week four, and so on.
The sequencing matters as much as the tasks themselves. Asana's emphasis on "right tasks in the right order" is a useful frame here — dependencies need to be surfaced early so that a missed handoff in week two doesn't collapse the entire plan by week six. Each milestone should specify who owns it and what "done" looks like, not just what needs to happen in the abstract. When milestones are ambiguous, accountability diffuses and slippage becomes invisible until it's too late to course-correct.
The review and adjustment cadence
The most common failure mode in goal management isn't poor goal-setting — it's treating the action plan as a static document reviewed once a year during performance cycles. An action plan is a living artifact. It should be revisited regularly, with a weekly or bi-weekly check-in cadence representing practitioner best practice for goals with meaningful complexity or interdependencies.
Regular review serves two functions. First, it surfaces drift early — tasks that slipped, dependencies that shifted, or assumptions that turned out to be wrong. Second, it creates a forcing function for honest progress assessment rather than optimistic projection. The goal isn't to hold every milestone perfectly on schedule; it's to catch misalignment while there's still time to adjust the plan rather than miss the goal.
Choosing a tracking structure that keeps the action plan a living document
The right tool is the one your team will actually use consistently. Project management platforms like Asana are purpose-built for exactly this use case — linking tasks to goals, assigning owners, setting due dates, and tracking completion over time. For smaller teams or simpler goals, a shared spreadsheet with columns for task, owner, due date, status, and blockers can serve the same function with less overhead.
What matters more than the tool is the structure: every action plan should have a single source of truth that all stakeholders can access and update, a defined review rhythm, and a clear owner responsible for keeping it current. Without that, even the best-designed plan becomes a document rather than a system.
Balancing realism with stretch: the two STAR acronyms are complementary instruments, not contradictions
The two STAR acronyms introduced earlier in this article can appear to contradict each other. One asks whether a goal is "Realistic and Achievable." The other asks whether it is "Stretching, Tremendous, Amazing, and Rich." These are not competing philosophies — they are complementary instruments, and understanding how to use both simultaneously is where goal-setting practice matures from administrative exercise into genuine performance driver.
The case for stretch goals: why ambitious targets produce better outcomes even when missed
Andy Grove, the Intel CEO widely credited as the father of OKRs, described stretch goals as "high-effort, high-risk" targets that are "often assumed to be impossible until a goal becomes possible." That framing is worth sitting with. The stretch goal is not a prediction — it is a deliberate expansion of what a team believes is within reach.
Google, Pinterest, Allbirds, and the National Academy of Engineering all use stretch goals as a formal part of their planning. The National Academy's goal of reverse-engineering the human brain is a useful illustration: no one expects that goal to be completed on a quarterly review cycle. What it does is orient research priorities, attract talent, and generate intermediate breakthroughs that a more conservative framing would never have produced.
The most important calibration tool here is the 70% benchmark. For organizations using OKR-style stretch goals, success is typically defined as achieving roughly 70% of what was set. This is not a consolation prize — it is the design. A stretch goal where 100% achievement feels likely is not stretching enough.
Conversely, when a team hits 70% of an ambitious target, they have frequently outperformed what 100% of a conservative target would have produced. As WhatMatters.com puts it: "even failed goals can result in substantial advancements."
This reframes the skeptic's objection. The concern with stretch goals is usually that they demoralize teams when missed. That concern is valid when stretch goals are set without structure, without milestones, and without a shared understanding that partial achievement is the expected outcome. With those elements in place, the risk profile changes substantially.
Running operational and aspirational goals in parallel
The practical resolution to the realistic-versus-ambitious tension is not to choose between them — it is to run both tracks simultaneously. Operational goals, whether framed as SMART goals or the "Realistic and Achievable" variant of STAR, provide the reliability baseline that teams need to function. Stretch goals layer on top of that foundation, targeting the breakthrough outcomes that operational planning alone will not reach.
Indeed.com describes one clean implementation of this structure: a stretch goal as "an optional extra goal a company can work toward if they exceed their original goal." Their crowdfunding example makes this concrete — an organization raises $10,000 for hurricane relief as its primary goal, then sets a stretch target of an additional $1,500 for food aid. The primary goal is achievable and meaningful on its own. The stretch goal creates upside without undermining the baseline.
This dual-track logic maps directly onto how OKRs distinguish between committed and aspirational objectives. The committed objective is what the team is accountable for delivering. The aspirational objective is what they are aiming toward. Both exist simultaneously, and the presence of the aspirational objective does not dilute accountability for the committed one.
Writing a stretch goal that inspires rather than discourages
The failure mode in stretch goal-setting is not ambition — it is ambiguity. A goal that is large and vague gives a team nowhere to start. The corrective is to pair the ambitious target with defined milestones and explicit thresholds for what partial success looks like.
WhatMatters.com frames this well: the best way to meet a stretch goal is through "defining the milestones to get there" and "communicating specific thresholds for success." The 70% benchmark functions as one such threshold — teams know in advance that hitting 70% of a well-calibrated stretch goal represents strong performance, not shortfall.
There is also a cultural dimension worth naming. Stretch goals only work in environments where partial achievement is treated as progress rather than failure. If a team hits 68% of an ambitious target and is penalized for missing it, the stretch goal framework collapses — not because the goal was wrong, but because the organizational response to near-achievement was wrong. Setting the cultural expectation explicitly, before the goal cycle begins, is as important as setting the goal itself.
Putting STAR goals into practice: which framework fits your context
The frameworks covered in this article are not interchangeable, and applying the wrong one to your context produces goals that miss the motivational mechanisms the framework was designed to activate. The decision is straightforward once you identify what problem you're actually trying to solve.
The ACE framework and Knutson's variant solve different problems — match the tool to the context
Use the ACE framework (Self-Created, Tied to a Greater Good, Action-Oriented, Realistic and Achievable) when:
- You are setting goals collaboratively with a direct report, student, or team member and want to activate genuine ownership rather than compliance
- Your goals need to connect individual performance targets to team or organizational outcomes in a way that sustains motivation through difficulty
- You are replacing a SMART goal process that produces technically correct goals but low engagement
Use Knutson's STAR variant (Stretching, Tremendous, Amazing, Rich) when:
- You are building a personal goal practice and want to run aspirational targets alongside your operational SMART goals
- You want to create deliberate space for outcomes that exceed what current circumstances appear to support
- You are not constrained by organizational alignment requirements and can afford to set goals without timelines or attainability filters
Use standard SMART goals when:
- The goal is purely operational and measurability is the primary requirement
- You need a format that integrates cleanly with existing performance management systems
- The goal-setter already has strong intrinsic motivation and the missing ingredient is structure, not ownership
These three approaches are not mutually exclusive. The most effective goal practices typically run all three simultaneously: SMART goals for operational accountability, ACE-structured STAR goals for collaborative performance development, and Knutson's STAR goals for personal aspiration.
Writing your first STAR goal using the ACE framework's connecting sentence
If you are implementing the ACE framework for the first time, the connecting sentence formula introduced earlier in this article is the fastest path to a well-formed goal: "I will [metric goal] so that [team outcome], which supports [organizational priority]."
Start there. Write the sentence. Then build the action plan around it — identifying the specific tasks, owners, deadlines, and dependencies that turn the commitment into an execution roadmap. Review the plan weekly. Adjust when assumptions prove wrong.
The formula is a starting constraint, not a ceiling. Once the connecting sentence is written, the goal has already satisfied three of the four ACE criteria: it is action-oriented by having a measurable commitment, tied to a greater good by naming the downstream outcome, and realistic by requiring you to state what you're actually committing to. The fourth criterion — self-created — is satisfied by the process of writing it yourself rather than receiving it from above.
Three implementation failures that undermine STAR goals before they start
The most common ways STAR goals fail in practice are not about goal quality — they are about process failures that occur before or after the goal is written.
The first failure is skipping the self-created step. A manager writes the goal, presents it to the employee as a STAR goal, and expects the ownership psychology to activate. It doesn't. The self-created criterion is not a formatting requirement — it is a process requirement. The employee must draft the goal. The manager's role is to establish context and review for alignment, not to write the objective.
The second failure is writing the connecting sentence but never building the action plan. A purpose-connected goal without an execution roadmap is motivating but inert. The "Tied to a Greater Good" criterion sustains motivation; the "Action-Oriented" criterion is what converts that motivation into measurable progress. Both are required. Neither substitutes for the other.
The third failure is treating stretch goals as performance commitments. When a Knutson-style STAR goal — aspirational, timeline-free, explicitly beyond current reach — gets evaluated as if it were a committed OKR, the framework breaks. Stretch goals are instruments for expanding what a team believes is possible. They are not accountability targets. Conflating the two produces the demoralization that stretch goal skeptics correctly warn against.
What to do next
If you manage a team, the highest-leverage first step is to restructure your next goal-setting conversation. Instead of presenting goals for acknowledgment, bring the organizational context — the OKRs, the strategic priorities, the constraints — and ask your direct report to draft their own goal within that space. Review together. The goal that emerges will be more specific, more owned, and more likely to survive the first difficult quarter than anything you could have written for them.
If you are an individual contributor, apply the connecting sentence formula to one current goal this week. Take a goal you already have — a metric you're responsible for — and write the sentence: "I will [metric goal] so that [team outcome], which supports [organizational priority]." If you can't complete the sentence, that's diagnostic information: the goal lacks the purpose connection that will sustain your effort when progress stalls.
If you work on a product team and your goals involve experimentation — conversion rate improvements, feature adoption, retention — GrowthBook's unified platform connects the metrics in your STAR goals directly to the feature flags and experiments driving them, so progress is visible and traceable in the same place you're running tests, rather than inferred from separate dashboards. That kind of direct connection between goal and evidence is exactly what the "Action-Oriented" criterion is designed to produce.
The frameworks in this article are not complicated. What makes them effective is consistent application — writing the goal with the right structure, building the action plan that follows from it, and reviewing both regularly enough to catch drift before it becomes failure.
Related insights
Related Articles
In healthcare, “Can we randomize it?” is the wrong first question. Start with “Could either experience change care, rights, privacy, or access?”
A/B testing can improve digital intake, appointment access, patient education, clinician workflows, and administrative operations. It can also create unacceptable risk when teams treat a clinical or consent decision like an ordinary conversion funnel.
The difference is not the label on the method. A/B tests are randomized experiments. What matters is the treatment, purpose, affected population, data flow, and oversight required in the organization and jurisdiction. This guide provides a practical product framework, not a substitute for legal, clinical, privacy, security, or institutional review.
Draw the boundary before designing variants
Create an intake step that classifies the proposed change before anyone builds a treatment. At minimum, ask:
- Can the change alter diagnosis, treatment, triage, dosage, or clinical recommendations?
- Can it delay or discourage access to care, accommodations, or urgent help?
- Does it change informed consent, privacy choice, required disclosure, or patient cost?
- Does it use protected or sensitive health information for assignment or measurement?
- Does it include children, people in crisis, or another population requiring added protection?
- Is the purpose internal quality improvement, or is it designed to contribute to generalizable knowledge?
- Could the software function fall within medical-device or clinical decision-support oversight?
The HHS quality-improvement guidance says many activities limited to improving patient care and collecting operational data are not research under the cited human-subjects regulations. It also states that some quality-improvement activities can have a research purpose, in which case human-subject protections may apply. A product team should not make that determination informally; route it to the organization’s authorized office.
Likewise, software that influences clinical decisions is not automatically an ordinary product surface. The FDA’s January 2026 clinical decision-support guidance explains that some software functions are excluded from the device definition while other patient- or caregiver-facing functions can remain subject to digital-health policy. Clinical and regulatory owners need to classify the function before experimentation.
Start with lower-risk operational questions
The safest early program tests reversible changes where both variants meet the same clinical, accessibility, privacy, and disclosure requirements.
Appointment reminder timing
Compare 2 approved reminder schedules or message structures to reduce missed appointments. Keep required details, opt-out behavior, language support, and urgent-contact instructions constant.
Use completed appointments or timely rescheduling as the primary outcome. Track cancellations, patient contacts, message delivery, opt-outs, wrong-recipient risk, and differences across language, age, disability, or access groups. A higher click rate is not enough if no-show rates or trust worsen.
Patient portal navigation
Test whether a clearer information architecture helps people complete a high-value administrative task, such as finding results, updating insurance, or sending a non-urgent message. Preserve emergency guidance and clinical escalation paths in both variants.
Measure successful task completion and time to completion. Guard against repeated navigation, abandonment, accessibility failures, mistaken message routing, and increased call-center burden. Use usability testing before the A/B test to catch failures randomization should never expose.
Administrative form sequence
Compare a long form with a staged flow, or test the order of non-clinical fields. Do not omit information needed for safe care, billing transparency, consent, or legal compliance.
Measure accurate completion, not just submission. Track validation errors, correction rates, staff rework, abandonment, and time to appointment. If the treatment collects sensitive data, confirm necessity and access controls before launch.
Educational content layout
Test 2 ways to present the same clinician-approved information: summary-first versus stepwise, text plus illustration versus text alone, or a clear action checklist versus a dense paragraph. Keep the medical meaning, risks, contraindications, and escalation advice equivalent.
Use a comprehension or appropriate next-action metric when feasible. Page time and clicks can be misleading. Accessibility, language quality, and comprehension across health-literacy levels belong in the guardrail plan.
Review the design before launch
Use a trustworthy experiment-design session to pressure-test metrics, safety checks, and decision rules before exposing patients or clinicians.
Watch the Experiment Design Session
Use stronger controls for care-adjacent products
Some product changes are not clinical interventions but can still influence care. They need clinical ownership, narrower eligibility, conservative ramps, and explicit stopping criteria.
Clinician workflow support
A test might compare how a work queue prioritizes administrative follow-up, how a note template reduces documentation work, or how a non-diagnostic alert is presented. The treatment should not silently alter the clinical standard of care.
Randomize at the unit that prevents contamination. Individual clinician assignment may fail when teams share queues and handoffs; clinic- or unit-level clusters may better match the workflow. Measure task completion and time saved, with guardrails for missed work, overrides, escalations, documentation quality, and staff workload.
Preventive-care outreach
Compare approved outreach content or channels for people already eligible under the same clinical rule. Do not experiment with whether one group receives necessary care or required notice.
Use completed appropriate follow-up as the primary outcome. Track opt-outs, unreachable patients, scheduling capacity, disparities, complaints, and downstream cancellations. If the treatment drives demand beyond operational capacity, a messaging lift can make access worse.
Digital adherence support
Test the presentation or timing of an approved reminder, checklist, or educational cue. Avoid treatment changes that could be interpreted as personalized medical advice without the corresponding validation and oversight.
Measure the intended behavior with caution. Self-reported completion or app engagement is not a clinical outcome. Include adverse-event reporting, escalation pathways, disengagement, and privacy events where relevant.
Feature rollout in health software
Use feature flags to separate deployment from release, start with internal or trained cohorts, and expand only when technical and clinical guardrails remain healthy. GrowthBook’s feature flag platform supports targeted rollouts and kill switches, while the experiment layer measures impact.
The rollback plan must describe more than turning off a flag. Determine whether the old experience remains clinically and operationally safe, how queued work is reconciled, what happens to partial workflows, and who is authorized to stop exposure.
Protect data by design
Do not send a broad event stream to an experimentation vendor and decide later which fields were unnecessary. Inventory the data before implementation:
| Data question | Required decision |
|---|---|
| Assignment | What is the least identifiable stable unit that works? |
| Eligibility | Which sensitive attributes are truly needed? |
| Exposure | What event proves the treatment was delivered? |
| Outcomes | Can metrics be computed inside the governed data environment? |
| Access | Which roles can view assignments, segments, and results? |
| Retention | When are raw records, logs, and exports removed? |
The HHS minimum-necessary guidance describes limiting uses, disclosures, and requests for protected health information to what is needed for the intended purpose, with policies based on roles and recurring versus non-routine access. Apply that principle to experiment attributes, debugging logs, dashboards, and downloaded readouts.
Pseudonymous identifiers reduce exposure but do not automatically make a dataset non-sensitive or outside applicable rules. Review linkability, small cohorts, free-text fields, URLs, device metadata, and combinations that can reveal a condition. Never put clinical details or identifiers in feature names, variation labels, or URLs.
A warehouse-native experimentation approach can query approved metrics where the organization already governs them. Architecture does not create compliance on its own; teams still need contracts, access control, auditability, retention rules, security review, and configuration that matches the approved data flow.
Keep unsafe questions out of product experimentation
An experimentation policy should name prohibited or separately governed categories. Product teams should not discover the boundary only after a proposal reaches launch review.
Do not use an ordinary product A/B test to withhold a clinically indicated service, emergency direction, safety warning, accessibility accommodation, required disclosure, or legally protected choice. Do not reduce the visibility of risks to improve completion. Do not randomize a diagnostic or treatment recommendation without the clinical, regulatory, and research framework appropriate to that intervention.
Avoid treatments that exploit fear, urgency, shame, or uncertainty about health. A message can increase appointment conversion while undermining informed choice. Likewise, do not test whether patients tolerate a harder cancellation, more confusing privacy control, or hidden cost. Both variants must meet the organization’s baseline standard for respectful and comprehensible communication.
Clinical AI and decision-support changes need an evaluation program beyond a click-based A/B test. Validate the model offline, examine performance and failure modes across relevant populations, review human factors, and stage deployment with clinical monitoring. An online comparison may contribute evidence only after both treatments meet the safety threshold for exposure.
When an activity may be human-subjects research, follow the institution’s process before enrolling or exposing anyone. HHS research-oversight training states that covered non-exempt human-subjects research requires the applicable review and that informed consent requirements apply unless the IRB authorizes otherwise. The product team should preserve the determination, protocol version, approved treatment, and reporting obligations with the experiment record.
Finally, do not interpret lack of detected harm as proof of safety. Rare adverse events, small vulnerable groups, and outcomes that occur after the experiment window may be underpowered. Use prior evidence, incident monitoring, qualitative reports, and post-rollout surveillance alongside the randomized estimate.
Define patient-centered metrics and guardrails
Healthcare teams need more than a conversion scorecard. Build a measurement hierarchy:
- Primary outcome: the operational or patient-facing result that answers the decision.
- Process diagnostics: steps that explain why the treatment worked or failed.
- Safety guardrails: outcomes that trigger a stop or clinical review.
- Equity checks: predeclared groups where access or benefit could differ.
- Operational guardrails: staffing, wait time, rework, cost, and downstream capacity.
Define the practical threshold before launch. A statistically detectable change may be too small to justify implementation, and a neutral aggregate can hide meaningful harm in a protected or vulnerable group. At the same time, slicing results across many small subgroups increases false-positive risk and can expose sensitive attributes. Predeclare the equity questions that matter and use appropriate privacy and multiple-testing controls.
GrowthBook supports reusable fact tables and metrics so teams can keep definitions reviewable. Use a power analysis for the primary outcome and critical guardrails. If the required sample or duration is unrealistic, do not weaken the standard; use usability research, simulation, staged quality improvement, or a larger treatment contrast.
Create a healthcare experiment review packet
Before launch, the owner should provide one reviewable packet:
- purpose, hypothesis, and operational decision
- classification and required oversight determination
- affected population and exclusion criteria
- clinical, privacy, security, accessibility, and compliance approvals
- treatment screenshots or workflow diagrams
- assignment, exposure, and data-flow design
- primary outcome, diagnostics, guardrails, and equity checks
- sample plan and stopping rule
- rollout stages, monitoring owner, and rollback procedure
- patient or clinician communication plan, if applicable
- documentation and retention plan
Use an approval matrix that names accountable people. Product approval does not replace clinical approval; a privacy review does not settle human-subjects research status; and an IRB determination does not automatically approve the production security architecture.
The WHO clinical-trial best-practices guidance emphasizes ethical standards, regulatory considerations, patient-centered research, transparency, and stakeholder collaboration. Not every healthcare product experiment is a clinical trial, but high-risk work should inherit the same respect for people and evidence.
Build trust into the experimentation program
Start with reversible operational improvements where both experiences are already acceptable. Prove that the team can classify risk, minimize data, validate assignment, monitor safety, and document decisions before expanding scope.
Publish internal rules for what teams may test, what requires added review, and what is out of bounds. Maintain an experiment registry and audit trail. Record neutral and negative results so a new team does not repeat the same risky idea.
GrowthBook can support the controlled delivery and analysis layer through experimentation, feature flags, permissions, and warehouse-defined metrics. The organization remains responsible for the clinical, ethical, legal, privacy, and operational framework around every test.
In healthcare, speed is valuable only when the learning process protects the people whose behavior creates the data.
Build a governed test workflow
Connect controlled releases to reviewable metrics and decision rules while keeping healthcare data in your approved architecture.
Get Started With GrowthBook
The right statistical test is determined by the question and data-generating process, not by which function is easiest to run. Start with the outcome, groups, and dependence structure; the test name comes later.
Z-tests, t-tests, chi-square tests, and analysis of variance (ANOVA) all compare observed data with a null model. They differ in the kind of outcome they model, the uncertainty they estimate, and the number or structure of groups they can compare.
For a simple product experiment, a useful first pass is:
- continuous outcome, two independent groups: usually a Welch two-sample t-test
- binary proportion, two large independent groups: a two-proportion z-test is common
- categorical counts across groups: chi-square test, if expected counts are adequate
- continuous outcome across three or more groups: one-way ANOVA or Welch ANOVA
Those rules are a starting point. Paired observations, clusters, ratios, repeated measures, heavy tails, covariate adjustment, or sequential monitoring require a model that reflects the design.
Choose from the outcome and hypothesis
Write the estimand before choosing a test. An estimand is the quantity the experiment is trying to estimate: a difference in mean revenue, a difference in conversion probability, or an association between two categorical variables.
| Question | Outcome | Common test |
|---|---|---|
| Did average order value change between A and B? | Continuous | Welch two-sample t-test |
| Did signup probability change between A and B? | Binary | Two-proportion z-test |
| Is plan choice associated with variant? | Categorical, 3+ levels | Chi-square test of independence |
| Do mean task times differ across four variants? | Continuous | One-way ANOVA |
| Did the same users' scores change before and after? | Paired continuous | Paired t-test |
The number of groups alone is insufficient. Conversion in four variants is still categorical data; a chi-square or binomial model may fit. Revenue in two groups is continuous; a t-test or regression is more natural.
The University of Michigan's statistical-test guide uses the same sequence: identify variable types and the relationship being tested before selecting a method.
When to use a z-test
A z-test compares a standardized estimate with the standard normal distribution. The classical one-sample z-test for a mean assumes the population standard deviation is known. That condition is unusual in product analytics, where variability is estimated from the current sample.
Z-tests remain common for proportions. In a two-arm conversion experiment, the estimate is:
Under the null of equal proportions and with adequate counts, the standardized difference is approximately normal. This yields a two-proportion z-test.
Use it when:
- the outcome is a binary count summarized as successes and failures
- assignment groups are independent
- sample sizes make the normal approximation credible
- the hypothesis and one- or two-sided direction were set before analysis
Do not rely on a universal “n greater than 30” rule. For rare events, 30 observations can produce almost no successes; for balanced common events, approximation quality can be good. Inspect expected successes and failures and use an exact or model-based method when counts are sparse.
In high-volume online experiments, a normal approximation is also used for many sample means through the central limit theorem. The important question is whether the estimator's sampling distribution and variance calculation are valid for the metric, not whether the raw user values look perfectly normal.
When to use a t-test
A t-test is designed for inference about means when the variance is estimated from sample data. That extra variance uncertainty produces a t distribution with heavier tails than the standard normal, especially at small sample sizes.
For two independent groups, default to Welch's t-test unless equal variance is justified. Welch's version does not assume the two population variances are equal and handles unequal group sizes. NIST's two-sample t-test reference shows the unequal-variance standard error based on each group's sample variance and size.
Use an independent two-sample t-test when:
- the outcome is numeric and the mean is the target
- the two groups contain different experimental units
- observations are independent within the model
- the mean and standard error behave well enough for the sample size
Use a paired t-test when each value has a meaningful partner: the same user's before-and-after score, or deliberately matched units. The analysis reduces each pair to a difference and tests the mean of those differences. Treating paired data as independent discards information and computes the wrong standard error.
The t-test can be sensitive to extreme values because the sample mean and variance are sensitive to them. Product metrics such as revenue or session duration are often skewed. At scale, the mean may still have a usable sampling distribution, but inspect outliers, data quality, and the estimand. Robust inference, transformations, winsorization policies, or bootstrap methods may be more appropriate when a few observations dominate the result.
Reduce variance before launch
Learn how CUPED and covariate adjustment can sharpen experiment estimates without changing the randomized comparison.
Explore Variance Reduction
When to use a chi-square test
Pearson's chi-square statistic compares observed category counts with counts expected under a null hypothesis. Two common forms are:
- goodness of fit: does one categorical distribution match specified probabilities?
- independence or homogeneity: is a categorical outcome distributed the same way across groups?
Suppose an onboarding experiment records three outcomes: completed, skipped, and abandoned. Cross-tabulate outcome by variant. A chi-square test asks whether the outcome distribution is independent of variant.
The test statistic sums (observed - expected)^2 / expected across cells. NIST's chi-square documentation describes the same comparison of binned frequency distributions.
Use a chi-square test when observations contribute counts to mutually exclusive categories and expected cell counts are large enough for the asymptotic approximation. With sparse cells, combine categories only when substantively justified or use an exact method such as Fisher's exact test for a two-by-two table.
A chi-square result says the distributions differ somewhere. It does not provide the most decision-friendly effect estimate by itself. Report category proportions, absolute differences, uncertainty intervals, and the cells contributing to the pattern.
For a binary two-arm experiment, the Pearson chi-square test and a two-sided two-proportion z-test are closely related: under standard conditions, the chi-square statistic with one degree of freedom equals the squared z statistic. Choose the representation that matches the hypothesis and reporting needs.
When to use ANOVA
ANOVA compares variation between group means with unexplained variation within groups. A one-way ANOVA tests the null that all population means are equal across levels of one factor.
Use it for a continuous outcome across three or more independent groups when the global question is whether any mean differs. Classical ANOVA assumes independent errors, normally distributed residuals within the model, and equal variances. Welch ANOVA relaxes the equal-variance assumption; R's 0 implements that approximation.
ANOVA's F-test is an omnibus test. A significant result means at least one mean differs, but it does not identify which one. Use planned contrasts or multiplicity-aware post-hoc comparisons to answer the product question.
ANOVA is more than a rule for “three or more groups.” Multi-factor ANOVA can estimate main effects and interactions in multivariate or factorial experiments. Repeated-measures or clustered data need corresponding error structures rather than a basic one-way calculation.
Why several t-tests are not a substitute for ANOVA
With four variants there are six pairwise comparisons. Testing each at 0.05 creates multiple opportunities for a false positive. An omnibus ANOVA tests one global null first, and planned follow-ups can use Tukey, Holm, Bonferroni, or another procedure appropriate to the family of claims.
The Bonferroni correction is simple and conservative. The right procedure depends on whether the goal is all pairwise comparisons, treatments versus one control, or a small set of preplanned contrasts. Define that family before looking at the ranking.
ANOVA and regression are also two views of the same linear-model machinery. R's 0 documentation describes aov as a wrapper around linear models for experimental designs. Regression is often more flexible when the analysis includes covariates, interactions, or unbalanced data.
Assumptions that change the choice
Before running any of the four tests, verify:
Independence and assignment unit
If the experiment randomizes accounts but analyzes users as independent observations, standard errors will usually be too small. Analyze at the randomization unit or use cluster-aware inference. If users can appear in both groups, repair the assignment or use a model that represents the dependence.
Paired or repeated observations
The same user measured twice is not two independent users. Use a paired test or repeated-measures model. For experiments with many events per user, aggregate to the user level or use appropriate clustered methods.
Outcome distribution and metric construction
Check missingness, zero inflation, extreme tails, ratio denominators, and censoring. A test can be mathematically correct for the supplied numbers while the metric itself misrepresents the user outcome.
Variance assumptions
Prefer Welch's t-test or Welch ANOVA when group variances may differ. Equal sample sizes do not prove equal variance, and a preliminary variance test can introduce another decision layer.
Sample size and sparse cells
Approximate z and chi-square methods need enough information in the relevant cells. Low-frequency guardrails and small segments may need exact methods or longer collection.
A product experimentation decision tree
Use this sequence before opening a statistics package:
- What unit was randomized: user, account, device, session, or region?
- What is the primary estimand: mean, proportion, category distribution, or model coefficient?
- Are groups independent, paired, repeated, or clustered?
- Are there two groups, several groups, or multiple factors?
- Do expected counts and sample sizes support the approximation?
- Are variances, tails, or outliers likely to break the default model?
- How many confirmatory hypotheses can trigger the decision?
- Was the test direction and stopping rule declared before launch?
Then choose the simplest model that answers the exact question. A two-proportion z-test may be perfect for signup conversion, while a t-test handles mean revenue and a chi-square test handles plan mix in the same experiment. Different metrics can require different tests.
Report effects, not only test names
The test produces a statistic and p-value under a null model. The guide to interpreting a t-test p-value shows why that number needs the effect, interval, and degrees of freedom beside it. The product decision needs more:
- the effect estimate in business units
- a confidence or credible interval
- sample sizes and allocation
- baseline and treatment values
- assumption and data-quality checks
- the planned hypothesis family
- practical thresholds and guardrails
GrowthBook's statistics documentation explains the frequentist and Bayesian engines available for experiment analysis. Whichever framework is used, review effect magnitude and uncertainty together. A small p-value can accompany a trivial lift in a huge sample, while a valuable estimated lift can remain uncertain in a small one.
Choose the test by tracing the data back to the experiment design. For three or more continuous-outcome variants, the deeper ANOVA guide covers the omnibus F-test, planned contrasts, and Welch alternative. When the outcome, assignment unit, dependence, and hypothesis are explicit, the difference between z, t, chi-square, and ANOVA becomes a modeling decision rather than a memorization exercise.
Analyze tests with context
Connect experiment assignments to trusted metrics, inspect uncertainty, and keep decision rules visible to the whole team.
Get Started With GrowthBookAn experiment with control plus three variants creates more than one comparison. ANOVA gives the team one principled global test of whether the variants differ before it starts hunting for a winner.
Analysis of variance, or ANOVA, is a family of statistical models for comparing group means and decomposing sources of variation. In a one-way product experiment, the “factor” is the assigned variant and its “levels” are control, B, C, and D.
The basic ANOVA question is deliberately broad: if all variants had the same population mean, would the observed separation among their sample means be surprising relative to the noise within variants?
That question is useful, but incomplete. A significant ANOVA result does not say which variant won, whether the lift is large enough to ship, or whether assumptions and instrumentation are sound. Those conclusions require planned contrasts, uncertainty intervals, and experiment-quality checks.
How ANOVA compares means through variance
ANOVA separates total variability into components:
- between-group variation: how far each group mean is from the overall mean
- within-group variation: how far individual observations are from their group mean
Each sum of squares is divided by its degrees of freedom to produce a mean square. The F statistic is:
Under the null hypothesis that all group means are equal, both quantities estimate the same underlying error variance, so their ratio should often be near 1. When group means are separated relative to the residual noise, F grows.
NIST's one-way ANOVA explanation describes this as comparing the level mean square with the residual mean square. The p-value is the probability, under the null model and assumptions, of an F statistic at least as large as the observed one.
For k groups and N total observations, one-way ANOVA usually has:
The numerator asks how much the k means vary. The denominator pools information about variability inside the groups.
A four-variant experiment example
Suppose a SaaS team tests four onboarding flows and measures projects created per eligible account during the first week.
| Variant | Accounts | Mean projects | Standard deviation |
|---|---|---|---|
| Control | 1,000 | 2.30 | 1.80 |
| B | 1,020 | 2.42 | 1.84 |
| C | 990 | 2.61 | 1.91 |
| D | 1,010 | 2.36 | 1.79 |
The null hypothesis is:
The alternative is that not all four means are equal. Notice what it does not say: “C is best.” The global alternative includes any pattern where at least one mean differs.
If the F-test rejects the null, the team should evaluate the comparisons it planned. It might compare every treatment with control, or test one contrast between the current flow and the average of three new concepts. The comparison plan should reflect the decision, not the visual ranking in the finished dashboard.
Make multiple tests trustworthy
See how experimentation leaders plan hypotheses, guardrails, and review practices when a result surface contains many possible claims.
Watch the Trustworthy Experiments Talk
Why not run every pairwise t-test?
Four groups create six pairs. If the team runs six independent tests at alpha 0.05 and treats any significant result as proof, the probability of at least one false positive across the family can exceed 0.05.
ANOVA gives one global test of the equality of all means. It also estimates residual variation using all groups, which can be more efficient than estimating it afresh for each pair under the classical equal-variance model.
The global test does not eliminate multiplicity in follow-up comparisons. R's Tukey HSD documentation explicitly notes that ordinary t-tests inflate the probability of a false declaration across a family. Choose the follow-up procedure for the comparisons the decision actually needs:
- every pair: Tukey-style simultaneous comparisons
- every treatment versus control: Dunnett-style comparisons
- a few planned product questions: predeclared contrasts with a suitable adjustment
- a conservative small family: a Bonferroni or Holm correction
An omnibus test can also be nonsignificant while one carefully planned contrast is persuasive, because the hypotheses and power differ. Decide before launch whether the global null or a treatment-versus-control contrast is the primary decision test.
Unequal group sizes do not automatically invalidate ANOVA, but they make the variance assumption and contrast plan more consequential. If allocation is intentionally uneven, power the smallest comparison that drives the decision and preserve the assignment probabilities. When variances and sample sizes both differ, classical pooled ANOVA can behave poorly; Welch ANOVA or a regression with suitable standard errors is usually easier to defend.
Planned contrasts can also use product structure that the global test ignores. Instead of comparing every pair, a team might compare control with the average of three related treatments, or compare two low-intensity treatments with two high-intensity treatments. A small set of predeclared contrasts often answers the business question with more power and clearer multiplicity control than an exhaustive winner search.
ANOVA assumptions in experiments
The familiar one-way fixed-effects model can be written as:
Classical inference depends on the residuals and design, not on a requirement that the combined raw outcome form one bell curve. NIST's model reference assumes independent, normally distributed errors with mean zero and common variance.
Independent observations
The analysis unit must respect randomization. If accounts are assigned but every user within an account is treated as independent, the standard error ignores clustering. Aggregate at the account level or use cluster-robust or hierarchical methods.
Repeated events from one user create the same problem. Ten sessions from one user do not carry the same independent information as ten users.
Appropriate residual behavior
ANOVA is often robust to moderate non-normality with balanced, sufficiently large groups, but severe skew, outliers, censoring, or zero inflation can make the mean unstable or the F approximation unreliable. Diagnose residuals and assess whether the mean is still the business estimand.
Equal variance for classical one-way ANOVA
Classical ANOVA assumes a common population variance. This can fail when a treatment changes both the mean and spread, or when groups serve different traffic mixes. Unequal group sizes make the problem more consequential.
SciPy's 0 supports Welch ANOVA when equal_var=False. Welch's method relaxes equal population variances and adjusts the degrees of freedom.
Correct outcome model
ANOVA targets a continuous mean. Conversion is binary; event counts are discrete; time-to-churn can be censored. Large-sample mean inference can sometimes work, but logistic, Poisson or negative-binomial, survival, or other generalized models may better represent the outcome and produce interpretable effects.
One-way, two-way, and repeated-measures ANOVA
“ANOVA” names a family rather than one calculation.
One-way ANOVA
One categorical factor with multiple levels, such as four assigned onboarding variants. This is the usual A/B/n example.
Two-way or factorial ANOVA
Two controlled factors, such as headline and layout. The model estimates each main effect plus their interaction. The interaction asks whether one factor's effect changes with the other. This is central to a properly designed multivariate test.
Repeated-measures ANOVA
The same units are observed under multiple conditions or times. Dependence is part of the design and must be modeled. A basic independent one-way ANOVA is invalid for repeated measurements.
ANCOVA
Analysis of covariance adds continuous covariates to the group comparison. In randomized experiments, pre-experiment covariates can improve precision when they are chosen and measured without post-treatment contamination. GrowthBook's guide to variance reduction explains the same motivation in online experimentation.
Run one-way ANOVA in Python
At the action boundary, keep one numeric observation per independent analysis unit in each group. In SciPy:
Before running it, confirm that rows match the randomization unit and missing values have a documented policy. Afterward, inspect group summaries and residual behavior. The p-value alone cannot reveal a broken exposure join or a few enormous outliers.
In R, aov(outcome ~ variant, data = experiment) fits the classical model. R documents 1 as a linear-model interface, which helps explain why ANOVA, regression, and contrasts are closely connected.
Interpret the ANOVA table
A standard output contains:
- degrees of freedom
- sum of squares
- mean square
- F statistic
- p-value
Suppose the output reports F(3, 4016) = 6.8, p < 0.001. Under the model, the observed ratio of between-variant to within-variant variation is unlikely if all four population means are equal. It does not mean every treatment beats control or that any effect is commercially important.
Add the quantities the product decision needs:
- each mean and sample size
- differences from control in original units
- simultaneous or comparison-specific intervals
- an effect-size measure when useful
- guardrail and data-quality results
- the follow-up comparison method
Avoid ranking noisy means without uncertainty. The highest observed variant has benefited from both its true effect and sampling variation, especially when many variants were screened.
Common ANOVA mistakes
Treating events as independent users
Repeated events make the nominal sample size huge and uncertainty too narrow. Preserve the assignment unit.
Using ANOVA for every metric shape
The word “variant” does not imply ANOVA. Match the outcome distribution and estimand to a model.
Checking assumptions after selecting a winner
Write the model, outlier policy, transformation, and variance choice before the ranking is visible. Result-driven switching creates hidden researcher degrees of freedom.
Treating a significant F-test as a winner declaration
Follow with the planned contrasts. The omnibus test only rejects equality of all means.
Ignoring practical significance
A very large experiment can detect a tiny difference. Compare intervals with a minimum practical effect and account for implementation cost and guardrails.
Use ANOVA as part of an experiment plan
Before launch, specify the factor and levels, independent unit, primary continuous outcome, minimum effect, sample-size plan, variance assumption, global or contrast hypothesis, comparison family, and stopping rule.
Then verify assignment and exposure before interpreting the model. A sample ratio mismatch can signal that observed group counts no longer reflect the planned randomization. No F-test can repair biased exposure data.
ANOVA is valuable because it turns a field of variant means into a structured model of signal and noise. The broader z-test, t-test, chi-square, and ANOVA guide shows when the outcome and hypothesis call for another member of that family. Use the omnibus test for the global question, planned contrasts for the decision, and effect estimates for practical judgment. That sequence makes a multiple-variant test easier to defend than a dashboard full of uncoordinated p-values.
Compare variants with discipline
Run controlled experiments, connect trusted metrics, and review treatment effects and uncertainty in one shared workflow.
Start With GrowthBookReady to ship faster?
No credit card required. Start with feature flags, experimentation, and product analytics—free.
