We talked to 4 product leaders about when not to run a test — here are their top takeaways

“Let's test it” is a strong default for resolvable uncertainty. It is a poor ritual when the test cannot change the decision.

A/B testing is the most reliable way to estimate the causal effect of many product changes. That does not make it the right method for every question. A team can waste traffic on an underpowered split, expose users to avoidable harm, or perform measurement theater after leadership has already committed to the launch.

Four product and experimentation leaders offer useful decision boundaries: Ronny Kohavi on actionability and power, Lina Blackman on predetermined AI launches, Andrew Willingham on testing assumptions before products, and Dave Massey on combining research with controlled evidence.

The decision filter

Before creating variants, ask 6 questions:

1. Will the result change an action? If no plausible outcome changes the plan, do not pretend the test is a decision tool.
2. Can the design identify the effect? Randomization, exposure, and analysis must match the causal question.
3. Can the sample detect the smallest effect worth acting on? Calculate this before implementation.
4. Is withholding or exposing the treatment acceptable? Security, legal, accessibility, and safety obligations constrain randomization.
5. Is an A/B test the cheapest credible method? A prototype, interview, log analysis, or staged rollout may answer the current uncertainty sooner.
6. Can the team preserve treatment integrity? Spillovers, network effects, and inconsistent execution may require a different design.

If those answers support a controlled test, run one. If they do not, choosing another method is statistical maturity, not a rejection of evidence.

Ronny Kohavi: Do not test without a decision rule

Ronny Kohavi warns against shipping flat results simply because a team already built the feature. A result that is not statistically significant does not prove zero effect or safety. If the confidence interval still includes a meaningful loss, “we did not detect harm” is not equivalent to “the treatment is harmless.”

The deeper boundary comes before launch. If the team will ship regardless of a negative, positive, or inconclusive result, an A/B test cannot serve its stated purpose. It may still be useful to estimate impact or monitor risk, but the team should name that objective honestly and choose a design for it.

Kohavi's long-term growth guidance recommends explicit shipping criteria that combine an overall evaluation criterion with guardrails. His research with Microsoft also documents how tiny product effects require large samples and careful diagnostics in online controlled experiments.

Do not run a conventional A/B test when: no outcome changes an action, the required sample is unavailable, or the team is using “inconclusive” as permission it never predeclared.

Use instead: a safe rollout with operational guardrails, a smaller prototype test, or a written strategic decision with post-launch measurement.

Check whether the test can decide

Review the power, SRM, stopping, and causal checks that determine whether a live experiment can support the decision.

Read the Prevention Playbook

Lina Blackman: Do not test a launch decision already made

Lina Blackman notes that many companies have already decided to launch an AI assistant because the market expects one. In that situation, framing an experiment as “should this product exist?” may be dishonest. The strategic decision has been made.

Squarespace instead can experiment on the parts that remain uncertain: entry points, prompt guidance, onboarding, output quality, and long-term value. This narrows the test to a decision the team is willing to change. Her Squarespace conversation also shows why early funnel movement can mislead; a blank template increased CMS entry while reducing downstream conversion.

AI products add another reason not to rely on a single A/B test. A treatment may generate an enormous variety of outputs. Mean engagement cannot certify that rare responses are accurate, safe, or policy-compliant. GrowthBook's guide to AI product experimentation recommends a layered system: offline evals for known behavior and online tests for real user outcomes.

Do not run one broad A/B test when: the launch itself is fixed or the average result cannot evaluate the output distribution.

Use instead: offline evals, red-team and safety review, gradual exposure, qualitative inspection, and focused A/B tests for reversible product choices.

Andrew Willingham: Do not build the whole idea to test its riskiest assumption

Andrew Willingham learned from Amazon and Atlassian that a polished product can still fail because users do not understand it or do not value the underlying job. Building both A and B may be an expensive way to discover a problem that a prototype or user session could have exposed.

His assumption-testing framework asks teams to list what must be true, rank assumptions by risk, and select the cheapest credible test. That might be a concierge workflow, clickable prototype, fake-door test with clear expectations, or structured interview before production code.

The UK Government Service Manual similarly recommends testing prototypes with users early through moderated usability research when the question is whether people understand and can use a design. An A/B test is better suited to causal comparison at scale after treatment and metrics are operationally meaningful.

Do not run a production split when: the key uncertainty is comprehension, desirability, feasibility, or whether the proposed problem exists.

Use instead: interviews for motivation, usability tests for comprehension, technical spikes for feasibility, and prototypes for workflow evidence.

Dave Massey: Do not ask behavior data to explain itself

Dave Massey's experimentation and UX research teams at UPS operate together. Controlled tests reveal what changed. Customer research helps explain why. That division prevents teams from forcing a causal story out of secondary metrics after the result arrives.

When a required recipient-email field damaged shipping conversion, the team learned that unexplained friction was costly. A later international-shipping version explained that the email could help resolve customs issues and did not produce the same penalty. The UPS account combines qualitative context and quantitative evidence rather than asking either method to do both jobs.

The National Academies' discussion of causal inference and study design reflects the broader principle: methods answer different questions under different assumptions. Randomization is powerful for average causal effects, but it does not automatically identify motivation or mechanism.

Do not run another variant when: the current uncertainty is why users reacted, what language they use, or which problem deserves investment.

Use instead: interviews, support-ticket analysis, session review, surveys designed for the question, and then a new confirmatory experiment.

Seven situations where another method is better

1. Mandatory security, legal, or accessibility work

Do not withhold a required fix from a randomized control merely to prove it helps conversion. Test safe implementation choices if genuine uncertainty remains, and monitor guardrails during rollout. NIST's Secure Software Development Framework treats security practices as risk requirements, not optional growth treatments.

2. Catastrophic downside

If even limited exposure could cause irreversible financial, physical, privacy, or reputational harm, use simulation, expert review, sandboxing, or tightly controlled pilots. Randomization does not make an unethical exposure acceptable.

3. Insufficient sample

Power calculations may show that a meaningful effect cannot be distinguished within the business horizon. Bayesian analysis does not manufacture information from tiny samples. Use stronger evidence from research, aggregate longer where stationarity is plausible, or choose a closer metric with validated linkage.

4. Severe interference

Marketplaces, social networks, logistics systems, and pricing changes can make one unit's treatment affect another unit's outcome. A user-level split may violate the no-interference assumption. DoorDash has used switchback designs that randomize time or market for marketplace questions.

5. One obvious bug fix

If behavior violates an agreed specification and the fix restores it, use software tests and a controlled rollout. You may measure business impact for learning, but do not leave known-broken behavior in control solely to create an experiment story.

6. Pure exploration among many options

When the goal is to allocate traffic toward the currently better option and cumulative reward matters more than an unbiased fixed-horizon comparison, a bandit may fit. GrowthBook distinguishes experiments, safe rollouts, and bandits by decision objective.

7. A decision too small to justify the test

Every test consumes implementation, traffic, analysis, and attention. If both options are cheap, reversible, and practically equivalent, choose one, instrument the outcome, and move on. Save controlled traffic for uncertainty that could change an important decision.

Use evidence without performing test theater

A mature experimentation culture does not test everything. It makes the measurement method proportional to uncertainty, risk, reversibility, sample, and decision value.

Document why a test was skipped and what evidence will govern the decision instead. That keeps “we cannot A/B test this” from becoming permission to rely on opinion. GrowthBook's warehouse-native experimentation supports controlled tests when they fit, while feature flags support staged delivery and rollback when the primary need is operational safety.

The best product leaders are not loyal to one method. They are loyal to answerable questions and decisions that can change.

Design for long-term learning

Hear how experienced leaders match experiment design, metrics, and shipping criteria to consequential product decisions.

Watch the Leadership Session