Statistical power: Definition and how to increase it

Statistical power is the probability that a test will detect a specified real effect under its design and decision rule.

An underpowered A/B test can run correctly and still fail to answer the product question. If a useful treatment effect exists but the test has only a 40% chance of detecting it, most repetitions will return a non-significant result. The problem is not that the effect vanished; the design could not reliably distinguish it from noise.

Power is therefore a planning property, not a score awarded after results arrive. It depends on a specific effect size, baseline or variance, sample size, test, significance threshold, allocation, and analysis behavior. Saying "the experiment has 80% power" is incomplete unless those inputs are stated.

Product teams can improve power by collecting more independent units, reducing legitimate noise, choosing a sensitive metric, balancing allocation, and using valid covariate adjustment. They should not manufacture power by excluding inconvenient users after looking at results, weakening error controls without discussion, or replacing the real outcome with a flattering proxy.

Statistical power and Type II error

In a frequentist hypothesis test, four outcomes are possible:

Power is:

power = 1 - beta

If a test has 80% power for a two-percentage-point lift, it will reject the null in about 80% of repeated experiments where that exact lift and all planning assumptions hold. It misses the effect about 20% of the time.

Penn State's power and sample-size lesson defines the same relationship and shows how alpha, beta, effect size, variability, and sample size interact.

Power is conditional on an effect

There is no single power value for an experiment across all possible alternatives. A design can have:

• Low power for a 0.2-point conversion lift.
• 80% power for a 1-point lift.
• Nearly 100% power for a 5-point lift.

This is a power curve: detection probability as a function of the true effect. Smaller effects resemble sampling noise and require more data. Larger effects are easier to distinguish.

For planning, choose a minimum practical effect: the smallest effect that would change the product decision. GrowthBook's guide to power analysis explains how that threshold turns a vague traffic question into a sample-size calculation.

Power is not the probability a result is true

Power is calculated under a hypothetical alternative before or independently of the result. It is not:

• The probability the treatment works.
• The probability a significant result will replicate.
• One minus the p-value.
• The probability the current null result is a false negative.

Those quantities require other assumptions or analyses. A test planned at 80% power can still produce a false positive, an exaggerated winning estimate, or an inconclusive interval.

What determines statistical power

Several inputs move power together. Changing one usually changes cost or error elsewhere.

The National Academies' treatment of evaluation power summarizes the same dependency on the number and nature of units, outcome variability, measurement, and the effect the study is designed to detect.

Effect size

Large true effects are easier to detect than small ones. Teams do not control the actual treatment effect, but they control which minimum effect the design targets.

Do not set the minimum detectable effect to whatever makes the traffic estimate convenient. Start from economics and user impact. If a 0.5% relative lift would not justify implementation, there is little value in powering the test to find it. If a 0.5 percentage-point loss would be dangerous, the guardrail may need power for that smaller margin.

Sample size

More independent units reduce standard error and increase power. The important word is independent. Ten thousand pageviews from 500 assigned users do not provide the same information as 10,000 independently assigned users.

The NIST sample-size formula for proportions makes the dependence explicit: required sample grows as the target change gets smaller and as alpha and beta become more stringent.

Variability

For a fixed effect and sample size, lower variance gives a larger standardized signal. Metric noise comes from real user heterogeneity, measurement error, event definitions, outliers, and mismatched observation windows.

Reducing noise is powerful when it preserves the estimand. Removing high-value customers because their revenue is variable changes the population and may remove the treatment effect the business cares about. Improving event quality or adjusting for pre-experiment behavior can increase precision without redefining success.

Significance level

A higher alpha moves the rejection boundary closer and increases power, but raises the probability of a Type I error. A lower alpha protects more strongly against false positives and reduces power at a fixed sample size.

Choose alpha based on decision stakes and multiplicity. Do not raise it after a result lands at p = 0.07. That is changing the rules after observing the outcome.

One-sided versus two-sided testing

A one-sided test places the rejection region in one direction and has more power there at the same alpha. It is justified only when an effect in the opposite direction would be treated the same as no effect and the direction was chosen before data.

Most product changes can cause harm. A two-sided primary test is usually honest. Guardrails may use one-sided non-inferiority or harm tests when only degradation triggers action.

Allocation across variations

For two groups with similar per-unit variance and cost, equal allocation usually maximizes power for a fixed total sample. Unequal allocation can be sensible when treatment is costly, risky, capacity constrained, or when control data support several simultaneous comparisons.

The decision should be included in the power calculation. A 90/10 rollout has fewer treatment observations than a 50/50 experiment with the same total traffic.

Test and analysis method

Methods differ in efficiency. A model using pre-experiment covariates can reduce residual variance. A paired design can remove stable between-unit differences. A cluster design often loses power because outcomes within a cluster are correlated.

The method must match the design. A smaller standard error achieved by pretending correlated observations are independent is not power; it is miscalibration.

How to increase statistical power responsibly

Collect more eligible independent units

The direct option is more traffic or time. Estimate duration using only eligible units expected to reach the exposure and mature outcome. Total monthly visitors overstates the available sample if half never encounter the feature.

Run long enough to cover relevant cycles and delayed outcomes. More days are not useful if treatment changes during the run or seasonal drift makes the result irrelevant.

Improve the metric

A powerful metric is sensitive to the treatment mechanism, reliably measured, and aligned with value. Improve it by:

• Defining the denominator at the assignment unit.
• Logging exposure when the user can experience treatment.
• Using a consistent maturity window.
• Removing duplicate and invalid events through rules fixed in advance.
• Reducing missingness and timestamp errors.
• Selecting a continuous measure when it faithfully captures more information than a coarse binary threshold.

Do not optimize a metric for statistical sensitivity alone. A click proxy may move quickly while long-term retention does not. Use it as the primary outcome only when evidence supports the relationship to the decision goal.

Use pre-experiment covariates

If prior user behavior predicts the outcome and cannot be affected by treatment, regression adjustment or CUPED can explain baseline variation. The residual treatment comparison becomes more precise.

GrowthBook's experimentation platform supports CUPED and variance reduction. Pre-period definitions and covariates should be fixed without using post-assignment information; otherwise adjustment can create bias.

Stratify or post-stratify on predictive attributes

Randomization within important strata can improve balance, especially with limited samples. Post-stratification can adjust group estimates to a common population composition. Use attributes measured before treatment and specify the plan before outcome-driven exploration.

Too many sparse strata can make estimates unstable. Focus on a small number of predictors with evidence of outcome relevance.

Target users who can receive the mechanism

An experiment about a new search ranking has its effect diluted if most assigned users never search. Activation metrics or trigger-based analysis can focus inference on users who reach a pre-treatment eligibility condition.

The trigger cannot be affected by treatment. Conditioning on a post-treatment action, such as clicking the new search box, selects different populations across variations and can bias the result.

GrowthBook's guide to A/B testing LLMs discusses the same dilution problem when many assigned users never invoke an AI feature.

Reduce unnecessary variants and confirmatory metrics

Every variant divides traffic, and every confirmatory comparison creates a multiple-testing burden. Test variants that correspond to distinct, valuable hypotheses. Designate one primary metric and a limited set of guardrails.

GrowthBook's article on why high-variant testing becomes difficult explains how the family of false-positive opportunities grows. Pruning weak ideas before launch often improves the information returned per user.

Use a suitable sequential design

Sequential testing does not create more information from the same data, but it can use information more efficiently across time by allowing valid early decisions. It may stop large effects sooner or end for futility under a planned rule.

Do not repeatedly apply a fixed-horizon test. GrowthBook's comparison of fixed-horizon, Bayesian, and sequential methods emphasizes that the sequential method and stopping behavior must be selected from the start.

Strategies that only appear to increase power

Post-hoc exclusions

Removing segments, dates, or outliers because they weaken the result changes the analysis after observing outcomes. Apply data-quality rules symmetrically and before unblinding, or label the result exploratory.

Treating events as independent users

Row counts rise and standard errors shrink, but repeated events from one user remain correlated. Analyze the randomized unit or use cluster-aware methods.

Switching metrics after launch

Selecting the most significant metric among many increases false-positive risk. It does not make the experiment more sensitive to the original hypothesis.

Raising alpha quietly

Moving from 0.05 to 0.10 increases detection probability by accepting more false positives. That tradeoff may sometimes be defensible, but it is not a free efficiency improvement and must be decided before results.

Using observed post-hoc power

Calculating power from the observed effect after a completed test mostly restates the p-value and adds little. Interpret the effect estimate and confidence interval. Use prospective or design power for the next decision.

Power for guardrails, non-inferiority, and equivalence

Not every test seeks improvement over zero.

• A guardrail may need power to detect a harmful degradation.
• A non-inferiority test asks whether treatment is not worse than control by more than a margin.
• An equivalence test asks whether the effect lies within a practically negligible range.

Each needs its own margin, alpha allocation, variance, and sample size. Powering the primary conversion metric does not guarantee useful sensitivity for a rare error-rate guardrail. For critical rare events, statistical detection may be infeasible; combine narrow rollout, absolute incident thresholds, and operational monitoring.

Read null results through the planned sensitivity

A non-significant result can mean several things:

1. The effect is near zero.
2. The effect exists but is smaller than the design could reliably detect.
3. Variance or missingness was worse than planned.
4. Exposure diluted the treatment contrast.
5. The experiment stopped before the required information accrued.

Inspect the confidence interval relative to the practical threshold. If it excludes all effects worth shipping, the null result is decision-useful. If it contains meaningful benefit and harm, the test is inconclusive.

Penn State's definition of power ties it directly to a specified alternative and rejection rule. That is why "we reached 80% power" cannot be retrofitted from a dashboard badge without checking the original inputs.

A statistical power planning checklist

Before launch, document:

• Primary metric and independent assignment unit.
• Baseline or variance estimated from comparable eligible users.
• Minimum practical effect in absolute and relative units.
• Alpha, desired power, sidedness, and multiple-testing correction.
• Allocation and number of variants.
• Eligible daily traffic and data-maturity delay.
• Covariate adjustment or variance-reduction method.
• Fixed-horizon or sequential stopping rule.
• Separate sensitivity for important guardrails.
• Decision for positive, negative, and inconclusive outcomes.

Run sensitivity scenarios around baseline and variance uncertainty. The NIST sample-size guidance for means notes that sample planning necessarily depends on assumed alpha, beta, standard deviation, and detectable shift. A single point estimate can create false certainty.

Power the decision you actually need to make

Statistical power tells you how reliably a planned test detects a specified effect. It is not a universal quality score and does not certify an observed result.

Increase power by adding independent information or using existing information more efficiently: more eligible units, better measurement, balanced allocation, predictive pre-period covariates, appropriate targeting, and a design matched to the data. Preserve the actual business estimand and error tradeoffs while doing so.

To plan and analyze product experiments with shared metrics, variance reduction, and multiple statistical approaches, explore GrowthBook experimentation.