Power analysis: What it is and how to perform one

An experiment can run exactly as planned and still have almost no chance of answering the question that justified it.

That happens when a team launches without checking whether its traffic, metric variance, and expected effect can produce a decisive result. The test accumulates data for weeks. The variation looks slightly better, then slightly worse. Nobody knows whether the feature had no effect or the experiment simply could not detect it.

Power analysis prevents that ambiguity from being a surprise. It connects the effect you care about, the error rates you will tolerate, and the sample you can collect. The output is not a promise that an experiment will succeed. It is a design constraint: under the assumptions you chose, this is how much information the test needs to have a specified chance of detecting a real effect.

This guide explains the mechanics, shows a practical A/B testing example, and turns the calculation into a planning workflow product and data teams can use before launch.

What power analysis tells you

Statistical power is the probability that a hypothesis test rejects the null hypothesis when a specific alternative is true. Equivalently, power is 1 - beta, where beta is the probability of a Type II error: failing to detect an effect that really exists. Penn State's statistical curriculum illustrates the relationship by comparing the null and alternative sampling distributions and measuring how often the test correctly rejects the null.

A power analysis solves for one unknown while the other design inputs are fixed. Teams most often use it to estimate required sample size, but it can answer several questions:

• How many experimental units does the test need?
• What effect size can the available sample reliably detect?
• What power will the test have at a fixed sample size?
• How does a stricter significance threshold change the required sample?
• How long will it take to accumulate enough eligible users or events?

In product experimentation, the sample-size question is usually the most operationally useful. If a checkout experiment needs 40,000 eligible users per variation and the product sees 5,000 eligible users per week, the team can estimate an eight-week minimum collection period before launch. That estimate can expose an infeasible test while there is still time to change the metric, population, effect threshold, or product plan.

This is why power belongs in experiment design, not in the final analysis. GrowthBook's guide to A/B testing places power analysis before the test starts: both Bayesian and frequentist workflows benefit from understanding what sample is required to detect the effect the team cares about.

Power does not measure the probability that your hypothesis is true

An 80% powered test does not mean there is an 80% chance the treatment works. It means that if the true effect equals the effect used in the calculation, the planned statistical procedure will detect it about 80% of the time over repeated experiments.

That conditional wording matters. Power depends on an assumed effect. A test designed for a five-percentage-point increase may have strong power for a five-point effect and weak power for a one-point effect. The calculation describes the sensitivity of a design, not the truth of a product hypothesis.

Power also does not guarantee a statistically significant result. Even a well-powered experiment can be inconclusive. At 80% power, a test still has a 20% Type II error rate for the assumed alternative. The point is to make that risk explicit and acceptable before spending traffic and engineering time.

Power connects practical and statistical significance

Teams often choose an effect size by asking what lift they hope to see. That is the wrong starting point. The better question is: what is the smallest effect that would change the decision?

This minimum detectable effect, or MDE, turns business judgment into a statistical input. If a 0.2% relative lift would never repay implementation and maintenance costs, powering a test to detect it wastes traffic. If a 1% decline in payment completion would create unacceptable financial or customer harm, the experiment may need enough power to detect a change that small.

The MDE therefore acts as a contract between product value and statistical sensitivity. GrowthBook's article on statistical validity demonstrates the time-dependent tradeoff: early in a test, only large effects are detectable; as the sample grows, the MDE shrinks.

The four inputs that drive statistical power

For a standard two-group test, four quantities determine one another: effect size, significance level, power, and sample size. Fix any three and you can solve for the fourth.

These inputs are not interchangeable business preferences. Each represents a different risk or resource constraint.

Effect size must match the metric

Effect size can be expressed as an absolute difference, relative lift, or standardized difference. The calculation must match the outcome and statistical test.

For a binary conversion metric, moving from 10% to 11% is a one-percentage-point absolute change and a 10% relative lift. For a continuous metric such as revenue per user, power depends on the difference in means relative to the metric's variance. For a count or ratio metric, a normal approximation may not fit, especially with heavy tails or many zeros.

Libraries encode those distinctions. For example, the official 0 proportion effect-size function transforms two proportions for use in a normal-approximation power calculation. A t-test power function expects a standardized mean difference instead. Feeding a relative percentage lift directly into the wrong function can produce a precise but meaningless sample size.

Alpha and power represent different errors

Alpha controls Type I error: declaring an effect when the null model is true. Beta controls Type II error: missing the specified effect when it is real. Reducing one error without changing anything else usually increases the other. Penn State's sample-size treatment makes the escape hatch explicit: increase the sample if you want both a low alpha and a low beta.

Common planning defaults are a two-sided alpha of 0.05 and power of 0.80. They are conventions, not laws. A test governing a low-risk interface preference may tolerate different tradeoffs than a test affecting fraud detection, loan approvals, or medication adherence. The important practice is to choose the thresholds before observing results and justify them based on the decision.

Sample size means independent experimental units

Power calculations operate on independent units, not arbitrary rows. If users are randomized but the dataset contains sessions, clicks, or purchases, counting every event as a separate sample inflates the apparent information. The experiment may have one million events but only 10,000 independently assigned users.

The distinction becomes more important in cluster-randomized designs. If a team randomizes companies, classrooms, or geographic regions, observations within each cluster tend to be correlated. The effective sample size is closer to the number and diversity of clusters than the raw event count. GrowthBook's guide to experimental units explains why the entity receiving treatment must match the unit used for inference.

How to perform a power analysis before an A/B test

A useful power analysis is less about typing numbers into a calculator and more about defending the inputs. The following workflow works for a conventional two-variation product experiment.

1. State the decision and hypothesis

Write down what the team will do if the treatment wins, loses, or remains inconclusive. Then specify the primary metric and the population eligible for the change.

Suppose a team is testing a new onboarding sequence. The primary outcome is activation within seven days. The existing activation rate is 20%. The team would ship the sequence if it improves activation by at least two percentage points without harming retention or support contacts.

That produces a clear planning statement:

• Control rate: 20%.
• Smallest worthwhile treatment rate: 22%.
• Absolute MDE: 2 percentage points.
• Relative MDE: 10%.
• Randomization unit: user.
• Test: two-sided comparison of two independent proportions.

The two-sided choice allows the experiment to detect a meaningful decline as well as an improvement. A one-sided test can require less sample, but it is only defensible when effects in the untested direction genuinely would not affect the decision.

2. Estimate the baseline and variance from relevant history

Use recent data from the same metric definition, eligibility rule, and unit of analysis. A sitewide conversion rate is a weak baseline for an experiment shown only to new mobile users. A revenue metric from holiday traffic may not represent a test planned for February.

Pull enough history to see weekly patterns and estimate the eligible population. Check whether metric variance changes substantially across segments or time. If the planned treatment affects a new surface with no direct history, use the closest analog and run a sensitivity analysis instead of pretending the estimate is exact.

This is one advantage of warehouse-connected planning. GrowthBook's experimentation product can use existing metric definitions and historical data rather than making a team re-create its measurement logic in a disconnected calculator.

3. Choose alpha, desired power, and allocation

For a conventional planning scenario, the team might choose:

• Two-sided alpha: 0.05.
• Desired power: 0.80.
• Allocation: 50% control and 50% treatment.

An equal split generally maximizes power for a fixed total sample when observations have similar cost and variance in both groups. Unequal allocation may be justified when treatment exposure is risky or expensive, but the total required sample will usually increase.

If the experiment tracks many primary outcomes or variations, adjust the design for multiple testing. A sample-size calculation for one clean comparison does not automatically protect a dashboard containing twenty chances to find a favorable result.

4. Calculate sample size with the correct test

For two proportions, software typically uses a normal approximation or a closely related method. For two means, the calculation uses the expected standard deviation and mean difference. Paired tests, cluster designs, survival outcomes, and noninferiority tests need their own procedures.

Using a standard two-sided calculation for the onboarding example—20% control, 22% treatment, alpha 0.05, 80% power, and equal groups—produces a requirement of roughly 6,500 users per variation, or about 13,000 total. Exact results vary slightly by method and continuity correction, which is why the team should record the software, function, and assumptions used.

The code can be short:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.22, 0.20)
n_per_group = NormalIndPower().solve_power(
    effect_size=effect,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative="two-sided",
)

print(round(n_per_group))

Treat that output as the start of planning, not the end. Confirm that the approximation fits the metric, units are independent, and expected counts are large enough. The broader SAS guide to power and sample-size analysis recommends what-if analysis because required sample size can be highly sensitive to uncertain inputs.

5. Convert the sample into a realistic duration

If 4,000 eligible users enter onboarding per week, a 13,000-user test might appear to need 3.25 weeks. Real duration planning should also account for:

• Traffic allocated to the experiment.
• Users excluded by targeting or activation rules.
• Delayed outcomes, such as seven-day activation.
• Weekly seasonality and minimum exposure cycles.
• Data latency and validation time.
• Expected sample loss from bot filtering or missing identifiers.

If only 80% of eligible users trigger the experiment and the metric needs seven days to mature, the calendar duration extends beyond the simple traffic division. Record both the assignment target and the earliest responsible analysis date.

Reddit's engineering team describes a similar sequence in its ads experiment process: define the hypothesis and audience, conduct pre-experiment power analysis, and use it to determine minimum duration before validating the setup with an A/A test.

6. Run sensitivity scenarios

Effect size is usually the least certain input. Calculate at least three scenarios:

• Optimistic: the lift the strongest evidence supports.
• Expected: the team's best defensible planning estimate.
• Conservative: a smaller effect that would still matter.

The output may reveal that a 10% relative lift is detectable in three weeks, while a 5% lift requires twelve. That is valuable product information. It forces the team to decide whether the smaller effect is worth waiting for and whether the experiment surface has enough traffic.

GrowthBook's analysis of traffic requirements for AI features recommends this scenario-based approach because variance and effect estimates are often uncertain. The lesson applies beyond AI: a single assumed lift hides how fragile the plan is.

Common power analysis mistakes

The calculation is easy to automate. The failure modes are mostly conceptual.

Choosing the effect after seeing the result

An observed effect is a noisy estimate of the population effect. Plugging it into a post-hoc power calculation largely recycles the same information already expressed by the p-value. It can also make a lucky, exaggerated estimate look like evidence that the study was well designed.

After an experiment, report the effect estimate and confidence or credible interval. Ask whether the interval excludes effects that would change the decision. Do not use observed power to retroactively certify the design.

Treating “not significant” as “no effect”

An underpowered test that fails to reject the null does not show equivalence. It may mean the effect is zero, smaller than the MDE, or simply obscured by noise. GrowthBook's explanation of experimental probability frames sample size as a design input precisely because an inconclusive result otherwise leaves these possibilities tangled together.

If the business question is whether two experiences are sufficiently similar, use an equivalence or noninferiority design with an explicit margin. A superiority test that fails is not a substitute.

Using the wrong baseline or denominator

Powering on pageviews when users are randomized, using all-site traffic for a targeted experiment, or using overall conversion for a rare segment can understate duration dramatically. Align the historical query with the actual exposure and analysis population.

Also distinguish assigned users from activated users. If many assigned users never encounter the changed feature, intent-to-treat dilution can shrink the observed effect and increase the sample needed. Decide how activation will be handled before launch rather than filtering opportunistically after results arrive.

Ignoring metric variance and clustering

Binary conversion calculators are convenient, but not every product metric is binary. Revenue per user is often skewed. Repeated observations within a user are correlated. Teams and accounts create clusters. A generic calculator that assumes independent, normally distributed observations may be optimistic.

Simulation is often the better approach for complex metrics. Generate synthetic assignments from historical units, inject plausible treatment effects, apply the planned estimator, and measure how often the decision rule detects the effect. This preserves more of the real metric distribution and analysis pipeline than a closed-form approximation.

Peeking without a sequential design

A fixed-horizon calculation assumes the team follows the planned analysis rule. Repeatedly checking a conventional p-value and stopping as soon as it crosses 0.05 changes the false-positive behavior. Reaching the sample target eventually does not repair a stopping rule that depended on interim results.

If continuous monitoring is operationally important, use a valid sequential method and power the actual decision procedure. GrowthBook supports sequential testing alongside Bayesian and frequentist analysis, but the method still needs a pre-specified decision policy. “Stop when the chart looks convincing” is not a statistical design.

How to improve power when traffic is limited

“Get more users” is only one way to improve power. Teams can often make the signal cleaner.

First, choose one primary metric closely connected to the treatment. A noisy or distant business outcome may be important as a guardrail but inefficient as the main decision metric. Second, improve instrumentation and assignment integrity. Missing exposures and inconsistent identifiers add noise that no formula can remove.

Third, reduce unexplained variance. Covariate adjustment can use pre-experiment behavior that predicts the outcome. CUPED, for example, adjusts the analysis using correlated pre-period data. Matched or blocked designs can help in settings where strong confounders are known, although they change the analysis and must be planned correctly. GrowthBook's guide to matched-pairs experiments explains why design-based balance can improve precision for smaller samples.

Fourth, test a larger product difference. When traffic cannot support detecting a tiny lift, the rational response may be to stop testing cosmetic changes and develop a stronger hypothesis. Power analysis is useful even when it says “do not run this test.” It prevents a low-information experiment from consuming weeks of attention.

Finally, narrow the decision. If the team cannot power every segment interaction, pre-specify the population and effect that matter most. Exploratory segment analysis can generate future hypotheses, but it should not quietly redefine the original success criterion.

Make power part of the experiment contract

A power analysis should leave behind a short, reviewable record:

• Primary metric and exact definition.
• Experimental unit and eligible population.
• Baseline estimate and historical window.
• Minimum detectable effect in absolute and relative terms.
• Alpha, target power, test direction, and allocation.
• Required sample per variation.
• Expected enrollment rate and minimum duration.
• Statistical test or simulation procedure.
• Sensitivity scenarios and major assumptions.
• Planned handling of interim monitoring and delayed outcomes.

That record turns “we need more data” from a vague reaction into a decision the team made before launch. It also makes disagreements productive. Product can challenge whether the MDE is commercially meaningful. Data can challenge the variance estimate or test choice. Engineering can challenge whether the eligible traffic estimate matches actual exposure.

Power analysis will not rescue a weak hypothesis, broken instrumentation, or biased assignment. It does something more foundational: it tells you whether the experiment you designed has a reasonable chance of answering its own question.

If your team wants planning and analysis to use the same warehouse-defined metrics, explore GrowthBook experimentation.