Z-test explained: When and how to use it in A/B testing

A z-test asks whether an observed difference is large relative to the sampling noise expected under a null hypothesis.

In A/B testing, the phrase usually refers to a two-proportion z-test: a method for comparing a binary outcome such as converted/not converted between independently assigned control and treatment groups. The test converts the observed difference into a z-score and uses the standard normal distribution to calculate a p-value.

The arithmetic is compact. Choosing and interpreting the test requires more care. The outcome must match the method, observations must be independent at the assignment unit, each group needs enough successes and failures for the normal approximation, and the team must follow a valid stopping rule. Even then, the p-value answers a narrow question; it does not say whether the effect is large enough to matter or whether the product should ship.

This guide focuses on the two-proportion z-test because that is its most common product-experiment use. It also explains other z-tests and when a different method is more appropriate.

What a z-test measures

A z-test belongs to frequentist hypothesis testing. The analyst specifies:

• A null hypothesis, usually no population difference.
• An alternative hypothesis, such as some difference in either direction.
• A significance level, alpha, selected before results are viewed.
• A test statistic whose distribution under the null is known or well approximated.

The generic idea is:

z = (observed estimate - null value) / standard error under the null

The numerator is the effect observed in the sample. The denominator is the amount of sample-to-sample variation the null model expects. A z-score near zero means the estimate is close to the null relative to its noise. A large positive or negative score means the estimate is more unusual under that model.

For a two-sided test, the p-value is the probability, assuming the null model, of observing a z-score at least as far from zero in either direction. GrowthBook's p-value guide covers the interpretation: it is not the probability that the null is true, nor the probability that the result occurred "by chance."

The standard normal reference

The z statistic is compared with a standard normal distribution, which has mean zero and standard deviation one. At alpha 0.05 in a two-sided test, the familiar critical values are about -1.96 and 1.96. A score beyond either boundary has a two-sided p-value below 0.05.

That threshold is a decision convention, not a law of nature. A reversible interface change and a safety-critical workflow may warrant different error tradeoffs. Define alpha alongside the practical effect and costs of false-positive and false-negative decisions.

The two-proportion z-test for A/B testing

Suppose an experiment compares activation rates:

H0: p_treatment - p_control = 0
Ha: p_treatment - p_control != 0

Let:

• x_c and x_t be the numbers of successes.
• n_c and n_t be the assigned units with mature outcomes.
• p_hat_c = x_c / n_c and p_hat_t = x_t / n_t be the sample rates.

Under the null that the population rates are equal, both groups estimate one common rate. The test uses the pooled estimate:

p_pooled = (x_c + x_t) / (n_c + n_t)

The null standard error is:

SE_null = sqrt(
  p_pooled * (1 - p_pooled) * (1/n_c + 1/n_t)
)

The z statistic is:

z = (p_hat_t - p_hat_c) / SE_null

The NIST two-proportion reference derives this normal approximation, and Penn State's treatment shows why the pooled proportion belongs in the null standard error.

Worked A/B test example

Imagine an onboarding test with these results:

The treatment effect is 1.5 percentage points, or a 15% relative lift over control.

The pooled rate is:

p_pooled = (500 + 575) / (5,000 + 5,000) = 0.1075

The null standard error is approximately:

SE_null = sqrt(0.1075 * 0.8925 * (1/5000 + 1/5000))
        = 0.00620

Then:

z = (0.115 - 0.100) / 0.00620
  = 2.42

A two-sided z-score of about 2.42 corresponds to a p-value near 0.016. At a pre-specified alpha of 0.05, the result is statistically significant.

In Python, the official statsmodels API exposes this calculation through 0. A production analysis should still preserve the inputs, alternative hypothesis, and experiment plan rather than treating a library return value as a complete readout.

That conclusion means the observed difference crossed the chosen evidence threshold under the null model and planned procedure. It does not establish that the exact lift will repeat, that the treatment caused no harm, or that a 1.5-point change justifies rollout.

Add a confidence interval

For decisions, calculate an interval for the effect. A conventional large-sample confidence interval for the difference in two proportions uses an unpooled standard error because it estimates the difference rather than imposing equality under the null:

SE_effect = sqrt(
  p_hat_c * (1 - p_hat_c) / n_c
  + p_hat_t * (1 - p_hat_t) / n_t
)

For this example, the approximate 95% interval is 0.29 to 2.71 percentage points. It excludes zero, matching the two-sided test at alpha 0.05, and it shows the range of effect sizes the data support under the procedure.

Compare the entire interval with the minimum practical effect. If the team needs at least a two-point increase, this result is statistically significant but does not yet establish that the effect clears that bar. GrowthBook's statistical significance guide explains why evidence against zero and product value are separate questions.

When a z-test is appropriate

The metric name is not enough. Check the outcome, design, sample size, and analysis behavior.

Binary outcome

A two-proportion z-test fits one success/failure outcome per independent unit: activated, converted, retained, clicked, or did not. Define success and the observation window before the experiment starts.

A conversion rate built from events rather than independent users may violate this structure. If one user can contribute many denominator events, those observations share behavior. Either aggregate to one value per assignment unit or use a model that handles repeated observations.

Independent groups and units

Control and treatment must be separate at the unit being analyzed. A user assigned to treatment should not also appear in control. If randomization occurs by account but analysis treats members as independent, the test overstates the effective sample size.

Network effects and marketplace interactions can also create dependence across groups. A seller-side change may alter buyer outcomes in both variations. Cluster randomization or a marketplace-specific design may be necessary.

GrowthBook's guide to experimental units shows how assignment, exposure, and inference need to line up.

Large enough success and failure counts

The test relies on a normal approximation to binomial sampling. A common rule checks for at least 10 successes and 10 failures in each group. Penn State's two-independent-proportions guide recommends an exact or resampling method when that condition fails.

The issue is especially important for rare conversions. A test can have thousands of users but only three fraud events or two enterprise upgrades. The total sample looks large while the outcome information remains sparse.

A valid sampling and stopping procedure

The formula assumes the analysis follows the plan. Repeatedly checking a fixed-horizon z-test and stopping when p < 0.05 gives random variation multiple opportunities to produce a false win.

Either analyze once at a sample size planned through power analysis, or use a sequential method designed for continuous monitoring. Do not repair optional stopping by simply waiting another day after an unfavorable result.

Different types of z-tests

"Z-test" describes a family, so identify the parameter and sampling structure.

For product experiments, two-proportion z-tests are far more common than mean z-tests with known population variance. A t-test is typically used for means when the population variance is unknown. GrowthBook's t-test versus z-test guide covers that choice in detail.

When not to use a basic z-test

A different method is warranted when the data or design do not match the approximation.

Sparse binary outcomes

For small counts or extreme rates, consider Fisher's exact test, an exact binomial method, or an appropriate model. The normal approximation can produce unreliable p-values and confidence intervals near zero or one.

Continuous, skewed, or heavy-tailed metrics

Revenue per user, latency, session duration, and order value are not binary. Their distributions can be skewed or dominated by a few observations. Depending on the estimand and sample size, use a t-based method, bootstrap, robust estimator, regression, or transformation. Do not convert a continuous metric into success/failure merely to use a familiar test unless that threshold is itself the business outcome.

Paired and repeated data

Pre/post measurements on the same users are paired. Multiple observations from one unit are correlated. A basic independent-groups z-test ignores that structure. Use paired methods, aggregate at the randomized unit, or fit a model with suitable standard errors.

Multiple metrics, variants, and segments

Testing many opportunities to win increases the chance that at least one looks significant by accident. Define one primary comparison and use multiple-testing controls for confirmatory secondary analyses. Treat unplanned segment discoveries as hypotheses to test later.

Nonrandomized comparisons

A z-test can quantify a difference between two samples, but it cannot create causal identification. If treatment assignment follows geography, user choice, device, or rollout timing, confounding may explain the result. Use randomized assignment when possible or a defensible quasi-experimental design when it is not.

Common z-test interpretation mistakes

"P = 0.016 means a 1.6% chance the null is true"

It does not. The calculation assumes the null and asks how extreme the statistic is under that assumption. It does not assign a probability to the hypothesis.

"Significant means important"

Large experiments can detect tiny effects. Report absolute and relative lift, the interval, and the practical threshold. The American Statistical Association's statement warns against using statistical significance as a measure of effect size or importance.

"Not significant means no effect"

A wide interval may include both meaningful benefit and harm. That experiment is inconclusive, not proof of equivalence. To support "no important difference," define an equivalence or non-inferiority margin and use a corresponding procedure.

"The test validates the experiment"

Correct arithmetic cannot fix sample-ratio mismatch, inconsistent exposure, missing events, changed metrics, or interference. Run assignment and data-quality checks before interpreting the z-score.

Report the test as part of a decision

A useful A/B test readout contains:

• Hypothesis, primary metric, and assignment unit.
• Control and treatment counts and rates.
• Absolute and relative effect.
• Confidence interval.
• Test statistic, p-value, alpha, and sidedness.
• Sample-size plan and stopping rule.
• Minimum practical effect.
• Guardrail and data-quality results.
• Decision and follow-up.

For the worked example, a concise statement is:

Activation increased from 10.0% to 11.5%, an absolute effect of 1.5 percentage points. The two-proportion z-test returned z = 2.42 and a two-sided p-value of about 0.016. The approximate 95% confidence interval is 0.29 to 2.71 points. The result is statistically significant at our pre-specified 0.05 level, but the interval still includes effects below our two-point practical target.

That conclusion is more informative than "treatment won." It states what the test found and what remains unresolved.

Use the z-test when the design earns it

The two-proportion z-test is useful because it turns a difference in binary rates into a calibrated measure of evidence under a null model. It is fast, transparent, and appropriate for many large randomized product experiments.

Its reliability comes from the design around the formula: independent assignment, sufficient counts, stable metrics, valid stopping, and honest interpretation. Pair the p-value with effect size, uncertainty, practical value, guardrails, and data-quality checks.

To analyze product experiments with transparent metrics and Bayesian, frequentist, or sequential methods, explore GrowthBook experimentation.