What is statistical significance? Definition and how to calculate it

“Statistically significant” is an evidence threshold, not a synonym for important, true, or ready to ship.

In a conventional hypothesis test, a result is statistically significant when it would be sufficiently unusual under a specified null model to cross a threshold chosen before the data were examined. The calculation can help distinguish signal from sampling variation. It cannot determine whether the effect is valuable, whether the experiment was implemented correctly, or whether the same result will replicate.

Product teams need both parts of that statement. Without significance testing, random fluctuations can look like product wins. Without practical and design context, a tiny or invalid effect can wear the label “significant” and receive more confidence than it deserves.

This guide explains the frequentist definition, walks through a two-proportion A/B test, and shows how significance fits into a responsible shipping decision.

Statistical significance starts with a null model

A statistical test begins with two competing statements:

• The null hypothesis (H0) describes the reference condition, often no treatment effect or no difference between groups.
• The alternative hypothesis (Ha) describes the departure the test is designed to detect.

Suppose a product team changes an onboarding flow. The primary metric is seven-day activation. A common two-sided setup is:

H0: treatment activation rate = control activation rate
Ha: treatment activation rate ≠ control activation rate

The experiment produces an observed difference. The test asks whether that difference is large relative to the variation expected if the null were true. The NIST hypothesis-testing overview describes this as a quantitative procedure for deciding whether the evidence is sufficient to reject a conjecture about a process.

Alpha defines the rejection rule

Before collecting data, the team chooses a significance level, alpha. Alpha controls the long-run probability of a Type I error under the planned test: rejecting the null when it is true.

At alpha = 0.05, a correctly calibrated procedure rejects a true null about 5% of the time over repeated uses, assuming the model and testing rules hold. This does not mean every significant result has a 5% probability of being false. It describes the behavior of the procedure under the null.

The distinction between alpha, beta, and power is easier to see in Penn State's power-analysis treatment, which shows how the rejection region overlaps the null and alternative sampling distributions.

Values such as 0.05 and 0.01 are conventions. The threshold should reflect the cost of a false positive, the number of comparisons, and the decision. A reversible UI change and a change to fraud controls may deserve different evidence standards.

The p-value measures compatibility with the null

After collecting data, the test returns a p-value: the probability, under the null model, of obtaining a test statistic at least as extreme as the one observed. If p <= alpha, the result is statistically significant at the selected level.

GrowthBook's p-value guide covers the interpretation in detail. The essential point is that a p-value conditions on the null. It does not calculate the probability that the null hypothesis is true.

The distinction matters because significance is always relative to a model and procedure. A low p-value may reflect a treatment effect, an assumption failure, biased assignment, a data pipeline defect, or an unusually extreme sample. Statistical significance is evidence to investigate and interpret, not a diagnosis of causality by itself.

How to calculate statistical significance

The appropriate calculation depends on the outcome and experiment design. A test of two proportions works for a large-sample comparison of binary outcomes such as activated/not activated. A t-test may fit a continuous outcome. Paired observations, clusters, survival outcomes, and count metrics require methods that represent those structures.

The general workflow is consistent:

1. Define the null and alternative hypotheses.
2. Choose alpha before viewing results.
3. Select a test that matches the metric, assignment unit, and design.
4. Calculate the effect estimate and its standard error.
5. Convert the standardized effect into a test statistic.
6. Calculate the p-value from the statistic's null distribution.
7. Compare p with alpha.

Example: two conversion rates

Imagine an onboarding A/B test with these results:

The observed absolute effect is 0.22 - 0.20 = 0.02, or two percentage points. The relative lift is 0.02 / 0.20 = 10%.

For the null hypothesis that both population rates are equal, a two-proportion z-test pools the observed rates:

pooled rate = (2,000 + 2,200) / (10,000 + 10,000) = 0.21

The null standard error is:

SE = sqrt(0.21 × 0.79 × (1/10,000 + 1/10,000))

Dividing the two-point difference by that standard error yields the z-statistic. A two-sided p-value includes equally extreme deviations in either direction. The NIST two-proportion reference provides the full formula and rejection regions.

For these numbers, the p-value is well below 0.05, so the difference is statistically significant under the planned two-sided test. But the analysis is not finished. The team must still ask whether two points is valuable, whether the confidence interval excludes harmful or trivial effects, whether assignment was clean, and whether guardrails remained stable.

Confidence intervals provide the missing scale

A p-value compares the data with a null value. A confidence interval shows a range of effect sizes compatible with the data under the interval procedure.

For a two-sided test at alpha 0.05, a 95% confidence interval that excludes zero corresponds to statistical significance at the same level. The interval is usually more useful for decisions because it displays magnitude and precision. An estimated lift of two points with a tight interval from 1.2 to 2.8 points supports a different decision from the same point estimate with an interval from -1 to 5 points.

Report the interval in the metric's natural units. “Activation increased by 2 percentage points” is easier to evaluate than a standardized test statistic.

When reporting a test, include the group estimates, their difference, the confidence interval, and then the p-value. A review of common p-value misinterpretations recommends this fuller presentation instead of treating the threshold as the result.

Statistical significance is not practical significance

Statistical significance asks whether the data are sufficiently inconsistent with a null model. Practical significance asks whether the effect is large enough to matter.

These questions diverge at scale. If a product has tens of millions of observations, a 0.02 percentage-point change may produce a very small p-value. The effect can be statistically detectable while being too small to justify engineering work, user disruption, or operational complexity.

The American Statistical Association's p-value statement warns that statistical significance does not measure effect size or importance and that decisions should not depend only on crossing a threshold.

Define a minimum practical effect before launch

The remedy is to define the smallest effect that would change the decision. In experiment planning, this is often called a minimum detectable effect or minimum practical effect, depending on whether the focus is statistical sensitivity or business relevance.

Suppose a new ranking service adds infrastructure cost. The team determines that activation must increase by at least 0.5 percentage points to cover the cost and complexity. A statistically significant 0.1-point lift is not a win under that decision rule.

Define the threshold before results arrive. Otherwise the team can rationalize any observed lift as important after the fact. GrowthBook's guide to power analysis explains how the practical threshold connects to sample size and the probability of detecting a real effect.

Read significance and effect size together

An experiment can land in four broad states:

This framework prevents “not significant” from being translated automatically into “no effect.” A low-powered experiment can miss a valuable effect. Conversely, an enormous experiment can detect a negligible one.

What can make a significant result invalid

The threshold only has its advertised meaning when the design and analysis rules hold.

Peeking at fixed-horizon results

If a team repeatedly checks a conventional p-value and stops the first time p falls below 0.05, it gives random variation many chances to trigger a win. The actual false-positive rate becomes larger than the nominal alpha.

Use the pre-calculated fixed sample and analyze once, or use a valid sequential method. GrowthBook's statistical-validity guide describes sequential testing as the structural answer when continuous monitoring and early decisions are operational requirements.

Testing many metrics, segments, and variants

At a 5% threshold, one isolated test has a controlled false-positive rate. Twenty independent opportunities to win create a much larger probability that at least one crosses the line by chance. Selecting only the favorable metric after seeing results is p-hacking even when every individual calculation is correct.

Pre-specify one primary decision metric. Use suitable multiple-testing corrections for confirmatory secondary comparisons, and label exploratory segment findings as hypotheses for future tests. GrowthBook's discussion of why high-variant experimentation requires stronger controls illustrates how variant proliferation changes the error rate.

Violating the experimental unit

If assignment occurs by user but analysis treats pageviews as independent, the sample size is overstated. Repeated observations from the same person share behavior and exposure. Clustered assignment introduces the same issue at the account, classroom, or geographic level.

Count and analyze the independent units that received treatment. GrowthBook's guide to experimental units explains why the randomization and inference levels must align.

Ignoring data-quality failures

A low p-value does not certify instrumentation. Sample ratio mismatch, delayed events, missing exposure logs, inconsistent identifiers, and changed metric definitions can produce convincing output from invalid inputs.

Before interpreting significance, confirm:

• Actual allocation matches the planned split.
• Units appear in only one variation.
• Exposure represents a real opportunity to experience treatment.
• Metric definitions stayed stable.
• Data windows matured equally across variations.
• Incidents and product changes were documented.

An A/A test—two identical experiences assigned as if they were different—can reveal systematic problems before a real experiment. Even a healthy A/A program will occasionally produce significance at alpha 0.05; the calibration is about the long-run distribution and suspicious patterns across many metrics.

Statistical significance across testing methods

The familiar p <= alpha rule belongs to frequentist hypothesis testing. Other methods express evidence differently.

Fixed-horizon frequentist testing

The team chooses sample size, metric, alpha, and stopping rule in advance, then analyzes once at the horizon. Results use p-values and confidence intervals. This method is transparent and well understood, but it requires discipline about peeking.

Sequential frequentist testing

Sequential methods adjust boundaries to allow interim looks while preserving error guarantees. They are appropriate when teams will monitor results and may stop early. The output may still include p-values, but those values come from the sequential procedure rather than a fixed-horizon test applied repeatedly.

Bayesian testing

Bayesian analysis combines a prior and likelihood to produce a posterior distribution. It can answer questions such as the probability treatment beats control or the probability of harm under the model. A posterior probability threshold is not a p-value and should not be described as “95% significance.”

GrowthBook compares these options in its A/B testing methodology guide. The right method depends on how the organization plans to monitor, communicate, and decide—not on which method makes the current result look strongest.

Turn significance into a decision-ready readout

A useful experiment summary should make the statistical claim auditable and the product decision understandable. Include:

• The pre-specified hypothesis and primary metric.
• Control and treatment values.
• Absolute and relative effects.
• A confidence or credible interval.
• P-value and alpha for frequentist tests.
• Sample size, power target, and stopping rule.
• Minimum practical effect.
• Guardrail results and data-quality checks.
• Planned decision for win, loss, or inconclusive outcome.

Avoid “Variant B is significant, so we should ship.” A better readout is:

Treatment increased activation by 2.0 percentage points, from 20.0% to 22.0%. The 95% confidence interval excludes zero and the two-sided p-value is below our pre-specified alpha of 0.05. The full plausible range exceeds our 0.5-point practical threshold, assignment checks passed, and guardrails were stable. We recommend rollout.

That statement makes significance one link in the chain. It does not ask the p-value to stand in for effect size, business value, or experiment integrity.

Treat significance as a filter, not a verdict

Statistical significance is valuable because it disciplines how teams respond to noisy samples. It defines what counts as sufficiently unusual under a null model and connects a decision rule to a controlled long-run error rate.

Its limits are equally important:

• It does not prove the alternative.
• It does not measure the size or value of an effect.
• It does not repair biased assignment or bad instrumentation.
• It does not turn an exploratory finding into a confirmatory one.
• It does not make an underpowered null result equivalent to no effect.

Set the threshold before launch, calculate it with a method that matches the design, and interpret it beside effect size, uncertainty, practical value, and data quality. That is what turns statistical significance from a dashboard badge into useful evidence.

To run warehouse-native experiments with Bayesian, frequentist, and sequential analysis, explore GrowthBook experimentation.