P-value explained: What it means and how to interpret it

A p-value can be calculated to four decimal places and still be interpreted completely wrong.

The most common mistake is deceptively simple: “p = 0.03 means there is a 3% chance the result happened by accident.” That sentence sounds intuitive. It is also not what the calculation says.

A p-value is a probability computed under a statistical model. It asks how surprising the observed test statistic—or something more extreme—would be if a specified null hypothesis were true. It does not tell you the probability that the null is true, the probability that the treatment works, or whether the observed effect is large enough to matter.

For product teams, this distinction is operational. A misread p-value can turn a tiny effect into a rollout mandate, an inconclusive result into a declaration of “no impact,” or a random fluctuation into a winner. This guide builds the interpretation from the hypothesis-test mechanics and then shows how to use p-values responsibly in A/B testing.

What a p-value actually means

A p-value is the probability, assuming the null hypothesis and statistical model are correct, of obtaining a test statistic at least as extreme as the statistic observed.

The NIST definition contains the three parts that every correct interpretation needs:

1. The null hypothesis is assumed for the calculation.
2. The probability refers to a test statistic, not directly to the hypothesis.
3. The tail includes the observed result and results considered more extreme.

Suppose an A/B test compares checkout completion between a control and a new design. The null hypothesis states that the underlying completion rates are equal. A suitable test converts the observed difference into a test statistic. The p-value then measures how often that null model would generate a statistic at least as far from its expected value as the one observed.

If p = 0.03, the correct interpretation is:

Under the null model and the test's assumptions, results at least this extreme would occur about 3% of the time.

That result may be evidence against the null model. It is not a 97% probability that the treatment works.

The null hypothesis is the reference model

Every p-value is attached to a hypothesis test. The null hypothesis often represents no difference, no association, or a particular parameter value. The alternative represents the departure the test is designed to detect.

For a two-sided A/B test of conversion rates:

• H0: treatment conversion equals control conversion.
• Ha: treatment conversion differs from control conversion.

The p-value depends on the observed data, the test statistic, the null model, and the sampling or randomization assumptions. Change the test direction, variance estimator, unit of analysis, or stopping rule and the p-value can change even when the headline conversion rates do not.

This is why a p-value is not a universal “chance” attached to a dataset. It is the output of a specified procedure. Penn State's p-value approach to hypothesis testing begins by defining the hypotheses and test statistic before calculating the relevant tail probability.

“At least as extreme” depends on the alternative

In a one-sided test, “more extreme” points in one pre-specified direction. In an upper-tailed test, larger statistics count against the null. In a lower-tailed test, smaller statistics do.

A two-sided test counts departures in both directions. For a symmetric z-test, a result at either +|z| or -|z| contributes to the p-value. That makes two-sided tests appropriate when both improvement and harm matter—which is true for most product experiments.

Choosing a one-sided test after seeing the direction of the result is not a legitimate shortcut. The direction must come from the decision and be set before observing outcomes.

How a p-value is calculated

The exact formula depends on the statistical test, but the workflow is stable:

1. Define the null and alternative hypotheses.
2. Choose a test whose assumptions match the outcome and design.
3. Calculate a test statistic from the observed effect and its standard error.
4. Use the statistic's null distribution to calculate a tail probability.

For two large independent samples with binary outcomes, teams often use a two-proportion z-test. If the control has x1 conversions among n1 users and treatment has x2 among n2, the observed rates are:

p1 = x1 / n1
p2 = x2 / n2

Under the null of equal rates, the test pools both groups to estimate a common rate. The observed difference is divided by its null standard error to produce a z-statistic. The NIST two-proportion test reference gives the full statistic and the one- and two-tailed p-value formulas.

Consider a simplified example:

• Control: 1,000 conversions from 10,000 users, or 10.0%.
• Treatment: 1,100 conversions from 10,000 users, or 11.0%.
• Observed absolute lift: 1 percentage point.
• Observed relative lift: 10%.

A two-proportion z-test calculates whether a one-point gap is large relative to the sampling variability expected at these rates and sample sizes. Software may return a p-value near 0.02. Under the planned null model, a difference at least this extreme would be unusual enough to cross an alpha threshold of 0.05.

The p-value alone still does not answer whether a one-point lift pays for the change or whether the experiment was valid. Those are separate questions.

P-value versus significance level

The p-value is calculated from the data. The significance level, alpha, is selected before the experiment. Alpha defines the test's long-run Type I error tolerance under the planned procedure.

If p <= alpha, the result is statistically significant at that level and the procedure rejects the null. If p > alpha, the procedure does not reject the null.

“Do not reject” is deliberate language. It does not mean “accept the null” or “prove there is no effect.” The experiment may have insufficient power to distinguish a meaningful effect from noise. GrowthBook's guide to power analysis connects that ambiguity to sample size and minimum detectable effect.

Why 0.05 is not a law of nature

The 0.05 threshold is conventional. NIST notes that 0.10, 0.05, and 0.01 are all used in practice and that choosing alpha involves judgment. The appropriate threshold depends on the cost of a false positive, the number of tests, the decision context, and the rest of the evidence.

More importantly, p = 0.049 and p = 0.051 are not fundamentally different scientific states. Treating one as truth and the other as failure discards uncertainty. Report exact p-values, effect estimates, and intervals rather than replacing the analysis with a binary label.

The American Statistical Association's statement is direct: decisions should not be based only on whether a p-value passes a threshold, and statistical significance does not measure effect size or importance.

Six things a p-value does not tell you

Most p-value failures come from asking the number to answer a question it was not designed to answer.

It is not the probability that the null hypothesis is true

The calculation assumes the null and evaluates data under that assumption. It does not assign prior probabilities to hypotheses or reverse the conditional probability.

P(data | null) is not P(null | data).

Bayesian analysis can estimate posterior probabilities for parameters or decisions, but it requires a likelihood and prior model. GrowthBook supports both Bayesian and frequentist methods, as outlined in its comparison of A/B testing methodologies. Use the language of the method actually run.

It is not the probability that chance caused the result

Random variation is part of the null model, but “chance caused the result” is not a precise hypothesis. Data can be inconsistent with the null because the treatment had an effect, because an assumption failed, because assignment was biased, because instrumentation broke, or because an unlikely sample occurred.

A small p-value tells you the data are surprising under the model. It does not diagnose why.

It does not measure effect size

Large samples can make tiny effects statistically significant. Small samples can leave large estimated effects uncertain. The same p-value can accompany very different business outcomes.

Always report the absolute difference, relative lift, and uncertainty interval. GrowthBook's article on statistical significance in experiments frames significance as a filter for noise, not a mandate to ship.

That reporting pattern is consistent with the practical recommendation to present group estimates, their difference, a confidence interval, and then the p-value rather than publishing the p-value alone, as summarized in this review of what p-values do and do not mean.

It does not measure data quality

A calculation can return p < 0.001 on biased or corrupted data. Sample ratio mismatch, inconsistent exposure logging, bot traffic, unit-of-analysis errors, and novelty effects can all invalidate the interpretation without preventing software from producing a number.

Read experiment health checks before inferential outputs. A precise answer to the wrong dataset is still wrong.

It does not prove replication

A low p-value in one experiment does not guarantee the same effect will recur. Effect estimates vary from sample to sample, and selective publication or winner reporting makes the visible evidence look stronger than the full record.

Replication depends on design quality, treatment stability, population, measurement, and accumulated evidence—not on one threshold crossing.

It does not tell you what decision to make

A rollout decision includes implementation cost, guardrail effects, downside risk, strategic value, and reversibility. A statistically significant improvement may be too small to justify maintenance. An inconclusive result may still rule out catastrophic harm or support a limited rollout.

The p-value is one input to a decision framework, not the framework itself.

How to interpret p-values in A/B testing

A good experiment readout starts with the effect, then explains uncertainty and decision criteria.

Use this order:

1. State the control and treatment metric values.
2. Report absolute and relative effect estimates.
3. Report a confidence interval.
4. Report the p-value and planned alpha.
5. Confirm sample size, stopping rule, and test assumptions.
6. Review guardrails and data-quality diagnostics.
7. Compare the plausible effect range with the minimum practical effect.

For example:

Treatment increased seven-day activation from 20.0% to 21.2%, an absolute lift of 1.2 percentage points. The 95% confidence interval for the lift is 0.3 to 2.1 points, and p = 0.02 under the planned two-sided fixed-horizon analysis. The result clears alpha = 0.05, the lower interval bound exceeds our 0.2-point minimum practical effect, guardrails were stable, and no sample-ratio issue was detected. We recommend rollout.

This language is stronger than “the treatment has a 98% chance of winning” because it does not invent a Bayesian probability from a frequentist output. It also connects the evidence to the business threshold.

GrowthBook's hypothesis-testing workflow puts the decision before the statistic: define the primary metric, evidence threshold, stopping rule, and rollout action before running the test.

Interpret a large p-value carefully

Suppose a test estimates a 1.2-point lift but returns p = 0.20. There are at least three plausible interpretations:

• The true effect is near zero.
• A real effect exists but is smaller than the design could reliably detect.
• Measurement noise or limited sample produced an imprecise estimate.

The confidence interval and power plan distinguish these cases better than the p-value alone. If the interval is narrow and excludes every effect worth shipping, the result supports a “no practically important benefit” conclusion. If the interval is wide and includes meaningful gains and losses, the result is inconclusive.

Avoid unplanned peeking

Repeatedly checking a fixed-horizon p-value creates repeated opportunities for noise to cross the threshold. Stopping when p first falls below 0.05 inflates the actual false-positive rate.

Either analyze once at the pre-specified sample or use a sequential method designed for interim monitoring. GrowthBook's discussion of statistical validity describes sequential testing as the structural solution when teams need valid early stopping.

Correct for multiple opportunities to win

An experiment with five variants, twenty metrics, and ten segments creates many possible comparisons. Even if each individual test uses alpha = 0.05, the chance of at least one false positive across the full search can become substantial.

Pre-specify one primary metric and the main comparison. Use appropriate family-wise error or false-discovery controls when multiple confirmatory tests are necessary. Treat unplanned segment discoveries as hypotheses for follow-up rather than automatic rollout evidence.

Use p-values as calibrated evidence

P-values remain useful when the experiment and language match their assumptions. They offer a standardized way to measure how incompatible data are with a null model, and they connect directly to long-run error control in a pre-specified testing procedure.

The discipline is to keep the number in its lane:

• Define the hypothesis before examining outcomes.
• Choose alpha, sample size, and stopping rules before launch.
• Use the correct experimental unit and test.
• Pair the p-value with effect size and an uncertainty interval.
• Distinguish statistical significance from practical value.
• Correct for repeated looks and multiple tests.
• Report inconclusive results honestly.

When teams follow that sequence, p = 0.03 becomes a precise statement about the data under a model—not a vague declaration that the feature “probably works.”

To analyze experiments with transparent Bayesian or frequentist methods and warehouse-defined metrics, explore GrowthBook experimentation.