P-value interpretation: A step-by-step guide for analysts

Interpret a p-value as evidence under a specified null procedure, then decide with the effect size, uncertainty, and experiment context.

A dashboard shows p = 0.032. Is the treatment a winner? Not yet.

The p-value says how unusual the test statistic is under a null model and testing procedure. To turn that number into an analysis, you must reconstruct the question it belongs to: the null and alternative hypotheses, whether the test is one- or two-sided, the metric and independent unit, the stopping rule, the number of comparisons, and the assumptions used to calculate the statistic.

Then move beyond the p-value. A statistically significant effect can be too small to matter. A non-significant result can be precise evidence against any useful lift or an inconclusive test with a wide interval. A tiny p-value from a broken exposure pipeline is still a broken experiment.

This guide gives analysts a repeatable sequence for reading and reporting the number without asking it to answer questions it was not designed to answer.

The definition to keep in view

A p-value is the probability, assuming the null hypothesis and statistical model, of observing a test statistic at least as extreme as the one calculated from the data.

Symbolically:

p-value = P(test statistic at least as extreme as observed | null model)

The NIST overview of statistical tests places the p-value inside a procedure that specifies a null, test statistic, significance level, and rejection rule.

Penn State's p-value lesson likewise frames the value as a conditional probability under the null rather than a probability assigned to the hypothesis itself.

Every phrase in the definition matters:

• "Assuming the null" means the calculation does not estimate the probability the null is true.
• "Test statistic" means extremeness is defined on a chosen scale, such as z, t, or chi-square.
• "At least as extreme" includes the observed statistic and more incompatible values.
• "Model" includes assumptions about independence, outcome distribution, and sampling.
• "Procedure" includes sidedness, stopping, and multiplicity.

GrowthBook's foundational p-value explainer develops that definition. The rest of this article focuses on the analyst's interpretation workflow after a result appears.

Step 1: Restate the hypotheses

Do not interpret p = 0.032 until you know what null was tested.

For a two-sided conversion experiment:

H0: treatment conversion - control conversion = 0
Ha: treatment conversion - control conversion != 0

For a one-sided degradation guardrail:

H0: treatment error rate is no worse than control by the margin
Ha: treatment error rate exceeds the allowed margin

Those tests can use the same data and produce different p-values because they ask different questions.

Confirm that the hypotheses were defined before results. If the team saw a negative effect and then changed a "treatment improves" one-sided test into a two-sided discovery claim, the reported procedure is post hoc.

Also name the estimand in product units. "Difference in seven-day activation among eligible assigned users" is interpretable. "Coefficient beta three" is not enough for a stakeholder or future analyst to reproduce the decision.

Step 2: Confirm the test and sidedness

The p-value inherits meaning from the test statistic. Check:

• Which test or model produced it?
• What is the independent unit?
• Was variance pooled, unpooled, clustered, or adjusted?
• Was the test one- or two-sided?
• Was it fixed-horizon or sequential?

A two-proportion z-test can fit a large independent binary outcome. A Welch t-test can fit independent user-level means. Repeated observations, clustered assignment, censored outcomes, and sparse events require methods that represent those structures.

GrowthBook's z-test guide and t-test guide show how the p-value changes with the metric and design. The test name should be part of the readout, not hidden inside platform configuration.

Sidedness affects the tail area. A two-sided test counts extreme departures in either direction; a one-sided test counts one prespecified direction. Do not halve a two-sided p-value after seeing the sign.

Step 3: Check the analysis clock

A fixed-horizon p-value assumes the experiment followed its planned stopping rule. If analysts checked daily and stopped on the first favorable crossing, p = 0.032 is not calibrated as a one-time 3.2% tail probability under that operational procedure.

Determine:

• Planned sample size or information horizon.
• Actual sample at analysis.
• Number and timing of interim looks.
• Whether a valid sequential test was selected before launch.
• Whether the metric's observation window had matured.

GrowthBook's comparison of fixed-horizon and sequential testing explains why a fixed test cannot be converted into a sequential one after peeking. If the procedure was violated, rerun under a defensible method where possible and label the original result exploratory.

Safety monitoring is different. Stop a harmful release when an operational guardrail fails; do not expose more users merely to protect a planned p-value. The statistical consequence should be documented in the final inference.

Step 4: Compare p with a pre-specified alpha

Alpha is the rule's Type I error threshold under the planned null procedure. If alpha = 0.05 and p = 0.032, the result crosses that decision boundary.

Use precise language:

The result is statistically significant at the pre-specified 0.05 level under the two-sided test.

Avoid:

The hypothesis is 96.8% proven.

Alpha should reflect decision costs, not habit alone. A reversible interface test and a fraud-control change may warrant different error tolerances. Multiple confirmatory comparisons may require a stricter per-comparison threshold.

Treat the evidence as continuous even when the decision rule is binary. p = 0.049 and p = 0.051 are nearly identical evidence. The rule may make different formal decisions, but the underlying estimates and uncertainty should prevent radically different narratives.

Step 5: Read the effect estimate in natural units

A p-value has no direction or business magnitude by itself. Find:

• Control estimate.
• Treatment estimate.
• Absolute difference.
• Relative difference.

Suppose:

The observed effect is +0.8 percentage points or +8% relative. Say both. Relative lift sounds large when the baseline is small; absolute lift connects more directly to incremental activated users.

Now ask whether 0.8 points clears the minimum practical effect defined before launch. If implementation cost requires at least 1.0 point, statistical significance alone does not establish a win.

GrowthBook's guide to statistical significance separates the null threshold from practical importance.

Step 6: Interpret the confidence interval

Suppose the approximate 95% confidence interval for the activation effect is +0.1 to +1.5 percentage points. The interval excludes zero, which aligns with significance at alpha 0.05 for a corresponding two-sided test.

The interval answers the more useful scale question. Under the interval procedure and model, the data are compatible with effects from a small 0.1-point lift to a substantial 1.5-point lift. If the practical threshold is 1.0 point, the interval spans both below- and above-threshold effects.

Do not say there is a 95% probability the fixed true effect lies in one realized frequentist interval. The 95% refers to the long-run coverage of the interval-building procedure.

A review of common p-value misconceptions recommends interpreting p-values with estimates and intervals rather than treating the threshold crossing as the complete result.

Step 7: Review power and sensitivity

For a significant result, low power can contribute to exaggerated winning estimates because only unusually large observed effects cross the threshold. For a non-significant result, planned sensitivity determines whether the outcome is informative.

Compare the interval with the minimum practical effect:

• Interval excludes all useful benefit: evidence supports no decision-relevant improvement.
• Interval includes useful benefit and harm: inconclusive.
• Interval lies above the practical threshold: evidence supports a decision-relevant effect.
• Interval excludes zero but includes trivial effects: statistically detectable, not yet clearly valuable.

GrowthBook's guide to statistical power explains why a null result does not have a fixed interpretation without the alternative the test was designed to detect.

Do not compute "observed power" from the realized effect and present it as new evidence. It mainly reformulates the p-value. Use the estimate, interval, and prospective sensitivity.

Step 8: Account for multiple testing

One p-value is different from the minimum p-value among 40 metrics, 10 segments, and 4 variants. If each opportunity uses a 5% threshold, the chance of at least one false positive under a family of nulls can be much larger than 5%.

Identify:

• One pre-specified primary metric.
• All confirmatory variants and comparisons.
• Planned secondary outcomes.
• Exploratory segments and diagnostics.

Apply a family-wise or false-discovery method appropriate to the decision. R's official 0 documentation describes Bonferroni, Holm, Benjamini-Hochberg, and related corrections. The corrected value or adjusted alpha belongs in the readout.

Unplanned segment findings can still be useful. Label them exploratory and run a fresh confirmatory test rather than presenting them as if they were the only question asked.

Step 9: Check experiment and data integrity

Before giving the p-value causal meaning, verify:

• Observed allocation matches the plan.
• Units appear in only one variation.
• Exposure indicates a real opportunity to experience treatment.
• Metric definitions remained stable.
• Observation windows matured equally.
• Missingness and exclusions are balanced and documented.
• Other launches or incidents did not differentially affect groups.
• Randomization occurred at the unit analyzed.

A low p-value is compatible with a treatment effect, a biased sample, dependence, a logging bug, or a model failure. It measures incompatibility with the specified null model; it does not diagnose which assumption failed.

Sample-ratio mismatch is especially important. If a planned 50/50 split becomes 55/45, do not continue to the outcome p-value as if assignment were healthy. Investigate bucketing, eligibility, exposure logging, and missing data first.

Step 10: Integrate guardrails and make the decision

The primary metric is not the whole product. Review error rate, latency, cost, retention, complaints, or safety outcomes selected before launch.

A treatment can improve activation significantly while increasing payment failures. A p-value for activation cannot trade away that guardrail automatically. Define whether each guardrail blocks rollout, triggers a follow-up, or is informational.

Then write the decision in complete form:

Treatment increased seven-day activation from 10.0% to 10.8%, an absolute effect of 0.8 points. The two-sided fixed-horizon test returned p = 0.032 against our pre-specified alpha of 0.05. The 95% confidence interval is 0.1 to 1.5 points, so the result excludes zero but remains uncertain relative to our 1.0-point practical threshold. Assignment checks passed and guardrails were stable. We recommend a staged rollout with continued measurement rather than declaring the full expected lift established.

That statement says what the data support and what they do not.

How to interpret common p-value ranges

P-values are continuous evidence under a model, not universal verbal categories. Still, analysts often face recurring situations.

Very small p-value

p < 0.001 indicates the observed statistic is highly incompatible with the null procedure. Check effect size and data quality; enormous samples and systematic bugs can both produce tiny values.

P-value just below alpha

p = 0.049 crosses a 0.05 rule but is sensitive to assumptions, analysis choices, and random variation. Report the exact value and interval. Do not use "highly significant."

P-value just above alpha

p = 0.051 does not cross the rule. It is not proof of no effect and is not meaningfully different evidence from 0.049. Read the interval and sensitivity.

Large p-value

p = 0.70 says the statistic is not unusual under the null procedure. It does not show the null is 70% likely or prove equivalence. A narrow interval within a practical equivalence range can support a no-material-difference conclusion; the p-value alone cannot.

P-value displayed as zero

Software may round an extremely small value to 0.000. Probability is not literally zero. Report a bound such as p < 0.001 or use scientific notation at the software's precision.

Misinterpretations to remove from reports

The American Statistical Association's p-value statement directly cautions against probability-of-hypothesis, effect-size, and threshold-only interpretations.

A reusable p-value review template

For every reported p-value, record:

1. Null and alternative hypotheses.
2. Metric, population, and independent unit.
3. Test, model assumptions, and sidedness.
4. Alpha and multiple-testing family.
5. Sample-size and stopping plan.
6. Control and treatment estimates.
7. Absolute and relative effect.
8. Confidence interval.
9. Minimum practical effect.
10. Assignment, exposure, and data-quality checks.
11. Guardrail results.
12. Decision and remaining uncertainty.

This template prevents a dashboard number from becoming a conclusion without context.

Interpret the experiment, not only the tail area

A p-value is a calibrated measure of how incompatible a test statistic is with a specified null procedure. Correct interpretation starts there, but useful analysis continues through effect size, confidence interval, practical value, power, multiplicity, data quality, and guardrails.

Use the pre-specified threshold for the formal decision and treat the number itself as continuous evidence. Preserve the exact hypotheses and stopping rule. State uncertainty in the metric's natural units. When the interval spans materially different decisions, say the result is unresolved.

To run experiments with transparent frequentist, Bayesian, and sequential analysis against shared product metrics, explore GrowthBook experimentation.