How to calculate a p-value in an experiment

Calculating a p-value means converting an observed result into a test statistic, then measuring the appropriate tail area under its null distribution.

There is no universal formula that takes a difference and returns a p-value. The calculation depends on the hypothesis, outcome, experiment design, and test. A difference in conversion rates may use a two-proportion z statistic. A difference in user-level means may use Welch's t statistic. A clustered marketplace test, rare event, or repeatedly monitored experiment needs a method that represents those features.

The general workflow is consistent:

1. Define the null and alternative hypotheses.
2. Select a test statistic and its null distribution.
3. Calculate the observed effect and standard error.
4. Standardize the effect into z, t, chi-square, or another statistic.
5. Calculate the probability of values at least as extreme under the null.
6. Interpret the p-value with alpha, effect size, uncertainty, and the test procedure.

The arithmetic is the last mile. Most serious errors happen earlier: wrong unit, wrong tail, optional stopping, or a test that does not fit the data.

The general p-value calculation

Let T be a test statistic and t_obs its observed value. For a right-sided alternative:

p = P(T >= t_obs | H0)

For a left-sided alternative:

p = P(T <= t_obs | H0)

For a symmetric two-sided test:

p = 2 * P(T >= abs(t_obs) | H0)

This last shortcut only applies when the null distribution is symmetric and the chosen two-sided definition uses equal tails. For discrete exact tests, "as extreme" needs a specific ordering and is not always twice a one-sided probability.

The NIST hypothesis-testing overview shows the connection among hypothesis, statistic, critical region, and significance level.

CDF and survival function

A cumulative distribution function gives P(T <= t). The upper-tail probability is:

P(T >= t) = 1 - CDF(t)

Statistical software often provides a survival function directly. For very extreme values, sf(t) can be numerically more accurate than subtracting a CDF near one.

Step 1: Write the hypotheses and choose tails

Suppose an A/B test compares treatment and control activation rates.

Two-sided:

H0: p_treatment - p_control = 0
Ha: p_treatment - p_control != 0

Right-sided:

H0: p_treatment - p_control <= 0
Ha: p_treatment - p_control > 0

A two-sided test counts unusually positive and negative results. A right-sided test counts only positive departures. Choose before viewing outcomes. If harm in the opposite direction would matter, use two-sided testing for the primary product effect.

The p-value cannot be calculated correctly from a test statistic without the alternative. z = 2.0 yields about 0.0228 for a right-sided test and 0.0455 for a two-sided test.

Step 2: Select the test from the metric and design

These are starting points. Cluster randomization, repeated observations, heavy tails, censoring, count exposure, and sequential monitoring change the analysis.

GrowthBook's comparison of t-tests and z-tests explains why sample size alone does not choose between them.

Example 1: Calculate a p-value from a z-score

Suppose a process has a known population standard deviation of 10. An independent sample of 100 observations has mean 52, and the null reference mean is 50.

The standard error is:

SE = sigma / sqrt(n)
   = 10 / sqrt(100)
   = 1

The z statistic is:

z = (sample mean - null mean) / SE
  = (52 - 50) / 1
  = 2.00

For a two-sided test, find standard normal probability beyond |2.00| in both tails:

p = 2 * (1 - Phi(2.00))
  = 2 * (1 - 0.97725)
  = 0.0455

At a pre-specified alpha of 0.05, the test rejects the null. The interpretation is that under the null model and procedure, z statistics at least as far from zero occur about 4.55% of the time.

In Python, SciPy's 0 can evaluate the upper tail:

from scipy.stats import norm

z = 2.0
p_two_sided = 2 * norm.sf(abs(z))

Example 2: Calculate a p-value from a t statistic

Now suppose the population standard deviation is unknown. A sample of 25 independent units has mean 52 and sample standard deviation 10. Test against 50.

SE = s / sqrt(n)
   = 10 / sqrt(25)
   = 2

t = (52 - 50) / 2
  = 1.00

The one-sample t statistic has 25 - 1 = 24 degrees of freedom. The two-sided p-value is:

p = 2 * P(T_24 >= 1.00)
  approximately 0.327

The p-value is larger than in the z example because the sample is smaller, the standard error is larger, and variance is estimated. Do not attribute the difference only to the t distribution; the example's information changed too.

SciPy's 0 evaluates the tail:

from scipy.stats import t

t_stat = 1.0
df = 24
p_two_sided = 2 * t.sf(abs(t_stat), df)

GrowthBook's t-test explainer covers one-sample, independent, paired, and Welch forms.

Example 3: Calculate a two-proportion A/B test p-value

Suppose an onboarding experiment observes:

The observed absolute effect is 0.015. Under the null that both population rates are equal, calculate the pooled rate:

p_pool = (500 + 575) / (5000 + 5000)
       = 0.1075

Calculate the null standard error:

SE_null = sqrt(
  0.1075 * (1 - 0.1075) * (1/5000 + 1/5000)
)
        approximately 0.00620

Then:

z = (0.115 - 0.100) / 0.00620
  approximately 2.42

The two-sided p-value is:

p = 2 * (1 - Phi(2.42))
  approximately 0.0155

The NIST two-proportion formula derives the pooled null standard error. GrowthBook's z-test guide provides the A/B testing interpretation and assumption checks.

Why confidence-interval standard error can differ

The null hypothesis imposes equal rates, so the hypothesis test pools the rate when calculating its null variance. A common confidence interval for the rate difference estimates each group's variance separately. The test and interval can therefore use different standard-error formulas while still serving related inferential roles.

Do not copy the test's pooled standard error blindly into every interval calculation.

Example 4: Calculate a p-value by permutation

Sometimes the null distribution is generated from the experiment's randomization rather than a named z or t distribution.

Suppose users were randomly assigned and the statistic is the difference in mean revenue. Under the sharp null of no treatment effect:

1. Keep each user's observed outcome fixed.
2. Shuffle treatment labels according to the original assignment mechanism.
3. Recalculate the mean difference.
4. Repeat many times or enumerate all assignments when feasible.
5. Count the share of shuffled statistics at least as extreme as observed.

With B random shuffles, a common Monte Carlo estimate is:

p = (extreme_count + 1) / (B + 1)

The added one avoids reporting zero merely because a finite random sample did not reproduce an equally extreme statistic. The permutation scheme must preserve blocking, clustering, or unequal allocation from the actual randomization.

Permutation tests are not assumption-free. They rely on exchangeability or the known randomization mechanism and a clearly defined null and statistic.

One-sided, two-sided, and discrete p-values

One-sided

Use the tail corresponding to the pre-specified alternative. For a positive observed z-score under a right-sided alternative, p = 1 - Phi(z).

Two-sided symmetric

For z and symmetric t tests, double the probability beyond the absolute statistic. This counts equally extreme evidence in either direction.

Discrete exact tests

With binomial or Fisher exact tests, outcomes have probability masses rather than a continuous curve. A two-sided definition may sum all outcomes with null probability no greater than the observed outcome, or use another ordering. Different valid conventions can yield slightly different p-values. Record the software and method.

Chi-square and F tests

These statistics are nonnegative and often use an upper tail. A two-sided scientific hypothesis does not imply multiplying an F or chi-square upper tail by two; sidedness is encoded in how the statistic was constructed.

Calculate the effect and interval too

A p-value is incomplete without magnitude. For every calculation, report:

• Group estimates.
• Absolute and relative difference.
• Confidence interval.
• Test statistic and degrees of freedom.
• Exact p-value and alpha.
• Minimum practical effect.

In the conversion example, treatment improved activation by 1.5 percentage points or 15% relative. An interval shows whether plausible effects include trivial lift or harm. GrowthBook's statistical significance guide explains why a low p-value does not make an effect important.

Software examples

Python: two proportions

from statsmodels.stats.proportion import proportions_ztest

successes = [575, 500]
observations = [5000, 5000]
z_stat, p_value = proportions_ztest(
    count=successes,
    nobs=observations,
    alternative="two-sided",
)

The official statsmodels 0 reference documents its inputs and alternatives.

Python: independent means

from scipy.stats import ttest_ind

result = ttest_ind(treatment, control, equal_var=False)
print(result.statistic, result.pvalue, result.df)

equal_var=False requests Welch's test. Ensure each array contains one independent value per analysis unit.

Spreadsheet formulas

Spreadsheets provide normal and t distribution functions and built-in test functions. Preserve whether a function returns a CDF, density, or tail. A density value is not a p-value, and a one-tailed function should not be reported as two-sided without adjustment.

Whatever the tool, save the input summaries, function name, version, tail, and settings. Reproducibility requires more than the displayed number.

Checks before trusting the calculation

Independence

If assignment is by user, analyze users or use a cluster-aware method. Pageviews from one user are not independent experimental units.

Approximation quality

For proportion z-tests, check success and failure counts. For t-tests, inspect skew, tails, and outliers, especially in small samples. For rare outcomes, exact or model-based methods may be better.

Fixed metric and horizon

Do not try many metrics, segments, exclusions, and dates until one p-value crosses 0.05. Every search adds another false-positive opportunity.

Valid stopping

An ordinary fixed-horizon p-value is not valid for daily stop-on-significance behavior. Use the planned sample once or a sequential method chosen before launch.

Data quality

Check allocation, duplicate assignment, exposure, missing outcomes, and metric maturity before inference. Correct tail arithmetic cannot fix a logging bug.

Common calculation mistakes

• Using the observed effect as the null standard error without the test's formula.
• Reading a z-score as a p-value.
• Forgetting to double a symmetric one-tail probability for a two-sided test.
• Doubling an F or chi-square upper tail automatically.
• Using a paired test on unrelated groups or an independent test on paired data.
• Treating sessions as users.
• Rounding 0.000 as an exact zero probability.
• Calculating after an unplanned stopping point and reporting the original alpha.
• Selecting the smallest p-value among many tests without adjustment.

The American Statistical Association's guidance emphasizes that conclusions should not be based only on whether a p-value crosses a threshold.

A reproducible calculation template

Record these fields in the experiment readout:

Question:
Null hypothesis:
Alternative hypothesis:
Primary metric:
Assignment and analysis unit:
Test and model:
One- or two-sided:
Group summaries:
Observed effect:
Standard-error formula:
Test statistic and degrees of freedom:
P-value calculation:
Alpha and multiplicity adjustment:
Confidence interval:
Practical threshold:
Stopping rule:
Data-quality checks:

This makes the number auditable and prevents the test from being separated from its assumptions.

The tail area is only as valid as the experiment

To calculate a p-value, define the hypotheses, choose a statistic whose null distribution matches the design, standardize the observed effect, and calculate the appropriate one- or two-tail probability. Software should handle distribution arithmetic, but the analyst must provide the right test, unit, tail, and stopping rule.

Then report the effect and interval. The p-value describes compatibility with a null procedure; it does not measure product value or prove the experiment is healthy.

To calculate experiment results from shared warehouse metrics with frequentist, Bayesian, and sequential options, explore GrowthBook experimentation.