T-test vs z-test: Key differences and when to use each

The practical choice is not “z for large samples, t for small samples.” It depends on the parameter, variance treatment, and experiment design.

T-tests and z-tests share the same structure: both divide an observed difference by a standard error and compare the resulting statistic with a reference distribution. The difference is how uncertainty in that standard error is represented.

A t-test uses a Student's t distribution, usually because the variance of a mean is estimated from the same sample. A z-test uses the standard normal distribution, either because the relevant variance is treated as known or because a large-sample statistic has a normal approximation. As the t-test's degrees of freedom increase, its reference distribution approaches the standard normal, so the numerical results often become nearly identical.

For product teams, the metric and assignment structure matter more than a memorized cutoff. Conversion rates, average revenue, paired before-and-after outcomes, clustered accounts, and repeated user events are different statistical problems even when every table has two columns labeled A and B.

T-test vs z-test at a glance

This table describes common uses, not every possible statistic. Regression coefficients can have t or asymptotic z tests. Proportions use normal approximations even though their variance is estimated. Survey, cluster, and generalized models derive their reference distributions from more detailed assumptions.

What the two tests have in common

Both methods begin with a null hypothesis and alternative. For two independent means:

H0: mean_treatment - mean_control = 0
Ha: mean_treatment - mean_control != 0

Both form a standardized statistic:

statistic = observed difference / standard error under the model

Both can produce a p-value and confidence interval. Both require independent observations at the analysis unit and a sampling distribution justified by the design. Neither proves a hypothesis, measures practical value, repairs bad randomization, or makes repeated peeking valid.

GrowthBook's guide to statistical significance explains how the evidence threshold fits with effect size, confidence intervals, and product decisions.

Why the reference distributions differ

Imagine estimating a population mean. If the population standard deviation sigma were known, the standard error would be sigma / sqrt(n). Under normal sampling assumptions, standardizing the mean with that known quantity produces a z statistic.

In practice, sigma is usually unknown. Replacing it with the sample standard deviation s adds uncertainty. Under normal-population assumptions, the standardized mean follows a t distribution with degrees of freedom tied to sample size.

The t distribution has heavier tails to represent that extra variance-estimation uncertainty. At low degrees of freedom, a larger absolute statistic is needed to cross the same two-sided alpha threshold. As degrees of freedom increase, s becomes more stable and the t distribution converges to the normal distribution.

This is why a t-test does not become wrong when the sample grows. It simply becomes numerically similar to a z-test.

When to use a t-test

Use a t-based method when the estimand is a mean or difference in means, the relevant variance is estimated from sample data, and the design supports t inference.

Independent A/B test means

For average user-level revenue, latency, task time, or sessions per eligible user, Welch's two-sample t-test is a reasonable baseline. It compares independent group means without assuming equal population variances:

t = (mean_treatment - mean_control) /
    sqrt(s_treatment^2/n_treatment + s_control^2/n_control)

The NIST two-sample t-test guide distinguishes unpaired and paired data as well as equal- and unequal-variance forms. SciPy's 0 implementation uses equal_var=False for Welch's test.

Welch is generally a safer default than the pooled Student test because real variations can differ in variance. Equal sample sizes do not prove equal population variances.

One sample versus a reference

Use a one-sample t-test when one independent sample mean is compared with a fixed, defensible reference and the population standard deviation is unknown. Examples include mean processing time versus a service target or mean model score versus a preregistered benchmark.

Do not treat an estimated historical mean as a fixed constant while ignoring its uncertainty. If both values come from samples, the comparison may be a two-sample, time-series, or regression problem.

Paired or matched observations

Use a paired t-test for a true one-to-one relationship: before and after measurements on the same unit, matched accounts, or the same item evaluated by two systems. The test reduces each pair to a difference and tests whether the mean difference is zero.

The NIST paired-test explanation shows that the standard deviation of the pairwise differences drives the standard error. Two samples with equal row counts are not paired unless the rows have a meaningful correspondence.

GrowthBook's full t-test guide covers these forms and their assumptions.

When to use a z-test

Z-tests arise in two common situations: a test of means with a genuinely known population variance, or an asymptotic normal test for another statistic.

Means with known population variance

In a textbook one-sample z-test:

z = (sample mean - reference mean) / (sigma / sqrt(n))

The population standard deviation sigma must be known independently, not estimated from the same sample and renamed. This situation is uncommon in product experimentation. A long historical estimate is still an estimate and may not describe the current eligible population or treatment condition.

The NIST one-mean testing framework illustrates tests of a process mean and the importance of defining the standard-error model.

Two-proportion A/B tests

For binary outcomes such as conversion, activation, or retention, a two-proportion z-test is common when groups are independent and success and failure counts are sufficiently large.

Under the null that both population rates are equal, the test uses a pooled rate:

p_pooled = (successes_control + successes_treatment) /
           (n_control + n_treatment)

z = (p_treatment - p_control) /
    sqrt(p_pooled * (1-p_pooled) * (1/n_treatment + 1/n_control))

The NIST two-proportion reference derives the normal approximation. GrowthBook's z-test guide walks through the A/B testing interpretation.

The use of z here does not mean a population standard deviation was known. The binomial model and large-sample approximation produce an asymptotically normal statistic. With rare outcomes or small category counts, use an exact or better-suited method.

Why “n greater than 30 means z” is misleading

The shortcut collapses several separate ideas:

1. The central limit theorem can make a sample mean approximately normal as sample size grows under suitable conditions.
2. The t distribution approaches the standard normal as degrees of freedom increase.
3. Variance estimated from many observations becomes more stable.

None implies that a t-test becomes invalid at 31 observations. If the population variance is unknown and the mean-test assumptions are reasonable, the t-test remains appropriate. Its p-value and interval automatically approach the z result.

Nor does a sample of 30 guarantee a valid mean comparison. Thirty observations from an extremely heavy-tailed distribution, 30 repeated rows from five users, or 30 clusters analyzed as 30,000 events can all defeat the simple method.

Choose from the design and estimand; let the reference distribution follow.

A/B testing decision framework

Start with these questions.

1. What is the outcome?

• Binary user outcome: consider a two-proportion z-test for large independent groups.
• Continuous user-level outcome: consider Welch's t-test for independent groups.
• Paired continuous outcome: consider a paired t-test.
• Count, ratio, censored time, or repeated outcome: consider a model designed for that structure.

2. What was randomized?

If users were assigned, do not treat their pageviews as independent. If accounts were assigned, do not analyze members as if assignment occurred individually. GrowthBook's experimental-units guide explains why an incorrect unit produces standard errors that are too small.

3. Are the approximation and distribution credible?

For a proportion z-test, check success and failure counts in each group. For a t-test, inspect skew, outliers, and tails, especially with small or unbalanced groups. Large samples improve many approximations but do not cure infinite variance, dependence, or biased missingness.

4. Will the team monitor continuously?

Neither a fixed-horizon t-test nor z-test permits unlimited peeking. Plan a final sample through power analysis and analyze once, or use a valid sequential method.

5. Is zero the real decision boundary?

A significant but tiny effect may not be useful. Define a minimum practical effect before launch and report a confidence interval. If the goal is to show treatment is not meaningfully worse, use non-inferiority. If the goal is to establish that differences are negligible in either direction, use equivalence testing.

Worked comparison

Suppose an independent A/B test measures average task time:

The observed difference is -8 seconds and the standard error is:

SE = sqrt(20^2/25 + 20^2/25) = 5.66

Both a z calculation and pooled t calculation produce a standardized statistic near -1.41. The difference lies in the reference distribution. A two-sided z p-value is about 0.157; a t distribution with 48 degrees of freedom produces a slightly larger p-value around 0.165.

With hundreds or thousands of independent users, the numerical gap becomes tiny. That does not justify choosing the z result because it is smaller. Use the test supported by the variance assumptions. Here the population standard deviation was estimated from the sample, so a t-based analysis is the coherent choice.

Cases where neither basic test is enough

Highly skewed revenue

Revenue per user can contain many zeros and a few very large values. Welch's t-test may perform reasonably at scale for the mean, but validate it through simulation or bootstrap and consider robust estimators. Do not switch to a conversion z-test by discarding revenue magnitude unless conversion is the actual estimand.

Repeated observations

If every user contributes daily measurements, the rows are correlated. Aggregate to one user-level outcome or use a longitudinal or cluster-robust model.

Multiple variants and metrics

Every additional confirmatory comparison creates another opportunity for a false positive. Apply suitable multiple-testing controls and preserve one primary outcome. GrowthBook's discussion of high-variant testing shows how quickly the chance of at least one false signal can grow.

Nonrandomized rollout

Neither test removes confounding. A region released in June and another measured in May can differ because of time, market, or population composition. Use randomized concurrent assignment or an appropriate causal design.

Interpret either test the same disciplined way

Regardless of the reference distribution, report:

• Group estimates and sample sizes.
• Absolute and relative effect.
• Standard error and confidence interval.
• Test statistic, degrees of freedom when applicable, and p-value.
• Alpha, sidedness, and stopping rule.
• Minimum practical effect.
• Assignment and data-quality checks.
• Guardrail outcomes.

The American Statistical Association's p-value statement warns that conclusions should not be based only on whether a threshold is crossed. A result should explain the plausible effect sizes and what they mean for the product.

Choose the model, not the more favorable p-value

Use a t-test when analyzing means whose variance is estimated from the data, including Welch's test for most independent A/B mean comparisons and the paired test for linked observations. Use a two-proportion z-test for sufficiently large independent binary outcomes, or a mean z-test only when the population variance is genuinely known under the model.

Do not let 30 rows make the decision. Start with the outcome, independent unit, distribution, and monitoring plan. In large mean comparisons, t and z may agree to several decimal places; the t-test remains the honest representation of estimated variance.

To manage metrics and analyze randomized product experiments with methods suited to the design, explore GrowthBook experimentation.