What is a t-test? Purpose, types, and how to conduct it

A t-test compares a mean with a benchmark or another mean while accounting for uncertainty in the sample variance.

The t-test is one of the most widely used methods for testing hypotheses about averages. A product team might use it to compare average revenue per user between A/B test groups, an operations team might compare mean processing time before and after a change, or a researcher might compare a sample mean with a known target.

The name covers several related tests. A one-sample t-test compares one sample mean with a reference. An independent two-sample test compares unrelated groups. A paired test compares linked observations. Welch's t-test handles independent groups without assuming equal population variances.

Choosing among them depends on the design, not the number of rows in a spreadsheet. The important questions are what parameter you want to estimate, which units are independent, whether observations are paired, how the outcome is distributed, and how the data were collected.

How a t-test works

Every t-test forms the same general ratio:

t = (observed estimate - null value) / estimated standard error

The numerator is the effect: a sample mean minus a benchmark, a difference between two group means, or the mean difference within pairs. The denominator estimates how much that quantity would vary from sample to sample if the null hypothesis held.

A large absolute t statistic means the observed estimate is far from the null relative to its estimated uncertainty. The p-value measures how often a t statistic at least that extreme would occur under the null model and planned procedure.

GrowthBook's p-value explanation covers the common interpretation trap: a p-value is not the probability that the null hypothesis is true. It conditions on the null rather than assigning probabilities to hypotheses.

Why use a t distribution instead of a normal distribution?

For a mean z-test, the population standard deviation is treated as known. That is uncommon in real product analysis. A t-test estimates the standard deviation from the sample, adding uncertainty.

The t distribution accounts for that uncertainty with heavier tails than the standard normal distribution. Its shape depends on degrees of freedom. With few degrees of freedom, extreme statistics receive more probability. As sample size grows, the variance estimate stabilizes and the t distribution approaches the normal distribution.

The distinction is why "use z above 30 and t below 30" is an unreliable rule. The parameter, variance knowledge, design, and distribution matter more than one sample-size cutoff. GrowthBook's t-test versus z-test guide compares the methods directly.

The main types of t-test

One-sample t-test

A one-sample t-test asks whether a population mean differs from a fixed reference value.

Suppose a service-level target is 500 milliseconds and a team samples independent request-level measurements from a stable process. The hypotheses could be:

H0: population mean latency = 500 ms
Ha: population mean latency != 500 ms

The statistic is:

t = (sample mean - reference mean) / (sample SD / sqrt(n))

with n - 1 degrees of freedom. The reference must be meaningful and specified independently of the same data. Comparing this month's average with last month's observed average as if the latter were a fixed known constant understates uncertainty; that is generally a two-sample or time-series problem.

Independent two-sample t-test

An independent two-sample test compares means from two unrelated groups. A randomized A/B test assigning each user to one variation is a common example.

The NIST two-sample t-test reference distinguishes paired versus unpaired observations and equal versus unequal variance forms. For product experiments, the independent unit is usually the randomized user or account, not every event that unit generated.

Two versions are common:

• Student's pooled t-test assumes equal population variances and combines the two variance estimates.
• Welch's t-test estimates each group's variance separately and adjusts the degrees of freedom.

Welch's test is usually the safer default when equal variances are not established. Modern software removes the computational reason to prefer the pooled formula. SciPy's official 0 documentation exposes the choice through equal_var=False for Welch's method.

Paired t-test

A paired t-test is appropriate when observations have a meaningful one-to-one link: the same participant measured before and after a treatment, matched accounts, or the same item evaluated by two systems.

Calculate one difference per pair, then perform a one-sample t-test on those differences:

d_i = after_i - before_i
t = mean(d) / (SD(d) / sqrt(n_pairs))

The NIST paired-observations guide emphasizes that inference is based on the distribution of differences. Normality, when important, concerns those differences rather than the two raw measurement distributions separately.

Pairing can remove stable between-unit variation and improve precision. It cannot be invented after the fact. Two groups with the same sample size are not paired unless each row in one group has a defensible partner in the other.

How to conduct a t-test

A responsible analysis starts with the estimand and design rather than a software menu.

1. Define the question and hypotheses

State the population, outcome, comparison, and direction. A two-sided independent test for average revenue might use:

H0: mean revenue per eligible user is equal in treatment and control
Ha: mean revenue per eligible user differs between treatment and control

Choose one-sided testing only when an effect in the opposite direction would lead to the same decision as no effect and the direction was fixed before viewing results. Most product experiments should default to two-sided tests because unexpected harm matters.

2. Identify the independent unit

If the experiment assigns users, aggregate the outcome to one value per user for a simple independent t-test. Treating orders, sessions, or pageviews as independent when users contribute multiple observations produces a falsely small standard error.

If assignment is by company, classroom, market, or cluster, a basic row-level t-test is also wrong. Analyze cluster-level outcomes or use a model and standard errors that reflect clustered assignment. GrowthBook's guide to experimental units explains the alignment among randomization, exposure, and inference.

3. Inspect missingness, skew, and outliers

Plot the outcome by group and examine extreme values. Do not delete observations merely because they weaken significance. Apply a documented rule tied to data validity or use a robust metric definition chosen before results.

For revenue, a handful of very large customers may dominate the mean. That might be genuine business value rather than bad data. Consider robust standard errors, winsorization fixed in advance, a trimmed-mean test, a bootstrap, or a model appropriate to the outcome. Report how the method changes the estimand.

Missing data also need a causal explanation. If treatment changes whether revenue or latency is observed, complete-case comparison can be biased.

4. Choose one-sample, Welch, pooled, or paired

Use this decision table:

This is a starting point, not an automatic answer. Count, binary, censored, longitudinal, clustered, or highly zero-inflated outcomes may need another model.

5. Calculate the statistic and degrees of freedom

For Welch's independent test:

t = (mean_treatment - mean_control) /
    sqrt(s_treatment^2/n_treatment + s_control^2/n_control)

The Welch-Satterthwaite equation supplies approximate degrees of freedom based on group variances and sample sizes. Statistical software should calculate it, but preserve the group summaries and settings so the result can be reproduced.

6. Calculate the p-value and confidence interval

Compare the t statistic with the t distribution at the calculated degrees of freedom. For a two-sided test, probability in both tails contributes to the p-value.

Also calculate a confidence interval for the mean difference:

observed difference +/- critical t * standard error

The interval communicates magnitude and precision. A result can cross zero and still leave important benefit and harm plausible; that is inconclusive, not evidence of no effect.

7. Interpret beside practical significance and guardrails

Compare the interval with a minimum practical effect. GrowthBook's statistical significance guide separates evidence against a null value from the value of the observed effect.

Then review experiment quality and guardrails: sample allocation, exposure logging, metric maturity, errors, retention, cost, and any user-safety outcome. The t-test does not validate those inputs.

Worked example: average task time in an A/B test

Suppose a team tests a new workflow and measures task-completion time for one mature task per independently assigned user.

The observed difference is 52 - 48 = 4 seconds. The Welch standard error is:

SE = sqrt(20^2/400 + 24^2/420)
   = sqrt(1.000 + 1.371)
   = 1.54 seconds

The statistic is approximately:

t = 4 / 1.54 = 2.60

With the Welch degrees of freedom, the two-sided p-value is around 0.01. The data provide evidence that mean completion time differs between groups under the test assumptions.

But "treatment is significant" is an incomplete conclusion. In this example a higher time may be harmful. The team should report that treatment increased mean task time by four seconds, include the confidence interval, compare it with the maximum tolerable increase, and review completion rate and error guardrails. Direction and product meaning come from the metric, not the statistic.

T-test assumptions and how they fail

Independence

This is the most important assumption and cannot be fixed by a large sample. Repeated user observations, cluster assignment, spillovers, or time dependence require an analysis that represents that structure.

Normality of the relevant mean or differences

Exact small-sample t inference assumes normally distributed observations for a one-sample test, normal group outcomes for independent tests, or normal pairwise differences for a paired test. With larger samples, the sampling distribution of the mean can be approximately normal under broad conditions.

"Large" depends on skew and tail behavior. Extreme heavy tails can converge slowly. Use simulations, resampling, robust estimators, and domain knowledge rather than a universal threshold.

Equal variance for the pooled test

Student's independent test assumes equal population variances. Welch's test relaxes this assumption and handles unequal sample sizes more reliably. Do not choose the form solely from a preliminary variance test; that adds another data-dependent decision. Use Welch by default unless the equal-variance model is substantively justified.

Stable measurement

Metric definitions, observation windows, and data pipelines must remain consistent across groups. A delayed outcome can make recently enrolled users appear worse. Instrumentation errors can produce a precise answer to the wrong question.

When a t-test is not the right tool

• Use a two-proportion method for a binary conversion rate when its assumptions fit.
• Use count or rate models for event counts with varying exposure.
• Use survival analysis for time-to-event outcomes with censoring.
• Use mixed or cluster-robust models for repeated or clustered observations.
• Use randomization inference when the assignment mechanism should define the test directly.
• Use equivalence or non-inferiority procedures when the goal is to rule out a meaningful difference.
• Use a valid sequential method when the team will monitor continuously and stop early.

GrowthBook's A/B testing methodology guide compares frequentist, Bayesian, and sequential decision frameworks. The best method is the one that matches the design and operational behavior, not the one that produces the smallest p-value.

Report more than the p-value

A complete t-test report includes:

• Test type and whether it was one- or two-sided.
• Unit of analysis and group sample sizes.
• Means and standard deviations.
• Mean difference and confidence interval.
• T statistic, degrees of freedom, and p-value.
• Missing-data and outlier rules.
• Minimum practical effect and stopping rule.
• Data-quality and guardrail results.

The American Statistical Association's p-value statement cautions that scientific and business conclusions should not depend only on whether a threshold was crossed. The same applies to product readouts: the estimate and uncertainty are the result; the p-value is one piece of evidence.

Match the test to the structure of the data

A t-test is not one procedure applied whenever a table contains averages. It is a family of methods for means, with different forms for one sample, independent groups, unequal variances, and paired observations.

Define the independent unit first. Use Welch's method for independent groups unless equal variance is defensible. Analyze within-pair differences for genuinely paired data. Inspect skew, tails, missingness, and persistent dependence. Then interpret the mean difference and interval beside practical value and experiment integrity.

To analyze product metrics with shared definitions, reproducible queries, and multiple statistical approaches, explore GrowthBook experimentation.