What is sequential testing? A guide for product teams

Sequential testing lets a product team examine accumulating evidence and act early without pretending each dashboard look was the only analysis.

Most teams watch experiments while they run. They check for regressions, answer stakeholder questions, and want to ship a clear win or stop a clear loss before a fixed calendar date. Ordinary fixed-horizon testing does not support stop-when-significant behavior: every additional look gives random variation another chance to cross the threshold.

Sequential testing designs the monitoring into the method. It can use stricter early boundaries, spend error probability across planned looks, or produce anytime-valid p-values and confidence sequences. The team can then stop for success, harm, or futility according to rules chosen before outcomes.

This flexibility is not free and does not replace sound experiment design. Sequential tests still need random assignment, stable exposure, mature metrics, sufficient operational coverage, and practical effect thresholds. They solve a specific problem: inference when sample size or stopping time depends on accumulating data.

Why ordinary peeking increases false positives

Suppose a team plans a two-sided test at alpha 0.05 but checks an ordinary p-value every day. It ships treatment on the first day p < 0.05.

Under the null, each daily statistic is noisy and correlated with earlier days. Even if no effect exists, the path can temporarily cross 0.05. The more opportunities to stop, the greater the overall chance of a false launch compared with one pre-specified final analysis.

The nominal p-value describes one fixed procedure. Change the procedure and its calibration changes. GrowthBook's p-value interpretation guide explains why the stopping rule is part of the result's meaning.

The NIST discussion of p-values and critical regions likewise treats alpha as a property of a test rule chosen in advance, not a probability attached to whichever interim result appears.

An analogy is repeatedly rolling a die and declaring it loaded as soon as a favorable streak appears. One streak may be rare at a fixed roll count; watching indefinitely makes a streak much less surprising.

Monitoring guardrails is not the same as declaring a winner

Teams should monitor serious safety and reliability metrics continuously. If treatment duplicates payments or crashes the app, stop it. No statistical method requires continued harm.

The distinction is between operational containment and confirmatory product inference. A rollout may stop on an absolute incident threshold while the primary outcome follows its sequential decision rule. Document the reason for stopping and avoid translating an emergency stop into a conventional final p-value without adjustment.

What makes a test sequential

A sequential design specifies how evidence will be evaluated over time and which actions are allowed.

Common families include:

• Group-sequential tests with a finite set of interim looks.
• Alpha-spending designs that allocate Type I error as information accrues.
• Sequential probability ratio tests based on accumulating likelihood ratios.
• Anytime-valid p-values that remain calibrated at allowed stopping times.
• Confidence sequences that provide time-uniform interval coverage.
• Bayesian decision policies with posterior thresholds and stopping rules.

The research on always-valid inference for A/B testing formalizes p-values and intervals intended to remain valid when users choose a stopping time from accumulating data.

These methods are not interchangeable. They make different assumptions and optimize different tradeoffs. Use the platform's documented implementation rather than labeling any repeated analysis "sequential."

Group-sequential testing

A group-sequential design schedules a limited number of analyses, such as at 25%, 50%, 75%, and 100% of planned information. Each look has a critical boundary.

Early boundaries are usually more demanding because a decision with little data must be unusually clear. Later boundaries can relax as more evidence accumulates. The complete set controls the desired Type I error under the plan.

Two broad boundary styles illustrate the choice:

• Conservative early boundaries preserve most alpha for the final analysis and require overwhelming early evidence.
• More even spending makes early stopping easier but changes later thresholds and may increase expected sample under some alternatives.

The right schedule follows traffic, outcome delay, and business cadence. Daily analyses are not useful when the primary metric takes 14 days to mature.

Anytime-valid quantities

An anytime-valid p-value is constructed so that, under the null, the probability it ever crosses alpha within the allowed process is controlled. A confidence sequence is an interval process designed to cover the parameter simultaneously over time with the stated probability.

The advantage is operational: the dashboard can remain visible, and the team can make a decision at an unscheduled time allowed by the method. The tradeoff can be wider intervals or more conservative evidence than a fixed-horizon analysis at the same current sample.

Recent work on safe testing for experimentation platforms addresses continuous monitoring without elevating incorrect decisions in the way naive repeated testing does.

Anytime-valid does not mean every adaptation is valid. Adding metrics after seeing trends, changing eligibility, launching new variants against a reused control, or switching the estimand may fall outside the guarantee.

Sequential probability ratio testing

The sequential probability ratio test compares how likely accumulating data are under two specified hypotheses. Evidence is summarized as a likelihood ratio. The process continues while the ratio lies between decision boundaries and stops when it crosses an upper or lower boundary.

This can be efficient when the alternative is well specified. Product effects are rarely known exactly, so composite alternatives or modern extensions are often needed.

NIST's overview of a sequential sampling plan describes efficiency through the average sample number required for stated Type I and Type II errors. That is the key sequential benefit: reducing average, not necessarily maximum, observations.

Plan a sequential A/B test

1. Define the decision and estimand

State the population, assignment unit, treatment, primary metric, and product action. Define an absolute minimum practical effect and any harm margin.

For example:

Among newly eligible accounts, does guided setup increase 14-day activation by at least one percentage point without increasing setup errors by more than 0.2 points?

That produces two decision dimensions: evidence of useful benefit and protection against material harm.

2. Select the error or loss policy

For a frequentist design, choose alpha, desired power, sidedness, and any multiplicity correction. For a Bayesian policy, specify priors, posterior or expected-loss thresholds, and evaluate its operating behavior.

GrowthBook's comparison of frequentist, Bayesian, and sequential methods emphasizes matching the method to how the team will monitor and communicate.

3. Set minimum and maximum information

Minimum exposure protects against operational blind spots. Even an extreme early effect may be based on one device, one day, or a small unrepresentative cohort.

Maximum information prevents indefinite tests. It can be a sample size, variance-based information level, or deadline. Define what an inconclusive result means at the maximum: no rollout, limited rollout, further research, or a new test.

Use power analysis to estimate the maximum sample needed for the practical effect under the sequential design, not a fixed-horizon calculator applied without adjustment.

4. Match looks to metric maturity

If activation needs seven days, a user assigned yesterday cannot contribute a mature outcome today. Interim datasets should use consistent cohorts and windows, or a model that explicitly handles partial observation.

Checking daily can add cost without information. Schedule looks after enough newly matured independent units accumulate.

5. Define stopping rules

A complete plan includes:

Do not let a statistical success rule override data-quality failure or operational harm.

6. Simulate the policy

Simulate no effect, practical benefit, small benefit, and harm. Include baseline and variance uncertainty, metric delay, unequal allocation, and realistic traffic.

Measure:

• False-positive and false-harm decisions.
• Power for the practical effect.
• Average and maximum sample.
• Probability and timing of futility stops.
• Winning-effect exaggeration after early stopping.
• Sensitivity to delayed and missing outcomes.

The policy is the combination of method and team behavior. Simulation tests that combination.

Recent work on robust sequential experimental design for A/B testing is one example of why operating characteristics should be studied under realistic uncertainty rather than only an idealized point model.

Worked example: planned interim looks

Suppose a fixed-horizon design needs 40,000 mature users for 80% power. The team chooses group-sequential analyses at 10,000, 20,000, 30,000, and 40,000 users.

The statistical design supplies a two-sided critical boundary for each look. An illustrative pattern might require a much smaller p-value at 10,000 than at 40,000. Do not invent these numbers manually; calculate them from the selected spending function or platform method.

At 10,000 users:

• Primary effect is positive but does not cross the early boundary.
• Error guardrail is healthy.
• The team continues.

At 20,000 users:

• The efficacy boundary is crossed.
• The confidence sequence excludes both zero and the practical threshold.
• Assignment, exposure, and guardrails pass.
• The team stops and begins a staged rollout.

The key is not that the ordinary p-value happened to be below 0.05 at 20,000. The pre-specified sequential boundary was crossed at an allowed look.

Sequential testing, progressive rollout, and canaries

These concepts solve different problems:

• Sequential testing governs statistical evidence as observations accumulate.
• Progressive rollout controls how many users receive a change.
• Canary deployment checks operational behavior on limited production traffic.

They can work together. A feature flag starts at limited exposure, operational guardrails catch acute failures, and a sequential experiment evaluates product outcomes. Exposure can expand only when both technical and statistical policies allow it.

GrowthBook's guide to progressive delivery explains the release-control layer. Do not call percentage ramping "sequential testing" unless an actual sequential statistical design governs the outcome evidence.

Common sequential testing mistakes

Running A, then B over time

Showing control in January and treatment in February is not sequential hypothesis testing. It is a historical comparison confounded by time unless a defensible time-series design addresses the change. A/B groups should generally run concurrently.

Turning sequential mode on after peeking

The early data and stopping opportunities are part of the procedure. A fixed test that looked promising cannot be retroactively made sequential.

Confusing an early boundary with a guaranteed final effect

Early stopping selects unusually strong observed effects. The point estimate can overstate the true lift. Report a sequentially valid interval or adjusted estimate where available and continue post-rollout measurement.

Ignoring futility

Teams often plan only a success stop. A futility rule can save traffic when the practical effect is unlikely, but it needs calibration and a clear distinction between "no useful effect" and "not enough information."

Checking immature outcomes

Partial conversion windows can create artificial trends as cohorts age. Freeze mature cohorts or model delay explicitly.

Adding comparisons midstream

New variants, metrics, and segments create new error opportunities. Update the multiple-testing design or treat them as exploratory future hypotheses.

Assuming sequential always uses less data

Strong effects may stop early. Effects near the boundary can use the full maximum, and some sequential designs require more maximum sample than a one-look test for the same target power.

Report a sequential experiment clearly

Include:

• Method and implementation version.
• Primary metric, assignment unit, and effect scale.
• Alpha or decision thresholds and practical margin.
• Planned minimum, maximum, and look schedule.
• Actual stopping time and reason.
• Sequential p-value or confidence sequence, not an ordinary fixed p-value.
• Effect estimate and interval suited to the design.
• Multiplicity handling.
• Data-quality and guardrail checks.
• Post-stop rollout and monitoring plan.

Avoid "we stopped early because p was significant." Say which sequential rule was met and at what information level.

Use a method that matches how the team behaves

Sequential testing is the honest choice when a product team will monitor an experiment and may act before a fixed horizon. It makes the stopping behavior part of the design instead of treating every look as invisible.

Define the practical effect, error policy, minimum and maximum exposure, metric-maturity rules, and success, harm, and futility actions before launch. Simulate the policy. Use sequentially valid outputs, and keep operational guardrails separate from the product winner rule.

To monitor warehouse-native experiments with sequential frequentist analysis alongside Bayesian and fixed-horizon options, explore GrowthBook experimentation.