Bayesian statistics: What it is and how it applies to A/B testing

Bayesian statistics updates uncertainty about an effect by combining a prior model with the evidence in observed data.

In a Bayesian A/B test, the output is not a single p-value. It is a posterior distribution describing plausible values of the treatment effect under the model and prior. From that distribution, a team can calculate the probability treatment beats control, the chance it exceeds a minimum useful lift, the chance of harm, and the expected loss from choosing either variation.

That language often matches product decisions better than "reject" or "fail to reject" a null hypothesis. It does not make the analysis automatic. Priors encode assumptions, likelihoods can be wrong, assignment can be broken, metrics can be immature, and intuitive probabilities can invite premature decisions.

The value of the Bayesian approach is that assumptions and uncertainty live in one explicit probability model. The discipline is to specify, check, and use that model honestly.

The core Bayesian update

Bayesian inference has three main components:

• The prior distribution represents uncertainty about a parameter before the current data.
• The likelihood describes how probable the observed data are for different parameter values.
• The posterior distribution combines the prior and likelihood after the data are observed.

Bayes' rule is often summarized as:

posterior is proportional to likelihood times prior

In notation:

p(theta | data) proportional to p(data | theta) * p(theta)

Here theta might be a conversion rate, mean revenue, or difference between treatment and control. The posterior is conditional on both the observed data and the modeling choices. A statement such as "there is a 94% probability that treatment is better" means 94% of the posterior mass for the effect is above zero under that prior and likelihood.

The prior is part of the model

A prior can be:

• Weakly informative, limiting implausible extremes without asserting a precise expected effect.
• Informative, using credible historical or domain evidence.
• Hierarchical, learning a distribution of effects across related experiments.
• A default mathematical prior chosen for routine analysis.

"Uninformative" does not mean assumption-free. Parameterization, scale, and distribution family still matter. Research on objective priors describes attempts to formalize defaults when prior information is limited, but a default must still be evaluated for the actual estimand.

For product experiments, prior predictive checks are practical. Simulate effects and outcomes from the prior before seeing experiment results. If the model assigns meaningful probability to impossible conversion rates or absurd revenue lifts, revise it.

The likelihood connects the parameter to data

The likelihood should match the outcome and sampling process:

• Bernoulli or binomial models for one binary outcome per independent unit.
• Normal or robust continuous models for user-level values.
• Count models for event totals with appropriate exposure.
• Survival models for time-to-event outcomes with censoring.
• Hierarchical models for clustered or repeated observations.

A convenient likelihood is not automatically a correct one. Treating sessions as independent when users generate many sessions makes the posterior too confident. Ignoring delayed conversions can bias the treatment effect. The Bayesian framework represents modeled uncertainty, not every uncertainty the analyst forgot to model.

The posterior drives inference and decisions

The posterior can be summarized with a mean, median, mode, credible interval, or probability of a region. For complex models, software such as Stan samples from or approximates the posterior, and diagnostics determine whether the computation adequately explored it.

The posterior also supports prediction. Stan's posterior predictive guide integrates over parameter uncertainty to describe future outcomes. In A/B testing, predictive quantities can help answer a business question such as the likely incremental conversions from a full rollout, not only the underlying rate difference.

Bayesian A/B testing with conversion rates

The beta-binomial model is a useful teaching example because its update has a closed form.

Suppose each eligible user either converts or does not. Let a variation's true conversion rate be p. Use a beta prior:

p ~ Beta(alpha, beta)

After observing x conversions among n independent users, the posterior is:

p | data ~ Beta(alpha + x, beta + n - x)

The same update is performed separately for control and treatment. Samples from the two posterior distributions can be subtracted to form a posterior distribution of lift.

Worked example

Assume a weak Beta(1, 1) prior for each conversion rate and these observations:

The posterior mean rates are approximately 10.1% and 12.1%. Sampling paired draws from those posteriors and calculating p_treatment - p_control answers several questions:

• What is the posterior probability treatment beats control?
• What is the probability lift exceeds one percentage point?
• What is the probability treatment is harmful by more than a chosen margin?
• What is the expected loss if the team chooses treatment or control?

With these approximate numbers, most posterior mass favors treatment, but the plausible effect range still includes small or possibly negative differences. The exact probability depends on the stated prior and calculation. A responsible readout reports that dependency rather than presenting "chance to win" as a property of the raw counts alone.

GrowthBook's article on Bayesian model updates describes the same workflow from prior and data to percent change, chance to win, and credible interval.

Credible intervals versus confidence intervals

A 95% Bayesian credible interval contains 95% of the posterior probability for the parameter under the model, prior, and observed data. It supports a conditional probability statement about the parameter.

A 95% frequentist confidence interval is produced by a procedure that covers the fixed true parameter in 95% of repeated samples under its assumptions. After one interval is calculated, the frequentist framework does not assign a 95% probability to that fixed parameter lying inside it.

The numerical intervals can look similar with large samples and weak priors. Their interpretations remain different.

GrowthBook's comparison of A/B testing methodologies sets Bayesian posterior probability and credible intervals beside fixed-horizon p-values and confidence intervals. Teams should use the language of the selected method rather than calling every 95% output "confidence."

Bayesian decision rules for product experiments

"Chance to win above 95%" is a possible rule, but it ignores effect size and decision costs. A richer policy uses practical thresholds and expected loss.

Probability of practical improvement

Let delta be treatment minus control and m be the minimum useful lift. Calculate:

P(delta > m | data)

This asks whether treatment is likely to beat the actual business threshold, not merely zero. A treatment can have a 98% probability of being positive while having little probability of clearing the cost of implementation.

Probability of material harm

Define a harm boundary -h and monitor:

P(delta < -h | data)

This is useful for guardrails and non-inferiority decisions. The threshold should reflect impact and reversibility. Technical incidents can still demand immediate rollback without waiting for posterior evidence.

Expected loss

For each decision, calculate the cost of being wrong across posterior draws. Choosing treatment when it is worse loses the counterfactual advantage of control; choosing control when treatment is better loses the treatment uplift.

Expected-loss rules make asymmetric costs explicit. A checkout regression and a missed color-copy improvement should not use identical policies simply because both are A/B tests.

Minimum exposure and maturity

Even when a posterior threshold is crossed early, require enough exposure to validate assignment, instrumentation, important segments, and delayed outcomes. The first 100 users may contain no mobile devices or enterprise accounts. A probability calculation cannot compensate for missing operational coverage.

Does Bayesian testing solve peeking?

Bayesian posteriors can be updated whenever new data arrive. There is no p-value whose fixed-horizon false-positive rate is directly inflated by each look. That does not mean any stopping behavior is harmless.

A team that stops every experiment the moment a noisy early posterior looks favorable can create selection bias in shipped effects, unstable decisions, and poor long-run business outcomes. Repeatedly testing new variants against the same control, changing priors after seeing results, or switching the primary metric creates other problems.

Research on informed Bayesian A/B testing highlights monitoring as a benefit while still treating prior selection and evidence for the null as substantive modeling choices.

Define before launch:

• Earliest decision time or minimum exposure.
• Scheduled review cadence.
• Probability or expected-loss thresholds.
• Practical improvement and harm boundaries.
• Maximum duration and inconclusive outcome.
• Guardrails that stop the release independently.

Then simulate the policy under no effect, small effects, and meaningful effects. Measure how often it ships harm, how long it runs, and how much regret it produces. This evaluates the complete decision system rather than one posterior snapshot.

Choosing and checking priors

Priors matter most when data are sparse and less as a regular likelihood accumulates. That is exactly when early stopping makes their influence operationally important.

Use historical experiments carefully

Historical effects can support an empirical or hierarchical prior, but account for selection. A database containing only shipped winners exaggerates expected lift. Include losses, nulls, and abandoned tests, and group genuinely comparable metrics and surfaces.

Run sensitivity analysis

Recalculate the decision under several plausible priors:

• Weakly informative and centered near zero.
• Skeptical, concentrating more mass near no effect.
• Historically informed.
• A reasonable optimistic alternative.

If the shipping decision changes across plausible priors, the evidence is prior-sensitive. Report that and gather more data rather than hiding the choice.

Check model fit

Posterior predictive checks compare simulated outcomes from the fitted model with observed patterns. Look for mismatches in zero rates, tails, segment variation, time trends, and within-user dependence. A narrow credible interval from a visibly poor model is false precision.

For computational Bayesian models, also inspect chain convergence, effective sample size, divergences, and Monte Carlo error. The Stan posterior-analysis documentation explains why posterior draws require diagnostic review before summaries can be trusted.

Bayesian versus frequentist A/B testing

Neither framework repairs a weak experiment. Both require stable treatment, valid assignment, relevant metrics, and transparent decision rules.

GrowthBook's guide to frequentist versus Bayesian testing focuses on choosing a framework that matches how a team monitors and communicates. Consistency is more important than switching methods when one produces a preferred result.

Common Bayesian A/B testing mistakes

Calling a bootstrap Bayesian

Resampling observed rows creates a bootstrap distribution. It does not become a Bayesian posterior unless it is part of a specifically defined Bayesian procedure. State the prior, likelihood, and posterior computation.

Reporting chance to win without the prior

Posterior probabilities are conditional on modeling choices. Document the prior family and parameters, and show sensitivity when those choices could affect the decision.

Ignoring practical effect

Probability that treatment is greater than control can become high for a tiny effect. Report probability of exceeding the practical threshold and expected loss.

Treating the posterior as protection from bad data

Sample-ratio mismatch, inconsistent exposure, metric leakage, and missing outcomes can corrupt any analysis. GrowthBook's A/B testing guide puts assignment and metric definition before statistical interpretation for good reason.

Using intuitive language without uncertainty

"Treatment has a 96% chance to win" sounds decisive. Add the credible interval, practical threshold, harm probability, and expected loss so stakeholders see how large and consequential the effect may be.

A Bayesian A/B test checklist

Before launch, document:

• Population, assignment unit, treatment, and hypothesis.
• Primary metric and likelihood.
• Prior and prior predictive check.
• Practical improvement and harm thresholds.
• Minimum exposure, review schedule, and maximum duration.
• Promotion, rollback, and inconclusive rules.
• Guardrails and data-quality checks.
• Planned prior-sensitivity analysis.
• Posterior and computational diagnostics.
• Final reporting language.

After the test, preserve the model version, priors, queries, posterior summaries, decision, and actual post-rollout outcome. That archive can inform better hierarchical priors and policy calibration without relying on memory or winner-only examples.

Use posterior probabilities as inputs to a policy

Bayesian statistics offers a coherent way to update uncertainty and communicate product effects in probability language. Its strongest A/B testing outputs go beyond chance to win: probability of practical improvement, probability of material harm, credible intervals, predictions, and expected loss.

Those quantities remain conditional on the model, prior, data quality, and stopping policy. Make those inputs visible. Check prior and posterior predictions, run sensitivity analysis, and simulate the operating behavior of the decision rule.

To run warehouse-native experiments with Bayesian, frequentist, and sequential analysis, explore GrowthBook experimentation.