Experiments

Data Science

True Positive: Definition and Examples in Testing

Apr 29, 2026

min read

A graphic of a bar chart with an arrow pointing upward.

Most teams tracking model performance or running A/B experiments focus on whether their system is catching real positives — but that's only one piece of a four-part picture.

A true positive only means something useful when you understand it alongside false positives, false negatives, and true negatives. Miss that context, and you end up optimizing the wrong thing, sometimes badly.

This article is for engineers, product managers, and data teams who work with classifiers, ML models, or experimentation platforms and want a clear, practical grip on how true positives fit into real evaluation work. Here's what you'll learn:

What a true positive is and how it fits into the four-outcome binary classification framework
How to read a confusion matrix and why accuracy alone will mislead you on imbalanced data
How to calculate the true positive rate (TPR), what it actually measures, and where it goes by other names like sensitivity and recall
Why maximizing true positives isn't always the right goal — and how to think about the sensitivity-specificity trade-off
How true positives work specifically in A/B testing, including the common practices that inflate false discoveries and suppress real ones

Each section builds on the last. Start with the definition, work through the math, and finish with the practical implications for experimentation — including how GrowthBook handles peeking, multiple testing, and statistical power in ways that directly affect whether your test results are real.

A true positive has a two-part definition — and getting either part wrong breaks your analysis

Precision matters when you're evaluating whether a test, model, or experiment is actually working. The term "true positive" gets used loosely in practice — often as a shorthand for "a correct result" — but that framing is incomplete in a way that causes real problems when you're trying to reason about model performance or test quality.

A true positive has a specific, two-part definition, and understanding it exactly is the foundation for everything else in this space.

Why "a correct result" is not specific enough

A true positive occurs when a test predicts a positive outcome and the underlying condition is genuinely present. Both conditions must hold simultaneously: the test says positive, and the ground truth is positive.

As Wikipedia frames it in the context of diagnostic testing, sensitivity (the true positive rate) is "the probability of a positive test result, conditioned on the individual truly being positive". That word conditioned is doing important work — it means the ground truth is already established as positive, and the question is whether the test correctly reflects that reality.

This distinction matters because "a correct result" is not specific enough. A test that correctly identifies a negative case — someone who doesn't have a disease, a transaction that isn't fraudulent — is also producing a correct result. That outcome is a true negative, which is a different category entirely. A true positive is specifically a correct positive identification.

The four-outcome binary classification framework

No test outcome exists in isolation. A true positive only has meaning when you understand it as one of four possible outcomes in any binary classification system. When a test is applied to a case where the condition either exists or doesn't, and the test either flags it or doesn't, you get exactly four combinations:

	Condition Present	Condition Absent
Test Positive	True Positive	False Positive (Type I Error)
Test Negative	False Negative (Type II Error)	True Negative

The four cells of this table represent the complete universe of outcomes for any binary classifier. A test that labels everything as positive would accumulate a high count of true positives, but it would also generate false positives on every negative case. That's not a useful test. The true positive count only becomes meaningful when you can see it in context alongside the other three outcomes.

The definition holds across domains; what changes is the cost of getting it wrong

The definition is domain-agnostic. The underlying logic — test says positive, condition is real — applies identically whether you're working in healthcare, financial services, or software systems.

In medical screening, a cancer test that correctly flags a patient who actually has cancer is producing a true positive. The clinical stakes here are high: a test with a high true positive rate catches real cases early, enabling timely treatment. Missing those cases — producing false negatives instead — carries serious consequences for patient outcomes.

In fraud detection, a true positive occurs when a model flags a transaction as fraudulent and that transaction is, in fact, fraudulent. Compliance teams rely on these correct identifications to prevent financial losses and protect customer accounts. The system is doing exactly what it's supposed to do.

Software and model evaluation follow the same logic: a classification model produces a true positive when it correctly identifies a positive instance — a defective component on a production line, a spam email in a filtering system, a bug flagged by a static analysis tool that is a genuine defect. The condition exists, and the model found it.

Across all three domains, the definition holds. What changes is the cost structure around errors — how damaging it is to miss a real positive versus how damaging it is to flag a false one. But that's a question of trade-offs, not of the definition itself.

True positives only have meaning inside the full four-outcome confusion matrix

A true positive doesn't exist in isolation. Its meaning only becomes clear when you place it alongside the three other outcomes that any classifier or test can produce: true negatives, false positives, and false negatives.

The confusion matrix is the minimum unit of analysis for evaluating test quality — and if you're only tracking how often your model catches real positives, you're missing most of the picture.

How the four cells map every possible prediction against reality

A confusion matrix is a 2×2 table that maps every prediction a model makes against what actually occurred. One axis represents the actual class; the other represents the predicted class. The diagonal — top-left to bottom-right — captures correct predictions. The off-diagonal cells capture errors.

	Predicted Positive	Predicted Negative
Actually Positive	True Positive (TP) ✓	False Negative (FN), Type II Error
Actually Negative	False Positive (FP), Type I Error	True Negative (TN) ✓

A concrete example makes this tangible. Consider a cancer screening classifier evaluated on 12 individuals — 8 who actually have cancer and 4 who don't. If the model makes 9 correct predictions but misclassifies 3 — calling 2 cancer patients cancer-free and flagging 1 healthy person as having cancer — you have 2 false negatives and 1 false positive.

The model looks reasonably accurate at first glance, but the error breakdown reveals two very different failure modes with very different consequences.

One note on convention: some sources place actual classes on rows and predicted classes on columns; others reverse this. Both are valid. The table above follows the row-as-actual convention used by Wikipedia and most ML tooling, but you'll encounter both in practice.

False positives (Type I errors): the cost of crying wolf

A false positive occurs when a model predicts positive but the actual outcome is negative. In machine learning, this is formally called a Type I error. In spam detection, it's flagging a legitimate email as spam. In fraud detection, it's blocking a valid transaction.

In A/B testing, the framing is slightly different but structurally identical. GrowthBook's documentation defines a Type I error as a situation where "your metrics all appear to be winners, but in reality the experiment has no effect." The test fires a signal; the signal is wrong. Acting on that signal — shipping a feature, changing a product flow — means making a real decision based on noise.

False positives carry costs that depend entirely on context. In some domains, they're annoying but recoverable. In others, they trigger irreversible actions: unnecessary medical treatment, a blocked customer, a shipped feature that degrades the product.

False negatives (Type II errors): the cost of missing what's real

A false negative is the mirror image: the model predicts negative, but the actual outcome was positive. This is a Type II error. In medical screening, it's a missed diagnosis. In fraud detection, it's a fraudulent transaction that goes through undetected.

In A/B testing, a Type II error occurs when "the data aren't showing a clear winner or loser when actually a variation is much better or worse." The consequence is that teams either collect more data — extending an experiment that already has an answer — or make a blind decision without the signal they needed. Real improvements go undetected; real regressions go unaddressed.

The cost asymmetry between Type I and Type II errors is domain-specific and worth making explicit in any system you're evaluating. Missing a cancer diagnosis is not the same kind of failure as flagging a healthy patient. Missing a winning A/B test variant is not the same kind of failure as shipping a losing one.

Why no single cell in the confusion matrix tells you whether your model is good

No single cell in the confusion matrix tells you whether your model is good. The matrix has to be read as a whole, and the derived metrics — accuracy, precision, recall — each weight the four cells differently.

Accuracy is the most intuitive: (TP + TN) / all predictions. But it's also the most misleading on imbalanced datasets. A model that predicts "negative" for every single input on a dataset where positives appear only 1% of the time achieves 99% accuracy while being completely useless. It has a perfect true negative rate and a true positive rate of zero. As one practitioner put it, such a classifier "could be losslessly replaced by a rock."

Precision (TP / TP + FP) tells you how often a positive prediction is correct — critical when false positives are expensive. Recall, also called sensitivity (TP / TP + FN), tells you how much of the actual positive class you're capturing — critical when false negatives are expensive. These metrics pull in opposite directions, and optimizing one typically costs you the other.

All of these metrics are also threshold-dependent. Change the classification threshold and every cell in the matrix shifts. The confusion matrix you compute at one threshold is not the confusion matrix you'd compute at another, which is why evaluating a classifier at a single operating point is rarely sufficient for understanding its real-world behavior.

True positive rate: the metric that tells you how much of reality your system is actually catching

Understanding what a true positive is gets you halfway there. The more useful question for practitioners is: how do you measure how many real positives your system is actually catching? That's what True Positive Rate (TPR) answers — and it's a metric precise enough to calculate, compare, and optimize.

The TPR formula and what it actually measures

TPR is calculated as:

TPR = TP / (TP + FN)

The numerator is the count of true positives — cases where your model or test correctly identified a real positive. The denominator is the total universe of actual positive cases: every real positive that existed, whether your system caught it (TP) or missed it (FN). The result is a value between 0 and 1, where 1.0 means every real positive was detected and nothing slipped through.

You'll encounter this metric under different names depending on the field. In medicine and statistics, it's called sensitivity. In machine learning, it's called recall. Wikipedia's formal definition captures it cleanly: sensitivity is "the probability of a positive test result, conditioned on the individual truly being positive." All three terms refer to the same calculation.

One practical note: a high TPR tells you the test is sensitive — it catches most real positives. But it doesn't tell you that any specific positive result is correct. Here's why that matters: imagine a disease that affects 1 in 10,000 people. Even a test with 99% TPR will produce more false positives than true positives in that population, simply because there are so few actual cases to find.

The probability that a positive result is real depends on how common the condition is — a separate calculation called positive predictive value (PPV). Conflating TPR with PPV is a common mistake with real consequences.

What high and low TPR signals about your system

A high TPR means your model is catching most of the real positive cases and producing few false negatives. A low TPR means real positives are routinely slipping through — your system is missing what it's supposed to find.

The stakes of a low TPR vary dramatically by domain. In healthcare, a missed diagnosis is a false negative with potentially irreversible consequences. In fraud detection, a missed fraudulent transaction carries direct financial and reputational costs. These are the domains where TPR is typically the primary metric to optimize, because the cost of a false negative far outweighs the cost of a false positive.

In A/B testing, statistical power functions as the domain-specific analog to TPR — it represents the probability that a real effect will be detected, which is precisely what TPR measures in classification contexts.

That said, a high TPR is not universally the right target. Pushing TPR toward 1.0 typically requires lowering the classification threshold, which increases false positives. Whether that trade-off is acceptable depends entirely on the cost structure of your specific problem — a point the next section addresses directly.

TPR and the ROC curve

The ROC (Receiver Operating Characteristic) curve is the standard tool for visualizing how TPR behaves across different classification thresholds. It plots TPR on the y-axis against the false positive rate (FPR) on the x-axis. Each point on the curve represents a different threshold setting.

As you lower the classification threshold, more cases get flagged as positive. TPR rises — you catch more real positives — but FPR rises too, because more negatives get incorrectly flagged. Raise the threshold and the reverse happens: fewer false positives, but more real positives missed. The ROC curve makes this trade-off visible across the full range of possible thresholds rather than at a single fixed point.

A classifier with a curve that hugs the top-left corner of the plot is performing well: it achieves high TPR at low FPR. A curve that runs diagonally from bottom-left to top-right is no better than random guessing. The shape of the curve tells you how much flexibility you have in setting a threshold before sensitivity degrades meaningfully.

The sensitivity-specificity trade-off: why maximizing true positives isn't always the goal

There's an intuitive appeal to the idea that a good classifier should catch as many true positives as possible. In practice, that instinct leads teams astray. Maximizing sensitivity — your true positive rate — always comes at a cost, and understanding that cost is what separates a well-calibrated system from one that creates as many problems as it solves.

The fundamental trade-off: why you can't maximize both

Sensitivity and specificity move in opposite directions. As Wikipedia's treatment of the topic states directly: "higher sensitivities will mean lower specificities and vice versa".

The mechanism is straightforward. When you lower a classification threshold to catch more true positives, you inevitably sweep in more negatives along with them. More true positives means more false positives — which means lower specificity. There's no configuration that escapes this relationship. The question is never whether to accept this trade-off, but where to set it.

When high sensitivity is the right call

The case for prioritizing sensitivity is strongest when a missed positive carries severe consequences. Wikipedia's criterion is precise: prioritize sensitivity "when the consequence of failing to treat the condition is serious and/or the treatment is very effective and has minimal side effects."

Cancer screening is the canonical example. A false negative — a test that misses a malignancy — means a patient goes untreated while the disease progresses. The downstream cost of that miss dwarfs the cost of a false positive, which typically means an additional confirmatory test. The asymmetry is clear: one error is inconvenient, the other can be fatal.

Fraud detection follows similar logic. Missing a fraudulent transaction carries real financial and reputational consequences for a business, while a false positive — a legitimate transaction flagged incorrectly — creates customer friction but is recoverable. In domains like these, false negatives are the more expensive error, and systems should be tuned accordingly.

When specificity deserves priority

The calculus flips when false positives carry their own serious costs. Wikipedia identifies the relevant condition: specificity matters most "when people who are identified as having a condition may be subjected to more testing, expense, stigma, anxiety, etc."

Confirmatory diagnostic testing is the clearest case. A false positive diagnosis can trigger unnecessary treatment, psychological harm, or lasting stigma — costs that are real and sometimes irreversible. The initial screening test can afford to be sensitive; the confirmatory test needs to be specific.

The same principle applies in A/B testing. A false positive in an experiment means declaring a winning variant when no real effect exists, then shipping a change that doesn't actually improve the product. The cost here isn't just a wasted engineering cycle — it's the compounding effect of making product decisions on noise.

Threshold selection is a business decision, not a statistical one

The classification threshold determines where on the sensitivity-specificity trade-off curve your system operates. There is no universally correct setting. The right threshold depends entirely on the relative cost of each error type in your specific context, and that cost calculation belongs to the domain, not the algorithm.

This becomes especially concrete in A/B testing when multiple metrics are evaluated simultaneously. GrowthBook's documentation on experimentation pitfalls notes that testing the same hypothesis across 20 metrics at a 5% significance level produces roughly a 64% probability of finding at least one statistically significant result by chance alone.

That's what happens when sensitivity is implicitly maximized without any mechanism to control specificity. Correction methods like Benjamini-Hochberg and Bonferroni deliberately sacrifice some sensitivity — accepting that a few real effects may go undetected — in order to reduce the rate of false discoveries.

That trade-off isn't a flaw in the methodology. It's the methodology working as intended, calibrated to the cost structure of the problem. The teams that get this right aren't the ones chasing the highest possible true positive rate — they're the ones who have thought clearly about what a false positive actually costs them.

Related reading: Statistical Validity: What It Means in Research

True positives in A/B testing: correctly detecting real effects and avoiding false discoveries

In A/B testing, a true positive has a specific meaning: your experiment correctly identifies a variation that genuinely improves a metric, and you ship it. The test said it won. It actually won. That alignment between the statistical signal and the underlying reality is exactly what experimentation is designed to produce — and it's rarer than most teams assume.

According to GrowthBook's documentation, only about one-third of experiments successfully improve the metrics they're designed to move. Another third have no effect, and the remaining third actually hurt performance. That distribution means the majority of experiments you run will not produce true positives. In that environment, correctly identifying the real winners matters enormously — and so does avoiding the false ones.

Where true positives sit in the A/B testing decision space

The full A/B testing decision space maps directly onto the confusion matrix framework. When you decide to ship a variation and it actually won, that's a correct inference — a true positive. Every other combination where "ship" and "actual outcome" don't align is a Type I error. Shipping a variation that had no real effect, or shutting down one that actually would have won, are both failures of the classification system.

The default significance threshold in most experimentation platforms is 95% confidence. That means even in a perfectly designed experiment with no methodological problems, you'll incorrectly flag a result as significant 5% of the time. As GrowthBook's documentation puts it plainly: "5% of the time, it isn't actually better." That baseline false positive risk is unavoidable — but several common practices make it dramatically worse.

The peeking problem and multiple testing

The peeking problem occurs when teams monitor experiment results continuously and stop a test the moment results look promising. This practice inflates the false positive rate substantially. Because statistical significance fluctuates throughout a test's runtime, repeatedly checking results and stopping early when p < 0.05 appears means you're much more likely to catch a random fluctuation than a real effect.

Sequential testing addresses this by allowing continuous monitoring and early stopping without inflating false positive rates — the statistical method adjusts for the repeated looks so the error rate stays controlled.

Multiple testing compounds the problem in a different way. When you test many metrics simultaneously, the probability that at least one will appear significant by chance increases with each metric added. Google's famous "41 shades of blue" experiment illustrates how cascade testing — running A vs. B, then B vs. C, and so on — can produce mathematically invalid conclusions when not handled correctly.

GrowthBook's documentation notes that adding many metrics to any test increases false positive risk, even in a correctly configured system. The solution is pre-registering your primary metric before the experiment runs, not selecting the most flattering result afterward.

P-hacking is the logical extension of this: continuing to slice data, add metrics, or adjust segments until statistical significance appears, then reporting only the significant finding. It's not always intentional, but the effect is the same — the result looks like a true positive and isn't.

Underpowered tests don't inflate false positives — they suppress true positives

Underpowered tests don't inflate false positives — they suppress true positives. If your experiment doesn't have enough users to detect the effect size you're looking for, real improvements will fail to reach significance. GrowthBook's documentation defines this through the concept of Minimal Detectable Effect (MDE): if the actual effect is smaller than the MDE given your sample size, the test cannot detect it even when it genuinely exists.

The result is a false negative — a true positive that never gets identified.

Variance reduction techniques directly address this. CUPED reduces variance by accounting for pre-experiment user behavior, which means experiments can reach statistical significance with fewer users. That speed matters because it reduces the temptation to peek early.

A/A testing eliminates infrastructure failures before they corrupt your true positive rate

Before you can trust that your A/B test results represent true positives, you need to confirm that your experimentation infrastructure isn't generating spurious signals on its own. A/A testing — running an experiment where both variations are identical — is the standard method for this validation. If an A/A test returns statistically significant results across multiple metrics, the system itself may be broken: traffic isn't splitting correctly, metrics are misconfigured, or the SDK integration has an error.

GrowthBook recommends running A/A tests after setting up a new SDK connection and after any significant changes to your integration, data warehouse, or tracking libraries. One important calibration note: even a correctly configured A/A test may show one or two marginally significant metrics due to random chance at the 5% threshold. That's expected. What's alarming is three or four metrics all showing significance above 99% — that's a signal the system is broken, not just unlucky.

A/A testing doesn't guarantee that your subsequent A/B results are true positives. But it eliminates a category of infrastructure-level failures that would make true positive detection impossible from the start.

Putting it together: calibrating your system to the actual cost of each error

The core insight running through this entire article is simple but easy to miss: a true positive only tells you something useful when you understand it in context. The count of things your system correctly identified means nothing without knowing how many real positives it missed, how many false alarms it generated, and what each of those errors actually costs you.

That four-part frame — not just the single number — is what separates a well-calibrated system from one that looks good on the surface and fails in practice.

Match your sensitivity-specificity balance to the cost of each error type

Before you tune a threshold or evaluate a metric, make the cost asymmetry explicit. In fraud detection, a missed fraudulent transaction is typically more expensive than a blocked legitimate one. In A/B testing, shipping a feature that had no real effect compounds quietly over time in ways a missed winner usually doesn't.

Neither of those cost structures is universal — they're specific to your domain, your users, and your business. The threshold decision follows from that analysis, not the other way around.

The confusion matrix and TPR reveal what accuracy alone conceals

If you're currently evaluating model performance with accuracy alone, the confusion matrix is the right place to start. Pull the full 2×2 breakdown, compute TPR and precision separately, and look at where your errors are concentrated. A model with 95% accuracy on an imbalanced dataset may have a TPR near zero — catching nothing that actually matters. The matrix makes that visible in a way a single aggregate metric never will.

Protect true positive rates in experimentation with rigorous statistical practices

In A/B testing, the practices that inflate false positives — peeking, running too many metrics, post-hoc segmentation — are the same ones that make your true positives harder to trust. Pre-register your primary metric, run an A/A test to validate your infrastructure, and size your experiments to detect the effect you actually care about. Sequential testing and variance reduction techniques are specifically designed to address these failure modes without forcing you to choose between speed and statistical integrity.

What to do next:

If you're evaluating a classifier: pull the full confusion matrix, compute TPR and precision separately, and compare performance at multiple thresholds before picking an operating point.
If you're running A/B tests: pre-register your primary metric, run an A/A test to validate your infrastructure, and calculate your MDE before launching.
If you're setting a classification threshold: write down the cost of a false positive and the cost of a false negative in your specific context before touching the threshold value.

One tension worth keeping in mind: when you add controls to reduce false positives — stricter significance thresholds, multiple testing corrections, higher sample size requirements — you will sometimes fail to detect real effects. That's not a flaw in the methodology. It's the trade-off working as intended. The goal isn't to catch every true positive at any cost. It's to build a system where the errors you make are the ones you've consciously decided are cheaper than the alternative.

Degrees of freedom in statistics, explained simply

Nov 5, 2026

min read

The same test statistic can support different conclusions when its degrees of freedom change.

You open an experiment report and see a t-statistic, a p-value, and a number labeled df. It might be 49, 118, or 19.48. That last value is not a typo. Degrees of freedom do not always equal the number of users minus 1, and they do not always come in whole numbers.

Degrees of freedom describe the independent variation available to a particular calculation or index the reference distribution used for inference. In familiar variance calculations, constraints explain them directly. In methods such as Welch's t-test, an approximation produces an effective value instead.

For experimentation, the useful question is specific: which calculation produced this df, and does that calculation match the data? Understanding that connection helps you spot wrong denominators, mismatched tests, and confidence intervals built from the wrong reference curve.

Start with the constraint behind n minus 1

Three observations, two freely varying deviations

Take a hypothetical sample of 3 measurements: 2, 4, and 6. Their mean is 4. Subtract that sample mean from each measurement and the deviations are -2, 0, and 2.

Those deviations sum to zero. They must: the mean was calculated from the same values. Once you know the mean and the first 2 deviations, the last deviation is determined. If the first 2 sum to -2, the last must be 2.

This is the intuition behind n - 1 degrees of freedom for the usual sample variance. There are n deviations but 1 independent linear constraint. Their possible patterns occupy n - 1 independent directions. You have not discarded a person or stopped using a measurement. All 3 values still contribute to the variance calculation.

You can see the arithmetic in this sample. Squaring the deviations gives 4, 0, and 4, for a total of 8. Divide by n - 1, which is 2, and the sample variance is 4. The sample standard deviation is its square root, or 2.

Why estimating the mean changes the denominator

The sample mean is fitted to the observed data. Deviations from that fitted mean tend to be smaller than deviations from the unknown population mean. For independent observations from the same distribution with finite variance, dividing their squared sum by n - 1 corrects the downward bias in the variance estimate.

The usual sample variance and one-sample t calculation therefore share the n - 1 quantity. This does not make the sample standard deviation itself exactly unbiased; taking the square root is a separate operation.

If your dataset is the entire finite population you want to describe, averaging squared deviations over n is a legitimate descriptive calculation. The denominator depends on the target. Estimating variation in a larger population is different from describing the values you already have.

Keep the intuition within its scope

“Independent observations minus fitted parameters” is useful for ordinary linear models, but it is not a universal df calculator. You must know what statistic is being calculated and what constraints or approximations apply.

Even a software parameter called ddof needs interpretation. The degrees-of-freedom adjustment in a standard-error calculation controls a variance denominator. It is not an instruction to use the same number as the reference df for every later test. A variance correction, a model's residual df, and a test's reference df are related concepts, not interchangeable labels.

See how df changes a statistical threshold

A t-statistic needs a reference curve

A t-statistic expresses an estimated difference relative to its estimated standard error. Its interpretation depends on the reference t distribution, whose shape is indexed by df. Low-df t distributions have heavier tails, reflecting uncertainty in the variance estimate under the model.

For a prespecified two-sided test at a 5% significance level, compare the absolute t-statistic with the appropriate 97.5th percentile. The t critical-value table gives approximately 2.776 at 4 df and 1.984 at 100 df.

Suppose two separate hypothetical analyses each produce t = 2.1. At 4 df, that value does not cross the 5% threshold. At 100 df, it does. The difference is not a judgment about the product feature. It follows from the reference distributions used to assess sampling uncertainty.

The reference distribution must itself be appropriate. Exact one-sample t inference assumes independent, normally distributed observations; large-sample approximations can cover broader settings. A small df does not automatically make a test invalid, but it gives you little protection against a poor distributional approximation. For highly skewed revenue measured on a handful of users, looking up the correct df is only one part of assessing the method.

Degrees of freedom are not the effect size

Larger df do not mean a bigger lift or a better variant. A tiny effect in a large sample can have high df. A large but imprecise effect in a small sample can have low df. You still need the effect's units, its interval, and a business threshold for deciding whether it matters.

Nor should you interpret df as statistical power. Power depends on the effect size you want to detect, variability, allocation, sample size, significance rule, and design. Use power analysis before launching an experiment to plan sensitivity. Reading a large df after the experiment does not replace that work.

Make small effects easier to detect

Degrees of freedom are only one part of uncertainty. Explore metric design and variance reduction techniques that can improve an experiment's sensitivity.

Explore Variance Reduction

Choose the formula for the method you actually use

One sample, pairs, and pooled groups

For a one-sample t-test, n independent observations give n - 1 df. For a paired t-test, calculate a difference within each pair, then analyze those independent differences. With 30 complete pairs, the usual paired test has 29 df, not 59.

For the pooled two-sample t-test, the common variance estimate combines the two groups' variance information. Its df are nA + nB - 2. The test assumes equal population variances as well as independent groups. You should not select that formula merely because you have an A/B experiment.

Calculation	Usual degrees of freedom	What must be counted correctly
One-sample t-test	n - 1	Independent observations in the sample
Paired t-test	Number of complete pairs - 1	Independent pairs, not both measurements separately
Pooled two-sample t-test	nA + nB - 2	Independent group sizes and the common-variance assumption
Welch's two-sample t-test	Welch-Satterthwaite approximation	Each group's size and variance
Ordinary least-squares residual variance	n - rank(X)	Usable rows and independent columns of the design matrix
Chi-square independence test	(Rows - 1) × (Columns - 1)	The contingency table's category structure

These are method-specific formulas, not options to try until a result becomes significant.

Welch can produce fractional df

Welch's t-test estimates each group's variance separately. Its reference degrees of freedom depend on how much each estimated variance contributes to uncertainty in the difference.

Consider a hypothetical revenue comparison with 20 independent units in group A and 100 in group B. The sample standard deviations are $20 and $5. For the absolute mean difference, the variance contributions are 20²/20 = 20 and 5²/100 = 0.25. Their sum is 20.25, giving a standard error of $4.50.

The Welch-Satterthwaite calculation is:

vA = sA^2 / nA
vB = sB^2 / nB
df = (vA + vB)^2 / (vA^2 / (nA - 1) + vB^2 / (nB - 1))

Here it gives approximately 19.48 df, even though there are 120 units in total. Most of the estimated uncertainty comes from the small, variable group A. Adding observations only to the already precise group B cannot remove that bottleneck.

Using 118 df because 20 + 100 - 2 = 118 would substitute the pooled method's assumptions. Rounding 19.48 up to 118 is not a harmless simplification. Let the chosen method calculate its df rather than overriding it from the total headcount.

Regression and count tables constrain different things

For ordinary least squares, residual df equal observations minus the design matrix's rank. With 100 usable observations, an intercept, and 2 nonredundant predictors, the rank is 3 and residual df are 97. Include the intercept in that count. Redundant predictor columns do not each add an independent fitted direction.

If the model fits an independent parameter for every usable observation, residual df can fall to zero. An exact fit then leaves no residual information for the usual error-variance estimate. Adding predictors until the observed data fit perfectly is not evidence that future outcomes will be predicted perfectly. A model's flexibility and its evidence for uncertainty are different things.

Ordinary one-way ANOVA illustrates another reason to read the label carefully: its F reference distribution has 2 df values. For k groups and N independent observations, the between-group and error degrees of freedom are k - 1 and N - k. A hypothetical comparison of 3 variants across 90 observations has F reference df of 2 and 87. The corrected total is 89, which equals their sum. Reporting only “df = 89” would omit the information needed to identify that F distribution. These formulas belong to the ordinary one-way model with its variance assumptions; they are not a recipe for every repeated-measures or unequal-variance ANOVA.

A chi-square test of independence uses a different structure. A 2-dimensional contingency table has (rows - 1) × (columns - 1) df. A 2-by-2 variant-by-conversion table has 1 df, whether it contains 200 users or 200,000. More traffic improves precision; it does not change the number of category relationships tested.

This is why df cannot serve as a universal substitute for sample size. Its meaning comes from the statistic and model.

Check the experimental unit before trusting a large df

Repeated events do not create independent users

Suppose you assign 1,000 users to a new experience and record 50,000 pageviews. A user-level treatment comparison does not gain 49,999 df just because the event table has that many rows. The same person's repeated events can be correlated.

Aggregate outcomes or model their dependence in a way that matches the design. A revenue-per-user metric, for example, can sum each user's eligible purchases over a fixed window before comparing groups. Explicit metric and Fact Table definitions help make that relationship between event rows and analyzed units inspectable.

Group assignment creates another case. If treatment is assigned to organizations, employees within an organization share an assignment and may influence one another. A cluster experiment needs inference that accounts for that structure. Plugging the number of employees into an ordinary independent-observation formula does not make the dependencies disappear.

Missingness changes the usable data

Check the data used for the specific metric, not only the experiment's top-line traffic count. Different metrics can have different valid windows, available measurements, or exclusions. In a paired analysis, incomplete pairs can reduce the number of usable differences.

That numerical reduction is only part of the issue. If treatment influences whether an outcome is observed, complete-case analysis can change who gets compared. A plausible df cannot tell you whether the missingness mechanism introduced bias.

Investigate a surprising value by tracing the inputs: which observations survived, what identifier defined an independent unit, what model was fitted, and whether the output uses an approximation. Treat df as a diagnostic clue rather than a quality certificate.

Also distinguish statistical df from the phrase “researcher degrees of freedom.” The latter refers to choices such as which metric, segment, exclusion, or analysis to report. Exploring those choices after seeing results can change the interpretation of evidence. It does not change the arithmetic identity n - 1 for a particular sample variance. Record the planned analysis and label exploratory follow-ups so a reader can distinguish the two issues.

Report the method behind the number

Keep df attached to the estimate and interval

A useful report names the test, the usable group sizes or pair count, the estimated effect, its standard error or interval, and the reference df when relevant. “Welch's t-test, df = 19.48” tells a reviewer more than “t-test, 120 observations.” Neither replaces an explanation of the experiment's assignment and measurement rules.

When reproducing a result, preserve the method's assumptions and software settings. A pooled t-test, a Welch test, and a model with clustered standard errors can analyze similar-looking columns while making different inferential calculations. Matching the displayed row count is not enough to reproduce the p-value.

GrowthBook's frequentist statistical calculations use a t reference distribution with degrees of freedom estimated through the Welch-Satterthwaite approximation. Its statistics overview also distinguishes frequentist and Bayesian engines. A df formula from one approach should not be inserted into the interpretation of another approach's output.

Resolve design problems before chasing precision

Before your next experiment, verify the unit of assignment and analysis, define the metric window, and choose the estimator and decision rule. Those decisions determine what information the analysis can use.

The stopping rule matters separately. A correct df does not authorize repeatedly inspecting a fixed-horizon p-value and stopping when it passes. If you need continuous decision-making, use a suitable method such as GrowthBook's frequentist sequential testing within its documented scope.

You rarely need to calculate degrees of freedom by hand in a production experiment. You do need to recognize when the reported value reflects the wrong test or the wrong independent unit. Start with the design, follow the calculation, and read the effect with its uncertainty. The df will then be an interpretable part of the result.

Build a test you can explain

Define your experiment's metrics and data source in GrowthBook, then review the treatment effect with its uncertainty and documented analysis method.

Get Started With GrowthBook

Experiments

Feature Flags

Server-side vs. client-side A/B testing

Nov 3, 2026

min read

Choosing where a variant runs is only the first architecture decision.

A checkout experiment can make its decision on the server, render in the browser, and measure purchases in a warehouse. Calling it “server-side” tells you little about whether the person received the assigned experience or whether their purchase joined to the right group.

Server-side testing evaluates or executes treatment logic in a backend service before returning its result. Client-side testing evaluates or applies treatment logic in the user's browser or application. Both can support randomized experiments. Both can fail through inconsistent identity, incorrect exposure events, or measurement that changes with the treatment.

The useful comparison starts with the change you want to test. A ranking algorithm belongs near the service that runs it. A browser interaction may belong in the application that renders it. Then work through the entire path from assignment to outcome, including the cache, navigation, and login transitions between them.

Separate the experiment's four locations

Assignment is different from treatment delivery

Assignment selects a variant for an eligible unit. Delivery makes the application behave according to that variant. They often happen together, but they do not have to.

For example, an edge worker can select a layout and a browser can render it. A backend can select a recommendation algorithm and return its results as JSON. A browser can select a button label without involving the server in that decision. The same statistical question can sit behind any of these implementations.

Describe the architecture with verbs: who assigns, who executes, who records exposure, and who computes the metric. That description is more useful than the label “full stack.” GrowthBook supports feature flags on the client and server, but you still choose which code consumes each flag and which system supplies the outcomes.

Exposure and analysis can live elsewhere

An exposure record should represent the experiment event your analysis is designed to use. An SDK evaluation callback tells you that an evaluation enrolled a user; it is not proof that the corresponding screen was visible or that a response reached a person.

If you evaluate a checkout flag during application startup but never use its value, the callback can precede any relevant treatment. If you log after the page finishes loading, a slower treatment can lose more events before the logger runs. Both mistakes change what the experiment population means.

Analysis has another location. With warehouse metric definitions, you can join assignment records to purchases or retention data already collected in your own data system. Running the branch on a server does not mean the server SDK has automatically captured or analyzed those outcomes.

Compare complete delivery paths

Decision	Browser implementation	Server implementation	Hybrid implementation
Where treatment executes	Browser UI or application behavior	Backend logic, API behavior, or rendered response	Different parts execute in different runtimes
Main implementation dependency	Frontend code or an eligible visual change	Backend deployment and service integration	Shared identity and explicit ownership across layers
First-render concern	Apply the intended experience before visible replacement	Return the intended response without avoidable delays	Preserve the server result during hydration
Measurement concern	Blocked, delayed, or missing client events	Join server assignment to actual delivery and outcomes	Avoid duplicate or contradictory exposure records
Cache concern	Stale configuration or application state	Variant-specific responses shared incorrectly	Cache keys and browser state disagree

These are engineering tradeoffs, not a universal speed ranking. A browser SDK can evaluate configuration already available locally. A server can add network waits if it fetches configuration on the request path. Measure the path you actually deploy.

Put the treatment where it can be implemented faithfully

Use the browser for changes the browser owns

Client-side testing suits interactions such as showing a different onboarding step, changing a filter interface, or selecting a message inside a web application. A developer can implement the variant directly in application code. A visual editing tool is another route for supported page changes, but client-side testing is not limited to editing the DOM after a page loads.

That distinction matters when teams compare ownership. Some changes can be authored by a growth team in a visual workflow; others require frontend engineering even though they run in the browser. The deployment and review process follows the implementation, not the statistical label.

Browser execution can create a visible replacement if the original content paints before the variant is applied. Hiding the entire page until configuration arrives can create a different problem: a long blank screen when the dependency is slow. Prefer an explicit rendering and fallback strategy over assuming every client-side test must flicker or that an anti-flicker snippet has no cost.

Use the server for backend behavior and controlled responses

Server-side testing fits a pricing calculation, search ranker, checkout service, recommendation pipeline, or response assembled before it reaches the browser. You can keep service credentials and confidential implementation details in that environment.

That benefit has a boundary. If you serialize sensitive rules, identifiers, or secrets into HTML or a browser configuration payload, their earlier evaluation on the server does not make the delivered data confidential. Review exactly what crosses the boundary.

Keep authorization independent of experimentation. A flag can select an experience for an already authorized user; it should not be the only check deciding whether that user can access a protected resource. Apply permission validation on requests regardless of the assigned variant. A client can alter browser state and requests, so browser-visible eligibility is especially unsuitable as a security boundary.

Measure performance instead of promising it

Server rendering can avoid a particular after-paint replacement, but it can also increase response time if the new branch is expensive. Client rendering can avoid a backend change while adding JavaScript work or delaying interactivity. Edge execution can shorten one network path while introducing cache and configuration-update concerns.

Compare the control and treatment on the experience people receive. Core Web Vitals distinguish loading, interaction responsiveness, and layout stability. Pair those measures with relevant backend latency and error metrics. A low flag-evaluation time does not describe the cost of the treatment, the configuration fetch, or the event pipeline.

Define fallback behavior before launch. If configuration is unavailable, decide which existing behavior remains appropriate, how the failure is observed, and whether that request belongs in the experiment analysis. Quietly mixing fallback users into an assigned treatment can dilute or distort the comparison.

Choose the smallest complete treatment

Consider 2 hypothetical checkout ideas. One changes the explanation beside a shipping option; the other changes the shipping-price calculation. The first can live in the frontend if that application already has the necessary context. The second belongs with the authoritative calculation so the displayed amount and the charged amount agree.

Combining both into one treatment may be a reasonable product decision, but it answers a broader question. If the experiment wins, you cannot attribute the change to the explanatory copy alone. Write down what differs across arms before deciding which team or runtime will implement it.

For a first integration, keep unrelated behavior identical. Use the existing purchase event and metric definition when they already represent the outcome you need. Changing rendering, pricing, identity and analytics together makes a surprising result harder to diagnose. A small experiment should still include the whole assignment-to-outcome path; small does not mean skipping instrumentation review.

Plan flags across your stack

Extend the architecture decision into ownership, lifecycle management, and production review. Use the feature flag guide to plan how flags will operate as your program grows.

Read the Feature Flag Guide

Keep identity and delivered variants consistent

Scope user attributes to the request

A shared configuration client is useful on a server. Shared mutable user attributes are a different matter. If one request replaces a global user's attributes while another request is evaluating a flag, users can receive each other's targeting context.

The current GrowthBook Node.js SDK supports a shared GrowthBookClient with a user context passed to evaluation, or a scoped instance for that context. Keep the authenticated or anonymous identifier in the request's scope. Do not use an arbitrary user-supplied identifier as authentication merely because it is convenient for experimentation.

Choose the randomization unit deliberately. A user-level experiment needs a stable user identifier; an account-level workflow may need an account identifier. Requests and pageviews are usually repeated measurements of those units, not new independent participants. Every service participating in the same treatment needs the agreed unit and identifier representation.

Treat login and configuration changes as design events

If a visitor is assigned using an anonymous identifier and switches to a different identifier after login, they can receive another variant. That is a design problem to resolve, not a reason to overwrite the old event history.

Document the identity transition, warehouse join, and intended assignment persistence. Sticky bucketing persists assignments for supported workflows; it is distinct from ordinary deterministic hashing with unchanged inputs. It does not make arbitrary identity changes or incompatible configurations equivalent.

For cross-runtime parity, verify the same experiment definition, hash attribute and its type, seed, hashing version, targeting context, and applicable rules. A string identifier and a numeric identifier should not be assumed interchangeable. Matching a few sample users is a useful diagnostic; it is not proof that every production context is equivalent.

Configuration versions also belong in the handoff. A browser tab can hold yesterday's payload while a server uses a newer one. Decide how a running experiment handles configuration refreshes and which changes require a new phase or a new experiment. Record enough metadata to diagnose disagreements without sending confidential configuration into public events.

Treat an SDK upgrade as a change to verify, too. The official JavaScript SDK changelog identifies behavior changes across versions, including attribute-type handling. Preserve your lockfile and run representative targeting and assignment checks when upgrading. A matching experiment key alone does not establish that two different SDK versions and payloads make equivalent decisions.

Make caching and hydration part of the experiment

A shared cache that keys only on URL can return the first cached variant to users assigned elsewhere. You need a cache design that preserves the intended partition without leaking personalized content between users.

Cache-Control directives distinguish private responses and storage restrictions. The Vary header describes request header fields used to select a response. Neither is a substitute for checking the actual CDN and framework cache configuration. A variant cookie does not help if the relevant cache ignores it.

With server-rendered React, the initial browser render should match the server HTML. Hydration expects matching content; changing the assigned variant on the client can create mismatches as well as a compromised treatment. Pass only the public state the browser needs, preserve the initial decision, and test navigation and login transitions separately.

Verify SDK parity before wiring production delivery

Start with a local fixture and explicit prerequisites

The following fixture compares the Node shared-client API with the GrowthBook class used in browser integrations. It runs both inside Node against synthetic configuration. It does not render a page, contact GrowthBook, send analytics, or implement production identity.

Use a currently maintained Node.js release and an empty local directory. The callback context used below requires GrowthBook SDK 1.7.0 or later; this example pins version 2.0.0. Install the package, keep the lockfile, and inspect the resolved version before continuing:

npm init -y
npm install @growthbook/growthbook@2.0.0
npm ls @growthbook/growthbook

If installation fails, fix that prerequisite before running the fixture. Do not replace the SDK with a homemade hash and treat that as a check of the SDK's behavior. The browser JavaScript SDK documents the GrowthBook class, payload initialization, and tracking callback used here.

Save this as parity.mjs. The payload is an illustrative 50/50 experiment, not a production SDK connection or a CMS-created experiment:

import assert from "node:assert/strict";
import { GrowthBook, GrowthBookClient } from "@growthbook/growthbook";

const payload = {
  features: {
    "checkout-layout": {
      defaultValue: "baseline",
      rules: [{
        key: "checkout-layout-parity",
        seed: "checkout-layout-parity",
        hashAttribute: "id",
        hashVersion: 2,
        variations: ["baseline", "compact"],
        weights: [0.5, 0.5],
        coverage: 1,
      }],
    },
  },
};

const diagnostics = [];
const record = (side) => (experiment, result, user) => {
  diagnostics.push({
    side,
    experiment: experiment.key,
    variation: result.key,
    id: user?.attributes.id,
  });
};

const server = new GrowthBookClient({ trackingCallback: record("server") });
await server.init({ payload });

const seen = new Set();
for (let i = 1; i <= 100; i++) {
  const attributes = { id: `qa-user-${i}` };
  const scoped = server.createScopedInstance({ attributes });
  const serverValue = scoped.getFeatureValue("checkout-layout", "baseline");

  const browser = new GrowthBook({
    attributes,
    trackingCallback: record("browser"),
  });
  browser.initSync({ payload });
  const browserValue = browser.getFeatureValue("checkout-layout", "baseline");

  assert.equal(browserValue, serverValue);
  assert.equal(scoped.getFeatureValue("checkout-layout", "baseline"), serverValue);
  seen.add(serverValue);
  browser.destroy();
}

assert.equal(seen.size, 2);
assert.ok(diagnostics.length > 0);
assert.ok(diagnostics.every((d) => /^qa-user-\d+$/.test(d.id)));
console.log("PASS: matching values, repeatability, both branches, attributed diagnostics");
server.destroy();

The callback's objects are local diagnostic records defined by this example. They are not a transport contract, and putting an object in this array does not insert an exposure into your analytics system.

Run the check and interpret its limits

Use strict assertions to fail visibly when these invariants do not hold. Check syntax, then execute the file:

node --check parity.mjs
node parity.mjs

If the assertions pass, the script prints the PASS line. An import error means the dependency is missing or cannot be resolved. An assertion failure requires inspection of the installed SDK, payload, identifiers, and context; do not change an expected value simply to silence it.

The fixture checks equal values for these synthetic identifiers, repeatability within a scoped context, both values appearing, and identifier-shaped callback diagnostics. It does not establish a precise 50/50 sample split. It also does not prove request concurrency isolation, browser compatibility, actual rendering, or production identity correctness. Add those checks in your application's environment.

Replace the fixture boundary deliberately

For a real application, initialize configuration using the documented SDK connection and its failure-handling path. Supply identity from your application rather than the synthetic loop. Consume the selected value at the point that controls the treatment, then verify the response or UI actually follows it.

Browser integration additionally needs the application's module bundler or supported script integration, configuration readiness, and rendering lifecycle. A Node console run cannot verify those. If you use React, follow the React SDK integration rather than assuming the fixture provides a provider, hooks, or hydration handoff.

Finally, connect the documented callback or supported tracking integration to your real collection system. Map its experiment key, variation key, evaluation-time identity and timestamps to your agreed schema. Keep the mapping and retry behavior inspectable; do not copy the diagnostic array into a production event pipeline and assume attribution is complete.

Validate delivery and measurement before interpreting lift

Trace one unit through the entire path

For an isolated QA user, capture the input identifier, configuration version, selected value, delivered response, exposure record, and eventual outcome join. Repeat for both branches, a returning visitor, a login transition, and a missing-configuration fallback. Use a QA environment or isolated event sink so these checks do not create production experiment outcomes.

Test the real cache path. Make requests as different units, revisit the same unit, and inspect whether cache hits preserve the intended behavior. Test server-rendered content before JavaScript runs, after hydration, and after client-side navigation. A successful flag evaluation alongside the wrong UI is a failed experiment integration.

Include failure cases rather than only a successful page load. Delay configuration, drop the analytics request in an isolated test, and open an old tab after publishing a new QA payload. Check whether each case uses the intended fallback, keeps the correct identity, and leaves enough evidence to explain its inclusion or exclusion from analysis.

For a backend service, exercise overlapping requests from different QA users while inspecting the selected values and evaluation contexts. The sequential fixture above cannot reveal a race in your own middleware. For a browser, test a slow device and a failed script request, not just a fast development laptop. These checks target the actual ways a delivery path can diverge from its assignment record.

Write the expected failure behavior before running the checks. If an event does not arrive, the acceptable result is an observable gap with a documented handling rule, not a fabricated conversion or a silent reassignment. Keep operational recovery separate from any decision to change the experiment's analytical population.

Evaluate only where the value is relevant. GrowthBook's feature rule types distinguish targeted value delivery from an experiment rule. A percentage release is not automatically a measured A/B comparison, and a callback for a value that was never consumed does not demonstrate treatment exposure.

Reconcile collection instead of trusting one callback

Server assignment can reduce reliance on a browser for one event, but client outcomes can still be missing because of blockers, consent state, navigation, or transport failure. Server events can also fail through queues, retries, schema errors, or joins. Decide what is collected and eligible under your actual consent and measurement policy, then test that implementation symmetrically across arms.

Beacon-style analytics delivery can help with page transitions, but a successful enqueue is not proof of warehouse receipt. Reconcile callback or server logs with received events and the analyzed population. Preserve meaningful timestamps and define how duplicates are handled at the randomization unit.

Look for variant-specific loss. If treatment slows the page and its exposure is logged only after an animation completes, the retained treatment cohort can look unusually engaged. Moving assignment to the server does not repair that later selection. Record the relevant eligibility event consistently and inspect where users disappear between stages.

Consider a hypothetical collection audit with 1,000 known eligible QA units in each arm and 120 purchasers in each. Both full cohorts convert at 12%. Now suppose the exposure table retains only 900 control units and 980 treatment units, while retaining every purchaser. An analysis restricted to those recorded exposures reports 13.33% versus 12.24%. The apparent difference comes from the denominator loss, not the treatment.

In production, you cannot assume you know the missing units' outcomes as you do in this constructed example. That is why reconciliation starts with the eligibility and collection definitions, rather than repairing counts until a result looks plausible. Find the stage that lost records and assess whether the original comparison remains recoverable. More sophisticated statistics cannot restore an unidentified missing cohort.

Keep the analysis and rollback decision explicit

GrowthBook's data quality checks include sample ratio mismatch and multiple exposures. Use them alongside raw event reconciliation. Passing a traffic-split check does not prove every assigned user received the right treatment, and balanced counts do not reveal all symmetric instrumentation errors.

Before the experiment starts, record the primary metric, observation window, guardrails, sample-size or duration plan, statistical method, and decision rule. Server and client delivery do not have different laws of inference. A larger event count still does not replace independent users or accounts, and repeatedly checking a fixed-horizon result still needs an appropriate decision procedure.

If you migrate an existing experiment from browser to server delivery, document the handoff before combining results. A new identity source or exposure definition can change the analyzed population even when the product treatment appears identical. Preserve the old and new collection definitions, compare them in an isolated environment, and decide whether a separate phase or a new experiment is needed.

An A/A integration check can help uncover allocation, rendering, or telemetry differences when both arms deliver the same behavior. It does not prove the absence of bias because the result is nonsignificant. Combine that diagnostic with known QA users, raw event comparisons, and code review. Check that temporary parallel collection does not cause the same assignment or purchase to be counted twice after the handoff.

Assign an owner to the rollback path as well as the launch path. Verify which configuration change restores the intended default, how cached configuration updates, and what happens to users already assigned. After a decision, preserve the experiment record and remove obsolete branches through normal engineering review.

The architecture is ready when the unit, delivered experience, and measured outcome tell the same story. Choose the runtime that can implement the treatment faithfully, then make that full path reviewable before interpreting its lift.

Build a measurable release path

Connect your application, define the outcomes that matter, and review assignment and data quality before using an experiment result to guide a release.

Get Started With GrowthBook

Experiments

Quasi-experiments: types, examples, and limits

Nov 2, 2026

min read

A rollout can create useful evidence even when you cannot randomize users. The comparison still has to earn a causal interpretation.

Your team introduces a new onboarding process in one market because its support team is ready first. Activation improves. That is encouraging, but the market also received a new acquisition campaign. A before-and-after chart cannot tell you how much improvement came from onboarding, the campaign, or changes that would have happened anyway.

A quasi-experiment uses a nonrandom assignment process and an explicit comparison strategy to estimate an intervention's effect. The comparison might come from another market, an eligibility cutoff, a policy rule, or a carefully modeled historical pattern. Its credibility depends on why that comparison represents what would have happened without the intervention.

This guide organizes the main designs around those opportunities. You'll learn which rollout situations support each design, what effects they can estimate, and which assumptions remain vulnerable even when a result looks statistically precise.

Identify the assignment mechanism before choosing a design

Quasi-experimental is a family of approaches

Definitions differ across fields. Some researchers use the label broadly for intervention evaluations without random allocation. Others reserve it for designs that exploit plausibly external variation in treatment. The practical question is the same: what makes the treated and comparison outcomes informative about a counterfactual?

The official guidance on quasi-experimental product evaluation describes several approaches and emphasizes the difficulty of choosing a comparison group. A label alone does not resolve that difficulty. Comparing users who voluntarily adopt a feature with users who ignore it may primarily measure motivation, need, or product familiarity.

An uncontrolled before-and-after study is especially weak when trends or other changes can explain the difference. Some taxonomies call this pre-experimental; others include it among weaker quasi-experimental designs. Describe the actual comparison instead of letting terminology imply more protection than the study provides. Design improvements for non-equivalent groups and time series help make those differences visible.

Group and time assignment can still be randomized

An experiment does not become quasi-experimental because you assign organizations instead of individual people. Randomly allocating organizations produces a cluster-randomized experiment. Randomly allocating eligible time blocks can produce a randomized switchback design, although carryover and time dependence need attention.

Conversely, assigning regions according to commercial priorities remains nonrandom even if the regions have similar population sizes. “Geographic” describes the unit, not the identification strategy.

GrowthBook's cluster experiment workflow supports group-level randomization with appropriately configured individual-level outcomes. Before accepting that randomization is impossible, investigate whether a different unit or a narrower eligible population would make it feasible. That is a design decision, not a reason to pretend account-level assignment creates independent user-level observations.

Specify the effect before opening the results

Write down the intervention, eligible population, assignment unit, start time, outcome window, and effect you need. “Did onboarding work?” is too vague. “How did offering the new onboarding process affect 28-day activation among eligible accounts in the first rollout market?” identifies a clearer target.

Distinguish an offer from actual adoption. Adoption may depend on a user's motivation, which also affects activation. Restricting the analysis to adopters after launch can destroy the comparison you intended to study.

In a randomized implementation, experiment configuration makes the assignment attribute, hypothesis, and variation setup explicit. Carry the same discipline into a quasi-experimental protocol, while recognizing that documentation does not make a nonrandom assignment rule ignorable.

Use rollout timing to construct a credible comparison

Controlled before-and-after designs compare changes

Suppose a hypothetical product rollout changes 28-day activation as follows. The outcome definition and enrollment windows are the same in both markets.

Market	Before rollout	After rollout	Change
First rollout market	60%	68%	+8 percentage points
Comparison market	58%	62%	+4 percentage points

The treated market improved by 8 points, but the comparison market improved by 4. A difference-in-differences calculation subtracts those changes: (68 - 60) - (62 - 58) = 4 percentage points.

The corresponding counterfactual for the treated market is 64%: its original 60% plus the comparison market's 4-point change. The estimated effect is 68% minus 64%. This is different from attributing the entire 8-point rise to the rollout.

The difference-in-differences identification argument requires that the groups' untreated outcomes would have followed parallel changes over the relevant period. Their baseline levels can differ. A simultaneous campaign confined to the rollout market threatens that argument because its effect enters the same 4-point estimate.

Inspect several pre-intervention periods and understand how rollout priority was chosen. A non-significant pretrend test does not prove the assumption; limited data may simply fail to reveal an important difference. Stable metric definitions and denominators are also necessary, since a new activation event or eligibility rule can manufacture an apparent change.

Staggered adoption needs explicit comparisons

When markets adopt in different months, “not treated yet” and “already treated” are different states. An already treated market can have an evolving treatment effect, making it an inappropriate stand-in for an untreated outcome.

Do not default to a single two-way fixed effects coefficient without understanding its comparisons. With heterogeneous or changing effects, it can mix effects that do not match the decision you want to make. Group-time treatment effect methods make treatment cohorts and comparison choices explicit.

For a product team, the operational task is to preserve adoption dates, eligibility, and untreated comparison periods before the rollout spreads. Once every market adopts, later effects may lack a credible untreated comparison. More post-rollout rows do not recreate that missing design element.

Interrupted time series uses the intervention date

An interrupted time series, or ITS, models the outcome trajectory before a known intervention and asks whether the subsequent level or slope departs from that trajectory. It is more informative than comparing two averages when a trend was already underway.

Consider a mandatory checkout change released everywhere on the same day. A daily series could assess an immediate discontinuity, a gradual slope change, or a delayed effect. Choose the expected pattern before searching for the most favorable model. The ITS tutorial and its updated materials address seasonality, autocorrelation, and concurrent events that complicate this analysis.

An interruption still needs a defensible counterfactual. If a holiday promotion begins on release day, an observed jump cannot distinguish the two changes without further information. An unaffected comparison series can strengthen the design when it shares relevant shocks, but a series touched by the same checkout change cannot serve that role.

Strengthen the design first

Review experiment-design mistakes, metric choices, and long-term impact in this session with Ronny Kohavi and Luke Sonnet before committing to a rollout comparison.

Explore Experiment Design

Use cutoffs and external rules without overstating their reach

Regression discontinuity identifies effects near a threshold

A regression discontinuity design, or RDD, can be useful when a pre-existing numerical rule determines eligibility. Imagine a hypothetical service offering proactive onboarding support to accounts with a risk score of at least 70. Accounts just above and below 70 may provide a useful local comparison if other determinants of activation change smoothly at the threshold.

The local cutoff identification argument does not require all high-risk and low-risk accounts to be comparable. It concerns the limit near the cutoff. The resulting effect should not automatically be generalized to accounts with scores of 20 or 95.

Check whether teams can manipulate the score or override eligibility. Also check whether the threshold triggers another intervention, such as a pricing change. If several benefits change at 70, the design may identify the effect of that bundle rather than onboarding support alone.

Implementation requires a defensible neighborhood and functional form. A raw difference between broad score bands is not the same as estimating a discontinuity. Local polynomial RD tools provide estimation, bandwidth selection, plotting, and bias-corrected inference; they cannot establish that the eligibility rule was uncompromised.

Fuzzy eligibility changes treatment probability

In a sharp design, the threshold determines treatment. In a fuzzy design, it changes the probability of treatment because some eligible accounts decline support or some ineligible accounts receive an exception.

Suppose, purely as a numerical illustration, fitted adoption probability jumps from 20% to 80% at the cutoff while fitted activation jumps from 63% to 69%. The ratio is 0.06 / 0.60 = 0.10, or 10 percentage points. Interpreting that ratio as a treatment effect requires additional instrumental-variable assumptions, including that eligibility affects activation only through the treatment and that treatment uptake moves in the required direction.

Those assumptions matter more than the attractive arithmetic. The effect concerns accounts whose treatment changes because of eligibility near the threshold. It is not automatically the effect for every account that used support.

Natural experiments need an assignment story

A natural experiment exploits an externally determined event or rule that generates useful treatment variation. The event is not necessarily a literal disaster or a sudden policy change. What matters is why its assignment is plausibly unrelated to the otherwise expected outcome, within the proposed comparison.

An instrumental-variable design can use such variation to study adoption. But an instrument must affect treatment, be suitably independent of outcome determinants, and satisfy an exclusion argument. Instrumental-variable assumptions and local effects explain why a predictor of adoption is not automatically a valid instrument.

For example, a support-team capacity rule may change whether an account receives onboarding. If that capacity also changes response times for unrelated support requests, the rule can affect activation through another path. Calling the variation external does not remove that problem. Preserve the operational records needed to examine those paths before staff assignments or exceptions are forgotten.

Construct comparison groups without hiding selection

Synthetic controls combine untreated units

When one market receives an intervention, a synthetic control can combine other markets into a weighted comparison chosen to resemble its pre-intervention trajectory and relevant predictors. It can be useful when no single market is a convincing comparison.

In a hypothetical illustration, weights of 60% and 40% on donor outcomes of 24 and 29 produce a synthetic outcome of 26. If the treated outcome is 30, the post-intervention gap is 4 outcome units. That gap becomes a causal estimate only if the donor construction and counterfactual assumptions are credible.

The feasibility requirements for synthetic controls include the availability of suitable donors and informative pre-intervention data. Donors affected by the intervention, related campaigns, or spillovers can contaminate the comparison. Good historical fit is useful evidence, but it does not guarantee that the treated market avoided a unique post-intervention shock.

Choose donors and fitting decisions without optimizing the desired post-intervention result. Compare alternative defensible donor pools and examine placebo analyses, while recognizing that a small, unusual donor set limits what those comparisons can establish.

Matching and weighting depend on measured information

Matching can align treated and comparison accounts on recorded characteristics. Weighting can reweight observations toward a specified population. These approaches may improve a non-equivalent-group design, but balance on available columns does not establish balance on unmeasured intent or readiness.

The unconfoundedness and overlap requirements are central to causal interpretations based on measured adjustment. If every enterprise account receives onboarding and no comparable enterprise account goes untreated, a model may be extrapolating rather than making a supported comparison.

Use pre-treatment variables appropriate to the causal question. Do not adjust for completed onboarding steps when those steps are part of how the intervention changes activation. That can remove part of the effect or introduce selection bias. Treat the choice of adjustment variables as a design decision, not an automatic instruction to include every warehouse field.

Let the assignment process narrow the candidates

Available variation	Candidate design	Main question to resolve
Some markets change while others remain untreated	Controlled before-and-after or DiD	Would untreated outcomes have changed similarly?
A known date interrupts a sufficiently informative series	ITS, preferably with a suitable control series	Could another event or trend explain the interruption?
Eligibility changes at a numerical threshold	Sharp or fuzzy RDD	Are potential outcomes continuous and the rule unmanipulated?
An external rule changes treatment uptake	Natural experiment or IV	Does the rule affect the outcome through another path?
One treated unit has suitable unaffected donors	Synthetic control	Can the donors support a credible post-intervention counterfactual?

If none of these arguments fits, a descriptive comparison may still be useful. Report it as descriptive rather than converting it into a causal claim through a more complicated model.

Audit threats that precise estimates can conceal

Keep population and exposure definitions stable

An activation rate can rise because activation improved or because the analyzed accounts changed. Record who was eligible at each date, who entered the data, and how incomplete outcome windows were handled. A later cohort with only 7 days of follow-up should not be compared with an earlier cohort's completed 28-day outcome.

Differential attrition creates another problem. If the intervention changes whether accounts remain observable, complete-case analysis can compare selected survivors. Similarly, filtering to users who revisit can condition on behavior caused by treatment. Count definitions and exclusions should follow the question, not the apparent direction of the result.

Also distinguish assigned availability from actual exposure. A rollout date does not mean every account encountered the new process that day. A per-protocol question about adoption needs its own assumptions; it cannot quietly replace an offer-based question after the analysis begins.

Trace spillovers, anticipation, and shared shocks

Markets can affect one another through referrals, shared sales teams, supply constraints, or users crossing boundaries. An untreated region may therefore receive part of the treatment's effect. A comparison that is administratively untreated is not necessarily causally unaffected.

People may also change behavior before the official launch because of an announcement or training. Decide whether the treatment includes that anticipation period and adjust the protocol accordingly. The date in the release calendar may be too late to define the intervention's true start.

For randomized work, GrowthBook's data quality checks surface issues such as sample ratio mismatch and multiple exposures. Those checks support an experiment's data review; they do not test the parallel trends, exclusion, or continuity assumptions of a quasi-experiment.

Match uncertainty to the source of variation

Thousands of user rows from one treated region do not create thousands of independently treated regions. Shared regional shocks and repeated measurements can make row-level standard errors misleading. The inference method needs to reflect the unit and time structure that generate the treatment comparison.

Few treated or comparison units are particularly challenging. Cluster-robust calculations, resampling, or placebo procedures have design-specific requirements; none creates independent information that was absent from the rollout. Report those limits alongside the interval instead of hiding them behind a large event count.

A falsification check should challenge a particular explanation. For example, a rollout should not change an outcome measured before any announcement or exposure. An apparent effect there can reveal a timing problem, selection, or model failure. A negative-control outcome can be useful when it shares the suspected confounding process but cannot plausibly respond to treatment. The logic of falsification testing helps distinguish that exercise from simply adding more favorable outcome charts.

Passing a check is weaker evidence than failing a well-chosen one. An unaffected outcome may not share the relevant confounder, and a placebo test may have little power. Explain why the check would reveal the threat you care about, rather than treating a collection of non-significant tests as certification.

Statistical precision also addresses a different question from identification. A narrow interval under a misspecified counterfactual can be precisely wrong. The interpretation of experiment results should keep uncertainty and data quality visible, while a quasi-experimental review must additionally evaluate the assumptions creating its comparison.

Write a protocol that can survive the rollout review

Record the causal argument before estimating the effect

A useful protocol begins with a decision: what rollout choice will this evidence inform, and what magnitude would matter? Then state the treatment version, target population, assignment mechanism, counterfactual, outcome scale, observation window, and primary analysis.

Add a threat-specific diagnostic plan. For a cutoff, inspect the assignment rule, score distribution, and other changes at the threshold. For a timed rollout, investigate pretrends, announcements, concurrent campaigns, and comparison-group exposure. Choose checks because they challenge the causal argument, not because they are easy to pass.

State what would change the conclusion. A missing historical period, an unexplained composition break, or a coincident market-specific campaign might downgrade the analysis to descriptive evidence. This makes the review useful even when the preferred causal interpretation fails.

Preserve operational evidence while the rollout is happening. Keep the original eligibility rule, announcement and deployment dates, exception log, support-capacity changes, and a record of campaigns affecting each comparison unit. Save the version of the score used for cutoff assignment rather than a later score updated after treatment. Historical outcomes alone cannot reconstruct those decisions reliably.

For the onboarding example, an audit-ready record might say: the offer began on a specified date; eligibility used account status recorded beforehand; accounts received a completed 28-day window; the comparison market retained its original process; and the concurrent campaign was either separately identified or acknowledged as unresolved. Each statement answers a concrete design question. A table containing only account IDs, a treatment flag, and activation cannot answer them all.

Report the scope and alternatives with the estimate

Share the estimate with its units, uncertainty, target population, comparison, and identifying assumptions. Separate prespecified analyses from sensitivity checks and later exploration. A result near an eligibility cutoff, an effect on treated markets, and an effect for instrument-induced adopters are different answers.

When several defensible specifications agree, explain what they share. Agreement is less reassuring if every specification assumes away the same unmeasured campaign. When results differ, show which assumption, window, or population change drives the difference rather than selecting the most favorable estimate.

Keep transport to the next rollout separate from identification of the first effect. A result for accounts near a risk threshold may not predict what happens when support is offered to everyone. Wider deployment can change staff capacity, user composition, or interactions between accounts. Even a credible local estimate may therefore support a staged follow-up rather than a universal rollout decision.

Treat that limitation as useful information for planning. Name which accounts and operational conditions the evidence covers, which extension is uncertain, and what additional comparison would reduce that uncertainty. This gives the next experiment a specific job instead of repeating the same broad question with another model.

Preserve opportunities for randomized evidence

Operational constraints may leave room for a randomized subpopulation, cluster assignment, or delayed release. A planned A/B design can sometimes answer a narrower question more credibly than a broad observational comparison. For long-term cumulative release questions, holdouts preserve a baseline across an explicitly defined feature scope.

GrowthBook's experiment decision framework helps connect randomized experiment evidence to success criteria and target precision. Its experimentation platform also supports reusable warehouse metrics and visible SQL. Those foundations help teams maintain consistent measurement; they do not turn a nonrandom rollout into a randomized study or supply a native estimator for every quasi-experimental design.

Choose the strongest comparison your assignment process can justify. Then make the limits clear enough that the next person reviewing the rollout can tell what the evidence supports and what still requires another study.

Build the randomized comparison

When your rollout allows random assignment, use GrowthBook to connect the experiment to explicit warehouse metrics and a reviewable decision process.

Get Started With GrowthBook

Ready to ship faster?

No credit card required. Start with feature flags, experimentation, and product analytics—free.

Get Started

Book a Demo

Simplified white illustration of a right angle ruler or carpenter's square tool.

White checkmark symbol with a scattered pixelated effect around its edges on a transparent background.

Most teams tracking model performance or running A/B experiments focus on whether their system is catching real positives — but that's only one piece of a four-part picture.

A true positive has a two-part definition — and getting either part wrong breaks your analysis

Why "a correct result" is not specific enough

The four-outcome binary classification framework

The definition holds across domains; what changes is the cost of getting it wrong

True positives only have meaning inside the full four-outcome confusion matrix

How the four cells map every possible prediction against reality

False positives (Type I errors): the cost of crying wolf

False negatives (Type II errors): the cost of missing what's real

Why no single cell in the confusion matrix tells you whether your model is good

True positive rate: the metric that tells you how much of reality your system is actually catching

The TPR formula and what it actually measures

What high and low TPR signals about your system

TPR and the ROC curve

The sensitivity-specificity trade-off: why maximizing true positives isn't always the goal

The fundamental trade-off: why you can't maximize both

When high sensitivity is the right call

When specificity deserves priority

Threshold selection is a business decision, not a statistical one

True positives in A/B testing: correctly detecting real effects and avoiding false discoveries

Where true positives sit in the A/B testing decision space

The peeking problem and multiple testing

Underpowered tests don't inflate false positives — they suppress true positives

A/A testing eliminates infrastructure failures before they corrupt your true positive rate

Putting it together: calibrating your system to the actual cost of each error

Match your sensitivity-specificity balance to the cost of each error type

The confusion matrix and TPR reveal what accuracy alone conceals

Protect true positive rates in experimentation with rigorous statistical practices

Related reading

Table of Contents

Related Articles

Degrees of freedom in statistics, explained simply

The same test statistic can support different conclusions when its degrees of freedom change.

Start with the constraint behind n minus 1

Three observations, two freely varying deviations

Why estimating the mean changes the denominator

Keep the intuition within its scope

See how df changes a statistical threshold

A t-statistic needs a reference curve

Degrees of freedom are not the effect size

Make small effects easier to detect

Choose the formula for the method you actually use

One sample, pairs, and pooled groups

Welch can produce fractional df

Regression and count tables constrain different things

Check the experimental unit before trusting a large df

Repeated events do not create independent users

Missingness changes the usable data

Report the method behind the number

Keep df attached to the estimate and interval

Resolve design problems before chasing precision

Build a test you can explain

Server-side vs. client-side A/B testing

Choosing where a variant runs is only the first architecture decision.

Separate the experiment's four locations

Assignment is different from treatment delivery

Exposure and analysis can live elsewhere

Compare complete delivery paths

Put the treatment where it can be implemented faithfully

Use the browser for changes the browser owns

Use the server for backend behavior and controlled responses

Measure performance instead of promising it

Choose the smallest complete treatment

Plan flags across your stack

Keep identity and delivered variants consistent

Scope user attributes to the request

Treat login and configuration changes as design events

Make caching and hydration part of the experiment

Verify SDK parity before wiring production delivery

Start with a local fixture and explicit prerequisites

Run the check and interpret its limits

Replace the fixture boundary deliberately

Validate delivery and measurement before interpreting lift

Trace one unit through the entire path

Reconcile collection instead of trusting one callback

Keep the analysis and rollback decision explicit

Build a measurable release path

Quasi-experiments: types, examples, and limits

A rollout can create useful evidence even when you cannot randomize users. The comparison still has to earn a causal interpretation.

Identify the assignment mechanism before choosing a design