What are confounding variables in experiments?

A confounding variable gives a rival explanation for an apparent treatment effect.

Suppose a redesigned checkout is released to mobile users on Monday and the original checkout stays on desktop. Revenue per visitor rises. Did the redesign cause the increase, or do mobile shoppers differ from desktop shoppers in ways that also affect revenue? Device type is entangled with treatment assignment and outcome, so the comparison cannot isolate the redesign's effect.

That entanglement is confounding. It is one of the main reasons an association is not automatically causal. In a well-run randomized experiment, assignment severs the systematic connection between pre-existing user characteristics and treatment. In an observational comparison, rollout, or broken experiment, the connection can remain.

For product teams, the practical lesson is not to collect every imaginable user attribute. It is to protect the assignment and measurement process that makes a causal comparison credible, then use diagnostics to find places where that process failed.

A confounder is related to assignment and outcome

A variable is a confounder when it satisfies three conditions:

1. It exists before the treatment is assigned.
2. It is associated with which treatment a unit receives.
3. It independently affects, predicts, or shares a cause with the outcome.

Penn State's statistics definition of a confounding variable describes it as a characteristic related to both the explanatory and response variables. In an experiment, those are the treatment and outcome.

Consider a pricing-page experiment in which enterprise accounts disproportionately enter treatment because the new page was first enabled for a particular sales segment. Account size predicts both assignment and contract value. A naive revenue comparison attributes the account-mix difference to the page.

The direction of bias is not fixed. A confounder can exaggerate a real effect, hide one, reverse its apparent sign, or create an effect where none exists. That is why a large sample or a small p-value cannot rescue a structurally biased comparison. More biased data can produce a more precise answer to the wrong question.

Confounders are not the same as every nuisance variable

A nuisance variable influences the outcome but is not necessarily associated with treatment. Day of week may affect conversion, for example. If both variants run concurrently with randomized assignment, each day should contribute to both arms, and the daily variation adds noise rather than systematic bias.

The distinction changes the response:

• A prognostic but balanced variable is often a precision opportunity. CUPED, stratification, or regression adjustment may reduce variance.
• A pre-treatment variable imbalanced by chance deserves investigation, but it does not prove the randomizer failed.
• A variable linked to assignment through targeting, logging, attrition, or eligibility is a plausible source of bias.
• A variable caused by treatment is not a pre-treatment confounder. Conditioning on it can introduce new bias.

The CDC's guidance on interpreting associations separates confounding from chance, selection bias, and information bias. Product experiment reviews benefit from the same discipline: name the failure mechanism instead of calling every suspicious pattern “a confounder.”

Randomization prevents systematic confounding

Random assignment gives every eligible unit a known chance of receiving each variation. Before treatment, assignment is independent of user characteristics in expectation. That protects the comparison from both measured and unmeasured pre-treatment confounders.

GrowthBook's guide to designing an A/B test starts with a clear population, randomization unit, exposure definition, and metric plan. Those choices determine what causal comparison the test can support.

Randomize at the unit that can receive the treatment independently

If a treatment is experienced by an organization, randomizing individual users within that organization may cause spillover. If it affects a shared marketplace, one buyer's treatment can change another seller's outcome. GrowthBook's guide to trustworthy experiment design explains why interference and mismatched randomization units must be addressed in the design rather than patched after analysis.

Choose the unit at which treatment can be assigned without contamination: user, account, household, device, team, region, or time block. Then use that unit consistently in assignment, exposure, and metric aggregation.

Run control and treatment concurrently

Before-after comparisons are vulnerable to time as a confounder. Marketing campaigns, holidays, outages, competitor changes, and user maturation can coincide with the release. A concurrent randomized control experiences the same calendar conditions and supplies the counterfactual the treated group needs.

When concurrency is impossible, designs such as switchbacks or randomized blocks can separate treatment from predictable time effects. NIST's randomized block design guidance treats blocking as a way to hold important nuisance conditions homogeneous while randomizing within each block.

Design experiments you can trust

Learn how to choose units, metrics, and validity checks before a rollout turns into an ambiguous comparison.

Watch the Webinar

How confounding enters a product experiment

Randomization is necessary, but the entire experiment pipeline must preserve it. Many apparent “statistical” problems are implementation problems downstream of assignment.

Targeting is coupled to treatment

If eligibility rules differ across variants, treatment becomes associated with the targeting attribute. A common version is enabling a new experience only in one country or on one app version, then comparing it with everyone else.

Target the eligible population first, then randomize within it. If the business question concerns a segment, predeclare the segment and ensure both arms contain eligible units from it. GrowthBook's feature flag experiments documentation describes how a flag can define targeting and variation assignment while the experiment analyzes outcomes.

Exposure logging differs by variation

Imagine that control logs exposure when the page starts loading, while treatment logs only after a new component renders. Slow or failing treatment sessions disappear before entering the analysis. The observed treatment group is now selected for successful rendering, which is related to both recorded treatment and conversion.

Log exposure at the same logical decision point for every variation. Audit missing identifiers, duplicate events, client-versus-server timestamps, and the join from exposures to metrics. GrowthBook's data source model keeps raw experiment data in the warehouse, which makes these joins inspectable rather than hiding them behind an opaque event pipeline.

Attrition or missingness depends on treatment

Treatment can affect whether an outcome is observable. An onboarding flow that makes account creation harder may cause users to leave before an ID is created. If downstream analysis requires that ID, the most negatively affected treatment users vanish.

Define the analysis population from assignment or the earliest symmetric exposure event, not from a downstream action the treatment can influence. Missing outcomes need explicit treatment, not silent inner joins.

Analysts filter on a post-treatment variable

Filtering to “users who completed onboarding,” “sessions that loaded successfully,” or “customers who remained subscribed” can condition on something caused by treatment. This can create collider or selection bias, even if the original assignment was perfect.

Keep the intent-to-treat population as the primary analysis. Triggered analysis is appropriate only when the trigger is measured before treatment can affect it or occurs at a symmetric exposure boundary.

Multiple experiments or releases move together

If every user assigned to experiment A's treatment is also assigned to experiment B's treatment, the two effects are aliased. A concurrent infrastructure migration can create the same problem if it reaches only one variation. Use independent assignment seeds, record overlapping experiment membership, and isolate coupled changes when the interaction would change the decision.

Diagnose confounding without data dredging

No single chart proves that an experiment is unconfounded. Use a chain of checks tied to plausible failure modes.

Start with the assignment and exposure funnel

Verify expected allocation, eligible counts, exposure timing, identifier coverage, duplicate rates, and metric join rates. A sample ratio mismatch indicates that observed counts differ more than expected from the planned split. SRM does not identify the cause, but it is a reason to stop interpreting lift until the assignment-to-analysis pipeline is understood.

Break those checks down by dimensions that could affect routing or logging: platform, app version, geography, browser, enrollment date, and service path. A global 50/50 split can conceal a mobile imbalance that cancels a desktop imbalance.

Compare pre-treatment variables

Inspect pre-period activity, account age, historical spend, locale, plan, and other strong predictors measured before assignment. Large or systematic differences may reveal targeting leaks, identifier collisions, or non-random enrollment.

Do not turn this into hundreds of significance tests and reject every experiment with one nominally significant difference. With enough covariates, chance findings are inevitable. Look for effect sizes, consistent patterns, and a technical path that explains the imbalance.

Use negative controls and invariant metrics

A treatment cannot cause an outcome that occurred before assignment. If treatment appears to move last week's purchases, acquisition source, or account creation date, the groups or joins differ. Likewise, identifiers or device properties that should be invariant can expose instrumentation problems.

Segment to investigate, not to manufacture a winner

Simpson's paradox can make an aggregate relationship reverse within meaningful subgroups. Segmenting by pre-treatment device or country may reveal a mix imbalance. But slicing the result until one group looks positive is not a correction. The segment should correspond to a predeclared mechanism or a reproducible diagnosis.

GrowthBook's experiment analysis and metric framework let teams define analysis around warehouse data, guardrails, and dimensions. The tooling supports the investigation; it does not replace causal judgment about which variables are safe to condition on.

Prevention and adjustment are different jobs

Design prevents bias. Analysis can improve precision and sometimes adjust for measured differences, but it cannot recreate a valid random experiment from a broken one.

Prevent confounding before launch

Use this preflight:

1. State the causal contrast and eligible population.
2. Choose the randomization unit and stable identifier.
3. Randomize within any necessary blocks or strata.
4. Define one symmetric exposure boundary.
5. Freeze primary metrics and inclusion rules before reading results.
6. Confirm control and treatment run over the same time window.
7. Log assignment, exposure, relevant pre-treatment attributes, and data versions.
8. Test allocation, exposure, and joins with an A/A test or limited rollout.

Adjust only with pre-treatment information

Regression adjustment, post-stratification, and CUPED can account for measured pre-treatment predictors and improve sensitivity. GrowthBook's guide to variance reduction techniques covers CUPED and post-stratification alongside metric design and winsorization.

Adjustment is safest when covariates are defined before the experiment, measured consistently, and unaffected by treatment. Never control for a mediator merely because it correlates with the outcome. If the new checkout changes page latency and latency changes conversion, latency is part of the treatment pathway, not an ordinary confounder to subtract away.

A causal diagram can make those assumptions reviewable before anyone writes a model. The BMJ guide to directed acyclic graphs shows how diagrams distinguish causal paths from backdoor paths that require adjustment. The NIH review of causal-diagram pitfalls also cautions that diagrams encode qualitative assumptions; they do not determine a valid adjustment from data alone.

Rerun when the causal comparison is no longer credible

If one arm lost exposure events, eligibility changed mid-test, or a coupled release reached only treatment, a fresh run is often cheaper than debating a fragile correction. Fix the mechanism, start a new phase with independent assignment, and preserve the incident as an experiment-quality lesson.

The decision rule is simple: if a reasonable reviewer can still explain the result through a variable tied to assignment and outcome, do not call the observed difference causal. A trustworthy experiment makes the alternative explanations implausible by design.

Work through the causal story before choosing an adjustment

Suppose a checkout treatment is enabled only for signed-in users while control includes anonymous visitors. Account status predicts both treatment and purchase probability, so the variation comparison mixes the interface effect with a population difference. Adding account status to a regression may reduce imbalance, but it does not recover anonymous users who could never receive treatment. The clean fix is to define a shared eligible population and randomize within it.

Now suppose both arms begin balanced, but treatment makes sign-in more likely and the analyst filters the result to signed-in users. Sign-in is post-treatment. Conditioning on it selects populations partly created by the variants and can bias the comparison. The solution is to analyze everyone randomized, with sign-in as an outcome or diagnostic rather than an eligibility filter.

These cases look similar in a dashboard because both involve account status. Their remedies differ because timing and causal position differ. For every proposed control variable, ask when it was measured, whether treatment could change it, and which causal path the adjustment is intended to block. That review is more valuable than automatically adding every predictive column to a model.

Build cleaner experiment evidence

Connect assignment and warehouse metrics in one workflow so your team can inspect quality checks before making a rollout decision.

Start Building Free