Designing experiments that produce trustworthy results: a pre-launch guide to validity threats

Every experimentation team has had this experience: An A/B test showed a statistically significant lift, but after the change was rolled out, the primary metric didn't budge. It turned out the intervention didn't have an effect and the lift was caused by a flaw in the experimental design that introduced bias. By the time you're analyzing an experiment's results, many of the decisions that determine whether those results are reliable have already been made. 

We've already looked at several components of trustworthy experiments in depth: reducing false positive rates for more reliable wins, reducing variance to run conclusive experiments more often, and diagnosing sample ratio mismatch when you detect sample bias.

This article is a pre-launch framework for the decisions you need to make and the checks you should run before your experiment starts collecting data to ensure your results are trustworthy.

The 4 validity types that matter for experiment design

Validity describes the trustworthiness of an experiment's conclusion, whether the effect it reports is real or just a flaw in how the test was built. The standard experimental-design framework, from Shadish, Cook, and Campbell, breaks it into 4 types.

1. Internal validity

Randomization is supposed to guarantee internal validity, meaning the change you tested, rather than some other factor, caused the observed difference between the test and control groups. But two randomized groups stop being comparable when the intervention itself causes a change in their composition.

For example, if a new checkout flow loads half a second slower, impatient users may abandon it before they're counted in the analysis, leaving the test group without the users who would drag conversion down. Novelty, network effects, selective enrollment, and attrition can all threaten internal validity when an experiment isn't designed properly.

2. External validity

A 2-week test run over the winter holidays can show a statistically significant lift that disappears in January when normal user behavior returns. When that happens, you have an issue with external validity.

External validity requires that an experiment's measured effect apply to more than just the users and the time period you tested. An effect won't generalize when the conditions during the test differ from the conditions after rollout. Plan to run your experiment during a representative period on a representative sample to get reliable results.

3. Construct validity

An experiment can be internally and externally valid while still measuring the wrong thing. Construct validity means the metric you measure actually reflects the outcome you care about. Metrics are often just proxies for something harder to measure, and they can improve even when the outcome they represent does not.

For example, a recommendation carousel test might measure click-through as a stand-in for purchases, but a variation can lift clicks while purchases stay flat. Choosing the right metric up front prevents issues with construct validity.

4. Statistical conclusion validity

Statistical conclusion validity depends on your analysis supporting your conclusions. Checking the dashboard each morning and stopping when the result looks significant raises the chance that the winner you declare is a false positive, even when the design itself is sound. You can protect statistical conclusion validity by setting the sample size in advance and pre-specifying the analysis details before launch.

GrowthBook's statistical validity guide covers 2 more types as well as common threats to validity like p-hacking, peeking, and regression to the mean.

How to design for experiment validity before you collect any data

The decisions that determine an experiment's validity happen at the planning stage. Pre-registering an experiment means fixing all the details in advance, such as the hypothesis, the primary metric, the sample size, and stopping criteria. Committing to them before launch means you can't go looking for a significant result by peeking at the data or choosing what to measure after you've seen the numbers.

Start with a hypothesis that has a mechanism

Without a stated mechanism in your hypothesis, you have no way to pick the right metric or to judge whether a result makes sense. Write your hypothesis so the reason is explicit. A clear research hypothesis names the change, the metric, the expected direction, and the mechanism: we believe [change] will move [metric] [direction] because [mechanism]. The because is easy to leave out, and it's the part that makes a null result informative. When the metric doesn't move, you can use the mechanism to decide which link in the causal chain to check first. Stating it up front also constrains the metric you pick next, because it names the specific behavior the feature is supposed to change.

Match your randomization unit to your analysis unit

Randomization creates balanced groups, but this balance only applies to the unit you make assignments on. For example, if you randomized by user, the user groups would be balanced. If you then analyzed that experiment using sessions instead, you would no longer be comparing the same groups you balanced. Since one user can generate multiple sessions, a variation that alters user behavior (like visit frequency) can skew the session count in one arm for reasons independent of the metric you are testing.

A mismatch between identifiers can cause the same problem. Many metrics rely on anonymous IDs, because the tracking events fire before users sign in. If your SDK assigns on a user ID, one user can map to several anonymous IDs in your data, and those IDs may not split evenly across your groups. To prevent this, assign and analyze on the same identifier. You’ll need to choose the one that's available when users first see the change: the user ID on signed-in surfaces or the anonymous ID before sign-in.

Randomize whole groups when users affect each other

Some treatments change how users affect each other. Standard randomization assumes that one user's assignment doesn't change another user's outcome. Statisticians call this the stable unit treatment value assumption, or SUTVA.

For example, imagine a referral test where the treatment arm sees an offer for $10 if they refer a friend, and the control arm doesn't. Treatment users send invitations. When the invited friends arrive, some are assigned to the control group, and of course they sign up. The control group's conversion rate rises because of the treatment, so the gap between the arms shrinks and the measured effect understates what the feature does. The split is still 50/50, and every health check passes, but the estimate is wrong. 

Marketplaces where buyers and sellers draw on shared inventory have the same problem, as does any social feature where treated and control users interact. The measured effect is still biased, and you can't easily predict the direction.

There's no way to correct for interference between groups in the analysis afterward, so you have to design for it in advance. Cluster randomization assigns whole groups to the same variation, like every user in a city or a connected social graph, so the effect stays contained inside a cluster instead of leaking between groups. Geographic holdouts and time-based designs, where the treatment switches on and off across time periods, do the same job when the groups don't divide neatly. The best option depends on how your users connect to each other. Whichever design you choose, the decision has to happen before launch.

Re-randomize when you reuse a surface

Reusing the same randomization across back-to-back experiments lets users keep their old assignment, so anyone in the treatment arm last time stays in it for the new test. If the earlier treatment caused a lasting effect, that effect is now correlated with the new assignment and biases the result. For example, if a previous feature redesign confused users, their temporary frustration will artificially drag down engagement with the new tool. Re-randomize with a fresh seed for every experiment, so each user's new assignment is independent of any prior test. GrowthBook does this automatically when you start a new phase with re-randomization or create a fresh experiment.

Keep the arms identical except for the change you're testing

A new variant introduces new code, and new code can differ from the control in ways you didn't intend. It might degrade performance or have a bug the control doesn't, and suddenly you’re not just measuring the intervention, but also everything else that came along with it.

For example, a redesign might appear to fail simply because the new page loads more slowly, confounding the design change with added latency. Compare latency and error rate across the arms so that a performance issue doesn’t bias the treatment effect.

You also need to measure all experiment arms the same way. They have to fire the exposure event (which records that a user entered the experiment) at the same point and use identical metric definitions. If the exposure event fires after variation-specific code runs, the slower arm will lose users before they're counted, creating a sample ratio mismatch (SRM) and a biased result. 

An A/A test, where both arms serve the same experience, can help you check your measurement setup. With no real difference between the arms, any difference the analysis reports came from your assignment or tracking. But it can't catch problems that only appear when the real variant runs, so monitor latency, error rates, and SRM warnings during the live test.

Choose a single primary metric that measures the right thing

The more metrics you analyze, the more likely you are to find a false positive. You can control that risk with a multiple comparisons correction, but you lose some sensitivity when you do. Declare a single primary metric that directly tests your hypothesis. Ideally, it should connect to your Overall Evaluation Criterion (OEC), the organization-wide definition of success that experiments are judged against.

Three questions help with primary metric selection: 

1. Is it proximate to the feature, meaning it measures something the feature directly changes? 
2. Does it predict the long-term outcome you want, rather than a short-term proxy that can rise while the real goal stays flat? 
3. Is it a rate or a binary metric, rather than a total that a handful of high-spending users can disproportionately influence? 

With a proximate, predictive, outlier-resistant metric, you can detect a real effect with far less data than with a noisy one.

In addition to your primary metric, you can also declare guardrail metrics. These are the metrics that must not degrade. Revenue, support tickets, latency, and error rate are all common here. You don't use a guardrail metric to call a winner, even when it improves. Instead, its purpose is to detect when the variant makes something important worse that the primary metric wouldn't catch. When you declare guardrails before launch, you see the decline during the test instead of after rollout.

Set your decision criteria before launch, too. Your metric choices describe what to measure, not when a result is good enough to ship, so document which combination of outcomes means you ship, roll back, or iterate. For example, you might only ship if the primary metric improves with 80%+ confidence and your guardrail metrics don’t degrade. Committing to your criteria up front is part of pre-registration, and it helps make sure all your stakeholders are aligned on what success looks like. 

Calculate your sample size and set a minimum duration 

Peeking at results early and acting on what you see is one of the easiest ways to accidentally increase your false positive rate. Without a sample-size target set in advance, every early look triggers a decision about whether or not to stop. As you're designing your experiment, run a power analysis to determine your sample-size target.

Power analysis is built on 3 inputs: 

• Minimum detectable effect: the smallest increase in a metric's performance that would be meaningful for your business.
• Baseline value: your primary metric's current value for the same audience, from 2-4 weeks of pre-experiment data.
• Power target: the chance of detecting a real effect if there is one. 80% is standard for most decisions, but you may want to go higher if missing a real effect would be costly.

Split the two arms evenly unless you have a reason not to, because a 50/50 allocation gets the most power out of a given sample. Using a ratio like 90/10 to limit exposure is a reasonable safety choice on riskier changes, but it will cost you sensitivity and lengthen the test.

If your test enrolls users continuously, like most product and website tests, it also needs a minimum duration, regardless of when you reach your sample-size target. A 2-week minimum is common. User behavior varies across the week, so running experiments in whole-week increments gives every day equal representation. Duration also needs to outlast the burst of first-time curiosity a new feature draws, and features that change habitual behavior, like navigation or a core workflow, need longer still.

Some metrics also take time to mature. If your primary metric is completing a 7-day trial, users who enroll in the test's final week haven't had time to complete it. Set a conversion window on the metric, which counts each user's events only within a fixed period after their own exposure, in this case 7 days, so a user who enrolled on day one and a user who enrolled in the final week are measured on the same terms. You’ll need to wait for the last enrollees' windows to close before analyzing. The window keeps the measurement period equal rather than shortening the wait.

Together, the sample size, the minimum duration, and metric maturity determine your analysis date. Plan to analyze the results once, when all 3 are met. If you know you'll want to act on results as they come in, enable sequential testing at design time, which adjusts the statistics so that repeated looks don't inflate the Type I error rate. 

Plan how concurrent experiments share users

When several experiments run on the same audience at the same time, they will inevitably share users. That's usually fine because experiments assign users independently, and shared users spread evenly across each experiment's arms. When Microsoft analyzed every pair of concurrent tests running across 4 of its products, 3 products showed no statistically significant interactions at all, and the fourth showed them in 1 in 50,000 test-pair metrics.

But when you expect two treatments to interact, for example if both are making changes to the same onboarding flow, you can design the experiments to be mutually exclusive so users only get enrolled in one. GrowthBook supports this with namespaces. Since mutual exclusion spreads your sample further, reserve it for experiments that actually interact.

Run pre-experiment checks before launch

A final round of checks catches potential validity issues while they are still relatively simple to correct. Before the experiment goes live, confirm the following:

• A/A test passed: Run an A/A test on any new experiment surface or after making tracking changes. Since both arms are identical, any statistically significant result points to a problem in your instrumentation or assignment.
• Pre-experiment balance check: If you’re assigning a fixed cohort before launch, confirm the groups don’t already differ on your primary metric (like revenue per user) before the treatment starts. An imbalance that predates the experiment means your results will be biased from the start. This is only possible for batch-assigned experiments on existing users who have historical data. 
• Enrollment overlap reviewed: Check which live experiments this one will overlap with. Having users in multiple experiments is fine as long as they change independent parts of the product. If two do interact, make them mutually exclusive so users only get enrolled in one experiment. You can do this in GrowthBook using a shared namespace.
• Runtime confirmed: Given your user volume and minimum detectable effect, confirm the experiment can reach its required sample size in a reasonable amount of time. A test projected to take 9 months at your current volume needs a different design.

Common threats to internal and external validity in experimentation

The design decisions above reduce the risk of a validity failure, but they can't eliminate it entirely. Once the experiment is live, watch for these common threats to validity, especially when the results don't seem to make sense.

Novelty and primacy effects

Users behave differently when they encounter something new, so the first week of data often shows a bigger effect than later weeks will. Novelty pushes the early effect up, and its opposite, the primacy effect, pushes it down. This is because experienced users need time to adapt before a change starts helping them. When a test is too short, it measures the reaction to a change rather than the change itself. You can spot novelty after rollout, when a winning result decays over the following weeks.

Network effects and SUTVA violations

Network effects are hard to spot because they pass standard health checks. You'll see a balanced split between groups, but if users are influencing one another, the results are skewed anyway. Beware of SUTVA violations anytime your experiment involves referrals, social interactions, or shared inventory, especially when the results are surprising. Since you can't fix this bias after the fact, it's best to catch it early and rerun the test using a group-based design, such as cluster randomization.

Interference from concurrent experiments

Concurrent experiments usually only contaminate each other when their treatments interact, like two changes to the same algorithm. As covered earlier, independent assignment keeps ordinary overlap from biasing either result. That said, you should suspect an interaction when two teams have shipped changes to the same surface at the same time, and either result looks implausible. To verify your results, make the experiments mutually exclusive with a namespace and rerun them, so that each test measures its own change.

The pre-launch checklist for trustworthy experiments

This checklist is your pre-registration record. Answer each item with a yes or no before the experiment goes live. Each item corresponds to an A/B testing best practice covered in the sections above.

Hypothesis written with its mechanism. We believe [change] will move [metric] [direction] because [mechanism].

• Decision-making criteria written down before launch, including which combination of goal and guardrail metrics means you ship, roll back, or iterate. 

• Guardrail metrics declared for anything that must not degrade, like revenue, support tickets, latency, or error rate.

• Randomization unit matched to the analysis unit, on both grain and identifier.

• Randomization re-seeded if reusing a surface or the same users from a prior experiment.

• Exposure event confirmed to fire at the same point in both arms, with latency and errors compared across arms.

• Network effects assessed: does this feature change how users affect each other? 

• User-sharing plan set: overlap allowed for independent experiments that change different parts of the product, or mutually exclusive namespaces for tests that interact.

• Sample size calculated from the minimum detectable effect, baseline, and power target, with an even split unless there's a reason otherwise.

• Minimum duration set for continuously enrolling tests, on a window without holidays or major promotions.

• A/A test passed on new surfaces and after tracking changes.

• Pre-experiment balance check run, with no significant imbalance between groups.

• Concurrent-experiment enrollment overlap reviewed.

• Stopping criteria written down. Analyze once at the committed sample size and date, or use sequential testing if you need to look early.

How GrowthBook supports pre-launch validity

GrowthBook's pre-launch checklist lets you encode these checks as required steps before an experiment moves to production. Use the power calculator to project power over time before launch and commit to a realistic runtime.

GrowthBook automatically flags SRM and pre-experiment imbalance on the Results tab once the test is live, the opt-in Health tab adds deeper checks and visualizations, and sequential testing keeps results valid if you need to monitor early.

GrowthBook’s Decision Framework lets teams define decision criteria directly in the experiment, so the readout includes a shipping recommendation. The stats engine is open source, so you can see exactly how each check is computed.

Trustworthy results begin during the design phase, long before data collection starts. GrowthBook's experimentation platform puts these safeguards in place by default. You can try it for free or book a demo with the team.

‍