What is a controlled experiment? Steps and examples

A controlled experiment compares what happened under a treatment with a credible estimate of what would have happened without it.

Suppose activation rises after a new onboarding flow launches. The flow may have caused the increase. Marketing traffic, seasonality, a pricing change, or a data-pipeline update may also have changed at the same time. A before-and-after chart cannot cleanly separate those explanations.

A controlled experiment creates a concurrent comparison. Eligible units are assigned to treatment and control groups, the treatment deliberately changes one aspect of their experience, and outcomes are measured under a predefined design. Randomization helps balance other influences between groups, allowing the outcome difference to estimate a causal effect.

In physical science, researchers may hold temperature, equipment, and materials constant. In digital products, teams cannot hold every user and circumstance constant. They use random assignment, shared eligibility, concurrent exposure, stable implementation, and statistical analysis to control alternative explanations.

The parts of a controlled experiment

Research question and hypothesis

The research question names the causal decision: “Does a guided setup checklist increase successful data connections within 24 hours for new workspace administrators?”

The hypothesis states the population, treatment, outcome, and mechanism. It should be written before results are available.

Independent variable

The independent variable is what the experimenter changes. In a product test, it is the assigned experience: existing onboarding versus the guided checklist.

An experiment can have more than two levels, such as control plus two treatment designs. More variants require more sample and create more comparisons.

Dependent variable

The dependent variable is the measured outcome. Examples include conversion, task completion, retention, error rate, latency, crop growth, blood pressure, or test score.

The metric must be operationally defined. “Activation” is ambiguous; “the proportion of eligible accounts that connect a valid data source within 24 hours” is reproducible.

Control group

The control group receives the baseline condition. It estimates the outcome that treatment units would have experienced without the change.

This counterfactual cannot be observed directly: the same unit cannot simultaneously receive treatment and control. Randomized groups provide an aggregate approximation.

Treatment group

The treatment group receives the intervention. Treatment should be implemented consistently enough that assignment means something. If some assigned units never encounter the change, exposure and analysis need careful definition.

Controlled variables and shared conditions

Physical experiments may keep temperature, time, equipment, and procedure constant. Product experiments run in a changing world, so groups operate concurrently under the same broad conditions.

Keep unrelated product changes from affecting only one variant. Preserve the control experience, eligibility, measurement pipeline, and experiment configuration while the test runs.

Randomization

Random assignment gives each eligible unit a known chance of receiving each condition. It tends to balance observed and unobserved characteristics across groups, within the limits of sample size.

Randomization can occur by user, account, device, request, store, classroom, region, or another unit. The unit should match how treatment spreads and how outcomes are shared.

Khan Academy's controlled-experiment overview explains the foundational relationship between changed factors and controlled conditions. In online products, randomization is the practical mechanism for balancing many factors that cannot be physically fixed.

Controlled experiment versus related methods

A before-and-after comparison may be useful for monitoring, but it confounds treatment with time. An observational study can find associations, but treated and untreated units may differ before treatment. Quasi-experimental methods use assumptions and designs such as difference-in-differences or regression discontinuity to construct a comparison when randomization is unavailable.

Not every randomized rollout is a useful controlled experiment. The design also needs a defined treatment, exposure, outcome, analysis, and decision rule.

The World Bank impact-evaluation handbook offers a broader comparison of randomized and quasi-experimental approaches when teams need to understand the assumptions behind a non-randomized design.

Steps to run a controlled experiment

1. Define the question

Start with a decision and evidence. Specify the population, intervention, comparison, and outcome. Reject questions that are too broad or cannot change action.

2. Write the hypothesis

State expected direction and mechanism:

A guided checklist will increase 24-hour data connection because it clarifies the remaining steps and accountable role.

List alternative explanations so the design can distinguish or monitor them.

3. Define eligibility

Specify who or what can enter, when eligibility occurs, and which exclusions are necessary. Avoid post-treatment filters—conditions influenced by treatment—because they bias the comparison.

4. Choose assignment unit

Randomize accounts when teammates share the feature or outcome. Randomize users when experiences are independent. Randomize requests only when users can safely see varying behavior.

Analyze uncertainty at a level consistent with assignment. Cluster assignment often requires cluster-aware analysis.

5. Specify control and treatment

Document exact differences and keep shared behavior stable. Review safety, privacy, accessibility, and technical correctness before exposure.

6. Define metrics

Choose one primary outcome tied to the hypothesis, guardrails for effects the team must not harm, and diagnostic metrics that explain behavior.

GrowthBook metrics can define conversion, count, duration, revenue, ratio, and other outcomes against supported data sources. Record units, windows, exclusions, and missing-data handling.

7. Plan sample and duration

Estimate baseline variance, minimum meaningful effect, desired power, error rate, eligible traffic, allocation, and delayed outcomes. Cover normal cycles and use a fixed horizon or validated sequential method.

The NIST engineering statistics handbook provides a canonical introduction to design-of-experiments principles, including objectives, factors, responses, and design choice.

The Penn State statistics material on power explains the relationship between sample, effect size, and the probability of detecting an effect.

8. Randomize and instrument

Use deterministic bucketing so units remain in one variant. Log exposure when the treatment can first affect behavior. Record experiment, variation, unit, time, and implementation version.

GrowthBook SDKs evaluate feature rules using supplied attributes, while feature flag experiments connect assignment to analysis.

9. QA before launch

Test both variants, targeting boundaries, assignment stability, exposure timing, events, metric queries, fallback behavior, and rollback. An A/A test with identical variants can reveal allocation or instrumentation defects.

10. Run according to plan

Monitor operational risk and data integrity. Do not stop a fixed-horizon experiment because an early result crosses a threshold. Do not change treatment or primary metric without documenting that the original experiment ended.

11. Analyze and decide

Check sample ratio mismatch and data quality first. Estimate treatment effect and uncertainty, then review guardrails and pre-specified secondary metrics. Compare statistical evidence with practical significance.

GrowthBook's experimentation platform supports Bayesian and frequentist methods, sequential testing, CUPED, dimensions, and guardrails. The method should match the decision policy chosen before launch.

The American Statistical Association's statement on p-values reinforces that a threshold alone does not measure effect importance or prove a hypothesis.

12. Record and follow through

Document design, dates, sample, estimates, limitations, and decision. Ship, revert, or plan a follow-up. Remove temporary treatment code after the decision while preserving the experiment record.

Three controlled experiment examples

Product onboarding

Question: Does a guided checklist improve 24-hour activation?

• Population: New administrator accounts reaching setup.
• Assignment: Account.
• Control: Existing setup page.
• Treatment: Checklist with owner and progress.
• Primary metric: Successful connection within 24 hours.
• Guardrails: Errors, support contacts, time to connection.

Account assignment avoids teammates receiving incompatible experiences. Exposure occurs when the setup page renders.

Backend recommendation model

Question: Does a new ranking model increase completed content sessions without harming latency?

• Population: Eligible recommendation requests.
• Assignment: User, if consistency matters across sessions.
• Control: Current model.
• Treatment: Candidate model.
• Primary metric: Completed content sessions per user.
• Guardrails: p95 latency, errors, compute cost, negative feedback.

Offline model evaluation should precede production exposure. The controlled online experiment measures user impact that offline accuracy cannot establish.

Plant growth

Question: Does fertilizer A increase growth relative to no fertilizer?

• Units: Comparable plants.
• Randomization: Plants assigned to fertilizer or control.
• Treatment: Defined fertilizer dose.
• Controlled conditions: Plant variety, soil, water, light, container, and observation period.
• Outcome: Change in height or biomass.

Physical control can hold many conditions constant. Randomization helps balance remaining differences.

Common mistakes

Comparing different time periods

Showing control in June and treatment in July allows seasonality and traffic changes to explain the result. Run groups concurrently.

Randomizing the wrong unit

User assignment fails when account members share state. Request assignment fails when users learn from inconsistent experiences. Match assignment to interference.

Logging exposure too early or late

Startup exposure includes users who never encounter treatment. Post-click exposure conditions on treatment-caused behavior. Log at the point variants can diverge.

Choosing metrics after seeing results

Searching many metrics and segments until one looks positive increases false discoveries. Predefine the hierarchy and label exploration.

Treating no significance as equivalence

An inconclusive result may reflect low power. Equivalence or non-inferiority requires margins and an appropriate design.

Ignoring average-effect limits

An average can hide harm to an important segment. Pre-specify decision-relevant dimensions and safety guardrails, but avoid unplanned segment mining.

When a controlled experiment is not appropriate

Use qualitative research when the problem is not understood. Use automated and usability testing for correctness. Use staged delivery for operational safety. Use quasi-experimental methods when randomization is impossible.

Do not randomize harmful, illegal, or nonconsensual treatment. High-impact domains require appropriate ethical, legal, privacy, and safety oversight. A valid statistical design does not make an unacceptable intervention permissible.

Turn a comparison into a decision

A controlled experiment earns trust through a credible control, appropriate randomization, consistent treatment, predefined outcomes, adequate sample, and a decision rule. The statistical result is the end of a chain; each earlier link must be sound.

For digital products, GrowthBook connects feature flags with transparent, warehouse-native experiment analysis. Start with a reversible treatment and a metric the team already trusts. You can start for free and use the first experiment to validate both the product idea and the measurement process.