How to Scale Your Experimentation Program

Getting your first experimentation win is one thing. Scaling A/B testing to hundreds of experiments a year across many teams is a distinct discipline with its own challenges. Programs that add velocity without changing how they operate often run into similar problems: teams rerun the same tests without realizing it, small mistakes in assignment and tracking distort results, different teams define the same metric differently and reach conflicting conclusions, false positives accumulate as volume increases, and new launches get backed up in the same engineering queue.

The pressure to scale is also rising. Kevin Yang, executive director and head of experimentation at JPMorgan Chase, described that pressure on The Experimentation Edge:

"With AI coming through, everybody is going to ship faster. But do you have the right infrastructure in place to measure the things you're shipping out? If you don't measure them right, your mistakes are going to compound."

Shipping faster only pays off if you can trust what you're measuring. And when teams optimize for test count, easy tests crowd out important ones, so the program learns less. The point of scaling is faster, more reliable learning that compounds across teams and keeps the product improving.

What "scaling experimentation" actually means

Scaling experimentation means moving from isolated, ad-hoc tests to a standardized, company-wide system. More teams run more experiments, and the data stays as trustworthy as it was when one analyst reviewed every result. The test count grows as a consequence of that system. You're building an operating model that lets measurement keep pace with how fast your teams ship. Shared metric definitions, automated health checks, and approval gates replace the quality control that used to live in a few experts' heads.The culture also has to scale alongside the system so that testing becomes the default for every new feature, and what team learn informs each other's next experiments.

How to scale your experimentation program: 5 pillars for success

Reaching hundreds (or even thousands) of experiments a year without losing trust in your results rests on 5 foundational pillars: infrastructure, prioritization, institutional memory, statistics, and governance. Whichever of the 5 is weakest sets the limit on how far your program can scale.

Pillar 1: A scalable technical foundation

Every experiment adds analysis work, so experiment volume is limited by how much of that work you can reuse. With a warehouse-native architecture, experiments are analyzed directly against the data warehouse you already maintain. There is no copied dataset in a vendor's system to reconcile. Fact tables and metrics are defined once in SQL against your warehouse tables, and that single metric definition serves feature flags rollouts, experiment analysis, and analytics. When the growth team and the checkout team both measure "conversion rate," they are running the same SQL, and you can compare their results.

Agreeing on which metrics to use is as important as agreeing on how they're defined. There are different schools of thought on the right structure. Some programs align the whole company on a single overall evaluation criterion (OEC) or North Star metric, while others set different primary metrics for different parts of the business. Many structures can work as long as the choice is deliberate. If every team picks its own success metrics, you lose the ability to compare results across the program. Whichever structure you choose, require experiments to declare their primary metrics up front, so results are judged against the metric the test was designed to move rather than whatever happened to improve.

It's also important to confirm that nothing in your experimentation pipeline is broken before you scale up. A/A tests (experiments where both groups receive the identical experience) are the gold standard for validating your pipeline. Any statistically significant difference between the groups is either chance or a bug. With GrowthBook's default thresholds, about 10% of A/A tests will flag a single metric as a winner or loser, so 1 flagged metric in 1 test is plausibly chance. Several flagged metrics in the same test, especially with large measured effects, point to broken assignment or tracking, a problem that’s much better to discover at 10 experiments a year rather than at 300.

Pillar 2: Ruthless prioritization

As a program grows, test ideas usually accumulate faster than the capacity to run them. A scoring framework such as ICE (impact, confidence, ease) or PIE (potential, importance, ease) forces every proposed test through the same filter. Most frameworks weigh how large of an effect is plausible, how strong the supporting evidence is, and what the test costs to build, run, and analyze. The scored ideas form a ranked experiment backlog. AI can help at both ends of that process, brainstorming test ideas and scoring them against your framework, with a human in the loop making the final determination. The exact score matters less than the habit of weighing expected value against cost. Low-value tests consume capacity and analyst time, so a mediocre idea that is cheap to run can still cost more than the information it returns.

Sometimes you can even measure an idea's impact without building it. Crystal Ammari, digital product optimization strategist at Disney, described on The Experimentation Edge how a customer service team she worked with tested demand for a video-chat support feature before committing to build it:

"We did what I call dry testing, which is essentially testing a feature without that feature actually being built."

The team added a button for the unbuilt feature to the help page and measured clicks.

"I believe there was somewhere around 4 million people that had entered the test, and only 106 people clicked on the button. I will never forget that 106 because it was such stark, obvious evidence that this is not something people wanted." She estimates the result saved the business millions of dollars in build, hiring, and training costs for a feature almost nobody would have used.

Pillar 3: A living experiment repository

As the experiment count grows, "didn't we already test this?" becomes harder to answer from memory. Some teams track past tests in a spreadsheet or shared doc, but this is only as good as your ability to keep it updated. A useful repository has to be part of the experimentation platform itself. When every experiment runs through the platform, the hypothesis, audience, dates, variations, and results are all captured automatically in a queryable record. That record is the single source of truth for what has been tested.

The repository has 2 uses: teams catch redundant tests before launch, and the accumulated record becomes a portfolio you can analyze for experiment velocity, win rates, average lift, and the scaled impact of shipped winners (the projected effect once a winning change rolls out to everyone). In GrowthBook 5.0, meta-analysis blocks on Product Analytics dashboards track what's running, win percentages, lift, and cumulative impact across the program. The new Learnings feature goes a step further and captures patterns across experiments, with each learning tied to its supporting and contradicting evidence, so people and AI agents can build on what the program has already discovered.

Insights still need to circulate outside the platform. Guests on the Experimentation Edge frequently describe the same two habits: a shared Slack channel where teams post results, and recurring meetings where they review experiment results as a group. Ilya Izrailevsky, the senior engineering manager leading DoorDash's experimentation platform, described how far that can go:

"After we run an experiment, no matter the results, whether ship or no ship, we send the results out across the company and have a discussion. Our leadership, including our CEO, Tony, would read and reply to those experiment emails and would congratulate folks, but also encourage them to try some alternative ways."

Pillar 4: Statistical rigor and clean data at volume

False positives multiply as experiment volume grows. Some of your winners are false, and the share depends on your program's true success rate. When the true success rate is around 10%, about 22% of statistically significant wins are false, roughly 1 in 5, even in a properly powered program. For a program running 300 experiments a year, that works out to about 30 winners, with roughly 7 of them being false positives.

Several techniques protect statistical rigor as experiment volume grows:

• Pre-committed experimental designs: Set the sample size and duration before launch. Stopping a test as soon as it crosses the significance threshold inflates the false positive rate, because at each look, random variation alone can produce a statistically significant result. When teams need the flexibility to monitor and act on results mid-experiment, sequential testing widens the confidence intervals so that repeated looks keep that rate under control.
• Health checks on every experiment: A sample ratio mismatch (SRM) (where the observed assignment ratio differs from what was configured) signals a data problem serious enough that the results can't be trusted. Other checks flag problems like multiple exposures and suspicious uplifts. At scale these checks must run automatically to catch problems before teams act on the results.
• Multiple testing corrections: Evaluating many metrics and variations per experiment multiplies the chances of a false positive. There are statistical corrections that you can implement to compensate for the extra comparisons. At scale, they should be a platform default rather than remembered analysis by analysis.
• Variance reduction: CUPED (Controlled-experiment Using Pre-Experiment Data) uses each unit's pre-experiment behavior to reduce noise in the outcome metric, so an experiment can detect the same effect with a smaller sample. When each test needs fewer users, the same user base can support more experiments at once.
• Isolation for concurrent tests: Concurrent experiments can contaminate each other's results when their changes interact, because some users experience both at once. Namespaces make conflicting experiments mutually exclusive, so a unit is only ever assigned to one of them, while unrelated experiments can continue to overlap freely. 

When a result is significant but can't be explained, rerunning the test is the best way to validate the results. Medha Umarji, VP of growth and experimentation at Fanatics, described catching a false winner: a change the team had tested 6 to 8 times before, always with flat results, suddenly showed a statistically significant revenue lift.

"Another thing that we do here to reduce our false positive risk is we replicate our outcomes. We try to replicate it if we are not able to explain it. And so we ended up turning off this test and rerunning it, and we weren't able to replicate it. It was really flat."

Pillar 5: Open access without losing governance

Engineering capacity limits experiment velocity in a way prioritization can't address. If every experiment requires engineers to implement variations, review the setup, and launch, velocity is capped by their sprint capacity. PMs, marketers, and analysts with testable ideas wait on engineering time. Self-serve tooling removes that dependence with features like a no-code Visual Editor for frontend changes and a simplified experiment creation flow.

Self-serve only works with a quality gate, because opening creation to more people also opens it to more setup mistakes. A draft-first approval flow provides that gate. Anyone can build an experiment, and a reviewer approves it before it reaches production. The same control extends to software agents. In GrowthBook 5.0, AI agents can brainstorm, create, and launch experiments, and their production changes wait for the same human review as everyone else's. When the review step is enforced by the platform, nothing slips through accidentally, even as the number of experiments grows.

What scaling experiments looks like in practice

These 5 pillars are visible in every large experimentation program. Izrailevsky put numbers on DoorDash's current scale:

"We run about 12,000 experiments per year. Over 42 million monthly active users on DoorDash's platform."

At peak, the platform evaluates experiment feature flags around 300 million times per second. No team can review 12,000 experiments a year manually. DoorDash balances a success metric against multiple guardrail metrics for each experiment and shares every result across the company.

The Home Depot's online business recently passed $25 billion in revenue. Kim Ting Li, senior manager of online experimentation there, described on The Experimentation Edge how much attention leadership pays to the tests:

"All the executives and leaders love to learn about the tests. They really truly understand A/B testing is the golden rule to understand incrementality, the real impact."

The same foundations show up in much smaller programs too:

• Chess.com ran about 400 experiments one year and set a goal of 1,000 for the next year, according to Nafis Shaikh, director of product management there. Each team aligns on and owns its own metric area, with the gameplay team optimizing the core experience and the monetization team optimizing revenue.
• Fyxer, an AI email assistant, ran 541 experiments in one year with a small team while growing from $1M to $35M in ARR. AI agents make the code changes and engineers sign off on video previews, so each developer can run several experiments in parallel.
• The Philadelphia Inquirer built a centralized program where marketing, the newsroom, and product all test under one set of standards.
• Lingokids, a kids' learning app with over 3 million weekly active users, roughly doubled its parallel experiment volume to about 15 per month after adopting namespaces to keep concurrent tests from colliding.

How to scale experimentation without sacrificing quality

An experimentation program can hit its velocity goals and still learn very little. Test count is only one input into learning, and once the count becomes the target, teams find ways to raise it that don't necessarily produce useful insights. Judge your program on its outputs instead: the winning changes it shipped, the harmful ones it caught before rollout, and what the results taught you about your users.

GrowthBook is built to implement these 5 pillars, with a warehouse-native foundation, SQL-defined metrics, health checks and false positive controls in the stats engine, a living portfolio view of every experiment, and governed self-serve access for people and agents alike. Ready to scale your experimentation program? Explore GrowthBook's experimentation platform or try it for free.