How 10 of the top companies scale to thousands of experiments a year

Running 1,000 experiments does not require 1,000 statisticians. It requires a system that makes the safe path the easy path.

Experiment volume is often presented as a cultural achievement. In practice, it is an operating-system achievement.

When a company moves from dozens of annual tests to hundreds or thousands, almost every manual step becomes a bottleneck. A central data scientist cannot review every metric. A platform team cannot configure every traffic split. Experiment names, overlapping treatments, exposure events, and decisions become too numerous to manage through spreadsheets and institutional memory.

The companies that scale do not remove rigor to move faster. They encode rigor into reusable infrastructure, then distribute ownership. Their exact architectures differ, but 5 patterns recur: self-service setup, shared metrics, automated quality checks, visible experiment history, and governance proportional to risk.

The public volume figures below come from different years and use different definitions. Some count launched experiments, some describe concurrent experiment configurations, and others cover only one product surface. Treat them as evidence of operating scale, not a leaderboard.

Scale comes from reducing marginal coordination

The first experiment at a company is expensive. Teams have to establish assignment, event logging, metrics, analysis, and a decision process. The thousandth experiment should reuse most of that foundation.

At scale, the marginal work should be the part specific to the hypothesis:

• What uncertainty is worth resolving?
• Which unit should be randomized?
• What change separates the treatment from the control?
• Which primary outcome and guardrails determine the decision?
• Which users or markets can safely enter the test?

Everything else should be standardized or automated. That includes stable hashing, exposure logging, common metric definitions, power checks, sample ratio mismatch detection, result computation, alerting, and the record of what happened.

This is why a high-volume experimentation platform is more than a results dashboard. It is a shared contract between product, engineering, and data science. Product teams gain autonomy because the platform prevents each team from inventing its own assignment and analysis conventions.

Centralize the platform, distribute the questions

Microsoft's Experimentation Platform supports product teams across a company whose intelligent systems reach billions of users. Its published description emphasizes a combination of large-scale infrastructure and developer-facing tooling, rather than a service bureau where one team runs every test. The Microsoft experimentation program illustrates the basic organizational split: a specialized team owns the trustworthy mechanism, while product teams own hypotheses and decisions.

The alternative breaks quickly. If every test requires a ticket to a central team, demand grows faster than specialist headcount. If every product team builds its own pipeline, definitions diverge and results become difficult to compare. Mature programs centralize the expensive, repeatable layer and distribute the context-heavy work.

Make quality checks automatic

More experiments create more opportunities for mundane failures: bad exposure logging, unexpected traffic splits, underpowered metrics, conflicting treatments, and repeated peeking. A program cannot scale if experts must discover each problem by hand.

Automated checks should stop or clearly flag a test when the data is not trustworthy. They should also explain the problem in language the experiment owner can act on. The goal is not to replace statistical expertise; it is to reserve that expertise for genuinely difficult designs and decisions.

Keep velocity trustworthy

Review the power, sequential-testing, multiple-comparison, and SRM practices that prevent high-volume programs from manufacturing false wins.

Read the Prevention Playbook

10 companies reveal A/B testing mechanisms behind volume

No company below reached scale through a single tactic. Each example highlights a different mechanism that reduces coordination cost without making experimentation casual.

1. Booking.com distributes authority and makes tests visible

Booking.com is the canonical example of experimentation as an organizational default. Harvard Business Review reported roughly 25,000 annual tests in 2020; a later discussion estimated more than 30,000. The exact number is less important than the governance model that makes such volume possible.

Teams can launch tests without waiting for a management committee, but experiments are transparent. Proposed tests are broadcast so colleagues can question the design, identify conflicts, or stop something unsafe. This is distributed authority paired with distributed oversight, not an absence of controls.

The Booking.com culture account also emphasizes how frequently plausible ideas fail. When most hypotheses do not win, the organization cannot punish every negative result or require executive certainty before testing. It has to reward learning while maintaining clear technical standards.

Mechanism to copy: Replace serial approval with visible defaults, peer challenge, and an accessible stop mechanism. Reserve centralized review for experiments with exceptional legal, financial, or user risk.

2. DoorDash treats velocity, toil, rigor, and cost as one system

DoorDash has described running thousands of experiments annually, while its GrowthBook interview put a more recent volume at 12,000 in a year. Those tests span consumers, Dashers, and merchants, creating network effects and competing outcomes that make simplistic optimization dangerous.

DoorDash's engineering team frames scale across 4 dimensions: increase velocity, reduce launch and analysis toil, preserve rigor, and control computational cost. Its experimentation framework allows teams to validate assumptions in the design itself rather than treating statistical review as something that begins after data arrives.

The program also makes completed results broadly visible. According to the DoorDash leader interview, company-wide readouts help experiment history influence future decisions instead of disappearing inside one team.

Mechanism to copy: Optimize experiment throughput as a multi-objective system. A workflow that launches more tests by increasing analyst toil or reducing rigor has not scaled.

3. Microsoft invests in a reusable science and engineering platform

Microsoft has reported thousands of experiments per month across major product surfaces. That volume requires the platform to serve both science and product delivery: stable assignment, metric computation, data-quality diagnosis, and interfaces ordinary development teams can use.

The company's published world-class experimentation platform principles focus on trust. Small product effects can be worth enormous sums at Microsoft's scale, but they are also easy to confuse with seasonality, logging errors, or random variation. The system has to make those failures observable.

Central expertise compounds because every improvement to the platform benefits many teams. Better variance reduction, assignment validation, or alerting is not applied to one experiment; it becomes part of the default path for the next thousand.

Mechanism to copy: Fund experimentation infrastructure as a product with internal users, not a one-time analytics project.

4. LinkedIn separates metric ownership from platform operation

LinkedIn has published that its platform could serve up to 41,000 A/B test configurations simultaneously across more than 700 million members. The number reflects a complex production assignment surface, not necessarily 41,000 newly launched strategic hypotheses, but it captures the engineering scale of the system.

The company's XLNT platform automated daily analysis while allowing teams to own the logic of their metrics. The experimentation group reviewed definitions, onboarded metrics, and operated common computation. That division prevents one central team from becoming responsible for every business definition while preserving a reviewable semantic layer.

LinkedIn later made its evaluation engine 20 times faster to keep analysis responsive as usage grew. Its experimentation infrastructure history shows that platform performance is a user-experience issue: if trustworthy results take too long or require custom work, teams route around the system.

Mechanism to copy: Let domain teams define what their metrics mean, but standardize how definitions are reviewed, computed, versioned, and reused.

5. Uber unifies feature delivery and experimentation

Uber reported more than 1,000 concurrent experiments on its platform as early as 2018. Its use cases span riders, drivers, delivery, freight, promotions, and machine-learning systems, with treatments that may affect a physical marketplace in different cities and time zones.

Uber's experimentation evolution exposed a crucial scaling failure: a platform can handle large traffic and still produce unreliable experiments. The company found that many tests in its earlier system had fatal design problems and required reruns. It rebuilt around stronger abstractions shared by engineering and data science.

The newer system unified feature configuration and experimentation, migrated thousands of developers and hundreds of services, and retired more than 50,000 stale experiment objects. The Uber platform rebuild shows why lifecycle management belongs in the scale conversation. Creating experiments cheaply without retiring them eventually makes the platform harder to understand and operate.

Mechanism to copy: Connect experiment assignment to the feature-delivery system, and treat cleanup as part of experiment completion.

6. Airbnb designs experimentation as a company-wide data product

Airbnb describes experimentation as critical to product decision-making and has advertised infrastructure supporting thousands of experiments each year. Its public work also shows that scale is not only about processing more rows. Teams need methods that detect smaller effects and reveal when averages hide different outcomes across hosts, guests, markets, or user cohorts.

This is why advanced experimentation platforms include tools for heterogeneous treatment effects, variance reduction, and alternative evaluation designs. Airbnb research on search ranking has reported substantial sensitivity gains from interleaving and counterfactual methods in contexts where standard A/B tests would be slow.

The organizational implication is important: the platform must support multiple valid designs without turning every experiment into custom analysis. Common abstractions should cover routine tests, while specialists extend the system for network effects, marketplace interference, and ranking problems.

Mechanism to copy: Standardize the common path, but give data scientists extensibility for designs that do not fit a basic user-level A/B test.

7. eBay automates randomization and SRM validation

Published research on eBay's experimentation platform describes hundreds of A/B tests running on a given day. At that level, sample ratio mismatch cannot be an occasional manual investigation. It has to be detected systematically.

SRM occurs when observed assignment counts differ unexpectedly from the planned split. It often indicates an implementation or logging failure that can invalidate the result. eBay's work on automated randomization validation and SRM detection focuses on maintaining sensitivity to real problems without flooding teams with false alarms.

This is a useful model for every automated quality check. A noisy safeguard will eventually be ignored. A scalable check needs a clear threshold, an understandable diagnosis, and a path to resolution.

Mechanism to copy: Treat experiment-quality alerts like production alerts. Measure their precision, make ownership clear, and improve them when teams repeatedly dismiss them.

8. Spotify measures learning quality, not only experiment count

Spotify's experimentation platform supports hundreds of teams. In one published snapshot, more than 58 teams ran 520 experiments on the mobile home surface alone in a year. The team then moved beyond raw velocity toward an “Experiments with Learning” framework.

That change addresses a common maturity problem. Once launching becomes easy, a company can generate many low-value tests. Test count may rise while hypothesis quality, decision usefulness, and organizational learning stagnate.

Spotify's Experiments with Learning framework evaluates whether experiments start with clear intent, use appropriate methods, and produce knowledge that affects a decision. It turns quality into something the program can inspect rather than a value teams merely endorse.

Mechanism to copy: Add a quality scorecard before rewarding teams for volume. Count experiments that reached a trustworthy decision and produced reusable learning.

9. HelloFresh standardizes templates as ownership spreads

HelloFresh has reported approaching 1,000 experiments a year and increasing the number of experiments that end with a decision. Its public case study emphasizes reusable templates and standardized setup as the program expands across teams.

Templates reduce two kinds of waste. They remove repetitive configuration for familiar experiment types, and they encode agreed practices so every new owner does not need to rediscover the right metrics, exposure event, or statistical setup.

The HelloFresh account also highlights decision rate. That is a better scale metric than launches alone. An experiment that cannot support a decision consumed traffic, engineering work, and attention without resolving the underlying uncertainty.

Mechanism to copy: Create templates around repeated decision contexts, not generic organizational departments. A checkout experiment and a search-ranking experiment need different defaults even if the same team owns both.

10. Fin treats every AI change as a candidate experiment

Fin has run thousands of experiments over several years and often maintains 12 to 24 concurrent tests. Its product produces non-deterministic answers across an effectively unlimited set of customer questions, so static test cases cannot cover the real behavior distribution.

The company can sustain high volume because it tests changes at many layers: prompts, context, latency, model behavior, and product presentation. It also uses confirmatory tests when a result is counterintuitive and guardrails when an apparent win creates dangerous behavior elsewhere.

The Fin experimentation story shows the difference between high velocity and indiscriminate shipping. Testing a comma may be cheap, but shipping a model change still requires evidence about resolution quality, customer satisfaction, hallucinations, and downstream promises.

Mechanism to copy: Make experiments cheap enough for small questions, but make the decision framework rich enough for multidimensional AI risk.

The shared architecture behind high-volume programs

The 10 programs suggest a practical architecture with 6 layers.

1. Stable assignment and exposure

Every analysis depends on knowing which unit received which treatment and when. The platform needs deterministic bucketing, explicit exposure events, namespaces or collision management, and support for the unit that matches the intervention: user, account, device, session, order, market, or another entity. GrowthBook's experiment assignment documentation explains how assignment and tracking fit together in a warehouse-backed setup.

2. A governed metric layer

Teams need reusable definitions for conversion, revenue, retention, latency, and guardrails. Domain owners should be able to inspect and improve those definitions without rebuilding an analysis pipeline for every test. Versioning matters because a metric that changes during a running experiment can change the meaning of the result. A shared GrowthBook metric layer is one implementation of this contract.

3. Automated trust checks

At minimum, automate SRM detection, minimum sample checks, suspicious uplift warnings, and exposure-health monitoring. Higher-scale programs also need collision visibility, multiple-comparison controls, and safeguards for repeated monitoring.

4. Self-service workflows

Product teams should be able to draft, implement, launch, and read routine experiments without queuing for a specialist. Self-service does not mean unlimited permissions. Use templates, scoped roles, launch checklists, peer review, and special gates for regulated or high-blast-radius changes.

5. Searchable institutional memory

Experiment history should answer: what did we test, why, what changed, which metrics moved, what decision followed, and what should a future team avoid repeating? DoorDash's company-wide readouts, Spotify's learning framework, and Booking.com's visible experiment proposals all make learning part of the operating system.

6. Lifecycle ownership

The program is not finished when a result is called. Teams still have to ship or revert the winning behavior, remove temporary experiment rules, clean up stale flag code, and preserve the decision. Uber's 50,000-object cleanup shows what happens when creation scales faster than closure.

GrowthBook packages these concerns in a warehouse-native experimentation platform with reusable metrics, feature-flag delivery, approvals, data-quality checks, and analysis against existing company data. The architectural point is broader than one tool: scale is possible when every experiment reuses a trustworthy system instead of assembling a new pipeline.

Measure learning throughput before chasing a bigger number

A goal such as “run 1,000 experiments” is easy to communicate and dangerous to optimize blindly. It can reward trivial changes, duplicated hypotheses, and underpowered tests.

A healthier program dashboard pairs volume with quality:

• Percentage of eligible product changes evaluated experimentally.
• Median time from approved hypothesis to trustworthy decision.
• Percentage of tests that pass assignment and data-quality checks.
• Percentage that reach a pre-declared decision threshold.
• Number of decisions that cite a prior experiment.
• Share of losing or neutral tests with a recorded learning and next action.
• Cumulative impact on North Star and guardrail metrics.
• Cleanup time from decision to removal of temporary experiment logic.

Start by identifying the manual queue that limits your current program. It may be metric creation, engineering implementation, analyst readouts, legal review, or simply the absence of stable exposure data. Standardize that queue before setting a larger volume target.

The lesson from the largest programs is not “test everything.” It is “make every reasonable question inexpensive to answer correctly.” When assignment, metrics, quality checks, visibility, and cleanup become shared infrastructure, experiment volume can grow without forcing teams to choose between speed and trust.

Learn the scaling pitfalls

Hear how experienced experimentation leaders approach metric choice, trustworthy setup, and the mistakes that appear as programs grow.

Watch the Scaling Session