How to Build a Culture of Experimentation

Companies that run successful experimentation programs understand their business better, improve the product and customer experience faster, and drive smarter decision-making company-wide.

But it’s hard for a program to get there without a strong experimentation culture, and the more a program scales, the more the culture matters. Successful programs rely on sustained investment, highly aligned teams that are able to ship quickly and feel safe taking risks, and leaders that are open to being proven wrong. A strong experimentation culture is what makes that possible.

What is a culture of experimentation?

A culture of experimentation is one where a company tests its assumptions before acting on them. In practice that means teams run tests to answer open questions, learn from every result, and let evidence carry more weight than the most senior opinion in the room.

Teams cannot reliably predict which changes will improve a metric. Even in mature experimentation programs like those at Booking.com, Google Ads, and Netflix, only about 10% of experiments move the metric they were designed to improve. Companies that accept that limitation are more likely to test by default rather than reserving experimentation for their largest bets. But testing by default also creates more opportunities for poorly designed tests, so a strong culture is crucial for teaching people how to design experiments that produce trustworthy results.

A strong experimentation culture is visible when a director whose hypothesis was disproven asks what the result implies for the next iteration. It’s visible when a product manager proposing to remove a field from the signup form can clearly articulate which metrics they expect to move and why, or when someone on another team can look up what signup tests have been run before and what the results were without asking the data team.

Building a culture of experimentation requires a change in how a company establishes confidence in a decision. An experimentation platform lowers the cost of producing measured results, but adopting them as the basis for decisions is an organizational change, and that change is generally the harder part of building a successful experimentation program.

7 practices that build a strong experimentation culture

Each of these 7 practices removes a reason someone would avoid running an experiment, whether the obstacle is the support and access needed to run one, the fear of being wrong, or doubt that the answer will be trusted and acted on.

1. Secure and sustain leadership buy-in

A sponsor in a leadership position is crucial to building a strong experimentation culture. That person champions the program, speaks for it in planning conversations, and brings experiment results into the decisions being made there. Their support is often what gets it funded. That person might be an executive who decides from the outset that the company will work this way, or a leader who is won over after one team starts running experiments and sharing the results.

Teams optimize for whatever executives pay attention to. When leadership rewards confident answers, no one is comfortable saying they don’t know yet. When they ask about assumptions and what’s being learned instead, teams run the tests that produce those insights. Asking those questions in public also sets an example. When a senior leader is openly uncertain and curious, everyone else can be too.

At DoorDash, which runs 12,000 experiments a year, results go out across the whole company whether or not the change ships, and leadership responds to them. Ilya Izrailevsky, the senior engineering manager who leads DoorDash’s experimentation platform, explained the practice on The Experimentation Edge:

“Our leadership, including our CEO, Tony, would read and reply to those experiment emails and would congratulate folks, but also encourage them to try some alternative ways or approaches to the same problem. This really builds a culture that experimentation is encouraged and everything we do should be through experiment._”

Leadership sponsorship also requires resource allocation. Executives fund the platform, the data work behind the metrics, the engineering hours that go into instrumentation, and the time teams spend building losing variations. Learning that an assumption is wrong before a team builds on it costs less than learning it after release.

2. Create psychological safety

Most experiments that are well designed and well executed do not improve the metric they were built to improve. Across Microsoft’s experimentation platform, only about 1 in 3 succeeded at improving the key metric they targeted. When a team treats a losing result as a mistake, most of its experiments reflect poorly on whoever ran them, and the rational response is to run fewer experiments, or take less risk. The tests that carry the least risk also offer the smallest potential gain, so the program returns less even when they win.

A program with psychological safety removes the fear of failure because learning and taking risks are valued over being right all the time. A team decides how an experiment is designed but not what it finds, so feedback belongs on the process. A review can ask whether the hypothesis was well informed, whether the design produces a trustworthy answer, and whether the question is one the business needs answered.

James Falzone, director of product management at Kargo, put the distinction plainly on The Experimentation Edge:

“There’s a difference between a bad result and a bad experiment, because to some extent, if you’re not getting those bad results, if you’re not failing, are you really trying anything new?”

A team that is rewarded for shipping will protect the launch at the cost of learning. Makram Mansour, head of a marketplace at ID.me, described the habit on The Experimentation Edge: “Our old habits are about I launch it, we do a major celebration, I put it in my performance review, I get promoted because of it.” When shipping is what earns the promotion, an experiment that could stop a launch is a threat to the person running it. Mansour worked with HR to revise the OKRs and KPIs teams were measured against, framing goals around learning quickly rather than launching.

3. Frame work as testable hypotheses

When work is framed as testable hypotheses, the assumption behind it is stated in advance, where the rest of the team can review it and the result can be measured against it. A roadmap item and a hypothesis describe the same change, but only the hypothesis describes what would have to be true for that change to work. A hypothesis template can be as simple as, “For [audience], changing [specific product behavior] will [expected behavior change] because [reason].” Filled in, that becomes something like “For new workspace admins, moving teammate invitations to after project setup will increase seven-day activation because users can complete the first meaningful task before being asked to collaborate.” Hypothesis-driven teams use a format like that because it forces 4 things into the open: who the change is for, the change itself, the outcome you expect, and the reason you think the change will produce that outcome.

At JPMorgan Chase, the hypothesis is recorded when a team registers the experiment, before it runs. Kevin Yang, executive director and head of experimentation there, described the setup on The Experimentation Edge:

“We have a platform where the teams will first come and register their experiment. And what we focus there on is making sure that we capture the hypothesis and what they’re trying to do, what they’re trying to measure. And that goes into our knowledge repository.”

When a feature ships without a stated hypothesis and the metric does not move, the team has no written expectation to compare the result against. An invalidated hypothesis shows that the assumption was wrong. Either the change does not affect the metric at all, or it affects it by less than the team expected, and the next experiment design can account for that.

4. Move decisions from HiPPO to evidence (and why trust matters)

Evidence can only outrank opinion if decision-makers trust it. When only the data team can see how a result was calculated, anyone who dislikes it can challenge the data quality. The decision then reverts to the HiPPO, the highest-paid person’s opinion.

Andrew Willingham, head of legal and people products at Atlassian, said what a trusted number does for a product manager on The Experimentation Edge:

“I can just say, well, here’s what the data shows. It’s not my opinion or your opinion. This is the data. And that helps short-circuit a lot of these political or opinion-based comments that I think can slow us down.”

Arun Bodapati, director of data science at Twitch, showed how quickly that credibility can be lost in his episode of The Experimentation Edge:

“There’s a lot of earned trust in that process. It’s one thing to say we do experiments, and then all you need is one misconfigured experiment to say, ‘You guys don’t know what you’re talking about.’”

How metrics are defined and computed determines how trustworthy they are:

• A single source of truth for metrics: Activation means the same thing in the experiment readout and in the weekly business review, because both read from the same definition.
• Visible computation: Anyone can read the SQL behind a result, so someone who doubts a number can verify the query that produced it.
• Warehouse-native analysis: The analysis runs in the data warehouse itself, on the same tables the rest of the business reports from, so experiment numbers reconcile with everything else.

A readable query can still run on top of a broken assignment, so data-quality checks should run alongside the analysis. For example, many data issues can be caught by a sample ratio mismatch (SRM) test, which verifies that users were divided between the variations as configured.

5. Open testing beyond engineering (without creating chaos)

Ideas on how to improve the product can come from across the company. But building an experiment to test those ideas usually requires technical work, most often from engineering. Those ideas then enter the engineering backlog, where they are prioritized against committed roadmap work.

Extending access to testing tools beyond engineering, often called democratization, lets the people closest to a problem test their own ideas without waiting on another team’s capacity, whether that is engineering, design, or data. Because those experiments run in production and affect real customers, broader access needs guardrails, such as limits on who can launch what and an approval step before anything goes live.

Self-serve access works better when there is somewhere to bring questions. Makram Mansour, head of a marketplace at ID.me, named 2 bottlenecks in experimentation programs on The Experimentation Edge. The first is engineering capacity. The second is access to data scientists. Teams with their own data scientist run a lot of experiments, while teams without one stay with simpler tests because they aren’t sure how to design a more involved one. His answer is a central team of data scientists anyone can go to: “If you have any questions, here’s a team or dedicated resources to help you.”

A no-code visual editor reduces engineering and design dependencies. For example, GrowthBook’s Visual Editor (Available on Pro and Enterprise) works on a live web page. The experiment owner can describe a change in plain language and let the editor propose the edits, or click an element directly to change its text, image, spacing, or styling. A Figma frame or mockup can also be imported as a variation.

Permissions, checklists, and approval steps keep broader access from producing overlapping tests, missing hypotheses, and results no one can interpret. In GrowthBook, this includes:

• Roles and projects: Everyone gets a role, from read-only up to admin, and roles can be scoped per project (Available on Pro and Enterprise), so a team’s access can be limited to the projects it works on.
• Pre-launch checklists: Every experiment goes through the same checklist (Available on Enterprise) before it launches, and the list can include a written hypothesis, a screenshot of each variation, a project, and a tag.
• Namespaces: Splitting the population into namespaces keeps overlapping experiments from assigning the same unit and interfering with each other.
• Approval flows: Changes to a feature flag or a saved group can require review before publishing (Available on Enterprise), and the person who wrote the change can’t be the one to approve it.

The same guardrails now cover AI agents. In GrowthBook 5.0, agents create their changes as drafts that someone has to approve before they go live, under the same policies that apply to humans.

6. Make learnings visible and shared

Results are useful well beyond the team that ran the experiment. Someone working on similar problems can see which questions have already been answered, a result from one part of the product can point to a hypothesis in another, and seeing how other teams designed their tests helps people learn what good experiment design looks like. For either of those to happen, experiments have to be documented and revisited regularly.

A searchable repository of past experiments is a great starting point, but there are many ways to distribute those results to extend their reach, including incorporating them into relevant dashboards, sharing individual readouts directly with the teams most likely to benefit, submitting them to internal newsletters, or even sending out an experimentation-focused newsletter that covers recent learnings and upcoming tests.

These communications should also include losing experiments in addition to winners. Subtractive learning, finding out what didn’t work, can be just as insightful as knowing what did work.

Crystal Ammari, digital product optimization strategist at Disney, described what she wants from a testing repository on The Experimentation Edge:

“Something where folks can just go in. It’s almost like a Google for our test results. And you go in there, and you can search for the site or the element you wanna change or the KPI you wanna move, right? And it would just pull up all those results.”

The Home Depot invites everyone interested in a test to the same session. Kim Ting Li, senior manager of online experimentation there, outlined what happens next on The Experimentation Edge: “They will all be invited to this public readout for about 30 minutes. And at the end of it, we send out a blast email with the final results to share across the stakeholders. But those results are also all stored in a centralized place, so anybody can go back into tests that ran maybe in 2020 to get the learnings and see if something has been done already.”

GrowthBook’s Learnings feature captures conclusions drawn from many experiments, such as shorter signup forms reliably increasing completion rates, or discount banners never improving revenue. Each learning lists the experiments that support it and the ones that contradict it. Product Analytics provides dashboards built on the same metric definitions used by the experiments, so the readout and the business review show the same numbers. Event webhooks push results out as they land, posting to Slack or Discord when a metric reaches significance or an experiment is ready to ship or roll back.

7. Build the rituals and cadence

Rituals support an experimentation culture by aligning teams and keeping people engaged with the program. The right set of rituals will look different for each organization, but these are a great starting point if you’re still building out your program.

• A weekly experiment review: Cover what’s planned, what’s in flight, and recent results, and keep the discussion on what the program is learning rather than how many tests it ran. A large program, or one spread across several product areas, can run more than one review so each one stays efficient and relevant for the people attending.
• A monthly prioritization session: Review the backlog, decide what to test next, and plan the roadmap. This can be done within teams or with cross-functional working groups of relevant stakeholders.
• A retrospective on bigger experiments: After a complex or high-stakes test, bring stakeholders together to assess what went well and how things could be done better next time. Kargo’s retros include a standing item called “Where did you fail?”
• Weekly office hours: Hold a recurring session where anyone can bring an experiment they are designing or a result they are trying to interpret. The questions that come up repeatedly show where the program needs clearer documentation, more training, or better tooling.
• A cross-team readout: Share recent results with the teams working on related problems. Focus on actionable insights and where help is needed with implementing the results.
• An executive readout: A high-signal summary of recent results and a review of what’s upcoming and in flight, geared toward the leadership team.
• A guess the winner game: Post experiment details in a Slack channel or other internal forum and ask participants to guess which variation won. Turning results into a game encourages people to challenge their assumptions and engage with new experiment insights.

Nick Beyler, data science manager at Stitch Fix, described the review his team runs each week on The Experimentation Edge:

“We have an experimentation review group. So this is a weekly forum where all the expert experimenters, data scientists, product managers, we meet weekly and talk about the upcoming launches. And it’s just a great venue that everyone can say ‘Oh, I tried that experiment or that kind of experiment a couple months ago or a year ago. Here’s what I learned there that this may be helpful for you.’”

Kevin Yang, executive director and head of experimentation at JPMorgan Chase, described the mix on The Experimentation Edge:

“We have champions across different teams that will hold their own experimentation forum. My own broader org does have an experimentation forum that we also host. We actually host a couple of them, and some of them are more executive-focused audience, and another one is more like general audience.” He also sends a written roundup to the whole firm, which he called an “experimentation wrapped newsletter.”

Danielle Olean, director of e-commerce at Box, described the executive readouts she runs there on The Experimentation Edge:

“I actually have a template for an impact report, and on a biweekly cadence, we meet with the COO. On a monthly cadence, we meet with the CEO, and we share, ‘This is what won, this is what lost. Based on the loss, this is what we learned, and this is what we’re going to do next.’”

What a culture of experimentation looks like in practice

In a company with a strong experimentation culture, testing is part of how the organization operates rather than a service one team provides.

The Philadelphia Inquirer runs one centralized testing program covering marketing, advertising, the newsroom, and product, with a shared set of standards and governance rules for all 4. Those departments have competing goals. For example, advertising wants to sell more ads, but the newsroom wants an uncluttered reading experience. A shared definition of success balances this tension. An intervention that increases ad revenue can only ship if engagement doesn’t decline.

DoorDash sends every experiment result to the whole company, where the CEO reads and replies. Izrailevsky, who leads DoorDash’s experimentation platform, named the mindset that goes with it:

“There’s no such thing as a failed experiment. Every experiment is a learning opportunity, no matter how it goes.”

At Lingokids, around 40 people across engineering, product, and data use GrowthBook inside a company of 150+. Filipa Batista, product manager for billing, pricing, and monetization, is one of the people launching those tests: “We don’t need any code changes, we don’t need an app release. We just configure the new tests and launch right away.” The team runs about 15 experiments a month, roughly double what it could run in parallel before, and has recorded a 58% increase in activation rate.

An experimentation platform can’t build a strong culture on its own, but it can support the practices that do by making experiments easier to run and trust. GrowthBook is an open-source, warehouse-native experimentation platform where new variations can be launched quickly with the no-code Visual Editor, every team reads from the same metric definitions, and changes can be switched on or rolled back without a new release. Try GrowthBook for free.