What 50 experimentation leaders had to say about the value of being wrong

Your next big idea probably won’t make a difference, and might even cost you. That was the most common refrain in the first 50 episodes of The Experimentation Edge podcast from GrowthBook.

We invited the people running experimentation at companies from startups to Fortune 500 companies to share their stories. In this season's wrap episode, GrowthBook co-founder Graham McNicoll and head of customer success and strategy Cris Dizon join me for a final take on the current state of experimentation and where it’s headed next.

About The Experimentation Edge

Our first 50 guests came from 49 companies worth more than $6 trillion combined, from JPMorgan Chase and UPS to Chess.com and Fyxer. Two-thirds of our speakers hold a title of director or above, and nearly a third have experimentation, testing, or CRO in their job title. (Get a quick look at our guest spotlights and takeaways.)

The biggest learning? You’re probably wrong 75% of the time. The only question is whether or not you're learning 100% of the time.

Most ideas won’t move the needle, or worse

Across season 1, guests estimated their win rates from 10% to almost 40%. The value in experimentation isn’t uncovering winners, but not shipping losers.

As Cris put it, first tests harvest the low-hanging fruit, the easy fixes, and the obvious wins. As a testing program matures, the deeper it goes into what can and can’t move the needle.

“It’s kind of a kick to the ego when you realize your best efforts are only winning 25% of the time,” said Graham in the wrap. He added that “no one builds a feature they think will not win.”

Crystal Ammari shared a painted door test that saved a financial services company millions of dollars. Instead of building a video customer support call center, they added a “video support” button. Four million users saw it, and only 106 clicked.

Losing tests are a learning asset

In 68% of Season 1 episodes, guests described a loss, a failure, or flat results as their insight. They gain trust in their programs by showing the value of experimentation, calculated by growth and revenue as well as loss avoidance: money not spent on development or maintenance.

PayPal estimates a $180 million impact in revenue growth and cost avoidance. JPMorgan Chase estimates a $1 billion value gained from experimentation and innovation.

“At GoPro, losing tests for us are jet fuel,” said Will Guyeskey. Crystal Ammari, now at Disney, stopped talking about wins and losses and started talking about gains and savings: “If a test lost, really we saved ourselves.”

Graham knows the high cost of not testing. At a previous company, a redesign shipped to 100% of users without a test so they could meet a marketing deadline. Six months later, the team found it had cut new-user signups by about 20%.

“If you’re not A/B testing, you still test with every release, you just don’t measure the results,” he advised.

Test everything, even deeply held assumptions

Our guests run testing programs across the experiences of millions of users. Nearly half of our episodes (46%) featured a confident belief that a test overturned. Examples of widely held beliefs proven wrong came from a host of companies, industries, and leaders.

Arun Bodapati, director of data science at Twitch, suggested testing a pricing model that had been fixed for years. A geo-fenced experiment showed elasticity in pricing and new revenue opportunities. Confidence in the experimentation team grew when leadership saw the results in the financials.

In eight episodes, adding friction improved outcomes, despite conventional wisdom.

• Forcing users to select a ring type at Signet Jewelers added a step, but reduced overwhelm and improved conversions.
• Offering a lower-tier option during cancellation reduced churn at Box, and drove users back to the mid-tier option.
• Providing detailed text and privacy disclaimers on a registration form actually increased conversions by building trust with users, according to Graham.

Mature experimentation teams consider every experiment a learning opportunity, and the accumulation of learnings is as important as any single aha moment.

The future of experimentation

When we asked guests about the future of experimentation, the conversation most often turned to AI. They expect AI to run more of the experimentation loop, and they still keep a human in charge of the decision.

The majority (78%) use AI in their experimentation program to speed analysis, ideate, or build tests. Kevin Yang of JPMorgan Chase notes that “with AI coming through, everybody is going to ship faster… if you don’t measure them right, your mistakes are going to compound.”

Just as AI speeds up the capacity to build and ship new features, it adds to the volume of output the experimentation team needs to review. Just under half of guests mentioned that they focus more on testing, verifying or checking AI test results for errors.

“The thing you need to do now…is master the art of evaluation,” said Marco Casalaina from Microsoft on the challenge of managing the Copilot CLI that ships twice a day. Will Guyeskey, GoPro, noted that the risk of running so many tests is not taking the time to understand the learnings.

Program maturity also shaped the answer. Newer programs talk about culture, foundations, and scaling up their programs. More mature programs referenced AI agents, bandits, and sequential testing in their programs.

Retail and consumer brands are betting on personalization and simulated customers as the next step with experimentation and AI.

Make learning a company-wide endeavor

After 50+ conversations, the programs and leaders doing incredible work have created a culture of experimentation within their organization. They reframe losses as learning, they make it safe for people to swing for the fences, and an executive in the room leads by example.

They reward the “save” by explicitly celebrating losses and calculating their value. When reviewing tests, Graham often asks, “Could we have learned this faster? What is the minimum we can build to get this data more quickly?”

Joe Yevoli of Charlie Health suggests a pre-mortem instead of a post-mortem: before a big test, ask the room to imagine it was a massive failure, then explain what went wrong.

Cris recommends sharing the experiment design prior to launch to get people invested: “Conduct prediction polls on Slack.” At DoorDash, CEO Tony Xu reads the company-wide experiment emails and replies, congratulating teams and suggesting other ideas to try.

The best advice? Be humble. People take risks and learn faster when leaders engage personally and demonstrate humility. They listen to data over opinion and trust the team to take chances and learn from their mistakes.

What we learned from 50 experimentation leaders

Everyone is wrong 75% of the time. Our guests have committed to measuring outcomes, having honest conversations about the results, and learning as an organization.

These are just a few of the learnings discussed in our season 1 wrap. Listen to the full episode for more insights and join us for season 2!

Every experiment is a story to be told and a lesson to be learned. What’s your story? Contact us and let us know.