From 22 clicks to 5: the zero impact experiment that shaped how Edd Saunders at JobLeads tests

Every experimentation program has a story it keeps coming back to. For Edd Saunders, product experimentation manager at JobLeads, it involves pizza.

On this episode of The Experimentation Edge, Edd walked through the experiment that looked like a guaranteed winner, failed completely, and permanently changed how he designs tests. He also shared the problem mapping framework he uses to train new experimenters and explained how JobLeads multiplied its experiment velocity almost tenfold in about a year.

The sure thing that went nowhere

A few years back, during his consulting days, Edd worked with a large, established pizza company that wanted to explore personalization as a way to improve its web experience. The research was thorough. Customer journey maps showed that the average customer needed around 22 distinct inputs from starting a session to completing an order. About half of all traffic came from returning users, and those returning users ordered the same pizza week after week. Previous experiments had shown that customers kept two or three competing delivery sites open at once, racing whichever basket filled first, hungry and increasingly irritated as their blood sugar dropped.

The logical conclusion practically wrote itself: reduce the number of clicks. The team built a database that captured every item a customer ordered, saved it against their user ID, and served their usual order back in a nicely designed widget on their next visit. One click added the entire order to the basket and sent the customer straight to checkout. Twenty-two steps became five or six.

"It had absolutely zero impact, zero impact on user behavior," Edd said. "It didn't increase purchases. It didn't decrease purchases. People saw it and just thought, nah, I'm not using that."

The team iterated a few times, then drew the honest conclusion: forcing personalization on these customers wasn't worth the money. But the deeper learning was about why the feature failed. "The exploration is still a massive part of customers' delight," Edd explained. Customers wanted to feel in control of their experience. "By making it easier to find their order again, we actually took away some of their control." People would tell themselves they wanted something different this time, browse the menu, and land on the same pizza they always ordered. The browsing was the point.

For Ashley, the story hit close to home. He admitted he personally would have wanted the feature, and that its failure would have kept him up at night. That is precisely the value of the experiment. It saved the company from continuing to invest in a direction that data, logic, and intuition all endorsed, and that customers quietly rejected.

Getting out of the solution space

The pizza story sets up the question every experimentation leader eventually faces: how do you guide someone who has never run a test before? Edd spent about six years consulting and training people who were new to experimentation but full of ideas, and he has a clear diagnosis of where they go wrong.

"The trickiest part is getting people out of the solution space thinking and moving them into the problem space thinking," he said. Solution space thinking means generating ideas and throwing them at the wall. And because everyone carries a natural bias that their own ideas are better, everyone new to testing eventually has the same humbling experience. "Everyone new to this has to go through this kind of ego death, where their brilliant idea actually has zero impact whatsoever."

His alternative starts with the customer. Edd runs an exercise called customer journey mapping that doubles as an introduction to analytics tools. First, understand where people come from, where they land, and every micro step between acquisition and activation, then calculate funnel drop-off rates in granular detail to find the biggest leak. Second, layer qualitative data on top: heatmaps, session recordings, click maps, and scroll maps that reveal the how behind the what. Third, add user research, from interviews to third party reviews. At JobLeads, social media reviews alone provide a huge amount of intelligence about where the product should go.

From those three sources, problems get written as statements: as a new user, I don't register for JobLeads because I don't trust the brand. Then comes the exercise Edd loves most, problem mapping. Each problem lands on a 2x2 matrix. The horizontal axis measures evidence, from pure assumption on the left to fully validated on the right. The vertical axis measures impact. The top right quadrant, high impact problems you know are real, becomes the first batch of hypotheses to test. High impact unknowns get logged for additional research, or simply tested if they're cheap to run.

The payoff goes beyond win rate. Validated problems are useful to marketing, customer support, and leadership, not just product teams. And the approach solves the fear that haunts every new experimenter: running out of ideas. "If you live in the solution space, your ideas run dry pretty quickly," Edd said. "If you understand the problems you're trying to solve, naturally you'll come up with multiple hypotheses for each problem. One problem can spring a nice tree of ideas."

Democratizing experimentation

JobLeads only started testing seriously in early 2025, and the program's growth has been steep. Velocity has climbed from roughly 0.3 experiments launched per month to about 2.8. Edd, who was the second dedicated experimentation hire, has spent much of his time on experiment operations: standardized workflows, making each departments roadmap accessible to everyone, and a company-wide knowledge base that synthesizes the learnings from every experiment for anyone in the organization, whether they sit in marketing, customer support, or product.

That knowledge base serves a second purpose: keeping people excited. Edd has watched interest in experimentation spike at launch and then dwindle as people discover that most of their ideas won't win. His counter is to reframe what a result means. "It isn't look at what's won, look at what's lost. It's look at what we've learnt and look at what better decisions we're empowered to make because we've run an experiment."

Ashley summarized the compounding effect with a baseball metaphor: experimentation isn't about home runs, it's about stacking singles. Edd agreed. "Your one test might not have a huge amount of impact, but when you look back over a year, the 100 or so tests you run might. The growth will have compounded over time."

Looking ahead, Edd's ambition is to push experimentation beyond the specialists. He wants content managers and everyday marketers to take an idea from problem to live experiment themselves, then watch it get validated and absorbed into production. "I guess you could call it democratizing experimentation, giving the power to the people."

The obstacle isn't primarily education. It's habit change. People are already under pressure, and mandating experimentation on top of their day jobs breeds resistance. Edd's answer is to make the value of each contribution visible: "Hey, you had that great idea a few weeks ago. We ran it as an experiment, and the company now knows X, Y, Z because of your insight."

AI plays a quiet supporting role throughout. Edd uses it to validate MVPs cheaply before features get built properly, and for the unglamorous automation that keeps the machine running: drafting tickets, standardizing documentation for the knowledge base, and small workflow automations like extracting Figma links between ticket statuses. Little stepping stones, as he put it, that ease the burden across the organization.

The takeaway

The thread running through the conversation is humility in the face of evidence. The most logical feature can have zero impact. The most brilliant idea can flop. What compounds isn't any single win but the accumulation of validated learnings, shared widely enough that the whole company gets smarter with every test.

Hear the full conversation with Edd Saunders on The Experimentation Edge. And if you're ready to raise your own experiment velocity, visit growthbook.io to see why the open source experimentation platform leader powers testing programs at any scale.