GoPro's case for treating losing tests as jet fuel

Most experimentation programs keep score the obvious way. Count the wins, divide by the tests, report the win rate. Will Guyeskey thinks that is the wrong scoreboard.

Will leads digital product at GoPro, where his team owns the e-commerce side of gopro.com. Before that he ran personalization across Gap, Banana Republic, Old Navy and Athleta, and before that he learned the craft at Brooks Bell, a consultancy focused entirely on A/B testing, experimentation and personalization. His clients there included Under Armour, Barnes & Noble, Ralph Lauren, Fidelity and IHG.

Across all of it, he says, one idea holds. "The through line for me is you've got to know your customer," he told Ashley Stirrup. "And that's so much of what the experimentation is about."

A humbling education in consulting

Consulting gave Will a fast, wide view of what works. It also taught him how often the expert is wrong.

"It's a very humbling experience to be on the consulting side and to be the expert," he said, "and then thinking that a big win on Under Armour might be translatable to Ralph Lauren and finding out that's not the case."

That lesson shaped how he approaches every program since. A result is a fact about one audience. It is not a rule for all of them.

The page nobody had tested

The clearest example came from Barnes & Noble. The retailer had a strong testing program, and Brooks Bell worked with it for years. The team tested every page type, many times: the homepage, product listing pages, product detail pages, landing pages, cart, checkout and search results.

One page had never been tested. The order confirmation page.

The team did what it always did before a test. It pulled the data on who visited that page, where they clicked, how long they stayed and whether they scrolled. It met in what Will calls "our little war room." The idea that came out was a recommendation module on the confirmation page.

Will did not expect much. "I was certain that it would be at best dead flat, probably a loser," he said. "And so we ran the test, and, of course, it was a big winner."

Why it worked for readers

The reason the test won says as much about the customer as about the page.

Barnes & Noble shoppers are voracious readers. Their purchases are small, a $15 or $20 book rather than a $600 camera. By the time a shopper reached the confirmation page, the site knew what they had browsed and where they had lingered. That meant the recommendations could be very good.

"Because they're a reader, they didn't want to lose that recommendation," Will said. "They wanted to purchase right then. And because the AOV wasn't super high, they felt like they could pull the trigger without too much hesitation."

The team measured the effect at the level of the visit. Often the shopper clicked the recommendation, went back and bought again in the same session. That visit ended with a higher average order value.

The same idea, everywhere else

Naturally, the team tried the idea with other clients. It did not work. "I don't remember it working on a single other client that we had," Will said.

At Ralph Lauren, the items were more expensive, and a shopper who had just bought something might not want more from that season. The recommendation that felt like a gift to a reader felt like noise to a clothing buyer.

Will does not see that as a disappointment. He sees it as the point. "It was unique to Barnes & Noble, which makes the insight all that much more powerful," he said. "This is a unique thing for your audience visiting your site. That's gold for them."

Testing friction on the GoPro Mission page

At GoPro, the same principle showed up in a recent launch. For the new Mission series of cameras, Will's team built a new kind of product page with a configurator. Shoppers choose a camera, then step through a series of decisions. Do they want a subscription, which automatically uploads footage and creates highlight reels? Do they want an SD card, which every GoPro needs? Do they want an accessory bundle?

Each step requires a choice. Shoppers must pick an option or say no thanks before they can add to cart.

That design drew some concern inside the company. Was it too much friction?

Will's team did not think so. "These are high price point items, high consideration purchases," he said. "We want to honor that by walking them through these pretty important decisions along the way." Still, he called it "a fair question to ask," and the team set out to answer it with a test.

In the variant, every option after the camera defaulted to no thanks, so shoppers could add to cart right away. In the control, a shopper who skipped the questions could not add to cart. The other side of the debate worried that auto-selecting no thanks would lower average order value.

The result was flat.

What a flat result revealed

For Will, flat is not the same as empty. "That is fantastic insight for us into the customer," he said. "Our customers, because it is a high consideration purchase, they're comfortable with a little bit of friction. They're comfortable having to make a few extra choices as long as there's value in those choices."

Ashley offered a picture of that buyer. Someone buying a GoPro may have a vacation coming up and a flight in a few days. They want to be sure they did not forget the battery or the SD card. The extra steps are a checklist, not a hurdle.

The team filed the learning away, and it now informs the next product page GoPro builds.

Sharing results before they are final

Learning only helps if people hear about it. GoPro runs a biweekly call that brings together engineering, product, program operations, merchandising, IT and marketing. That is where the digital product team shares its priorities and its A/B test learnings.

Recently the team started sharing interim readouts too. For a long time it held them back, worried people would see a result that was up one week and assume it would stay up. "We frequently will share out results that are way down one week, and they'll be way up the next week," Will said.

The team decided that seeing the volatility was itself useful. "That's precisely why we believe in rigorous experimentation design and not calling something until it reaches statistical significance," he said.

Why win rate is the wrong North Star

That brings Will back to the scoreboard. When someone new to experimentation arrives with an exciting feature, he steers the conversation away from whether it will win.

"You should not be worried about a losing test. You should not be worried about a flat test," he said. "You should absolutely be worried about designing a test in such a way that you will have impactful learnings regardless of the outcome."

His view of win rate is direct. "I really believe that programs that run an A/B testing program and choose win rate as their North Star are missing the boat," he said. "At GoPro, losing tests for us are jet fuel. That is what helps us get to the right experience way faster than we would have otherwise."

Part of designing for learning is the gap between control and variant. Will's team talks about that distance often. Sometimes the variant needs to be further from the control to produce a clear read.

AI, velocity and the intern

Will expects AI to change experimentation in significant ways, at GoPro and everywhere else. It already helps his team build test assets and experiments faster.

His concern is what that speed can do to learning. "The thing that I'm going to watch really carefully for is making sure that we're not running so many tests that we start running tests without taking time to understand and implement, crucially, the learnings," he said.

GoPro will keep monitoring win rate and trying to improve it. But more tests only help if the team understands what each one says about the customer.

Ashley sees AI opening experimentation to more people, acting like a data scientist that helps them set tests up correctly, while agreeing that AI is far from running tests and drawing conclusions on its own.

Will's team has a name for that balance. Someone on his team calls Claude "the intern." "This is a really potentially capable and powerful assistant to have," Will said, "but it still very much needs your guidance and your direction."

The takeaway

Will's career runs from Brooks Bell to Gap to GoPro, and the lesson has stayed the same. Winning ideas rarely travel between audiences. Flat results still carry answers. Losing tests are worth running when they are built to teach.

The win rate will take care of itself. The job is to know your customer, and every test is a chance to know them better.

🎧 Listen to the full episode →