Notable Quotes
"You should not be worried about a losing test. You should not be worried about a flat test. You should absolutely be worried about designing a test in such a way that you will have impactful learnings regardless of the outcome."
"I really believe that programs that run an A/B testing program and choose win rate as their North Star are missing the boat. At GoPro, losing tests for us are jet fuel. That is what helps us get to the right experience way faster than we would have otherwise."
"That's where I really learned it's so much about the audience and the uniqueness of your customer. The through line for me is you've got to know your customer, and that's so much of what the experimentation is about."
"What it tells us is our customers, because it is a high consideration purchase, they're comfortable with a little bit of friction. They're comfortable having to make a few extra choices as long as there's value in those choices that they're making."
"The thing that I'm going to watch really carefully for is making sure that we're not running so many tests that we start running tests without taking time to understand and implement, crucially, the learnings."
Takeaways
AI can speed up building and running experiments, but a team that runs more tests than it can learn from is not getting better.
Share interim readouts across the company, and use them to show how volatile results are before a test reaches statistical significance.
A flat result is still an answer. GoPro's configurator test showed that buyers of high consideration products accept extra steps when each choice adds value.
