How UPS's Experimentation Team Generated Half a Billion From 80+ Apps With A/B Testing
Chapters
00:00 Cold open and welcome to the show
01:45 What Stitch Fix actually does
04:15 Balancing AI with the human stylist
05:15 From public policy to the A/B testing adrenaline rush
07:15 Inside the weekly experimentation review group
08:45 The AI style assistant and listening to qualitative feedback
10:45 Why adoption friction beats product bugs
13:45 Testing for losers and building guardrails
15:45 Keep rate, successful fixes, and the holy grail metric
18:15 The new platform and the promise of agentic AI
Notable Quotes
"The holy grail for me of experimentation would be a North Star metric that accurately and reliably predicts long-term value."
"We can measure the short term, but we can only predict the long term."
"My mantra, if you will, is if you're only testing winners you're maybe not taking enough risks."
"The most impactful learnings aren't necessarily coming from a bug in the product or feature."
"When I transitioned to the A/B testing space I was blown away. It's just such an adrenaline rush."
Transcript
"The holy grail for me of experimentation would be a North Star metric that accurately and reliably predicts long-term value.""We can measure the short term, but we can only predict the long term.""My mantra, if you will, is if you're only testing winners you're maybe not taking enough risks.""The most impactful learnings aren't necessarily coming from a bug in the product or feature.""When I transitioned to the A/B testing space I was blown away. It's just such an adrenaline rush."
