How UPS's Experimentation Team Generated Half a Billion From 80+ Apps With A/B Testing
Chapters
00:00 Cold open
00:50 Meet Amir and the air booking platform at Expedia
03:10 DoorDash, growth, and a 70 experiment year
04:20 Three kinds of experimentation at Expedia
06:30 AI velocity and the new frontier model pace
08:30 What you cannot measure, you cannot ship
10:45 The DoorDash carousel and the price experiment
12:45 The three sided marketplace and competing metrics
16:55 There are no losing experiments
20:45 Predictability, LLMs, and Expedia's road ahead
Notable Quotes
"What you cannot measure, you cannot ship."
"I try not to look at experiment as winners or losers, because any experiment, the result is showing us something. It's teaching us something."
"Price by itself doesn't matter, whether it's cheaper or even more expensive. What matters is different customers, they want different things at different times."
"It's a three-sided marketplace because Dashers are also our customers. People who are delivering food are another type of customer."
"Experimentation is all about testing your hypothesis, and avoiding debating on opinions."
Transcript
"What you cannot measure, you cannot ship.""I try not to look at experiment as winners or losers, because any experiment, the result is showing us something. It's teaching us something.""Price by itself doesn't matter, whether it's cheaper or even more expensive. What matters is different customers, they want different things at different times.""It's a three-sided marketplace because Dashers are also our customers. People who are delivering food are another type of customer.""Experimentation is all about testing your hypothesis, and avoiding debating on opinions."
