Notable Quotes
"Your odds of winning at roulette or poker are probably higher than your odds of winning at experimentation. It's literally like gambling. And if you like a challenge, that makes it more fun."
"We don't have to convince people to test things anymore. It's more of how do we do it—not why. And that shift is exactly where you want to be as an experimentation team."
"But we have a rule: if you can't explain the causality through your micro-metrics, you replicate it."
"We reran it. Completely flat. That was a real instance of us running into that 5% false positive rate—and it would have been a wrong decision if we hadn't caught it."
Takeaways
Documenting experiments in a centralized Wiki creates a growth flywheel: Fanatics' Wiki feeds their roadmap with iterations on already-built features, reducing tech dependency and accelerating velocity.
Top-down buy-in shifts the conversation from "why test?" to "how do we test?": When leadership treats data as the tiebreaker, teams stop defending opinions and start building better experiments.
Micro-metrics establish causality beyond top-line KPIs: If revenue moves but scroll depth, cart adds, and product views don't follow the same pattern, question the result before declaring a win.
