Notable Quotes
"False negatives are the killer."
"If the intervention in plain English is very weak, just don't do the experiment. You're just wasting time."
"We only make money if our creators make money."
"The numerical result is less interesting than actually describing in plain English what the user behavior was."
"There's this whole idea generation, whose cost has just dropped to zero."
Takeaways
Chase estimates over a billion dollars of value from experimentation, and most of the lasting learning comes from the losing tests, not the winners.
Twitch used geo-fenced experiments with matched markets and causal inference to measure true price elasticity, turning a feared pricing decision into a measured, accretive one.
False negatives are more dangerous than false positives — they get institutionalized as "we tried that, it didn't work" and quietly kill good ideas for years.
Run a broad explore experiment first; small, over-narrowed populations lack power and raise the odds of a false negative. Find the responsive segment with heterogeneous treatment effects afterward.
