Notable Quotes
"The 30, 30, 30 rule basically means that 30% of your experiments are gonna be winners, 30% are going to be inconclusive or insignificant, and then 30% are gonna lose. The math is the math."
"When you forecast out the impact of that fallout compared to the impact of the attach, it was actually a loser. The revenue was actually a loser."
"Your winning metric is really important, but knowing what other impacting metrics and secondary metrics are in the mix and where the thresholds for success around those lies is so important, especially before you're ready to do your victory lap."
"Keep it at the simplest definition of a good test and then let them try. You learn by doing."
"Not making the experiment do all the work either. Go do the homework first. Do your journey map, do your usability testing, do your research, and pull in those qualitative insights."
Takeaways
AI can make anyone an aspirational data scientist for analyzing results and spotting opportunities, but it can be confidently wrong. Keep a human in the loop and sanity check output the way you'd peek at a freshly launched test.
You don't need a stats background to run good experiments. Teach the simplest definition of a good test, then let people learn by doing.
Start a new team on foundations: what a clean test and an A/A test look like, which surfaces should not be tested, and a peer review program that lets people graduate to more complex experiments.
