"A test without a specific falsifiable hypothesis is just like a fishing expedition in my mind."
"You have to understand the A/B test is a learning agenda. If you've learned something, it's good enough. If you're winning too many, I would be very skeptical, because really 85 to 90% of them are supposed to fail."
"You should have a pre-registered kill criteria. What would make you say out loud that this idea was wrong and we're not shipping it?"
"If you're just seeing flat or slightly negative changes, be skeptical. Dig deeper."
"Imagine if you didn't A/B test: if 85 to 90% fail, you'd actually be losing revenue. If you were simply to launch everything without testing, imagine the losses you missed. My sincere request is people do more A/B testing, not less."
Takeaways
Log every test and its learnings where the whole organization can see them; that reinforcement loop is what separates world-class experimentation programs.
Define kill criteria and success, failure, and guardrail-dip actions before launch, with leadership sign-off, so nobody chases a loss into a fake win.
Pre-register the full analysis plan, including hypothesis, mechanism, primary metric, exact statistical test, and subgroups, so p-hacking can't creep in when a test goes sideways.
