Notable Quotes
"Someone has a hypothesis and you wanna test it, great, but also test the opposite of that hypothesis."
"It was surprising how often the opposite direction actually won. But also even just knowing that something had an effect, it tells you that this thing, this knob matters."
"We set our goal to do 300 A/B tests by the end of the year, and we felt if we can do that, we'll get enough winners, and we can become viral."
"Their second merge request was very often an idea of their own to A/B test."
"The more focused your tests are, I think the more clear your learnings."
Takeaways
Analysis friction sets the ceiling on testing velocity. Three days of ad hoc SQL per test quietly discourages teams from running more.
Make experimentation part of hiring and onboarding. Every new engineer's second merge request was their own test idea.
Translate a growth goal into an execution count. One million subscribers is not actionable. 300 A/B tests by year end is, and everyone can influence it.
