Notable Quotes
"For me, one of the funnest things about experimentation is regardless of what happens, there is usually always a next step to test even if things don't go as planned. As cheesy as it may sound, even a loss is a win."
"Even though the test ended up positive, what we saw was, okay, there is another bottleneck being created here. We are letting too many people through."
"Going to everybody at first, you build something for everyone, you actually build it for nobody and you burn that first impression that is so important."
"We say, 'Okay, we launched this experiment. The experiment wrapped. It was a massive failure. What went wrong?' And then the room goes silent."
"What AI is leading to is this habit of thinking you have the answer when you don't fully understand it, and that is super dangerous."
Takeaways
Before launching, check the quantitative and qualitative data behind the hypothesis, map the full funnel, and measure toward the real business outcome, so a loss still leaves you with a next step.
Run a premortem before any large, risky experiment: ask the room "it was a massive failure, what went wrong?", have everyone write and share, and build the safeguards while there is still time.
Build it for everyone and you build it for nobody; Teachers Pay Teachers cut roughly 80% of the market for Easel, repositioned it around one job, and roughly quadrupled retention.
