Notable Quotes
"We also run some long-term experiments which span over months, in which we will just hold off a very small traffic, in which we won't show any ad, and we will see how much increment in the customer experience we are seeing."
"It's always something we have to pick and choose. This is important for the business. This is important for the customer. And intuitively, we have to think what's long-term good for us as a business. It's not a yes or no we are looking for, but mostly what's meaningful for us."
"We should not think about how to launch this product. We should think about how to make this product more usable, how to create a product which would be better for the customer. When this thinking comes in, the hypothesis automatically changes."
"It's not only about finding if we will launch or not, but what's the learning we will get. Can it be a stepping stone for our next product we are trying to build?"
Takeaways
A written process (test plan, no early peeking, end-of-test checks) catches bugs early and builds trust; self-serve tooling is what lets PMs experiment without an analyst.
Start every experiment by asking how to make the product more usable, not how to launch it; the hypothesis changes and so do the metrics.
Customers engaged with complementary recommendations, but that engagement did not always translate into purchases; the fix was reducing diversity and building themes around the customer's mission.
