Notable Quotes
"The hypothesis was that if we feature more subscription information on the registration flow, then account creation will increase. It led to a 38.7% uplift. The learning was that users wanted to be confident about what they are signing up for before they sign up."
"We all were sure it was going to win because we were going to test a localized payment method for our countries. But the experiment went totally in the different direction, the negative direction. Users were still opting for the credit card, still opting for PayPal, but not this one."
"We did cross verify different sources of data, Stripe data, different sources of truth, and shared the data to make them understand that it is not that the data we gathered is wrong. It's just that this payment method has not worked for us. Not every time something we localize is going to win."
"Is it the same for mobile or desktop? If it's not behaving the same, we are not going to implement it for both. Just for one, or for none if it doesn't work."
"70% of the work for an experiment is going to be done before it goes live, and then there's the analysis part that is just left."
Takeaways
About 70% of an experiment's work is done before it goes live: research, clear objectives, data quality, then the analysis is the small part.
Segment by device and traffic source; a change that works on desktop is not shipped on mobile if it did not win there.
