Notable Quotes
"If you go to paypal.com, as soon as you land on the page, you will be assigned a control or a treatment. That is the assignment or the intent that we are going to show you control or treatment. But that does not mean you actually saw the control or treatment."
"If the data is diluted, then you will reach stat sig very quickly, even if some of the users did not even participate in that experiment. That was one of the big reasons it was creating a false win rate for us."
"There are no winning or losing experiments. If your experiment is winning, that means you're going to have a direct revenue impact. But if your experiment is losing, that also is sort of a winner because that is helping you to not ship a bad piece of feature. One is a direct impact, one is a cost avoidance."
"Building an experimentation platform is just half of the job. If we put a good set of engineers together, maybe in six months you can build a decent enough experimentation platform. What took a lot of effort was educating teams on the best practices of experiment."
"I think by default, every agent that should go live in production should go behind a feature flag. If an agent has access to a lot of data and a lot of context and it is not governed, that could actually be pretty dangerous for some of the businesses."
Takeaways
About half of PayPal's $180 million impact in 2025 was cost avoidance from features that tested badly and never shipped, which is why Gaurav argues there are no losing experiments.
A six variant carousel test projected at 700 days was resolved by a multi armed bandit in 51 days, with a winner worth almost five basis points.
Standardized instrumentation and canonical metric definitions are what let the analysis pipeline run without human cleanup, taking experiment readouts from four weeks to 24 hours.
