Notable Quotes
"The holy grail for me of experimentation would be a North Star metric that accurately and reliably predicts long-term value."
"We can measure the short term, but we can only predict the long term."
"My mantra, if you will, is if you're only testing winners you're maybe not taking enough risks."
"The most impactful learnings aren't necessarily coming from a bug in the product or feature."
"When I transitioned to the A/B testing space I was blown away. It's just such an adrenaline rush."
Takeaways
A losing test is a finding, not a failure. If every experiment wins, you're not taking enough risk to learn anything new.
Quantitative results are only half the story. Direct, qualitative client feedback inside an experiment often reshapes the rollout more than the numbers do.
The most valuable North Star metric is the one you can't measure yet, long-term client value, and causal-inference modeling helps predict it from short-term behavior.
