The Experimentation Edge

Scale experimentation with AI: use Cursor desktop/cloud agents for parallel builds and visual QA; orchestrate docs/analysis via Claude; automate cleanups and reporting.

Simplification has a limit. Removing too much can strip away the cues and context buyers actually need to decide.

Win rate matters less than learnings per test — DoorDash ships company-wide experiment summaries (win or lose) that the CEO actively reads and responds to, creating cultural accountability around testing rigor.

The biggest thing that gets a team testing is to just do it. Stop designing the perfect experiment and get something simple live to take away the mystery.

False negatives are more dangerous than false positives — they get institutionalized as "we tried that, it didn't work" and quietly kill good ideas for years.

One centralized team of about 40 people tests every major change to Home Depot's $25B online business, serving 40–50 business teams with consistent hypothesis and analysis standards.

Shift from MVP to MVT: list leap-of-faith assumptions and design minimum viable tests before you build.

Chase estimates over a billion dollars of value from experimentation, and most of the lasting learning comes from the losing tests, not the winners.

A failed test can hold the real winner; contextual onboarding matched to user intent roughly doubled activation and became the default variant after the bundling experiment was rolled back.

Accept that being wrong is the point—experimentation only works when leadership embraces humility

Test metrics before you test features — usage time could signal engagement or just mean your product takes too long to do its job.

Share losses as openly as wins. Wins build credibility, and losses build the psychological safety a testing culture runs on.

Separate your two experimentation modes: high-volume CRO chases many small wins, while big uncertain bets deserve multiple shots to de-risk.

When you struggle to land a result, lead with the story of what the customer did, then bring the numbers.

Unblock teams: create a center of excellence for data science and enable rapid variants with AI-powered tooling.

Twitch used geo-fenced experiments with matched markets and causal inference to measure true price elasticity, turning a feared pricing decision into a measured, accretive one.

Reposition features around how users actually feel, not how you assume they should feel

A feature that fails early in a flow can succeed later; placement and timing often matter more than the idea itself.

DoorDash's price experiment proved price by itself doesn't predict orders. Different customers want different things at different times, which pushed the team toward personalization.

Prioritize like a pyramid: fix the widest-impact experiences first, then optimize down into smaller cohorts.

Top-down buy-in shifts the conversation from "why test?" to "how do we test?": When leadership treats data as the tiebreaker, teams stop defending opinions and start building better experiments.