The Experimentation Edge

Unblock teams: create a center of excellence for data science and enable rapid variants with AI-powered tooling.

Shift from MVP to MVT: list leap-of-faith assumptions and design minimum viable tests before you build.

Win rate matters less than learnings per test — DoorDash ships company-wide experiment summaries (win or lose) that the CEO actively reads and responds to, creating cultural accountability around testing rigor.

A feature that fails early in a flow can succeed later; placement and timing often matter more than the idea itself.

One centralized team of about 40 people tests every major change to Home Depot's $25B online business, serving 40–50 business teams with consistent hypothesis and analysis standards.

Separate your two experimentation modes: high-volume CRO chases many small wins, while big uncertain bets deserve multiple shots to de-risk.

DoorDash's price experiment proved price by itself doesn't predict orders. Different customers want different things at different times, which pushed the team toward personalization.

Documenting experiments in a centralized Wiki creates a growth flywheel: Fanatics' Wiki feeds their roadmap with iterations on already-built features, reducing tech dependency and accelerating velocity.

Reposition features around how users actually feel, not how you assume they should feel

Test metrics before you test features — usage time could signal engagement or just mean your product takes too long to do its job.

Scale experimentation with AI: use Cursor desktop/cloud agents for parallel builds and visual QA; orchestrate docs/analysis via Claude; automate cleanups and reporting.

False negatives are more dangerous than false positives — they get institutionalized as "we tried that, it didn't work" and quietly kill good ideas for years.

Top-down buy-in shifts the conversation from "why test?" to "how do we test?": When leadership treats data as the tiebreaker, teams stop defending opinions and start building better experiments.

The biggest thing that gets a team testing is to just do it. Stop designing the perfect experiment and get something simple live to take away the mystery.

Twitch used geo-fenced experiments with matched markets and causal inference to measure true price elasticity, turning a feared pricing decision into a measured, accretive one.

Massey's first test removed navigation from UPS's shipping checkout flow and delivered $35 million in incremental revenue—proving e-commerce best practices apply even when customers think "this is just a tool, not e-commerce."

Accept that being wrong is the point—experimentation only works when leadership embraces humility

Share losses as openly as wins. Wins build credibility, and losses build the psychological safety a testing culture runs on.

When you struggle to land a result, lead with the story of what the customer did, then bring the numbers.

A failed test can hold the real winner; contextual onboarding matched to user intent roughly doubled activation and became the default variant after the bundling experiment was rolled back.

Build the triad: pair an easy-to-use platform with training, top-down sponsorship, and clear launch processes.