The Experimentation Edge

One centralized team of about 40 people tests every major change to Home Depot's $25B online business, serving 40–50 business teams with consistent hypothesis and analysis standards.

Test metrics before you test features — usage time could signal engagement or just mean your product takes too long to do its job.

Accept that being wrong is the point—experimentation only works when leadership embraces humility

False negatives are more dangerous than false positives — they get institutionalized as "we tried that, it didn't work" and quietly kill good ideas for years.

Prioritize like a pyramid: fix the widest-impact experiences first, then optimize down into smaller cohorts.

Simplification has a limit. Removing too much can strip away the cues and context buyers actually need to decide.

Persistence pays: four months and three to four rounds of trial-model testing at Codecademy produced a 35% conversion increase.

Separate your two experimentation modes: high-volume CRO chases many small wins, while big uncertain bets deserve multiple shots to de-risk.

Share losses as openly as wins. Wins build credibility, and losses build the psychological safety a testing culture runs on.

DoorDash's price experiment proved price by itself doesn't predict orders. Different customers want different things at different times, which pushed the team toward personalization.

Friction can increase revenue. Blocking the "view all" grid and forcing a style choice sent shoppers deeper and lifted conversion and revenue, because the extra click added value.

A failed test can hold the real winner; contextual onboarding matched to user intent roughly doubled activation and became the default variant after the bundling experiment was rolled back.

Democratize experimentation with a centralized platform and self-serve tooling; reset baselines regularly.

Close every losing test with two questions: did it work for a granular segment, and is the idea worth further investment?

Documenting experiments in a centralized Wiki creates a growth flywheel: Fanatics' Wiki feeds their roadmap with iterations on already-built features, reducing tech dependency and accelerating velocity.

A feature that fails early in a flow can succeed later; placement and timing often matter more than the idea itself.

Purge “anti-knowledge” by standardizing design, instituting cross-functional reviews, and only codifying learnings supported by repeatable data.

Build composite metrics (e.g., CPQI) to align finance, engineering, and data science around shared outcomes.

Scale experimentation with AI: use Cursor desktop/cloud agents for parallel builds and visual QA; orchestrate docs/analysis via Claude; automate cleanups and reporting.

Twitch used geo-fenced experiments with matched markets and causal inference to measure true price elasticity, turning a feared pricing decision into a measured, accretive one.

Build the triad: pair an easy-to-use platform with training, top-down sponsorship, and clear launch processes.