Experimentation
Takeaways
Filter results
Design onboarding around the shortest path to value, not the longest path to personalization
Unblock teams: create a center of excellence for data science and enable rapid variants with AI-powered tooling.
A public Slack channel where execs question PMs about launches and kills keeps rigor and transparency high across 1,000 tests a year
Synthetic digital audiences rank 20 content options by predicted engagement before a single live impression; Principal's first selection beat the control.
.svg.avif){kind=logo}
Guardrail open‑world agents: add out‑of‑domain evals and explicit capability limits; set acceptable error rates based on the stakes of your use case.
Testing gets watered down when "let's try something" replaces a control group; a little pre-planning gets far more out of every experiment.
More conversation history made Fin more helpful and more prone to fake promises, until a targeted prompt fix removed the hallucinations.
Keep humans in the loop for AI-assisted coding and customer answers—trust but verify in regulated contexts.
A winning test is a data point, not a finish line; Charlie Health's form page removal won on top-of-funnel metrics and still exposed a downstream bottleneck that became the next experiment.
Persistence pays: four months and three to four rounds of trial-model testing at Codecademy produced a 35% conversion increase.
Both upstream de-risking and fast iteration work. The right choice depends on how quickly engineering can ship, and it should be made on purpose.
Scale test volume to learning speed, not just shipping speed
One centralized team of about 40 people tests every major change to Home Depot's $25B online business, serving 40–50 business teams with consistent hypothesis and analysis standards.
Make experimentation part of hiring and onboarding. Every new engineer's second merge request was their own test idea.
Start small and visible: rack up quick wins, over-communicate progress, and grow influence through relationships.
Require a hypothesis, a metric and a definition of success for every test idea so learning is the baseline outcome
AI is ushering in a golden era for experimentation, because shipping faster only compounds mistakes unless you measure what you ship.
Before launching, check the quantitative and qualitative data behind the hypothesis, map the full funnel, and measure toward the real business outcome, so a loss still leaves you with a next step.
Calibrate certainty to stakes — tight bounds on revenue and pricing tests, wider bounds on engagement tests so teams don't spin on noise.
More traffic entering the funnel means conversion rate goes down, even when more people reach the bottom; treat it as a law of physics and plan for it before you read the results.
AI lets Grubhub's design teams put clickable prototypes in front of users early, turning each iteration into a micro experiment.
Connect every experiment to the company North Star by cascading it down to sensitive proxy metrics and controllable inputs.
Localization is not a guaranteed win; a country-specific payment method lost to PayPal and cards, and only cross-verified data got the team to accept it.
Define kill criteria and success, failure, and guardrail-dip actions before launch, with leadership sign-off, so nobody chases a loss into a fake win.
A control group is non-negotiable: at scale, a change worth millions is invisible under noise and seasonality, and no one can spot it by eye.
Shift quality left with automated checks so developers catch issues early without human gatekeeping.
Test the opposite of every hypothesis. At Shop It To Me the inverse won surprisingly often, and even when it lost it proved the variable mattered.
Many ecommerce drop offs are structural. The basket and product page leak in roughly 80% of shops because it is ecommerce, not because of your product.
Purge “anti-knowledge” by standardizing design, instituting cross-functional reviews, and only codifying learnings supported by repeatable data.
Plan for failure before you run a test. A pre-built playbook for a loss prevents confirmation bias and keeps teams from gaming the metrics.
Define input and output metrics; ship only what improves core outcomes (retention, sign-ups), and roll back fast if not.
Route every experiment through one entry point. Farfetch's feature toggle connects segmentation, user systems, CMS and messaging.
Understanding the emotional context around a purchase earns more trust than tactical button tests.
Start a new team on foundations: what a clean test and an A/A test look like, which surfaces should not be tested, and a peer review program that lets people graduate to more complex experiments.
Scale experimentation with AI: use Cursor desktop/cloud agents for parallel builds and visual QA; orchestrate docs/analysis via Claude; automate cleanups and reporting.
Manage by learning rate, not win rate. The only failed test is one that was badly designed; every other test produces a learning.
AI saves real time in experiment analysis, but a human in the loop must validate anything AI produces before it goes live.
If an intervention sounds weak when you write it out in plain English, don't run the experiment — you're just wasting time.
Treat B2B as its own customer base with its own testing discipline; a single bulk order on one day can inflate a B2B test into a false positive unless order data is normalized before results are read.
.svg.avif){kind=logo}
A guardrail metric saved Atlassian from a costly mistake: bundling Jira Service Desk lifted trials more than 50 percent but tanked activation and paid conversion, forcing a rollback.
Every experiment on Ebb, Headspace's AI companion, is defined by inputs, outputs and guardrails, with clinicians doing clinical QA
Your customer and your user may not be the same person — building for HR specialists instead of the HRBPs who actually run talent reviews resulted in a feature nobody could use.
Pilot in one branch with a trained “feedback team,” iterate, then roll out—don’t scale too soon.
AI can make anyone an aspirational data scientist for analyzing results and spotting opportunities, but it can be confidently wrong. Keep a human in the loop and sanity check output the way you'd peek at a freshly launched test.
Announce experiments to users before they run; Supercell's community managers tell players what is being tested and why, which turns a skeptical community into a research partner.
About half of PayPal's $180 million impact in 2025 was cost avoidance from features that tested badly and never shipped, which is why Gaurav argues there are no losing experiments.
Match the interface to how people buy: new buyers need information, returning buyers want speed, and B2B buyers want an offer, not a catalog.
Conversion should be a do-no-harm guardrail for teams that do not own it, even when it is not their primary metric.
A culture of experimentation needs data-driven decision making first and a safe space to lose.
Kim's stakeholder filter: if you wouldn't do anything differently after a bad result, don't run the test.
The experiments that fail deliver the most valuable learnings, especially when you expected a slam dunk.
A losing test is not a failure; it prevents costly mistakes and shows what not to do next.
Strategic bets deserve a longer clock than fail fast allows. Farfetch iterated on Inspire for two years before it replaced the market leader.
Test metrics before you test features — usage time could signal engagement or just mean your product takes too long to do its job.
.avif)
Revenue per visitor is the honest north star. Conversion rate can be gamed to 100% by making everything free or cutting bounce-heavy traffic; revenue per visitor can't.
Revenue is not the only guardrail: relevance and latency are long-term metrics, and ad-free holdouts running for months measure what ads cost in trust.
Put failure on the agenda. A biweekly "where did you fail?" retro turns one person's dead end into the whole team's shortcut.
Productize adaptability with model routing to match tasks to the right model family as capabilities shift.
In a decentralized company, a central experimentation team earns its impact by providing the platform, partnering on rigor, and making sure learnings travel across independent game teams.
Treat AI as an activation lever: elevate AI-first CTAs and streamline onboarding to boost adoption.
Guardrails and stopping criteria are what make risk-taking safe, especially when the experience is as personal as shopping.
Treat engagement carefully. For a bank, more time in the app isn't a win; trust, fast task completion, and healthy repeat engagement are.
Run the four-question framework before any test: clean randomization, a plausible effect size for your traffic, a reversible and cheap change, and a falsifiable hypothesis.
The exposure event fixed it. Fire an event as close to render as possible so the platform knows exactly when a user entered the experiment, instead of logging on page load.
Implement a staged experimentation funnel—discovery, simulation, then customer A/B—to reduce risk.
Use AI to accelerate builds, detect incidents sooner, and evaluate models; watch MTTR and MTTD.
A surprising win from a simple change deserves a rerun; Preply unscaled a wording test and the lift disappeared
The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like.
A three sided marketplace (buyers, merchants, Dashers) makes metrics compete. Running the test is easy; deciding what to optimize when goals conflict is the real work.
When you can't test at scale, desk rides replace A/B tests — sitting with users and watching them struggle reveals failures faster than any dashboard.
Top-down buy-in shifts the conversation from "why test?" to "how do we test?": When leadership treats data as the tiebreaker, teams stop defending opinions and start building better experiments.
At Headspace the target is health outcomes, not time in the app, so engagement metrics sit alongside self-reported measures of how members feel
Shift from MVP to MVT: list leap-of-faith assumptions and design minimum viable tests before you build.
Twitch used geo-fenced experiments with matched markets and causal inference to measure true price elasticity, turning a feared pricing decision into a measured, accretive one.
In B2B the randomization unit is a design decision. User-level bucketing risks two colleagues at one account seeing different prices, and account-level avoids that but costs sample size.
Upskill teams in prompt engineering and AI oversight so developers can effectively direct and review AI “agents.”
Losing tests become coded constraints: two losing popular parlays tests turned into a 2.5x penalty weight and a hard floor in the next experiment's envelope.
Build hypotheses around user psychology, not just KPI movement
The agent builds its own features (756 candidates to 20), never sees the validation set, and will reframe a problem when the scoring rules make a wrong guess cost more than a right one earns.
Analysis friction sets the ceiling on testing velocity. Three days of ad hoc SQL per test quietly discourages teams from running more.
AI scales institutional knowledge, not just analysis speed — mining past experiment readouts to auto-generate new hypotheses turns your testing history into a compounding advantage.
Prioritize like a pyramid: fix the widest-impact experiences first, then optimize down into smaller cohorts.
A center of excellence that shares wins and losses turns tribal knowledge into shared knowledge and stops "we tried that years ago" from killing retests.
You cannot unit test a non-deterministic AI. A/B testing at scale, millions of samples in days, is the only reliable way to know a change helped.
Use LLM based agents as a cheap simulation layer to screen hypotheses, not as a replacement for real A/B tests.
Institutionalize learning: align OKRs with “fail forward,” and be willing to kill low-performing features quickly.
One-size metrics break in multi-dimensional marketplaces — DoorDash balances consumer retention, dasher utilization, and merchant inventory mix across verticals because optimizing one side degrades the ecosystem.
Log every test and its learnings where the whole organization can see them; that reinforcement loop is what separates world-class experimentation programs.
Moving new experimenters from solution space to problem space thinking raises win rates and produces learnings the whole organization can use.
Map where the decision happens, not just where the purchase happens; Samsung's business buyers decide on mobile and buy on desktop, and surfacing add-ons on mobile nearly doubled attach sales.
Measurement spans three live dimensions: spend (more with less), speed (sprints instead of quarters), and quality, with guardrail "do no harm" metrics on top.
Friction can increase revenue. Blocking the "view all" grid and forcing a style choice sent shoppers deeper and lifted conversion and revenue, because the extra click added value.
Translate a growth goal into an execution count. One million subscribers is not actionable. 300 A/B tests by year end is, and everyone can influence it.
Enforce experiment hygiene: change one variable at a time, randomize at the right unit (account vs. user), and run long enough for effect size.
A simple fallback, like a two second load rule, can save an ambitious experiment without sacrificing coverage or security.
Treat experimentation as a portfolio: balance confirmatory tests that protect the business with game-changing bets that can win big.
Assess risk tiers when building the roadmap so measurement rigor scales with the stakes instead of slowing every decision down.
The biggest thing that gets a team testing is to just do it. Stop designing the perfect experiment and get something simple live to take away the mystery.
