We talked to 4 leaders about the surprising reasons experiments win — here are their top takeaways

The result tells you whether behavior changed. It does not automatically tell you why.

Product teams often tell experiment stories in a clean sequence: we formed a hypothesis, built the treatment, and the result confirmed our reasoning. Real tests are messier. A winning treatment may work because users notice a different cue, interpret the choice differently, or change behavior at a point the team did not consider.

That gap between outcome and mechanism is not a statistical defect. Controlled experiments estimate the causal effect of an assigned treatment on measured outcomes. They do not isolate which detail inside a multi-part experience caused the effect unless the design does that explicitly.

Four leaders from Box, Fin, UPS, and Squarespace offer unusually clear examples. Their wins became more valuable when teams resisted the neat story and investigated what users were actually doing.

Four wins that challenged the original explanation

Box won by changing the reference point

Danielle Olean's ecommerce team at Box tested a cancellation-flow offer that let customers move to a cheaper plan. A straightforward hypothesis would predict that the treatment works when some customers accept the downgrade instead of canceling.

The result contained a more interesting behavior. Some customers stayed on their existing plan after seeing the lower-priced alternative. The offer changed the reference point: compared with the smaller plan, the current plan looked more valuable. The team called it the “wine effect,” analogous to how a menu can make one bottle feel appropriately priced by placing another beside it.

The lesson is not that every cancellation flow should add a decoy. The lesson is that choice architecture affects interpretation. Pricing pages, packaging, and save offers do not merely transmit facts; they establish comparisons. The Box experimentation interview also describes how one simplification won while additional simplification lost, revealing the point where reduced complexity removed information customers needed.

Research on choice architecture gives teams language for mechanisms such as defaults, framing, and reference dependence. But behavioral theory should generate hypotheses, not replace a controlled test on your product and audience.

What to inspect: movement between plan options, cancellation completion, retention after the save, support contacts, and longer-term revenue—not only the immediate “did not cancel” event.

Choose metrics that explain wins

Build primary, secondary, guardrail, and diagnostic metrics that connect a result to the customer behavior behind it.

Read the KPI Playbook

Fin won by making an AI response slower

Pedro Tabacof's team at Fin tests changes to an AI customer-support agent. One experiment added latency and unexpectedly improved positive feedback. That appears to contradict a familiar product rule: faster is better.

Several mechanisms could explain the result. A short delay might signal that the system is doing meaningful work. It might better match a customer's expectation of how long a considered answer takes. It might change the timing of the surrounding interface or give the model process more time. The controlled result establishes that this treatment changed feedback in that context; it does not prove a universal preference for slower software.

That qualification matters because performance is still a critical guardrail. Google's Interaction to Next Paint guidance explains why responsiveness affects the experienced quality of web interfaces. An artificial conversational pause and an unresponsive interface are different treatments. Teams should instrument them separately.

Fin's broader production experimentation account shows why AI products require both offline evaluation and online controlled tests. An eval can check whether a response is correct or policy-compliant in a known scenario. An A/B test can reveal whether real customers perceive the product differently across the distribution of live conversations.

What to inspect: time to first token, time to complete answer, resolution, repeat contact, explicit rating, abandonment, escalation, and whether the effect differs by issue complexity.

UPS won when it explained necessary friction

Dave Massey's UPS team tested requiring a recipient email in a shipping flow. The first version, motivated by customer-data collection, caused such a large conversion decline that the team stopped it within about a day.

Years later, the international shipping team had a different need. Customs delays often require contacting the recipient, so collecting the email could prevent a real delivery problem. This version explained the reason for the field. The earlier conversion penalty did not recur.

The surprising insight is not that users suddenly liked forms. The value exchange changed. Friction without a visible customer purpose looked extractive; the same action tied to successful delivery made sense. Baymard's checkout research repeatedly documents the importance of reducing avoidable form burden and clarifying why unusual information is required in checkout usability.

The UPS program story is also a warning against copying surface patterns. “Remove fields” and “add explanatory copy” are not universal prescriptions. Start from the user's job, the operational need, and the smallest treatment that makes the exchange understandable.

What to inspect: field-level drop-off, error rate, completion time, customs-contact success, support contacts, and delivery outcomes. A conversion-neutral change can still win if it prevents expensive downstream failure under pre-agreed criteria.

Squarespace won by learning from a launch that lost

Lina Blackman's Squarespace team launched a blank website template intended to give users complete creative freedom. More people entered the CMS, an encouraging leading signal, but fewer completed the journey. The blank canvas worked for technical builders and overwhelmed people who needed guidance.

The failed experience exposed two modes of use: builders who want control and learners who need structure. Squarespace killed the blank template, but the segmentation informed Blueprint, a guided builder. The Squarespace story shows why the “winning reason” sometimes emerges across a sequence of tests rather than within one treatment.

This pattern is common in onboarding. A treatment can improve an early funnel metric by attracting clicks while hurting the outcome that matters. The team's job is to map the causal chain and avoid promoting a proxy to the status of a business goal. Microsoft's experimentation guidance emphasizes an overall evaluation criterion that connects short-term movement to long-term value in its online controlled experiment research.

What to inspect: entry, first meaningful action, completion, time to value, publishing, retention, and segments defined by experience or intent. Do not select segments after the fact solely because one produces significance.

A winning result needs a mechanism review

Use this sequence before turning a surprising win into a company-wide rule.

1. Verify the test before explaining it

Check assignment, exposure timing, sample ratio, metric SQL, outliers, bot filters, runtime, and variation fidelity. Large effects deserve more scrutiny. The American Statistical Association cautions against reducing scientific conclusions to a single threshold in its statement on p-values.

GrowthBook automatically surfaces key experiment checks and lets data teams inspect warehouse queries. Its experiment analysis provides evidence; it does not remove the need to understand the implementation.

2. Draw the behavioral chain

Write the shortest plausible sequence from treatment to outcome. For Box, the alternative plan changes the reference point, which changes perceived value, which changes cancellation behavior. For UPS, the explanation changes perceived purpose, which changes willingness to provide information, which changes completion.

List a metric or qualitative signal for each step. If none of the intermediate behavior moved, the causal story may be wrong even when the top-line metric is positive.

3. Consider competing explanations

Novelty, selection, timing, technical side effects, and instrumentation can mimic the intended mechanism. State at least one alternative explanation and what evidence would distinguish it. This is a simple defense against turning the original hypothesis into a post-hoc certainty.

4. Replicate or isolate when the stakes justify it

A follow-up can reproduce the treatment, remove one component, change the audience, or extend the observation window. Fanatics once reran an apparently positive ad-removal test because the diagnostic metrics did not support the revenue lift; the second result was flat. Replication costs traffic, so reserve it for effects that are surprising, valuable, irreversible, or likely to become reusable doctrine.

5. Record the boundary of the lesson

Write “this treatment improved this outcome for this population in this period” before writing a general principle. Include known exceptions, guardrails, and what would cause the team to revisit the decision. GrowthBook's experiment documentation workflow keeps the result and analysis connected to the shipped change.

The best surprise improves the next hypothesis

Unexpected wins are valuable because they update the team's model of the user. Box learned about reference points, Fin about perceived response timing, UPS about purpose-driven friction, and Squarespace about different creation modes.

The practical payoff is not a clever story attached to one dashboard. It is a sharper next test: separate the response delay from processing time, isolate the explanation from the required field, test guidance by user mode, or measure whether a lower-tier comparison changes perceived value without creating harmful downgrades.

A controlled experiment can tell you that the package caused the outcome. Diagnostic data, research, and follow-up designs help unpack the package. Teams that do both turn surprising wins into durable product knowledge instead of folklore.

Investigate surprising results

Learn the statistical and causal checks experienced teams use before they trust an effect that looks too good—or too strange—to be true.

Read the Trustworthy Tests Recap