How 12 of the world's top product teams use AI to run experiments

AI is making it possible to build more experiments. The people running the strongest programs are using that speed to ask harder questions, not to skip judgment.

When Kameron Tanseli described the growth engineering workflow at Fyxer, he did not talk about handing a roadmap to an autonomous product manager. He talked about something more immediate: a coding agent that can implement several contained experiments at once, run the product, and bring back a video for review.

“With Cursor Desktop, it’s running the app. It’s showing me a video of what the experiment is going to look like. Then I can just sign it off without having to do the extra manual QA myself.”

That small scene captures the change taking place across product teams. AI does the waiting-room work: finding a prior test, reading hundreds of calls, drafting a variant, checking an interface, or assembling a first-pass analysis. A person still decides whether the idea is worth pursuing, whether the evidence is trustworthy, and whether the change should reach users.

Across conversations with leaders at Fyxer, Microsoft, Fanatics, Twitch, Diligent, DoorDash, Chess.com, Atlassian, Fin, Stitch Fix, Upwork, and Kargo, the same story appeared in different forms. First, AI made accumulated knowledge easier to retrieve. Then it moved into building and analyzing experiments. Now, for teams shipping AI products, the model itself has become the thing that needs to be tested.

The result is not autonomous experimentation. It is a faster learning loop with a more important human checkpoint.

The bottleneck moved before the process did

Fyxer shows what happens when an experimentation process is already explicit enough for agents to join it. Its growth team has a shared brief, a central data source, visible experiment updates, and a clear review point. AI can move between those parts because the parts already have names.

In the full conversation with Tanseli, he explained that Cursor can handle smaller variants in parallel, while Claude acts as a hub for shared skills, reporting, and analysis. An internal “AI data scientist” can answer complex questions against documented table schemas. Automations check for stale experiment code. The team is not asking one model to improvise the entire lifecycle; it is giving several tools bounded jobs.

“AI is definitely helping us go faster,” Tanseli said. “The main focus is developer performance. Can we break through a thousand experiments?”

The longer Fyxer story matters because the ambition is attached to a real operating system: reusable skills, a reviewable diff, visual QA, shared warehouse context, and a public changelog. Without those pieces, faster code would mostly produce a faster queue of things nobody has validated.

Marco Casalaina, VP of Products for Core AI at Microsoft, described the same acceleration from inside an organization building the tools themselves.

“We use Copilot CLI to help us build Copilot CLI,” he said in the Microsoft episode. “We’re doing like two releases a day. The velocity of this stuff is vastly increased.”

That velocity changes the cost of implementation, but not the need for evidence. Microsoft’s long-running experimentation platform was built around that distinction: trustworthy shared infrastructure is what lets a company learn at scale. AI increases the flow of candidate changes through the system. It does not repeal assignment integrity, statistical power, or the need for a control.

AI first made experiment history usable

Before an agent writes a variation, it can answer a question many companies are surprisingly bad at answering: Have we already tried this?

At Fanatics, Medha Umarji has spent years building a centralized experimentation wiki. It contains results, screenshots, causal interpretations, recommendations, and the next iterations that feed the roadmap. That work began long before the current wave of generative AI. Now it gives AI something worth retrieving.

“The wiki is really helping us as we move toward AI,” Umarji said in the Fanatics conversation. “People are able to build their own feature-based meta-analysis.”

The important word is meta-analysis. Fanatics does not ask a model to summarize a single isolated result and call that institutional knowledge. When a team has run several tests on a feature, it builds a compact view of the different messages, price points, or treatments and what happened across them. Tools such as Claude and Glean can then help someone find that body of evidence before proposing the next test.

Twitch has taken a similar idea closer to the product manager’s daily workspace. Arun Bodapati described an internal AI environment called Kiro, with connections to company data and tools.

“We built an experimentation MCP into it,” he said in the Twitch episode. “Now all of the experiments that are running, and all of the experiments of the past, are within that MCP.”

A product manager can ask for a summary of the advertising experiments currently running, inspect earlier results, and use that history to think about the next hypothesis. Bodapati was careful about hallucinations and the need for guardrails, but the practical unlock was already clear: people who once needed help locating the right notebook can begin with the organization’s accumulated evidence.

At Diligent, the source material is not only prior experiments. It is the customer conversation that should precede one. Dan Layfield contrasted the old research process—recruiting users, conducting interviews, transcribing them, and manually synthesizing the findings—with what his team can do now.

“We use Gong’s MCP, and you can get reasonably good synthesis from sales calls in probably an hour or two,” Layfield said in the Diligent episode.

That does not turn a sales call into a product decision. It lets a product manager reach the judgment stage while the evidence is still timely. The Diligent story also supplies an important counterweight: AI can accelerate the analysis of an inconclusive test, but it should not automatically kill a strategically important idea after the first disappointing result.

DoorDash has the scale version of this memory problem. Ilya Izrailevsky said the company runs about 12,000 experiments a year across a marketplace connecting consumers, merchants, and Dashers. Nobody can remember that history unaided.

“We’re using AI to mine the past experiments we already launched, see what worked and what didn’t, and use the past hypotheses to inform new hypotheses,” he said in the DoorDash episode.

The same agentic layer can help set up tests, diagnose imbalance, analyze results, and draft the company-wide summaries that DoorDash leaders read. But the DoorDash experimentation story makes the boundary visible: retrieving evidence is a computation problem; deciding how to balance consumer price and speed, merchant economics, and Dasher earnings is not.

Test AI Beyond Offline Evals

Learn how to combine offline evaluation, guarded rollout, and controlled experiments when shipping generative AI.

Compare Evals and A/B Tests

Then the agent moved into the build loop

Once research and prior learnings are available, the next bottleneck is turning a defensible hypothesis into something testable.

Chess.com is still working toward the most complete version of that loop. Nafis Shaikh described the destination plainly in the Chess.com conversation:

“We want to have a conversation in Slack about a particular feature, build out all of its requirements through conversation, and hand it off to an agent. An agent will build the feature, set up the experiment, run the experiment, and report back to that same Slack thread.”

Shaikh did not present that as current reality. Chess.com had already built a high-velocity experimentation practice, but the agentic workflow remained an infrastructure goal. That candor is useful. There is a large gap between generating code and closing a trustworthy experiment loop, and the gap contains requirements, targeting, instrumentation, metric selection, QA, runtime, and interpretation.

Atlassian’s Andrew Willingham begins one step earlier: with the reason a person should use an AI feature at all. A company mandate to “use AI” is not a value proposition.

“If I’m a frontline product manager, do I care about AI because the company says it’s important? Probably not,” he said in the Atlassian episode. “What I care about is: you’re about to type up that status report; we’ll generate it for you.”

That is an experimentation problem before it is an implementation problem. Teams can test the promise, placement, and moment of need rather than treating the presence of AI as the benefit. Atlassian’s separate AI Product Builders Week created more than 100 documented use cases, including an agent for retrieving prior growth experiments. Shared demos and failed prototypes gave those tools organizational context that a generic coding model would not possess.

Microsoft’s Casalaina described what happens after the first build. He gave a coding agent an evaluation threshold, let it revise an AI application, run the evals, inspect the result, and continue overnight until it crossed the bar.

“It hit my threshold at 3:35 in the morning,” he said. “That’s the funny thing about evals: you’re not going to hit 100% most of the time. You have to pick a threshold and say, at this point, this is good.”

This is where procedure matters. GrowthBook’s approach to AI coding agents and A/B testing separates capability from rules: an API or MCP server gives an agent access, while reusable skills describe how to find prior work, draft an experiment, check results, and clean up temporary flag logic. The AI-native development layer is useful precisely because the workflow remains inspectable. The goal is not to make the agent responsible for the product decision. It is to shorten the distance from an approved decision to verified execution.

Real users expose failures that offline evals cannot

The story changes when AI is not merely helping run the experiment. When the model, prompt, retrieval system, or agent behavior is the treatment, offline evaluation can screen candidates—but people still produce the surprises.

Pedro Tabacof, Principal Machine Learning Scientist at Fin, called production testing the “gold standard for decision-making.” Offline evals may cover thousands of examples. Fin’s live experiments can reach millions of samples in days and measure resolution rate, customer satisfaction, positive feedback, hallucinations, and other business or guardrail outcomes.

One test began with an assumption that almost everyone shared: faster AI responses should be better. The team could not easily make Fin faster for the experiment, so it carefully made some responses slower. The result went the other way.

“Higher latency only saw good stuff happening,” Tabacof said in the Fin episode. “What we didn’t expect was that positive feedback would increase as well.”

The team’s best hypothesis was psychological. An immediate answer can feel canned; a short pause can signal that work is being done. Fin did not deliberately slow the product after the test, because latency still mattered in demonstrations and other contexts. Instead, the team learned to separate latency from quality in later experiments so that an apparently better model was not credited for an effect caused by timing.

Another test was more consequential. Adding conversation history improved positive feedback, but also made Fin more likely to invent promises such as approving a refund. The headline metric went up while a guardrail got worse. The team stopped, isolated the new hallucination patterns, changed the prompt, and tested again. The second version kept much of the improvement without the unacceptable promises. The full account of the refund experiment, along with Intercom’s description of the testing behind a major Fin release, shows why “the treatment won” is not enough for an AI system.

Stitch Fix found a gentler version of the same lesson. Its AI style assistant helps clients clarify the note they send to a stylist. The quantitative results were promising, but direct feedback revealed that some clients wanted more flexibility or felt pushed toward the tool.

“It’s not just the quantitative results,” Nick Beyler said in the Stitch Fix episode. “We get feedback directly from our clients that are part of the experiment.”

That feedback changed how the feature should be introduced. Stitch Fix’s product combines recommendation algorithms with a human stylist, and Beyler emphasized that balance: the algorithm can narrow options, but a person still interprets a request for a summer wedding or a client’s discomfort with uploading an image. Controlled measurement can tell the team where the AI helps; qualitative evidence helps explain why someone resisted it.

Upwork’s Vinoj Kumar gave perhaps the cleanest warning about AI engagement metrics. His team built “Talk to Data,” a natural-language interface over company data. Counting questions made the product look busy, but a busy chatbot was not necessarily a useful one.

“If a product manager asks the bot the same question five different ways because the bot is giving bad answers, your dashboard shows high engagement,” Kumar said in the Upwork episode. “That’s a retry storm, not success.”

The team added explicit feedback and a “turns to success” metric, then looked downstream. As Talk to Data usage rose, basic ad hoc tickets to analysts fell by about 20%. That outcome was more convincing than message volume. Kumar also described a composite “cost per quality inference” metric that forces latency, cloud cost, and model quality into the same conversation. One number alone would invite one function to improve its target at everyone else’s expense.

These stories reinforce the layered approach in Intercom’s writing on designing AI products: use offline tests to catch obvious regressions, then combine production experiments, telemetry, and qualitative evidence. The same pattern appears in GrowthBook’s sessions with Khan Academy, in analysis of Character.AI’s user-centered model experiments, and in Dropbox’s staged AI rollouts. Each team needs a different outcome metric, but all of them need evidence from the environment where the product will actually be used.

The useful boundary is architectural

Kargo resolves ad auctions inside an unforgiving latency budget. James Falzone does not expect a large language model to sit in that live path simply because agents are fashionable.

“LLMs cannot operate in the ad-tech world just yet. They’re not fast enough,” he said in the Kargo episode. “But they can operate at an orchestration layer. We could have an agent running tests for us.”

That distinction is more useful than a generic mandate to “add AI.” Flexible reasoning belongs around the real-time system: opening access to data, investigating outcomes, proposing variants, and orchestrating test operations. Deterministic, low-latency software remains in control of the auction. The Kargo story also shows why broader access cannot mean weaker context. A bidding model that worked for direct demand initially failed when copied to third-party demand because different customer systems reacted differently.

DoorDash faces another kind of architectural boundary: no single metric can represent a three-sided marketplace. An agent can surface an imbalance, prepare a readout, or suggest an experiment design. It cannot decide how much consumer speed should be traded for Dasher utilization or merchant economics.

“AI can help you do a lot of research and give you insights,” Izrailevsky said. “At the end of the day, humans need to make the calls and prioritize the go or no-go decisions.”

Other conversations point to the same boundary. In the Box episode, a sequence of pricing-page experiments showed that “simpler” can become “oversimplified”; the written Box story is exactly the kind of context an ideation agent needs before recommending another simplification test. In the JPMorgan Chase episode, Kevin Yang argued that AI-driven shipping speed creates demand for more measurement, not less; the longer Chase account describes the self-service infrastructure and expert support around that speed.

The NIST AI Risk Management Framework uses broader governance language, but the operating idea is similar: organizations still need explicit ways to govern, map, measure, and manage AI risk. Google Cloud’s discussion of experimentation as an organizational capability makes the cultural counterpart. The technology can make testing easier; leaders still decide whether learning is valued and acted upon.

The human role is getting more specific

The guests did not defend a vague “human in the loop.” They named the work people still have to do.

People choose the problem. Fin can use Claude to inspect raw examples, generate hypotheses, draft code, and scaffold another test. Tabacof still sees a gap in taste.

“AI is very good at the low level—writing code, writing prompts, even churning through ideas,” he said. “But knowing exactly what to experiment on is the big challenge.”

People define acceptable error. Microsoft can run an agent through a self-correcting evaluation loop, but someone has to decide whether 90% task adherence is enough for that use case and whether an open-world agent should be capable of performing the requested action at all.

“Master the art of evaluation,” Casalaina advised. “There is no one way to do it. There is no one metric that’s right.”

People decide what a metric means. At Twitch, longer viewing may signal a healthy community or a poor experience. At Upwork, more chatbot turns may signal curiosity or failure. At Atlassian, more time in an internal tool might mean engagement—or that the task takes too long. Before optimizing a proxy, the team has to establish its relationship to the outcome that matters.

People approve exposure. A pricing treatment, a financial flow, a model with access to tools, or an AI agent that can promise a refund deserves a named launch boundary. Small cohorts, guardrails, rollback paths, and the discipline to stop are what make faster implementation compatible with responsible operation.

And people interpret tradeoffs. The agent can point out that resolution rose while hallucinations worsened, or that an inference became faster while cost and quality moved in opposite directions. It should not silently decide which constituency absorbs the downside.

Start with one slow, inspectable handoff

The wrong first project is “automate experimentation.” It is too broad to verify and too vague to govern.

Instead, map the path from question to decision and find one bounded wait:

1. Research: Can the team retrieve relevant calls and prior experiments with links back to the source?
2. Design: Can an assistant turn an approved hypothesis into a draft design without inventing the success metric?
3. Implementation: Can a coding agent return a diff, tests, a preview, and the feature-flag state for a contained variant?
4. Launch: Can the system prepare a draft while a person approves targeting, exposure, guardrails, and rollback?
5. Analysis: Can AI inspect predeclared metrics and raw examples while clearly separating observation from inference?
6. Learning: Can the result, caveats, screenshots, and next decision be found by a different team six months later?

Give AI read access before write access. Measure the automation itself: time from approved brief to launch-ready draft, the percentage of analyses requiring material correction, the number of tests that reuse prior evidence, and reviewer time saved. A process that launches more experiments but creates more cleanup has not become more effective.

The 12 teams do not share one tool stack. They share a shape of responsibility. AI compresses a defined part of the learning loop. A controlled experiment reconnects that speed to what users actually do. And a person remains accountable for the problem, the threshold, the exposure, and the decision.

Hear the Stories Behind the Tests

The Experimentation Edge features product, engineering, growth, and data leaders unpacking the experiments that changed how their teams work.

Explore the Podcast