How JP Morgan ships faster, measures better: experimentation in the age of AI

When Kevin Yang puts a number on what experimentation has been worth to JPMorgan Chase, it stops the conversation. Over the years, his team estimates the value driven by experimentation and innovation at more than a billion dollars. For most companies, that figure alone would be the headline. For Kevin, who has spent the last six years building experimentation across Chase's digital platforms, the more interesting story is hiding inside it.

"That's only from the winners of the experiments," he told Ashley Stirrup on The Experimentation Edge. "Not even the losers." And then the line that reframes the whole discipline: "A lot of the value, if you think about it, the losers are probably where the value is really coming from."

That idea runs against the instinct of almost every team that ships software. We are trained to celebrate wins and quietly bury losses. Kevin's argument is that the losses are the point. They are the cheapest insurance a business can buy, and the only reliable engine of learning it has.

Listen to the full episode →

Why the losers matter most

The logic is simple once you sit with it. When an experiment wins, you ship it and move on. You rarely interrogate it, because success doesn't demand an explanation. When an experiment loses, you are forced to ask the question that actually makes a team smarter: why did this lose? Was the assumption wrong, or was it the execution? Should we refine the idea, or abandon it?

A losing test also does something the win never does. It stops you from rolling out a change that would have quietly degraded the experience for millions of customers. At Chase's scale, where Kevin says nearly every line of business could be its own company, that protection is enormous. The billion-dollar figure counts the upside of the winners. It doesn't even count the losses avoided by catching bad ideas before they scaled.

That scale is real. Kevin's team supports roughly 100 product teams across Chase's consumer bank, spanning credit cards, checking and savings accounts, mortgages, a sizable travel business, and even call-center and backend systems. They run about 300 experiments a year now, up from just eight in the first year after the infrastructure was in place. The team is built on two pillars: a platform that lets product teams self-serve experiments without needing a data scientist embedded on every team, and a practice arm that embeds experts into strategic initiatives to instill the right culture and decision frameworks.

The chart nobody can read

The cultural work is harder than the technical work, and Kevin has a favorite exercise for it. He puts up a 90-day time series of an app completion rate, a metric where a small relative change is worth millions to the business. Somewhere in that chart, a real and significant change occurred. He asks the room to point to it.

"So far, every time I try this, I don't think I've ever had anybody get it right," he said.

People pick the spikes. They pick the drops. They pattern-match to whatever looks dramatic. But the dramatic movements are just noise and seasonality, the same up-on-Monday, down-on-Sunday rhythm that anyone who has stared at marketing data will recognize. The genuinely valuable change is invisible to the naked eye, buried under variance.

Then Kevin reveals the trick. Add a control group, and a second line appears. The orange control line and the shaded gap between it and the treatment are the impact, finally made visible. "Most of the stuff you're releasing to market is not going to create huge spikes that you can really observe," he explained, "especially with seasonality and everything. So having a control group is super important."

This is the first myth he busts in every training: the belief that you can simply monitor a metric and know whether your change worked. At Chase's scale, where a single percentage point can be worth millions and is almost impossible to see by eye, monitoring without a control is just trusting noise. As Kevin put it, once you start measuring everything, you don't just learn more about individual features. You start to see your whole portfolio at a higher level, and that comprehensive view is what moves an organization in the right direction.

Measuring what's easy to count and hard to interpret

Not every metric behaves. Kevin is careful about engagement in particular, calling it one of those things that is easy to measure and hard to interpret. For a bank, more time in the app is not automatically good. The goal is trust and fast task completion, not screen time.

He gave a concrete example. On the mobile home screen, there's a tile that leads to your credit score. The team debated whether to bring the score forward or tuck it away, worried that surfacing it might stop people from clicking deeper into the feature. They brought it forward, and what they saw was repeat engagement, the healthy kind, alongside satisfaction and retention holding up over time. That's the difference between real engagement and vanity engagement.

Sometimes the right move is to deliberately slow users down. Chase introduces speed bumps in payment flows, including on Zelle, to protect customers from being tricked into sending money to the wrong person. Raw engagement would say to remove the friction. Customer trust says add it. Only a balanced decision framework, agreed on before the results come in, keeps a team honest about that trade-off.

Plan for failure before you run the test

That phrase, "before the results come in," is where Kevin spends a lot of his energy. Confirmation bias, he warns, creeps in precisely when a team never expected to lose. When the loss arrives unplanned, people start hunting for evidence to support what they wanted to do, questioning the metric or the methodology after the fact.

His antidote is to plan for failure up front. "One of the things we want to preach when we work with teams is to plan for failure," he said. Build the playbook for a loss before you run the test. Decide in advance what a loss would mean for your assumptions and what you'll do next. When losing is just the trigger for an already-planned next step, it stops feeling like a defeat, and teams stop gaming the outcome. He points to the iPod, which didn't truly become a winner until its third version, as a reminder that expecting V1 to win is rarely realistic.

The golden era, and the measurement it demands

All of this becomes more urgent in the AI era, which is where Kevin sees experimentation entering a golden age. The reasoning is direct. AI lets everyone ship faster, and soon lets people build and customize their own features. But speed without measurement is dangerous. "If you don't measure them right, your mistakes are going to compound," he said. More output multiplies the cost of being wrong.

AI also pushes teams into a non-deterministic world, where the same input doesn't always produce the same output, and the most important outcomes are qualitative. Ashley offered the example of Khan Academy, a GrowthBook customer whose AI tutor optimizes for cognitive engagement, whether a student is genuinely trying to learn rather than just extracting an answer. They lifted it by 6%, an outcome no automated eval could grade. Evals are strong at QA, at confirming you got the expected answer. They are weak at predicting how millions of real people, asking the same question nine million different ways, will actually behave and whether they leave satisfied or frustrated.

That gap is exactly where experimentation earns its place. As more software gets built faster, by more people, the teams that win won't be the ones shipping the most. They'll be the ones who can still tell which of those ships actually worked. "Everybody becomes a builder," Kevin said. "It's a fun time." It will be a fun time for the teams that remember to measure.

‍