Even a loss is a win: Charlie Health's approach to failed experiments

Guest: Joe Yevoli, Director of Growth, Charlie Health. Host: Ashley Stirrup, CMO, GrowthBook. Show: The Experimentation Edge.

Running an experiment is the easy part. The hard part is knowing what to do when the result comes back, especially when the result is a win.

Joe Yevoli learned that again this year. As Director of Growth at Charlie Health, a virtual intensive outpatient program that sits between once-a-week therapy and hospitalization, he owns performance marketing, lifecycle marketing, and on-site and in-product experimentation. Every test on the landing pages and the intake form runs through his team. When he joined Ashley Stirrup on The Experimentation Edge, he brought three stories that share a single thread: the experiments that teach you the most are the ones that refuse to behave the way you expected.

🎧 Listen to the full episode →

The win that created a bottleneck

Charlie Health's intake form used to open with a page that, in Yevoli's words, "essentially functioned as almost a second landing page." A prospective client would click Get Started, land on the form, and be greeted with more information rather than the first question.

"Most people, when they click Get Started, is not expecting to get further information," he said. So the team removed the page and ran the test for about three months. The results were strong. More people entered the funnel, and the variant looked like an obvious ship.

Then the team looked further down. "What we also saw was that we were unnecessarily inflating the top of the funnel," Yevoli said. That first page had been quietly doing a job: it gave people information they needed before they committed. Without it, more people started the form, but a larger share of them were not the right fit, and conversion downstream dipped.

"Even though the test ended up positive, what we saw was, okay, there is another bottleneck being created here. We are letting too many people through."

The response was not to reverse the change. The removed page was still the wrong place for that information. Instead, the win produced the next hypothesis: an FAQ-style chatbot on the landing pages that surfaces the same information to anyone who wants it, before they ever enter the funnel. Inform people early, remove friction once they are inside.

Yevoli is candid that the dip surprised him even though it should not have. "You increase people in the top of the funnel, by definition, conversion is gonna get worse even if you are getting more people at the bottom of the funnel," he said. "It's almost like a law of thermodynamics." He had seen it play out many times before. "I had totally forgot that this was gonna happen. And another reminder of you don't know anything."

That phrase, "I don't know anything," comes up repeatedly in the conversation. It is not false modesty. It is the operating posture that makes the rest of his approach work.

Build it for everyone, build it for nobody

The second story comes from Yevoli's time at Teachers Pay Teachers, the marketplace where teachers sell supplemental resources to other teachers. During the pandemic, the company launched a digital-first product called Easel. The instinct, Yevoli admits, was the one most product teams have: "Let's see if we can build something for everyone, because we wanna make sure that the most amount of people use it."

So Easel shipped as a PDF overlay that could, in theory, work with every resource on the platform. Teachers tried it in large numbers. Then they stopped. In the middle of a pandemic, with instruction happening on screens, a digital teaching tool was not retaining.

The interviews explained why. "We built something way too wide, and teachers were almost putting their own meaning into what they were experiencing," Yevoli said. One teacher would call it a PDF overlay tool. Another would call it an iPad teaching tool. Each invented a definition, and none of them matched what the company was trying to do.

What teachers actually needed at that moment was narrower and more specific: a way to assign homework, confirm students did it, and track progress. So the team ran a lightweight test that repositioned Easel as exactly that, with a minimum set of features built to support the new promise.

"I think we cut out probably about 80% of the teacher market for that one feature," Yevoli said. "Retention probably quadrupled."

He credits the reframe to Crossing the Chasm and its beachhead strategy: pick a narrow niche, make the product as good as possible for that group, then use those early users as the reference point for where to expand next. The team never got to the expansion phase at Teachers Pay Teachers, but Yevoli knows what he would have looked for. Were teachers using the homework tool in class? For tests? Those signals would have shaped the next iteration and grown the market in the right direction rather than all directions at once.

The counterargument he heard internally is the one every growth leader hears. "We're cutting out so much of the market." Yevoli understands the instinct and still rejects it. "Going to everybody at first, you build something for everyone, you actually build it for nobody, and you burn that first impression that is so important."

The same logic now governs how his team reads results at Charlie Health. Some segments refer clients at high frequency; others refer rarely. "The solves for both of those segments are entirely different," he said. "If we approached the whole thing in the same way, we would get a very muddled view of what is actually working." An average hides the experiment. Segments reveal it.

Assume it failed, then launch

The third practice is the one Stirrup singled out during the interview, and it is the most transferable. For large, risky experiments, ones that touch a big share of the user experience and could do real damage if they go wrong, Charlie Health runs a premortem.

The mechanics are simple. Before launch, everyone involved gathers in a room. Then someone says: "Okay, we launched this experiment. The experiment wrapped. It was a massive failure. What went wrong?"

"And then the room goes silent," Yevoli said. "Everybody writes down everything that they can think of that could possibly go wrong on the day of launch or at the end of the experiment. And then we just go around the room and everybody reads what they wrote down."

The exercise does two things. The first is cultural. Everyone has sat in a meeting thinking a plan has a flaw and said nothing because the team seemed confident and the project was too far along. "And then sure enough, the thing happens and you were like, 'Oh my God, that was exactly what I was worried about.'" The premortem gives that person a floor to stand on. It matters most when the idea belongs to someone senior. "When the experiment is proposed by, say, the CPO or the CEO, who's the associate in the room that's gonna be like, 'I don't think so'?"

The second is practical. Naming the failure modes in advance gives the team time to build mitigations. "It's always led to some implementation of a safeguard that previously wouldn't have been there that has helped us avoid catastrophe," Yevoli said. Sometimes the brainstorm goes further and produces an idea worth testing on its own, a variant C or D added to the original design.

He is careful not to oversell it. "That doesn't mean that every test is a winner. Quite frequently they are not. But it does make sure that we're not missing stuff."

Designing for the loss before it happens

Underneath all three stories is a discipline Yevoli applies to every experiment proposal that crosses his desk. He starts with two questions. What is the quantitative data behind the hypothesis? And what is the qualitative data? "You may see that a certain subset of people are behaving a certain way with data, but you'd be surprised about what you hear once you actually talk to them."

If both hold up, the next question is the full-funnel user experience for the feature, and whether every step of it is being measured toward the outcome the business actually needs. "It's one thing if you build a feature that increases opportunities, but it doesn't matter if it's not producing the end result, which is more sales."

Only then does he talk about the likelihood of losing. He rarely launches anything he does not feel good about, and he still tells new teammates that most of what they ship will not win. If the test was launched for the right reasons, tracked to the right end goal, and measured at every step, a loss is not wasted. "We're gonna look at the data, and we're gonna say, 'Huh, that's interesting. What's going on here? Let's talk to some users here. Let's figure out something different that we can do.' And eventually, more often than not, you get to that win that way."

That is the sense in which even a loss is a win. Not because losing feels good, but because an experiment designed to teach you something will do so regardless of which direction the number moves. The form page test won and taught the team about a downstream bottleneck. Easel lost and taught the team what teachers actually needed. The premortem exists so the lesson arrives before the launch instead of after it.

Yevoli's closing thought on AI applies to all of it. Tools that let one person act as engineer, designer, and PM at once make it possible to move fast and to do damage fast. His antidote is the same one he brings to experimentation: write down what you think before you ask, think through the answer before you report it, and keep asking what you are not seeing. "What AI is leading to is this habit of thinking you have the answer when you don't fully understand it, and that is super dangerous."

The teams that compound learning are not the ones with the highest win rate. They are the ones who treat every result, including the wins, as the start of the next question.

Listen to the full conversation with Joe Yevoli on The Experimentation Edge, and learn more about running experiments that teach you something either way at growthbook.io.

🎧 Listen to the full episode →