Engagement with safety: Headspace's guardrails for AI tests

Most consumer subscription businesses share a simple instinct: more engagement is better. More minutes watched, more books read, more sessions started. Mary Alfheim spent years inside that instinct, leading product analytics at Prime Video, working at HBO when HBO Max launched and leading data and analytics at Scribd.

At Headspace, where she is now VP Data and Analytics, the instinct breaks.

When more is not always better

Headspace is best known for meditation and mindfulness, with sleep content, wake-up content and stress management tools. It has also expanded into clinical care, coaching, therapy and psychiatry, sold through employers and health plans as well as directly to consumers.

That mix creates what Mary calls a dual mandate. "More is not always better," she told host Ashley Stirrup. "We want people to get better if they are coming to us with actual symptoms."

So the optimization target is health outcomes, not time in the app. Engagement still matters, but it sits alongside self-reported measures of anxiety and depression, tracked as snapshots over time and correlated with the content and care a member used.

That creates a measurement problem most product teams never face. Headspace does not have 200 million users filling a sample overnight, and annual resubscription or a change in self-reported health can take a long time to read. Mary's team works backwards with clinical researchers and actuarial scientists to establish which patterns of engagement are associated with improvement, then builds composite and surrogate metrics that product managers can use to self-serve day to day.

Engagement with safety

The clearest example is Ebb, Headspace's AI companion, an empathetic conversational chatbot built into the app that helps members reflect and process their emotions.

Experimentation has been fundamental to growing engagement with Ebb. But the operative phrase is engagement with safety. "People can jailbreak chatbots," Mary said. "They could be having moments of crisis that don't call for AI, that you absolutely need humans in the loop."

That is existential for a mental health company. Headspace is not going to keep optimizing to push people toward a chatbot when what they need is human care.

So the team organized every Ebb experiment around three layers:

• Inputs: what is being changed. A new prompt, a new safety rule, a change to the infrastructure or architecture the agent runs on.
• Outputs: the success measures. More sessions, longer sessions, more messages per session, user feedback such as NPS.
• Guardrails: the protections. Clinicians are in the loop on every experiment, doing clinical QA and monitoring for interactions that need more action or a different level of care.

A year ago Headspace was not running much experimentation at all, and for one-way doors like a huge redesign that can be fine. Ebb was different. The safety standards, the rules, the prompts and even the infrastructure were going to keep evolving, which made it exactly the place to test.

Stop rules make self-serve possible

Defining those layers forced another decision: where the stop rules sit. Mary sees this as the most portable idea in the whole program.

Her everyday example has nothing to do with AI. Say a team wants to drive engagement but not at the expense of cost. Write the cost guardrail down before the test. "If you get positive lift on engagement, and there's no harm to your cost guardrail, feel free to roll it out," she said. "If you have harm to your guardrail, then there can be clear rules put into place to roll it back or go to some other sort of review mechanism."

Once the rules exist, teams do not need the data team to make every call. The guardrails are what make self-serve experimentation safe.

Mary is equally clear that the data team should not write those definitions alone. Product teams know the intention and the roadmap and are deeply empathetic to their users. Inputs, outputs and guardrails should come out of a conversation about what the team is truly trying to drive, whether it can be measured, and how to start collecting the data if it can't. And as the product evolves, the metrics need to evolve with it.

Test as you go

Some of Mary's strongest views come from experiments that did not go to plan.

At a previous media subscription business, a product team spent about a year and a half building an entirely new way to navigate and find something to watch. The measurement plan was aligned ahead of time: metrics, read dates, what success looked like. The team even planned a long monitoring window for the dip in engagement that comes when existing users dislike change.

It still did not work the way anyone planned. By the time the test ran, the business objectives had shifted, so the success criteria no longer matched where the company wanted to go over the next 18 months. And when results disappointed, it was nearly impossible to tell which part of the experience was underperforming, because the "redesign" was really many changes packed together. The team had to go back and test the pieces one at a time.

"Faster is better, more discrete changes is better, and don't wait for some big grand reveal," she said. "Truly test as you go."

Know your treatment

A second lesson came from payments. Mary's team partnered on a test that split traffic between the incumbent payment processor and a challenger, with guardrails on chargebacks and fraud.

The challenger underperformed so badly that the team wanted to roll it back. Only after peeling back the layers did they see the cause. The incumbent had years of payment history on the customer base and was better at predicting fraud and risky transactions. The challenger was learning the audience from scratch. The traffic split looked random. The treatment was not.

The fix turned one experiment into a portfolio: stratified samples, separate experiments for new customers and people resubscribing, and repeated runs over time to see whether the gap held or closed.

"You have really got to know what your treatment is and question and question again if you're getting your randomization right," Mary said. It was engineering and product partners, asking whether the result held for a given cohort or tenure, who uncovered it. That is why she insists experimentation is never just the data team's job, a point Ashley extended to AI features, where nobody fully knows how the system works.

A third bucket for learning

Looking ahead, Mary expects Headspace to test faster and with bolder treatments, with AI on both ends of the loop: helping build new experiences and speeding up the interpretation of results. The team is at a couple of dozen tests a quarter, and she watches for the plateau in effect size that says the program is running the right amount of tests. By her read, Headspace is still on the ascent.

Scale is not the goal on its own. "Don't test just so you can say you had a win and put it in your quarterly business review," she said. "Make it worth it."

Many practitioners describe a portfolio of experiments: small loop tests always on, plus a few big bets. Mary adds a third bucket, a learning bucket, where tests never have to show up in a business review and the team does not even need expectations. That, she argues, is what feeds curiosity and the next generation of big loop tests.

It fits a broader pattern Ashley sees across the best programs: leaders who demonstrate humility, ask what can be learned, and do not expect a high win rate. The most successful companies run at 10 to 20% win rates. Stacking those wins still has a massive impact.

The pattern

Headspace's approach comes down to a few habits any team can borrow:

1. Decide what "good" means for the user, not just for engagement, and measure it even when it takes science to connect short term signals to long term outcomes.
2. Structure every experiment as inputs, outputs and guardrails, with stop rules written before launch.
3. Build those definitions with the product team, and let them evolve with the product.
4. Test early and in small pieces, and know exactly how your treatment works before you trust the split.
5. Keep a learning bucket, and celebrate the tests that don't move the needle.

Full conversation with Mary Alfheim, VP Data and Analytics at Headspace, on The Experimentation Edge.

🎧 Listen to the full episode →