How Preply runs close to 1,000 experiments a year

At Preply, there is no such thing as a change that ships without a test. The learning marketplace connects millions of learners with more than 150,000 bookable tutors, and every product decision across its 25 to 30 product teams goes through an experiment or, at minimum, a staged A/B rollout. Sebastien Polis, Senior Experimentation Manager at Preply, runs the team that keeps that machine fast and trustworthy.

On The Experimentation Edge, Sebastien walked Ashley Stirrup through how Preply built a culture where close to 1,000 experiments a year is normal, why losing is expected, and what a 2-sided marketplace does to the statistics.

Experimentation is baked into how goals are set

Sebastien is quick to credit Preply's founders. Experimentation was part of the company from the very start, so when he joined about 2 and a half years ago, the culture and the tooling were already strong. His job is to keep pushing those capabilities forward while protecting what was built.

Two levers do most of the work. The first is tooling: experimentation should feel like a natural extension of every developer's toolkit, with as little friction as possible. The second is accountability. Every product team gets quarterly objectives, and those objectives can only be attained by launching experiments and claiming results from them. A claim only counts if the experiment uses an approved target metric.

"We're very much controlling the culture and controlling the rigor from how we are setting our OKRs," Sebastien explained.

Losing is the norm, and iteration is the habit

A high bar raises an obvious question: what happens to the experiments that fail? At Preply, most of them do. Sebastien estimates that around 80% of experiments never get scaled to production.

That is by design. "If you're building an experimentation program where you are embarrassed of failure, then your program's going to end up lying to you," he said. People will find ways to claim spurious wins.

Instead, Preply teams iterate. About 30 to 40% of the experiments run in a year are iterations of an earlier idea, so names ending in v1, v2 and v3 are common, sometimes all the way to v5. Teams refine the hypothesis until they either have strong evidence or decide the idea has no legs. "We accept losses, but we just don't give up," Sebastien said. "We learn from them and we move on."

The payoff is measurable. Preply runs a global control group to measure uplift across all experiments, and in a good year the overall growth on key metrics lands in the double digits. Then there are the avoided losses: the spaghetti code and product bloat that never shipped because the test said no.

When a control group goes negative

Sebastien's most useful lessons this year came from control groups. Within a month of launching one, Preply saw sharply negative results across business metrics, even though the team had only scaled a handful of winners. Senior leadership noticed, and his team ran a long, exhaustive deep dive.

The cause was not a broken coin flip. It was bad luck in allocation: by random chance, one side received a disproportionate share of weaker users. The lesson for any high-stakes test is to work hard on balance, whether through seeding, stratified sampling or something similar, and to have diagnostic tools ready before you need them.

The same investigation surfaced a second lesson. A few experiments that changed simple wording in the tutor-matching flow, such as replacing "budget" with "price," had shown a lift in new subscribers. After scaling, the lift was nowhere to be found. When Preply unscaled and reran the test, the result was flat.

Ashley pointed to GrowthBook's webinar with Ronny Kohavi on trustworthy experiments, and Sebastien named the principle: Twyman's law. If a result looks too good to be true, it probably is. "If you run something simple, you shouldn't expect big, big business changes," he said. Rerunning a surprising win takes time, but it saves a lot of headaches later.

A public channel where every test is explained

Rigor at Preply is not only a process. It is also visible. Every product team posts its launches and results in a shared Slack channel, explaining what they are testing, why they expect it to work and, when a test is killed, what they learned and how the next iteration will change.

The exec team reads that channel and asks questions. Sebastien recalled an executive messaging him about a small test with a low sample size that he had not even known was running. His reaction was delight. "This is fantastic," he remembered thinking, "that they care enough about going through the detail."

New PMs and data scientists join squads that already know the drill. They must show supporting evidence for a hypothesis, choose a well-reasoned target metric with supporting and guardrail metrics, and follow clear scaling rules: significance on the target metric and no negative impact on any guardrail, which double as the business North Stars.

From learnings to a learning system

Capturing learnings already happens at the squad and tribe level each quarter. The next step is a learning system: systematically recording each learning with metrics and a confidence level, and raising its status when new experiments confirm it.

That curated data set is what makes AI useful. "In today's world, context is key," Sebastien said. One of his pipe dreams is an experiment run end to end by AI, from ideation to code to a scaling recommendation, with a human approving the final call. As he put it, "ultimately it's us building the product, not the AIs."

Why marketplaces randomize the tutor

Preply's 2-sided marketplace adds complexity that single-sided products never face. In the virtual classroom, a tutor teaches many students, so randomizing students would give the same tutor a different experience in each lesson. That breaks experimentation assumptions and creates a poor user experience.

Instead, Preply randomizes tutors and measures student outcomes, using a clustered analysis to account for the tutor as a confounding variable, much like healthcare studies randomize hospitals rather than patients. With 150,000-plus tutors against millions of learners, power is lower and metrics are noisier. The practical lesson is to ship bug-free code: a buggy test exposed to half of all tutors can bias everything downstream.

Sebastien summed it up simply. These rules "come from a lot of battle scars," and they are what let Preply keep learning at the pace of close to 1,000 experiments a year.

🎧 Listen to the full episode →