Learneo on testing the opposite of every hypothesis

Guest: Rich Liebling, Senior Director of Engineering at Learneo. Host: Ashley Stirrup. Show: The Experimentation Edge, presented by GrowthBook.

Writing a hypothesis is the easy part. The hard part is accepting that the version you are most confident about might be exactly backwards.

Rich Liebling has the receipts. He is senior director of engineering at Learneo, the company behind Course Hero, and six years into that role. Before it, he was the first engineering hire at Shop It To Me, an email subscription service that sent people apparel matching their brand, size, and store preferences. He joined as VP of engineering when the founder had built the product in business school and grown it to roughly 50,000 subscribers. The goal for Rich's first year was one million. They got there in nine months.

On The Experimentation Edge, Ashley Stirrup asked Rich how that happened, what practice came out of it, and what changes when experimentation is not an organization's core value.

🎧 Listen to the full episode →

The test that lost, and its inverse that won

The story Rich returns to is small enough to sound trivial and consequential enough that it changed how his team worked.

Someone had a hypothesis about a particular page. Too much text. Too busy. Shorten it, simplify it, conversion goes up. They built it and ran it.

"The version with less text did worse," Rich said. "Not just didn't improve things, but it actually did worse."

At most companies that is where the test ends. The hypothesis was wrong, the change gets reverted, the team moves on to the next idea in the backlog. Instead, somebody proposed the inverse: what if the page needs more text, more explanation? They ran that. It won.

From then on it was standing practice. "Someone has a hypothesis and you wanna test it, great, but also test the opposite of that hypothesis," Rich said.

Two things came out of running it that way. The first surprised everyone: the opposite direction won often. Not occasionally, often. The second is the more useful one, because it holds even when the inverse loses. A result in either direction proves the variable has an effect on the metric you care about. "It tells you that this thing, this knob matters," Rich said. "It affects what you care about. And so you should use that as an indication to figure out how you can turn it in the right direction."

The failure mode this protects against is quiet and expensive: a team spends a quarter on a variable that does not move anything, because a single flat result was never distinguished from a signal pointing the wrong way.

New hires found the practice strange at first. They would bring an idea to a brainstorm, the team would agree to test it, and then add that they would also be testing its opposite. "Everybody felt a little weird about that," Rich said, "but it just became like, oh, yeah, okay, that makes sense. Let's do that."

Small tests are what make the opposite meaningful

There is a constraint on this that Rich is direct about, and it is the reason the practice does not scale to every test in a program.

"When you do very small focused tests, you can say what the opposite would be," he said. "But if you're completely redesigning a page and it does worse, you learn, okay, that version of the page doesn't work, but it's not clear what you do to try to make things better."

A wholesale redesign bundles a dozen good ideas with a few bad ones and returns a single number. There is no coherent inverse to run, and no way to separate what helped from what hurt. Ashley described a webinar earlier that day with the Philadelphia Inquirer, where a sign-up flow changed so much between A and B that the data scientist on the project could barely interpret the result. Rich's read: "They probably had a bunch of good ideas, but then some bad ideas mixed in, and where does it balance out and how do you tell the good from the bad? Very hard to say."

The rule he lands on is short. "The more focused your tests are, I think the more clear your learnings."

That does not mean only running small tests. Shop It To Me ran a portfolio: many small, low-investment tests changing a button's text or color, medium tests altering behavior and requiring back-end work, and larger bets on new product ideas. What the portfolio required was conviction. When a bigger bet did not win outright, the team's default was to keep iterating rather than declare the direction dead. "We wanna get onto a bigger hill," Rich said. "We recognize that it's gonna take some iteration to decide whether we actually made it to a bigger hill or not. And not just one guess and then say, oh no, that didn't work, we're not trying that again."

The same discipline applies to winners, and it is the one most teams skip. "When you find a winning variation, you do an experiment and you get a winner, is not to just move on to something else, but to say, hey, this has promise. Can we make it even better?" Rich said. "There's no reason to believe you're now at the optimal point."

A goal of 300 tests, not a goal of one million subscribers

The reason Shop It To Me had enough tests to learn any of this is a piece of goal-setting worth stealing.

One million subscribers is not something an individual contributor can act on. So the team decomposed it. The strategy was virality through a refer-a-friend program. The execution goal, the number everyone in the company could see and influence, was 300 A/B tests by the end of the year. Run enough tests, get enough winners, and the loop compounds.

"That's a good example of how at a strategic level you can translate a high-level business goal down to something that's purely execution-oriented," Rich said. "Everybody that we hired knew our goal and knew how we were gonna get there, and there was something that they had control over."

It worked. Nine months to a million, with a peak of 30,000 new subscribers in a single day.

The compounding did not stop at the subscriber count. As traffic grew, tests resolved faster, which meant the team learned faster than the copycat sites that started appearing. "Our big advantage was we have a headstart in our user base and our traffic, and that lets us move quicker," Rich said. "As you get bigger, the A/B tests can be resolved more quickly, and that gives you an ongoing advantage over someone starting smaller."

The second merge request

Engineering is not usually where an experimentation program finds its champion. Engineers want to ship the thing and move to the next problem. Ashley named that directly, and Rich agreed, which makes what happened at Shop It To Me more interesting.

A/B testing was a stated core value, discussed in interviews and used as a recruiting argument. Candidates did not have to care about fashion to matter here. "You're a part of driving the product and your ideas will matter," was the pitch.

Then the company backed it with a mechanic. A new engineer's first merge request was housekeeping, updating the onboarding documentation they had just used. Their second merge request was very often their own A/B test idea. Within two weeks, a new hire had shipped an experiment and read its result.

The brainstorms were open to everyone. That mattered because the engineering team was, in Rich's words, very disjoint from a user base of young women deeply into fashion. Nobody got dismissed with "what do you know about fashionistas?" Ideas were evaluated on the merits, and engineers' ideas won frequently enough to keep the whole company invested. Everyone watched the metrics as the growth curve steepened. "It was like just constant excitement," Rich said.

What it looks like when testing is not the core value

Course Hero is the contrast case, and Rich is candid that the practice looks different there.

Course Hero is a platform for sharing study materials, mostly for college students. Some content is free, some is gated, and there are two ways past the gate: pay for a subscription, or upload content to earn access. That created two teams with opposing incentives, one chasing uploads and one chasing subscribers, each able to win at the other's expense. The fix was a model that estimates a customer's lifetime value early, roughly at sign-up, so both teams optimize the same number. The dynamic stays messy, since paying subscribers also upload and their content tends to be better quality.

The larger difference is friction. There was no standard analysis process, so a data analyst would write ad hoc SQL and sometimes spend three days interpreting a single test. That cost alone suppressed how many tests got run. "Just that level of friction tended to discourage a lot of AB tests," Rich said. His conclusion covers both halves of the pipeline: lowering the friction to run a test matters, and lowering the friction to analyze one matters just as much.

Two other constraints shape the program. Concurrent tests on the same surface interact, multiplying code paths and edge cases, particularly when tests change behavior rather than copy. And traffic is seasonal, tied to the school year, with demand spiking around exams. The team's answer to the second is a multi-armed bandit framework that continuously shifts traffic toward the leading variation to minimize regret, balancing exploration against exploitation. It reduces the cost of a losing variation and makes it practical to let a test run across different parts of the calendar.

Steps to take from Learneo's playbook

• Run the inverse of your hypothesis alongside the hypothesis. Budget for it in your test plan rather than treating it as a consolation experiment after a loss.
• Keep tests focused enough that the opposite is a coherent thing to build. If you cannot state the inverse in one sentence, the test is too broad to teach you much.
• Translate the growth goal into a count your team controls. A subscriber target is a scoreboard. A test target is a job.
• Put a test in a new engineer's second merge request. Onboarding is where a testing culture is either established or lost.
• Measure your analysis friction, not just your shipping friction. If interpreting one test takes three days of ad hoc SQL, that number is your real ceiling on testing velocity.

🎧 Listen to the full episode →