Diligent reveals the PM's most costly mistake in experimentation

Running an experiment is the easy part. Knowing what it's telling you, and what to do when it tells you nothing clearly, is the hard part. That's the thread running through Dan Layfield's fifteen years in product management, from a mid-2010s startup to one of the largest consumer apps in the world to the boardrooms of the Fortune 1000.

Layfield is Director of Product Management at Diligent, a roughly 2,000-person company that serves around 70% of the Fortune 1000 on bespoke corporate governance problems. Before Diligent, he was head of growth at Codecademy, where he took the company from about $10M to $50M ARR, and a backend-focused PM on Uber Eats' home feed ranking system. Across all three, the constant has been experimentation, and lately, a new collaborator in the work: AI.

The losing experiment that became a 35% win

The story product teams should sit with is the one that didn't work right away. At Codecademy, Layfield's team set out to rebuild the trial model, the single biggest growth lever for most consumer subscription products. A predecessor had already shipped a successful reverse trial, where new users get the paid product automatically and then have to decide whether to keep it. Layfield's team went further, removing the auto-enroll step and asking people to choose the trial and enter a credit card upfront.

The first rounds did not produce a clean win. They produced the thing most experiments produce when they lose: inconclusive results. Not statistically significant positives, not statistically significant negatives, just noise that tells you very little.

This is where most teams move on. Layfield calls moving on too early one of the biggest mistakes he made in his career. A PM's life is a roadmap of things to ship, and every extra week on one project is a week stolen from the others. The pull toward the next thing is constant.

His team stayed. Over roughly four months and three to four rounds of tests, they kept refining the paywall structure, the question of how and where free users come into contact with the paid product, and whether they hit it at a moment that relieves real pain. The tool that broke it open was almost embarrassingly simple: they put every screen of the user experience onto a giant Figma board and overlaid the metrics at each decision point. With the entire funnel laid out, they could see how users flowed through the product and exactly where to move the gates. The payoff was a 35% increase in conversion, a massive result for a business where monetization compounds.

The lesson is not "never give up." It's that a losing test is often a map, not a verdict, and the discipline is knowing when a problem is big enough, and your read of the data deep enough, to take another shot.

Two flavors of experimentation, and one feature factory to escape

Layfield draws a clean line between two kinds of experimentation. One is high-volume conversion rate optimization: run many small tests, expect a quarter of them to deliver small wins, and let volume do the work. The other is using experiments to de-risk something big and genuinely uncertain. The trial-model rebuild was the second kind, which is exactly why taking multiple shots made sense. Email subject-line tests are the first kind; no single one matters much by the time it trickles down to a purchase.

That distinction gets sharper in B2B, where many teams aren't really experimenting at all. Layfield's description of the failure mode is precise: the feature factory. You have a thousand clients, the top 5% request things in every QBR, and the PM's job becomes shipping that list. It keeps the biggest accounts happy in the short term and produces a disjointed product over the long term.

The alternative is a foundation of disciplined, top-down product management: a top-down OKR system where leadership sets meaningful business goals, each team owns a thoughtfully chosen North Star metric that ladders up to those goals, and feature pick metrics that ladder up to the North Star. Layfield's most useful observation is about how this breaks. When a planning process goes wrong, it's usually not that any single layer is broken. Leadership picks good goals. Teams pick reasonable metrics. The failure lives in the connections between the layers, when a team's North Star only loosely relates to what the business actually cares about. Every layer can look healthy while the whole structure quietly drifts.

B2B makes this harder than B2C because of the feedback loop. In consumer products, an unhappy user leaves immediately, which is painful but fast. In B2B, clients sit on long-term contracts, so there's a long delay between weak product usage and the dollar retention hit at renewal. If you aren't watching usage and adoption closely, the bill arrives long after you could have done anything about it.

Anchor the North Star to the natural use case

At Diligent, Layfield's team thinks about North Star metrics by product. The flagship is a board-of-directors collaboration suite used by, in his estimate, nearly every famous board director you could name. The core value isn't receiving a document; if all you do is print a board deck, Diligent is an expensive way to move a file. The value shows up when directors actually collaborate, comment, take edits, and prepare on the plane. So the team anchors the board product on director-side usage.

The catch is rhythm. A director on one board has a meeting roughly once a quarter, so they use the product the week before and a few days after, and then nothing. Push for more engagement than that and it starts to feel spammy, because unless a board deck is waiting, there's genuinely nothing to do. Layfield's rule is worth posting on a wall: retention and engagement should always ride whatever the natural use case is, not fight it. Guardrail metrics like session length and abandonment matter, but the headline number has to respect how the product is actually meant to be used, especially in a regulated space where Diligent deliberately tracks far less than a consumer app like Uber ever would.

AI as the data scientist in the room

Which brings us back to the title. Ask Layfield where AI has earned its place so far, and the answer is research and data synthesis. The old workflow was weeks of labor: find a hundred users, email all of them, spend two to three weeks scheduling calls, write up every interview, then synthesize by hand. Now he points AI at the raw material. Gong's MCP delivers reasonably good synthesis from sales calls in an hour or two, and for simple A/B test analysis, in his words, Claude is a pretty good data scientist.

The point isn't that AI replaces product judgment. It's that AI collapses the weeks of grunt work between having a question and having an answer. For experimentation teams, that's the part that compounds: the faster you can synthesize what users are telling you and what a test actually did, the more shots you get at the problems that matter, and the less tempting it is to move on too early from the one that's about to pay off.

Listen to the full episode of The Experimentation Edge with Ashley Stirrup. How does your team decide when a test is worth one more round? Share below.

‍