Box uncovered these interesting surprises that reshaped how they run e-commerce experiments

When Danielle Oleen describes her job, she does not start with a feature roadmap. She starts with a number. As Director of E-commerce at Box, she is not measured by whether she shipped something on time. She is measured by revenue. That single distinction, she argues, changes everything about how a product team should work, and it is the thread running through one of the most practical conversations The Experimentation Edge has hosted.

Oleen has spent more than 15 years in e-commerce. She came up in the B2C world at companies like Wayfair and Drizly, then spent nearly six years scaling online sales at Zoom through the chaos of the pandemic, watching a one-person operation balloon into a 100-person department almost overnight. Now she is at Box, helping reposition the company from a cloud storage business into an AI platform for managing unstructured content. Across all of it, one tool has been her constant: the A/B test.

Experimentation is not an e-commerce luxury

The most important idea Oleen offers is also the one most product teams resist: experimentation is not just for people who own a checkout flow. It belongs to everyone who builds.

Her reasoning is hard to argue with. Time is finite. Engineering capacity is finite. Every feature you choose to build is an opportunity cost for one you chose not to. So the question that matters is never simply "did we launch it." It is "did this actually move the business, and was it a net positive."

You do not need a shopping cart to ask that question. As Oleen points out, every product has a flow. Even an AI chatbot has one. A user asks something, gets a response, and either abandons the interaction or comes back. Someone trying to create a document has to find the right place, start the action, and complete it with the formatting they wanted. All of that is a funnel, and a funnel can be measured.

That measurement reveals things intuition never would. Oleen describes a recurring pattern: a feature that very few people discover, but that the small group who find it use at a high rate. The instinct is to call the feature a failure. The data says the opposite. The feature is valuable. The placement is wrong. Catching that early turns what would have become a customer complaint or a pile of technical debt into a quick fix.

The humility underneath this is what makes it powerful. Host Ashley Stirrup framed it as the hard truth of experimentation: only two or three out of ten ideas tend to land the way you expected. That is not a sign that the other ideas were bad. It usually means a piece was missing, the wording was off, or the timing was wrong. For a B2B product manager, that is not discouraging. It is an enormous opportunity to double their impact by building what customers actually want to use.

The simplification tightrope

If the first lesson is that you should test, the second is that testing will humble you, even when you are winning. Oleen's pricing page saga is the clearest example.

Box's pricing page was, in her words, robust. Lots of colors, lots of elements, eight self-service plans, and a tangle of security, compliance, and AI features competing for attention. Her hypothesis was straightforward: simplify the page, reduce the cognitive overload, and more customers will be able to choose a plan and buy. The team did a large overhaul, even setting rules to keep feature descriptions to one line with tooltips for anything longer. The result beat expectations. A genuine, measurable win.

And here is where most teams go wrong. A win feels like a direction. "This is the right hypothesis, let's do more of it." So the team kept simplifying. They tested removing the slash-out pricing, the crossed-out higher number next to the discounted one, assuming it was just more clutter. It failed. That crossed-out price was quietly reinforcing the 25% annual discount, and customers valued the cue more than anyone realized.

Undeterred, they tried once more, trimming each plan's feature list to show only what differentiated it from the plan below. That failed too. Box's plans are complex, and customers wanted the full detail to feel confident about what they were buying. Stripping it away did not reduce overload. It removed information people needed.

One win, two losses. Oleen calls it the tightrope. There is a diminishing return where simplification stops helping and starts hurting, and the only way to find that edge is to test until you cross it. Tellingly, the team's next move was not more simplification but an addition: a third tab to serve a new persona, on the theory that the page was now simple enough to absorb it. And of course, they are testing that too.

The wine effect

The third lesson is Oleen's favorite kind of result: the test that wins for a reason you never hypothesized.

In a prior role, the team faced a churn problem. So they created a cheaper plan called Basic Plus and showed it only to customers in the cancellation flow, never on the public pricing page. The hypothesis was logical. These people think we are too expensive, so offer them something cheaper and convert a full churn into a partial one.

The test won. But not because people downgraded. The Basic Plus plan had roughly 70% fewer features, and simply seeing it made customers' current plan look far more valuable by comparison. So instead of switching down, more of them stayed exactly where they were.

Oleen calls it the wine effect. On a wine list, you rarely order the cheapest bottle. You order the second-cheapest because the cheapest one makes the next option up look like a deal. The experiment succeeded, the hypothesis was wrong, and both things were true at once. That gap between what you predicted and what actually happened is not noise. It is the entire reason to run the test.

Building a culture that can handle losses

None of this works if a team is afraid to lose. Oleen is deliberate about that. She presents every win and every loss to leadership, with a biweekly impact report to the COO and a monthly one to the CEO. The wins build credibility and prove the team knows what it is doing. The losses, presented just as openly, create the psychological safety that a testing culture depends on. Every loss comes with what was learned and what the team will try next.

The payoff has been cultural. A year into her role at Box, Oleen now has executives who proactively say "let's make sure we test that" before shipping a new feature, and a growing line of colleagues asking how to run experiments of their own. She thinks about it as a pyramid: start with the widest-impact experiences like the pricing page and checkout, then work up into smaller cohorts, like desktop versus mobile or high-context international markets such as Japan. And she is already looking ahead to monetization tests and AI agents that scan competitors and help ideate new experiments.

The throughline is simple. You will be wrong more often than you are right; you cannot reason your way to the balance point, and your most valuable insights will arrive disguised as surprises. The teams that win are not the ones with the best guesses. They are the ones willing to find out.

‍