A Culture of Experimentation Is a Relationship Problem, Not a Statistics Problem

Most advice on building a culture of experimentation focuses on evidence. Run more tests, raise your standards, get the statistics right, and the organization will come along.

Shiva Manjunath does not think that is where programs actually die.

Shiva is a lead strategist at Motive and the host of the From A to B podcast, and he has spent much of his career as a solo CRO, the one person in the building responsible for making testing happen. On a recent episode of The Experimentation Edge, almost none of his practical advice was about test design. It was about which teams you talk to before you need anything from them. Below: his bar for what a culture of experimentation actually is, two tests that changed how he reads customer research, and why the fastest way to unblock your A/B testing roadmap runs through the sales team.

Shiva Manjunath Sets a Very High Bar for a Culture of Experimentation

He is not grading on a curve. To him, a culture of experimentation means every decision runs through some form of evidence hierarchy, with experimentation as the default rather than the exception. He puts it in the same tier as the greats of any sport: Jordan, Brady, Gretzky. Very few companies live up to it.

"A lot of people might say they have a culture of experimentation, but when it comes to brass tacks, they don’t actually follow that through all the time," he says.

He is careful about what that standard does not mean. Not having the culture is no reason not to test. If you are routing as many decisions as you can through evidence, you are ahead of most teams.

The deeper problem is that the market around the role is still immature. In Shiva’s ideal world, CRO would not exist as a job at all, the same way you would not need peacekeeping bodies in a world that already had peace. Designers, marketers, and product managers would each carry the mindset themselves. The role exists because that mindset is not distributed, and product is where he sees the biggest gap. If product teams genuinely carried it, he says, that would be "the biggest unlock in a company."

Nobody agrees on what the job is either. Ask fifty companies what a CRO does, and you get sixty job descriptions. Product management and engineering are broadly standardized. Conversion rate optimization is not, so most CROs spend part of their first year defining their own role in public.

What Customers Say They Want Is Rarely What They Want

The pricing test that tanked

Shiva’s team ran on-page surveys and asked customers directly what was missing. The answer came back unanimous: show me prices. So they showed prices. Conversion rate tanked.

His read is that the word was pricing and the need was value. Customers can't evaluate the offer, and the closest available word for that is price. When the team rewrote the page around benefits, ROI, and time saved rather than a number, conversion went up.

The research was not wrong. It was literal. Customers reliably name the symptom, and diagnosing what sits underneath it is the researcher’s job. "Always validate through some means," he says.

Edible Arrangements and gifts under $50

When Shiva was at Edible Arrangements, the recurring hesitation was price. People love the product and love the reaction it gets, but "it’s just fruit," and customers were not always willing to pay the higher end prices, especially once delivery fees were added on top.

The response was not a blanket discount. It was a test. They built a dedicated gifts under $50 category for customers who want to send something but don't want to spend a hundred dollars doing it. Clicks into that category and purchases from it were, in his words, astronomical.

It did not stop at the website. The category became printed banners in the stores, and as far as he knows, it is still in use today. A small A/B test turned into a durable merchandising decision, which is also how a program earns its next round of trust. "That was a good idea. What else can we feed in that system?"

Fear, uncertainty, and doubt as a test idea engine

Both tests came from the same habit: asking what fear, uncertainty or doubt sits between the customer and the purchase. Shiva sources it explicitly by asking and implicitly by watching behavior. A spike in traffic to the returns page is a FUD signal even if no customer says the word out loud.

His own example is a custom-fit blind he bought for an awkward window. The anxiety was not price or quality, it was whether a made-to-measure product would fit a window he had measured himself. The company had one thorough install video on YouTube with roughly 100,000 views. He watched it five times before buying.

An install video with a hundred thousand views is a hundred thousand people looking for reassurance before they will spend the money. If you run that site, that is not a support asset. It should be on your highest-signal list of test ideas.

Building a Culture of Experimentation Is a Relationship Problem

This is where the conversation kept landing. The constraint on most programs is not statistical; it is political, and the work of loosening it looks nothing like test design.

The button color trap

If your first conversations with a product team are about green buttons and social proof while they are dealing with real disruption in their business, you lose them before you have run a single test. Shiva’s illustration is a CRO at PlayStation walking into a product org preoccupied with an enormous upcoming game launch, hardware costs and the coming end of disc-based games. Open with a button color test, and the reaction is obvious: "there are so many bigger problems we need to address."

The cost is not just that meeting. Reputations are sticky. "Because you said that button color thing, they’ll keep on thinking of you as just that button color person," he says. He tells every new CRO he mentors to establish authority early, before anyone forms a smaller picture of what you are for.

Build bridges before you need them

The antidote is building relationships before you need anything from them. His metaphor is Chuck E. Cheese tokens. Become the sales team’s best advocate, do the work they ask for, collect tokens, and hold on to them. Then, when a product manager who blocks something that matters, the sales director asking for the same thing moves it.

Sales and support are the two most underused sources of test ideas

Everyone mines analytics. Everyone runs customer interviews. Almost nobody systematically harvests what sales hears on calls and what support hears from frustrated customers.

That is free qualitative data, and it is fresher than anything in your warehouse. Sales knows which objections are trending this month. Support knows which problem generates the most complaints, which is often a fixable experience problem sitting in plain sight.

The trick is to attach a number to it. A support fix is thankless until someone measures it, so treat complaint volume or repeat contacts as a metric the test is accountable to. Cut the calls about the same failure by a few percent, and that team becomes your loudest advocate.

Both bridges do double duty: better test ideas, and stakeholders who defend the program when it is under pressure. As Shiva puts it, "your program will take bullets."

Key Takeaways

• Take research literally at your peril. Customers asked for pricing. What they could not articulate was value.
• Mine for FUD explicitly and implicitly. Returns page traffic and a heavily watched install video are research too.
• Lead with their problem, not your tactic. Open on button colors with a team facing bigger threats, and you become the button color person.
• Bank favors before you need them. Sales and support give you better test ideas, and stakeholders who defend the program.

You Do Not Need an Executive Sponsor, You Need a Data-Driven Organization

Not every company will set a CRO up to succeed, and an executive who visibly lives the culture makes everything easier. But Shiva pushes back on treating that as a prerequisite. Without one, it is harder, slower, and more frustrating. It is not impossible. And if the role exists at all, someone senior enough to approve it already believed in the work.

What matters more is whether the organization already makes decisions with data. Experimentation is data. If people do not trust data, they will not trust your results either. So screen for it when you are looking at your next role: listen for whether data comes up unprompted in how people describe their work.

David Massey is a good example of what happens when the foundation is there. When he joined UPS, the company was just starting to explore experimentation, but it was already data-driven. He sold the organization on the potential, and his first test generated $17 million in new revenue. That got everyone’s attention. Today the team has generated over $500 million, and testing new features is simply how the company works.

🎧 Listen to the full episode →