From Cross-Sell to Shopper Intent: What Experimentation Taught Us

Search for banana on Walmart and look at the ads. There are almost none. Hardly any advertiser sells bananas, or fresh food of any kind, so the pages that shoppers visit most for groceries are some of the emptiest for sponsored products.

For a retail media team, that looks like an open field. Arv Ramchandani's team at Walmart Connect set out to increase ad coverage on exactly those pages, and the first idea felt like a sure thing: cross-sell. If someone searches for banana, show them something that goes with banana. Chocolate. The makings of a smoothie.

"We were pretty sure that, okay, this is bound to work," Arv said.

The results prompted the team to rethink how closely the recommendations aligned with shopper intent. What the team learned offers a useful lesson in running experiments in an advertising business.

The thesis: an experiment is most valuable when it is designed to teach you something, and the lesson is usually about the customer, not the feature.

Three scoreboards on one page

Before the banana story makes sense, it helps to understand where Arv sits. Walmart Connect is Walmart's retail media business. Arv works on experimentation and analytics for sponsored products across pages that other teams own.

That means every test he runs lives on someone else's scoreboard. The search team is trying to drive customers to purchase more. The conversational AI team, which builds the shopping agents, weights trust and relevance. The ads side has advertiser returns and revenue to think about.

"Every team would have their own primary metrics," Arv said. "We have to work around that."

In practice, working around it has three parts. The ads team coordinates with page owners so experiments run in sequence and do not conflict. Both teams agree up front on the objective. And they negotiate a degradation threshold: how much of the page owner's metric can drop, and how much ad revenue the business earns in return. The equation is defined per page, and the ads team's job is to stay inside it.

Revenue is only one guardrail. Relevance of the ads and latency are the long-term metrics, the ones that do not show up in a two-week window. "We are building a product not for tomorrow, but maybe for the next few years," Arv said.

To put a number on the long-term cost, the team runs holdouts that last months. A very small slice of traffic sees no ads at all, and the team compares the customer experience for that slice against everyone else. It is a direct answer to a question most ad businesses would rather leave vague: how much trust are we spending?

The lesson from this beat: when you run experiments on shared real estate, agree on what you are allowed to break before you start.

What the first test revealed

Back to the banana. The cross-sell experiment went live with the confidence of a team that had already decided the answer.

While customers engaged with complementary recommendations, that engagement did not always translate into purchases. This prompted the team to rethink how closely the recommendations aligned with shoppers' original intent.

He put it plainly. "They are here to buy something. We are distracting them."

While the experience encouraged exploration, that engagement did not consistently translate into the conversion outcomes we had hoped for. This highlighted an opportunity to better align advertising experiences with shopper intent.

At this point a lot of teams would have filed the idea under "tried it, didn't work." Arv's team did the opposite. It was still a good idea, and shoppers clearly liked to explore. The question was why exploring did not end in a purchase.

The lesson: an unexpected result is data about the customer, and it deserves as much analysis as a win.

Building around the mission

The team went back to first principles and found the problem in the diversity of what they were showing.

"If you search for bacon and I'm showing you a frying pan, it makes sense," Arv said. "But not everyone is trying to come and buy a frying pan."

A complementary product is relevant in the abstract. For a specific shopper on a specific visit, it is often a detour. The fix was to stop guessing at complements and start from what the customer was trying to accomplish.

Customers do not type their mission into a search box. Someone searching for cocoa powder may be baking for the holidays, but they do not write "I'm baking." So the team identified those missions, built themes around them, and cut down the variety of items shown inside each theme. Search for taco and you see a Taco Tuesday night theme, the handful of things you actually need, instead of a scattered set of loosely related products.

That changed the job the ad was doing. It stopped competing for the shopper's attention and started shortening their trip. People could buy what they needed faster instead of searching for each item separately.

The rebuilt version delivered better CTR and better conversion, and the team submitted a white paper on the work to the AAAI 2027 Innovative Applications of AI conference for consideration.

The lesson: fewer, better-grouped options can beat more options, as long as the grouping follows what the customer came to do.

Designing for learnings, not launches

For Arv, the banana test is not interesting because it eventually shipped. It is interesting because of what it taught.

"It's not only about finding if we will launch or not, but what's the learning we will get?" he said. The insights that surprise people, the ones that become a stepping stone for the next product, are what motivate him and the product managers he works with.

That starts with the question at the top of the test plan. Arv's first question, to himself and to anyone who brings him an experiment, is not how to launch. "We should not think about how to launch this product. We should think about how to make this product more usable."

Change the question and the hypothesis follows. It stops being about ad revenue and starts being about what you are trying to improve, such as diversity or conversion, even when that improvement does not show up as ad revenue at the end.

The team backed that thinking with a written process that spans the whole life of a test. Before launch, a test plan. During the test, no p-hacking and no peeking at results too early. At the end, a defined set of checks. The payoff has been faster experiments and bugs caught early rather than discovered when the test is over. The team also trusts experimentation more, and has a shared definition of a successful test: one whose result ties back to its hypothesis.

Where does it go next? Self-serve. Arv's team already built self-serve tools that help analysts see results faster. The next step is to get that capability to people beyond the analyst, like PMs, the way AI is helping people outside engineering write code. An analyst still guides the work, but most of it is done by the person closest to the product. That is how a team runs many experiments at once without waiting in line.

The pattern

Four moves from the way Arv's team runs experiments on shared, revenue-bearing pages.

1. Negotiate the degradation threshold first. Agree with the page owner on what can drop and what the business gets in return, then stay inside it.
2. Track the guardrails that move slowly. Relevance and latency matter over years, and a months-long ad-free holdout is the honest way to measure what ads cost in trust.
3. Treat an unexpected result as research. When results differ from expectations, look for the customer behavior behind them; for Walmart Connect that meant building around the shopper's mission instead of the ad.
4. Ask how to make it usable, not how to launch it. Then write the process (test plan, no early peeking, end-of-test checks) that lets the whole team trust the answer, and open it up so people beyond the analyst can run tests too.

Full conversation with Arv Ramchandani, Senior Product Manager - Experimentation & Analytics at Walmart Connect, on The Experimentation Edge.

🎧 Listen to the full episode →