Transcript

The Experimentation Edge - Arv Ramchandani

Arv Ramchandani: We also try to prove our hypothesis, and prove if this is how much trust we are losing. We also run some long-term experiments which span over months, in which we will just hold off a very small traffic, in which we won't show any ad, and we will see how much increment in the customer experience we are seeing. These kinds of experiments tell us that, okay, we are hurting or maybe not hurting the customer experience that much, and we are within the bounds of what would be acceptable for our business.

Ashley Stirrup: Hello and welcome to today's episode. I'm excited to have Arv Ramchandani, product analyst at Walmart Connect. Welcome to the show, Arv.

Arv Ramchandani: Thanks for having me, Ashley. I am really excited to be here.

Ashley Stirrup: Yeah, excited to have you on. Maybe we could kick things off by having you just tell us a little bit about Walmart Connect.

Arv Ramchandani: Sure, yeah. I work as a product analyst in Walmart Connect, which is the advertising division of Walmart, or we call it the retail media network. A retail media network typically differs from traditional agencies or advertising because it has its own first-party data, and it also has a closed-loop measurement, in which the advertisers can trust and we are able to say that, okay, a customer is clicking on the ad and is able to purchase. So that gives advertisers more trust in the system, and that's why it's growing at a faster pace now. As a role, I work in the ad experience team, which essentially looks at the customer experience of the ads we are delivering to the customers.

Arv Ramchandani: The interesting thing about advertising, or this world where I work in, is it's a three-sided ecosystem. We look at the customers, we are trying to give them meaningful products. We are also trying to provide for advertisers, connecting them to the right customers and giving them the dollar value for them. And also we are building something which is sustainable for the business. A big part of my role would be to work with the product team and make sure we are achieving all these three objectives. And that would involve doing experimentation, deciding what's the hypothesis, measuring the impact, and maybe looking at the historical data and the tests and deciding how we can influence the strategy to build something which is more meaningful.

Ashley Stirrup: Yeah, boy, this is just an interesting topic, because it sounds like you're relatively new in this business for Walmart, but it has massive potential to be a huge business, and it's an incredibly data-driven business. Is that right?

Arv Ramchandani: That is absolutely true. We launched a few years back, and when I joined this team, we were like a four-member team, and we have grown to more than twenty members right now. So it has been growing at a very fast pace. And the core of it is data. We look at the experiments, we have to test everything. There's nothing which passes without an experiment, and everyone is very adapted to the system.

Ashley Stirrup: Got it. Yeah, super interesting. And so how does experimentation work there? Do you have a central team, or do you have data scientists in every group? How does that work?

Arv Ramchandani: For the advertising business, we have a central team, and we look across the whole Walmart advertising. It could be customer experience, which is my role. It could be advertiser experience. It could be CTV, which is the connected TVs, or maybe the off-site. We are trying to do some ads in the Walmart stores as well. So we look at everything as a whole, and that's the great part of it, because we have all the insights of all the experimentation going on in the ads business.

Ashley Stirrup: Yeah. Yeah, that's super interesting. And I can understand how you probably have to look at different experiments, and some might be a winner for you and might be a winner for your advertisers, but not necessarily the end user, and vice versa. And so you've kind of got to balance all of those, right?

Arv Ramchandani: That is true. And that's, I think, the core piece of this role, something we call trade-off. And something which is very tricky is we are a separate business of Walmart, but we still go and advertise on different pages, which are owned by different teams, and we're essentially renting out or taking over this space. So when we are testing, it's not a straightforward test. We do see some decline in the customer metrics. Sometimes we are optimizing for CTRs, which might decrease some conversions for advertisers. So it's always something we have to pick and choose. Okay, this is important for the business. This is important for the customer. And intuitively, we have to think what's long-term good for us as a business. It's not a yes or no we are looking for, but mostly what's meaningful for us.

Ashley Stirrup: Yeah, that makes total sense. So the last thing you were just talking about was balancing metrics for different types of users, right? For your customers, for you, and for your end customers, I guess.

Arv Ramchandani: Yeah. Yes, that's true. Because we interact with various teams across the site with different objectives. Like, I have been working on search, which is trying to drive customers to purchase more stuff, but I'm also working on the conversational AI, which is the shopping agents, in which the primary metrics could be more about trust and relevance. So every team would have their own primary metrics. We have to work around that, work around the strategies they are trying to run, and we try to match their strategies. We try to look at those metrics along with the advertiser metrics and also how much revenue we are generating for the business.

Ashley Stirrup: Yeah. Yeah, that's super interesting. And so do you actually jointly define experiments, and what the guardrails are, and what trade-offs you're looking for between metrics?

Arv Ramchandani: That is true. We are always in touch with the other owners of other pages, because they are also trying to run their tests. We want to run the test on their pages, so we have to figure out a way we can run them in a sequential manner in which they don't conflict with each other. And after that, we define what's the objective or the true goal for both of the teams. We have our goals defined. We work with them to define a threshold, or a degradation threshold, which they are able to accept, and how much ad revenue, because we still want to be positive for the business. If we are degrading some of their metrics, we still want to give some revenue to the business. So that's an equation we have defined for different pages, and we try to be within those thresholds.

Arv Ramchandani: And ultimately, there would also be different kinds of guardrails we'll look at. It's not only the revenue we want to be focusing on, but also other important metrics could be relevance of the ads, which is more long term, which might not show in the short term, and maybe the latency. So we try to focus as a product. Okay, we are building a product not for tomorrow, but maybe for the next few years.

Ashley Stirrup: Yeah. Yeah, that all makes a ton of sense. It's pretty interesting. We recently had the Philadelphia Inquirer on a webinar at GrowthBook, and they talked about how they used experimentation to build trust, because they had a very siloed advertising team. And originally that team was built for physical newspapers, right? And so obviously their world had changed a lot, and people really cared about the user experience. And so, thinking through the ad versus the content and all that, there were a lot of things to balance. And so they used the experiment definition as a way to come up with a shared scorecard. It really helped build trust. It sounds like you're doing something similar, because you're trying to really think about your partner teams and what their goals are as you're building yours.

Arv Ramchandani: That is true. And we also try to prove our hypothesis, and prove if this is how much trust we are losing. We also run some long-term experiments which span over months, in which we will just hold off a very small traffic, in which we won't show any ad, and we will see how much increment in the customer experience we are seeing. These kinds of experiments tell us that, okay, we are hurting or maybe not hurting the customer experience that much, and we are within the bounds of what would be acceptable for our business.

Ashley Stirrup: Yeah. Yeah, that's interesting. I would imagine there are times when ads are actually considered a value add, too.

Arv Ramchandani: We're still trying to make it a value add for the customers. We are trying to give them the relevant items, because ultimately, if they're purchasing, we will get the revenue at the end.

Ashley Stirrup: Right. Right. Right. Of course. Yeah. So do you have an example of an experiment you ran that had some kind of surprising results where you learned something?

Arv Ramchandani: Yeah. This is one which was an interesting problem, and it's an interesting problem across retail, especially in advertising. So we had a problem in which we were trying to increase the coverage of the ads. And what I mean is, show more ads on different pages, especially on pages which are more fresh foods. For example, banana. If you go and search banana, there are hardly any advertisers who are selling bananas or fresh items. So intuitively, what we thought, okay, the first step could be cross-selling. We could show them an item which a customer would find very similar to banana, or something related to their query.

Ashley Stirrup: Chocolate.

Arv Ramchandani: Chocolate. Chocolate to have with banana.

Ashley Stirrup: Chocolate. Yeah. Like a smoothie you're trying to make, right? There you go.

Arv Ramchandani: Yeah. And we were pretty sure that, okay, this is bound to work. This is something customers will use. Eventually, after running the test, we found some interesting results. We found that, okay, customers are actually clicking on and exploring the items, but they're not purchasing. So they are actually getting distracted by the items we are trying to show them. They are moving away from the journey of the shopping. They are here to buy something. We are distracting them, and they are roaming around the website, and they're ultimately abandoning the website. And also this reduces the returns for the advertisers, because they are clicking on the ads, but they are not purchasing. So it's reducing the returns. At this point, we did not want to think this is a bad product, because it's a great idea. It's a very standard idea.

Ashley Stirrup: It makes a ton of sense. Yeah.

Arv Ramchandani: It makes sense. Everyone is trying to explore, they love shopping. Then we did some analysis. We tried to figure out, okay, how can we improve this product? We had to go back to first-principles thinking. We have to think how it makes more sense for the customers. We did some analysis. We found out that there is an issue with the diversity of the products we are showing. If you search for bacon and I'm showing you a frying pan, it makes sense. It's a complementary product. But not everyone is trying to come and buy a frying pan. And you might be clicking on it, exploring, but you are ultimately abandoning.

Arv Ramchandani: So we just overhauled the process. We created something which was around the mission of the customer. Every time a customer comes, they don't write what they are trying to find. For example, if you're trying to search for cocoa powder, you might be interested in baking something for the holidays, but you are not writing, okay, I'm interested in baking. So we identified these missions of the customer, we created themes around them, and we reduced the diversity around that theme. For example, if you're searching for banana, you might be interested in breakfast items. So we can show you, okay, here are some breakfast items. Or maybe you're searching for taco, so we can show you Taco Tuesday night. And this improves the mission of your objective, and you are getting more valuable results out of it. You are able to purchase them faster instead of searching for all of these as separate items. Ultimately, this gave us better CTR, better conversions, and this was a successful product. We also published a small paper around it, because it was a very interesting idea. And yeah, it's a successful product for us.

Ashley Stirrup: Okay, terrific. Yeah, we'll be sure to share a link to that paper in the show notes. Yeah, that's such a great example, because it's so tempting: you run a test, it fails, okay, let's move on to the next thing. But no, it's important that you say, okay, the idea was great, there's something about the execution we need to evolve. And in that case, you'd never have guessed, we're actually going to hurt conversions with this. And they're not going to buy the original product, and they're not going to buy the other things we showed them. And so it sounds like you were able to flip that all around and improve conversions across the board.

Arv Ramchandani: That is true. And I think that's what motivates us as analysts, and also the product managers: these wow insights we try to provide from every experiment. It's not only about finding if we will launch or not, but what's the learning we will get. Can it be a stepping stone for our next product we are trying to build? And that's what I strive for, to provide such good insights, which will eventually define the strategy for the business.

Ashley Stirrup: Yes. Yes. Makes a ton of sense. And so let's say you're working with a new analyst, and they're working on a feature that they're all excited about. They are just sure it's going to be a winner, and maybe they're not thinking about, well, what do I do if this is a loser, and what would the next steps be? How would you help them think about designing an experiment to get the most learnings?

Arv Ramchandani: Yeah, I think great question. And this is where we try to preach or define some processes around it. I think the first question I always ask myself, and always ask someone else, is: we should not think about how to launch this product. We should think about how to make this product more usable, how to create a product which would be better for the customer. I think when this thinking comes in, the hypothesis automatically changes. It's not about the ad revenue anymore. It's about what we are trying to improve. Are we trying to improve, for example, the diversity in this case, or the conversions, which might not lead to ad revenue at the end? And this is the first thinking we try to incorporate. And to get this, we have created some processes around it, which start before the experiment, even when we're trying to build the test plan, and something during the experiment, which is avoiding p-hacking, looking at the results too early. And then when we're trying to end the test, what are the key things we should be looking at? And this has become a process. It has helped us to run faster experiments, catch bugs early in the process rather than finding bugs at the end of the experiment. And also, this has changed the thinking of the whole team. This has generated more trust in the team, to believe in experimentation, believe in the process. And ultimately, if the result ties back to the hypothesis, that's where we call it a successful experiment.

Ashley Stirrup: Yeah. Yeah, boy, that's so much great stuff in that answer. I love the fact that it's almost like you're saying, okay, the true North Star is a great customer experience, and so we can never lose sight of that. Yes, we're trying to grow revenue, but first let's make sure we're staying true to our North Star, and then how do we grow revenue?

Arv Ramchandani: That is true. Yeah.

Ashley Stirrup: And then adding in all the rigor, so that you end up having trustworthy experiments. That's such a great summary. Thank you. And so, in terms of what do you see as the future for experimentation for Walmart Connect? How do you see things evolving?

Arv Ramchandani: I think of the same question of how we evolved a decade back, or maybe a few years back. At that point, we created self-serve tools for experimentation, in which we are able to help an analyst see the results faster. Execution is still a challenge. You need to know your systems, your tools, your data well enough. But we will be able to do a faster test and see the results faster. I think the next step would be to evolve, and get this technique out to all the other people beyond the analyst, like PMs. The same thing AI is doing with software engineering, where it's able to help others to code. Similarly, in the experimentation world, we will move along. Everyone is able to test, everyone is able to trust the process and get meaningful insights, even without depending on an analyst. Of course, there is an analyst who is helping you or guiding you along, but most of the work is done by you. I think that's what I see the next steps could be, and it will help us to run many experiments at the same time.

Ashley Stirrup: Yeah, that just makes so much sense. I'm really passionate about that topic. I feel like obviously we're seeing a whole revolution in coding through AI, and I believe we have the same potential to do that for experimentation. It shows up a little differently, because there's just so much context. Like, do I have my data pipelines working correctly? Have I designed the experiment correctly? Have we run an experiment like this before? But I'm a big believer that we can enable more people to run experiments with an AI data scientist helping guide them along the way. So you can scale all that. So yeah. Well, thank you so much for joining the show today, Arv. You shared a lot of great insights, such fascinating use cases. I feel like we learned a lot today. Thank you so much.

Arv Ramchandani: Thank you actually for having me. I really enjoyed this.

Ashley Stirrup: Yes, me too.