Back to Podcast
A/B Testing
Scale
Testing AI

From Opinion to Evidence: Evaluating AI Ads at Walmart Connect

S2 | E11
Nov 17, 2026

Summary

How do you measure the quality of an AI-generated ad when designers, legal teams, and advertisers may all define “good” differently? Nishanth Kadiyala, Product Leader at HelloFresh and formerly an ad tech product leader at Walmart Connect, joins Ashley Stirrup on The Experimentation Edge to explain how the team shifted from subjective creative preferences to measurable acceptability, compliance, and performance.

Nishanth walks through the realities of building creative automation for smaller advertisers: segmenting users, validating AI output against a large evaluation dataset, running A/B tests to earn stakeholder buy-in, and letting real advertiser needs shape the product. He describes moving creative production from weeks to minutes as more than 250 advertisers tried the product, then shares how those lessons inform creative velocity and diversity in his current work at HelloFresh.

📝 Read the full blog post →

Chapters

00:00 Meet Nishanth Kadiyala
01:16 Inside Walmart Connect’s retail media stack
05:30 Why B2B experimentation requires a different playbook
07:30 Moving from subjective quality to acceptability
09:45 Helping smaller advertisers create better ads
12:21 Iterating with user research and phased launches
14:47 Testing AI creative against human-designed ads
15:57 Building a golden dataset for AI evaluation
18:05 Scaling across products, advertisers, and categories
22:18 Defining baseline quality and performance
24:50 Unexpected lessons from advertiser feedback
26:35 Choosing the right level of AI control
28:33 Results: 250+ advertisers and minutes to create
29:44 From manual reviews to AI evaluation pipelines
31:35 Creative velocity and diversity at HelloFresh
34:57 AI’s role in the next generation of testing
36:40 Closing reflections

Notable Quotes

“The same designer might feel differently about the same creative on two different days.”

“Now in this AI-first world, we are dealing with stochastic systems. You ask the same question, and the output may not always be the same.”

“Once we start moving from that quality to acceptability angle, that in itself became a massive exercise of learning and evolution of the models behind the scenes.”

“The creative production time has reduced from multiple weeks to a matter of minutes.”

“Let’s have a strong golden data set and a strong eval process and automate it at the right time so that we’re able to build things at scale at a faster pace.”

Transcript

The Experimentation Edge - Nishanth Kadiyala

Nishanth Kadiyala (00:00) picking the right model we thought was like the most important thing, only to later realize to be able to find a happy middle path between what compliance and legal cares about, what design cares about, and performance,

Nishanth Kadiyala (00:17) while giving them the performance, while giving them the coverage that end user is looking for. Because I cannot say,

Ashley Stirrup (00:21) Yeah.

Nishanth Kadiyala (00:22) hey, I can build a great creative for 50% of your catalog. End user will be able to understand that. I need to be able to do it at a scale. that

Nishanth Kadiyala (00:30) was the biggest learning. And the way we went about it was very incremental.

Nishanth Kadiyala (00:35) Hypothesizing that okay, let's first test if we can do it through solid backgrounds. Let's have complementary background colors while I figure out the nuances of what

Nishanth Kadiyala (00:45) Compliance cares about what design cares about. And over the course of time, it was also about improving our coverage. In the very first iteration, we were able to say, you know what, if you have a square product, square looking product or a cube product, we'd be able to do great.

Ashley Stirrup (01:32) Hello and welcome to today's episode. Today I'm excited to have Nishanth Kadiyala, product leader at HelloFresh. Welcome to the show, Nishanth.

Nishanth Kadiyala (01:41) I actually thanks for having me. Glad to be here.

Ashley Stirrup (01:44) Yeah, yeah. You've got a great career. maybe we start with a little bit of that. love to hear a little bit on your background and when you first got involved in A/B testing.

Nishanth Kadiyala (01:52) Yeah, thanks for that. I think A/B testing has been part of my life for a while now. but higher level, I'll focus on the last two stints that might be most relevant over here. at the moment, I'm a product leader at HelloFresh. I lead their paid social channels and all the automation that comes along with the creatives and the larger ad tech portfolio. That being said, for the past five years I've spent my time at Walmart Connect.

Nishanth Kadiyala (02:20) As their adtech leader across retargeting, dynamic creatives, and more recently automated creatives. So A/B testing has always been part of the product lifecycles across my career, initially as a product marketer and then as a product manager. But I'm happy to dive deep into any of the experiences that you're curious about.

Ashley Stirrup (02:42) Yeah, you mentioned those different types of things you were doing at Walmart Connect. Could you explain those a little bit more?

Nishanth Kadiyala (02:48) Yeah, of course. So at Walmart Connect, which is the retail media arm of Walmart, the business was about enabling suppliers, aka advertisers, to go after the shoppers who might be in the market for their products. Right. foundationally a couple of different technologies. One is targeting. So who do you go after? What do you tell them?

Nishanth Kadiyala (03:10) How do you bring them back to the website? Right. So my first half was focused on off site advertising, where let's say you search for a blue shoe, I search for a red shoe, and we both end up not buying the product. We go onto the larger internet, you're retargeted with the blue shoe, and I'm retargeted with the red shoe. So right time, right place, right product, you come back and I higher likelihood of you converting. Right. So that's what I was focused on initially.

Nishanth Kadiyala (03:38) And more recently, I've shifted my expertise onto the creative side. So creative is what you see as part of the ad. What imagery, what product, what messaging, what landing page, everything around it. Right. And in that landscape, there are different aspects to a creative generation. So there's the asset generation, there's the assembly, there's the moderation, and there's the optimization.

Nishanth Kadiyala (04:04) So that's the life cycle that my team owned. And in particular, over the last year and a half, we've been focused on automating this whole process for the sake of making the ground normalized and leveled for even the smaller advertisers to compete with the larger ones. With the advent of Gen AI, how would you make it possible for the smaller players to compete and have visually exciting creatives? But

Nishanth Kadiyala (04:30) within the guidelines of what is acceptable from a legal perspective, what is acceptable from a visual perspective, what is expected from a performance perspective. So that's where majority of the experimentation lied over the last couple of years.

Ashley Stirrup (04:44) Yeah, super interesting.

Ashley Stirrup (04:45) And were there certain types of lessons learned that you were, often providing to the people you were working with?

Nishanth Kadiyala (04:52) Totally. I would say especially from the lens of experimentation, it's good to have that type of B2B versus B2C because it foundationally translates to how much of an exposure can you get to your experiment at any given point. In case of B2C, there are millions of users, millions of views. So naturally there's more scope to do the standard A/B test. but if you're going with the B2B route, then there are probably hundreds and thousands of suppliers.

Nishanth Kadiyala (05:20) And spread out by multiple users within the same account. Right. So to be able to run such a controlled experiment is going to be slightly tricky. So that's where it's more spaced out. So

Ashley Stirrup (05:32) Yeah.

Nishanth Kadiyala (05:32) from a learning standpoint, to iterate towards the right product in a B2B environment, we obviously have to experiment, but we're doing it in a more phased approach.

Nishanth Kadiyala (05:43) Figure out your target audience, bucket them. In my case, from the lens of automated creatives, there were three different personas. The Pepsi's and Cokes of the World who are going to be very meticulous about the exact pixel that goes on to a creative. And on the extreme other side, on the long tail side, you have advertisers who are saying, you know what, I don't know much about creatives. All I care about is performance. And in between, you'll have folks who are saying,

Nishanth Kadiyala (06:10) You know what, we have some branding, but we're not gonna be pedantic about exact pixel, for example. Right. So

Ashley Stirrup (06:18) Right.

Nishanth Kadiyala (06:18) to be able to figure out who to go after, what version of the product to go after, so that we're able to balance speed versus quality versus customization, and then gradually find our product market fit with the larger players is something that was very interesting for us to find out over the course of like eighteen months.

Ashley Stirrup (06:32) Yeah. Yeah. Yeah.

Ashley Stirrup (06:34) Did you have certain rules of thumb that let's say, particularly when you're working with a smaller marketer who maybe doesn't have the same skill set, are there certain things you'd always kinda look at at an ad, Okay, here's five best practices that, you should always follow, make sure you show your product in a certain light

Ashley Stirrup (06:50) I don't know what the other things you might say, but were there certain rules of thumbs that when you were just evaluating somebody's ad that you would kinda check for those things before you ever got into the more nuanced stuff?

Nishanth Kadiyala (07:01) Great question.

Nishanth Kadiyala (07:01) This is very much in line with our own journey to the creative process. Initially,

Ashley Stirrup (07:06) Yeah.

Nishanth Kadiyala (07:07) we were very married to the idea of which model to pick. How do we make everything perfect? Right. And

Ashley Stirrup (07:13) Yeah.

Nishanth Kadiyala (07:14) that definition of perfect varies by the person you speak with. If you're talking to the legal and compliance team, they're looking at a different lens. And if you're talking to the design team, they're talking about a different lens. And more importantly,

Nishanth Kadiyala (07:28) Two designers might feel differently about the same creative.

Ashley Stirrup (07:31) Yeah.

Nishanth Kadiyala (07:32) Or this will be surprising for you. The same designer might feel differently about the same creative on two different days. Because there

Ashley Stirrup (07:39) Right.

Nishanth Kadiyala (07:40) is no absolute logic or math behind the scenes. It's almost like a feeling. So you can directionally

Ashley Stirrup (07:43) Yeah. Yeah.

Nishanth Kadiyala (07:45) get it right, but not perfect. So that's one of the things that we had to unlearn over the course of designing the product to move away from.

Nishanth Kadiyala (07:53) discussions of quality to the discussions of acceptability. And that

Ashley Stirrup (07:56) Mm-hmm.

Nishanth Kadiyala (07:57) then translated into your questions around okay, what is expected out of a creative, right? So then it came down to hey, is the product getting highlighted? Is it being sold to in a lens of the right context? Is the environment that we're putting this in meaningful? Is it respecting the

Nishanth Kadiyala (08:17) physics and gravity, man, does this look too small, too big, for example?

Ashley Stirrup (08:21) Right.

Nishanth Kadiyala (08:21) And then on the legal side, we were talking about, hey, are we modifying the product? Or are we injecting third-party brands that shouldn't be inside this ecosystem? Are we

Ashley Stirrup (08:31) Mm-hmm.

Nishanth Kadiyala (08:32) using celebrity faces, for instance?

Ashley Stirrup (08:34) Yeah.

Nishanth Kadiyala (08:34) So by moving the conversation from something as granular as is the shadow right? Is the lighting

Ashley Stirrup (08:40) Mm-hmm.

Nishanth Kadiyala (08:41) accurate? Is

Nishanth Kadiyala (08:42) Is the contextual background color complementary to the product. So these are like nuances that designers will have a strong opinion on. But thankfully the models have already crossed that chasm.

Nishanth Kadiyala (08:55) Exactly. the vision I had for my product when I kicked off this journey about an year and a half was ago was that

Nishanth Kadiyala (09:00) We have grown massively for a couple of years or more than a couple of years thanks to the large advertisers. But at some point we reached that point of inflection where the next phase of growth needs to come from the long tail advertisers. And when I think in terms of the larger flywheel of Walmart, the success of the Walmart ecosystem lies in enabling the new entrants to succeed in the

Nishanth Kadiyala (09:29) marketplace and the biggest

Ashley Stirrup (09:30) Yeah.

Nishanth Kadiyala (09:31) lever for them to succeed is imagine an advertiser not having any reviews. How would they get the attention of users? And that has to happen through the paid channels of advertising among other levers and that's where we're saying hey you have sponsored ads where you do not need to do any creative anything fancy but the moment you shift our focus to display which is a little bit more brand based

Nishanth Kadiyala (09:55) they did not know how to cross that lever. So

Ashley Stirrup (09:58) Yeah.

Nishanth Kadiyala (09:59) in one of the hypotheses that I had at that point was if I were to leverage AI to minimize the amount of friction it includes today to produce like an ad that would boost our revenue from a small and medium sized businesses perspective. And yeah,

Ashley Stirrup (10:17) Yeah.

Nishanth Kadiyala (10:17) to your question, AI based creatives is where it began in the context is

Nishanth Kadiyala (10:23) folks already have a strong place to start with in the form of the PDP. Right? The product description page. It has images, it has text. So how can we repurpose that to produce an exciting creative? That's the problem space

Ashley Stirrup (10:35) Interesting.

Nishanth Kadiyala (10:36) that we were working at.

Ashley Stirrup (10:37) Okay. Is that typically what you were doing? Is you were starting with the PDP to generate the ad? It's super interesting.

Nishanth Kadiyala (10:41) Yeah.

Nishanth Kadiyala (10:43) And that's the mental model, right? So let's not have people reinvent the wheel,

Ashley Stirrup (10:49) Yeah.

Nishanth Kadiyala (10:50) whether it is the SMBs or the system. Let's understand the language that the SMBs are already talking about their product in and then build on top of it and reimagine that in the setting that makes the most sense from a display perspective, display advertising

Ashley Stirrup (11:05) Yeah.

Nishanth Kadiyala (11:05) perspective.

Ashley Stirrup (11:06) Yeah, imagery in particular, you can get to ninety percent of the way there pretty quick, but that last ten percent can be pretty painful. Were you able to help people actually make it all the way to where they were ready to publish something?

Nishanth Kadiyala (11:18) Absolutely. And that took a lot of experimentation over the course of eighteen months, right?

Ashley Stirrup (11:24) Yeah.

Nishanth Kadiyala (11:25) To my earlier point, like picking the right model we thought was like the most important thing, only to later realize to be able to find a happy middle path between what compliance and legal cares about, what design cares about, and performance, right?

Nishanth Kadiyala (11:43) while giving them the performance, while giving them the coverage that end user is looking for. Because I cannot say,

Ashley Stirrup (11:47) Yeah.

Nishanth Kadiyala (11:48) hey, I can build a great creative for 50% of your catalog. End user will be able to understand that. I need to be able to do it at a scale. So that

Ashley Stirrup (11:56) Yeah.

Nishanth Kadiyala (11:57) was the biggest learning. And the way we went about it was very incremental.

Ashley Stirrup (12:01) Yeah.

Nishanth Kadiyala (12:02) Hypothesizing that okay, let's first test if we can do it through solid backgrounds. Let's have complementary background colors while I figure out the nuances of what

Nishanth Kadiyala (12:11) Compliance cares about what design cares about. And over the course of time, it was also about improving our coverage. In the very first iteration, we were able to say, you know what, if you have a square product, square looking product or a cube product, we'd be able to do great. But everything else around it, we're like, we don't know yet. And that was all right for an alpha.

Ashley Stirrup (12:34) And so were you running a lot of A/B tests there?

Nishanth Kadiyala (12:36) A/B test both there were two parts to it. One is the A/B test to prove that the creatives that were being produced are performing as well as the standard manual creatives. Because the whole gospel of truth if you speak to agencies is that okay, they have this human touch that is hard to mimic. And I'm sure we will be able to empathize better with like manually created produced creatives.

Nishanth Kadiyala (13:01) But that being said, at the scale of an ad that's launching 15 million impressions, where ads are first of all not paid attention to because the customer is in the mind space of like buying something, how important is an image is like a debatable topic. The

Ashley Stirrup (13:20) Right.

Nishanth Kadiyala (13:21) day you take the conversation to performance, we're saying, hey, like we agree that creative A looks better than B.

Nishanth Kadiyala (13:28) From an aesthetics perspective. But if we expand the conversation to performance, they're comparable. Right? So that's the kind of A/B test I had to launch to get internal buy-in to launch the product at scale. But that being said, there's another layer of experimentation that was going behind the scenes, and that is the entire AI evaluation process. The issue is that

Nishanth Kadiyala (13:52) Now in this AI first world, we are dealing with stochastic systems. You ask the same question, the output may not always be the same. So in such

Ashley Stirrup (14:00) Yeah.

Nishanth Kadiyala (14:00) a world, how would you build systems that are reliable more often than not? Right? So that entailed a lot of a massive golden data set that is truly representative of the catalogue.

Ashley Stirrup (14:14) Mm-hmm.

Nishanth Kadiyala (14:16) cube, is it a circular product? Is it small? Is it big? is it wide? So to be able to truly expand across the catalogue, we were talking about north of 20,000 plus items. And then evaluating each of these outputs as good as bad across five dimensions that we

Ashley Stirrup (14:33) Yeah.

Nishanth Kadiyala (14:33) were earlier alluding to. is it accurate? Is it contextual? Is it

Nishanth Kadiyala (14:39) being truthful to the product, is it avoiding third party branding, et cetera. Right. So once we start moving from that quality to acceptability angle, that in itself became like a massive exercise of learning and evolution of the models behind the scenes before it ever gets presented to the shop, the advertiser. That's the

Ashley Stirrup (14:59) Right.

Nishanth Kadiyala (15:00) exciting part about AI, where you can run a lot of internal testing before you expose it to the users.

Nishanth Kadiyala (15:07) And then we have guardrail metrics around what does the acceptability look like?

Ashley Stirrup (15:12) Mm-hmm.

Nishanth Kadiyala (15:13) Between every model, we're able to look at, hey, if a hundred advertisers tried the product, how many of them have submitted the creative? How many of them went live with the creative? That in itself becomes like a guardrail for are we headed in the right direction? Are we building the right thing?

Ashley Stirrup (15:30) Yeah. And so this is all so fascinating. And so not only got to a point where you're able to create new ads, but you were able to test them using AI. What

Nishanth Kadiyala (15:41) Yep. Yep.

Ashley Stirrup (15:42) were some of the parts that were the hardest to overcome in that whole process? Was it more on the creative side or more on the testing side?

Nishanth Kadiyala (15:50) I would say the reconciliation between what is good enough for the different stakeholders in the ecosystem was the most difficult part. And also

Ashley Stirrup (15:57) Yeah. Mm-hmm.

Nishanth Kadiyala (16:00) understanding who we are building for was another important piece. Given the state of GenAI technologies, it was important to be able to say, hey, here are people it makes sense to build for to begin with. And as the technology improves, as our internal muscle

Nishanth Kadiyala (16:16) understanding and processes and technologies improve to catch up with how to build an AI software and not just a regular software, that's when we could start affording to pay attention to the bigger players. So that

Ashley Stirrup (16:29) Yeah.

Nishanth Kadiyala (16:30) was the first and foremost important thing to level set the expectations across the board about who we're going after. The second piece is then driving that empathy towards the process.

Nishanth Kadiyala (16:42) Between the stakeholders who are generally used to the standard ways of testing.

Nishanth Kadiyala (16:46) used to deterministic outputs of, okay, tell me what is it gonna look like, is

Ashley Stirrup (16:51) Yeah.

Nishanth Kadiyala (16:51) an answer where I'm like, it depends, right? So how do

Ashley Stirrup (16:54) Yeah. Yeah.

Nishanth Kadiyala (16:56) we drive that empathy between a design team who can be very paradistic about that exact kind of shade they're looking for, or a legal

Ashley Stirrup (17:04) Mm-hmm.

Nishanth Kadiyala (17:04) team who can be very particular about

Nishanth Kadiyala (17:07) what's being told about the product and the copy and not. So that was particularly challenging.

Ashley Stirrup (17:10) Yeah. Yeah.

Nishanth Kadiyala (17:14) And then doing it at the scale of Walmart was the other big challenge. Right.

Ashley Stirrup (17:18) I'll bet. Yeah.

Nishanth Kadiyala (17:20) Yeah. I'm sure like if a smaller retailer or like a brand was trying to solve the same problem, especially if I'm a brand, I'm either solving for CPG or like fashion or hardline,

Ashley Stirrup (17:31) Mm-hmm.

Nishanth Kadiyala (17:32) right? So it becomes a much more scoped down problem.

Nishanth Kadiyala (17:35) The day you're starting

Ashley Stirrup (17:35) Yeah.

Nishanth Kadiyala (17:36) to say, I want to solve for everyone, that's another big challenge.

Ashley Stirrup (17:39) I was gonna say it's a wider variety of products for sure. Yeah. Yeah. Yeah.

Nishanth Kadiyala (17:43) yeah, probably the biggest catalogue that you'll have to deal with, right? So that was the luxury

Nishanth Kadiyala (17:47) as well as boon and bane, right?

Ashley Stirrup (17:48) Yeah.

Nishanth Kadiyala (17:49) I had the luxury and fortune to be able to build something at that scale, but

Ashley Stirrup (17:55) Mm-hmm.

Nishanth Kadiyala (17:56) also I had to go through the pain of building at that scale. And

Ashley Stirrup (17:58) Yeah.

Nishanth Kadiyala (17:59) that came at the cost of being intentional about who we are going after, what verticals are we building for. So from day

Ashley Stirrup (18:05) Yeah.

Nishanth Kadiyala (18:05) one we said, okay, here is the concentration of the assembly market.

Nishanth Kadiyala (18:08) will solve for this market first and then gradually expand to other markets. Because by the time we go GA, many SMBs they don't stick to a single vertical. They have all

Ashley Stirrup (18:18) Mm-hmm.

Nishanth Kadiyala (18:18) these adjacent items and when they start launching, we cannot say in some cases, yes I can solve for you, no I cannot do it.

Nishanth Kadiyala (18:25) And

Nishanth Kadiyala (18:26) to support this narrative, it was important to have the backing of data by running those tests ahead of time. A manual

Ashley Stirrup (18:32) Yeah.

Nishanth Kadiyala (18:32) creative, exciting AI based.

Nishanth Kadiyala (18:35) Lifestyle creative and a non-AI-based systemic creator that's like a simple background. And we were able

Ashley Stirrup (18:41) Mm-hmm.

Nishanth Kadiyala (18:42) to prove over a course of time that they're all going to be meaningful in some sense. It's hard

Ashley Stirrup (18:47) Mm-hmm.

Nishanth Kadiyala (18:47) to attribute performance to the specific excellence of the creative itself, which in itself becomes

Ashley Stirrup (18:53) Right.

Nishanth Kadiyala (18:54) an ammunition for accelerating your roadmap.

Ashley Stirrup (18:57) Right, right, right. Yeah. And you know, I've talked to a fair number of people about say using evals to test LLM powered, I don't know, chat bots type of thing. And they often said that, you know, the evals were fine for QA. Like, does it actually generate an output? But in terms of measuring which one is likely to resonate with humans, that that it was terrible at that part.

Ashley Stirrup (19:22) It was fine, at yes the words came out and yes they were talking about roughly the right thing. But in terms of did people engage with it more, did they buy more or were they satisfied with, you know, the experience, that kind of stuff, that it was much harder to do that. how about in in this case? Were you able to kind of use your AI powered tests? was it highly correlated to what you actually saw in real world human performance?

Nishanth Kadiyala (19:45) this is something that we had to learn the hard way to eventually

Ashley Stirrup (19:48) Mm-hmm.

Nishanth Kadiyala (19:49) abstract this to acceptability, to my earlier point, where we said, okay, the ground rules are ground rules. Are we doing good in that front? Is in itself the first line of defense that we have in place to make sure that we meet the baseline.

Nishanth Kadiyala (20:04) And everything else is a fast follow. Once we made the baseline, are people going to stop using the product because it doesn't meet their excellence bar? Is a different question from is the creative performing. So to ensure that the GTM is accounting for these nuance was extremely important. And the way we did this was twofold. One is what we call like a benevolent evaluator.

Ashley Stirrup (20:29) Mm-hmm.

Nishanth Kadiyala (20:30) So identify a X SME, in this case, like a design SME, that's able to represent a massive portion of the market through

Ashley Stirrup (20:42) Mm-hmm.

Nishanth Kadiyala (20:42) their historical understanding of the market and design principles. Right.

Ashley Stirrup (20:47) Mm-hmm.

Nishanth Kadiyala (20:47) And the same thing for compliance. Initially, we went after this model where we were trying to have

Nishanth Kadiyala (20:53) Multiple experts along with advertisers come to consensus, and that's far more difficult.

Ashley Stirrup (20:59) Mm-hmm.

Nishanth Kadiyala (20:59) To be able to come back and say, okay, here is this one person who understands the nuance of design, performance media, and AI systems, and they will help us improve the baseline to a point where we are agnostic to what's getting presented at that point. And then the performance drives the improvements of the product.

Ashley Stirrup (21:19) Yeah.

Nishanth Kadiyala (21:19) we know for a fact what's working, what's not working, and that in becomes a good like feedback loop for

Nishanth Kadiyala (21:25) the system.

Ashley Stirrup (21:38) Yep. Do you have an example of an experiment you ran where you had a lot of learnings?

Nishanth Kadiyala (21:43) I would say

Nishanth Kadiyala (21:44) The entire loop of like change through to the automated creative generation process, I

Ashley Stirrup (21:50) Mm-hmm.

Nishanth Kadiyala (21:50) would say the biggest learning for me was the nuances of what a small and medium business will care for compared to a bigger one.

Ashley Stirrup (21:58) Mm.

Ashley Stirrup (22:00) Right.

Nishanth Kadiyala (22:01) to lead with this idea that something as simple as a product on a background shot. And again, I I did that for a different reason. We were finding it difficult to get approvals internally to launch AI products. So I said, hey, let's start with a non AI product where I can still get the basic instrumentation in place, warm up to like what the market is looking for. And even in that state, there were several people who were like, this is great. This is exactly what I wanted.

Nishanth Kadiyala (22:29) Typically, even for this creative, if I had to get the buy-in from my creative team, it would take me a couple of weeks. And

Ashley Stirrup (22:36) Yeah.

Nishanth Kadiyala (22:36) more importantly, it was not even the visual baseline that was stopping people from trying the product. It

Ashley Stirrup (22:42) Mm-hmm.

Nishanth Kadiyala (22:43) was the flip side, which is a compliance piece of it. So for example,

Ashley Stirrup (22:45) Mm.

Nishanth Kadiyala (22:46) someone can say something like, best in the universe, and the creator will go through. No problems.

Ashley Stirrup (22:52) Mm-hmm.

Ashley Stirrup (22:53) Yeah.

Nishanth Kadiyala (22:53) But if

Nishanth Kadiyala (22:53) you say something like best in the US, now that's a claim. The first one was a puffery. The second one is a claim.

Ashley Stirrup (22:57) Mm-hmm. Right. Yeah.

Nishanth Kadiyala (23:01) And why do I know I will trade off a lot of times of my time to

Ashley Stirrup (23:02) That's so funny. Yeah. Right, right.

Nishanth Kadiyala (23:05) not know it. But the problem is the same thing is going on with the small and medium businesses.

Ashley Stirrup (23:10) Yeah.

Nishanth Kadiyala (23:11) So what they were saying is, hey, I have a twofold problem. I don't know how to produce a creative that is good looking, but I also don't know how to work around.

Nishanth Kadiyala (23:19) the nuances of what you consider is acceptable and not acceptable because Walmart has all these strict compliance rules.

Ashley Stirrup (23:27) Yeah.

Nishanth Kadiyala (23:28) So along the way, while I was also learning how to solve the visual process, it was a critical decision to break down the product into different sequenceable features. I said, okay, let's also solve for auto moderation before auto design.

Nishanth Kadiyala (23:46) So that in itself helped us accelerate the overall process. And

Ashley Stirrup (23:50) Yeah.

Nishanth Kadiyala (23:50) the choice of making it a black box was also extremely critical. What

Ashley Stirrup (23:55) Mm-hmm.

Nishanth Kadiyala (23:55) I mean by that is after doing some market research, there were two extremes. There was the Facebook side of the world which said, Hey, no control. Just say what you're trying to do. We're gonna generate the creatives, we're gonna optimize it, everything is in our hands.

Nishanth Kadiyala (24:10) And on the flip side, we had Amazon, which was saying, handles like this silver bullet. Do what you feel like.

Ashley Stirrup (24:16) Mm-hmm.

Nishanth Kadiyala (24:18) And during the analysis, I realized like it's not the right answer for either of them. Like in one case,

Ashley Stirrup (24:23) Mm-hmm.

Nishanth Kadiyala (24:24) it's over control

Ashley Stirrup (24:26) Yeah.

Nishanth Kadiyala (24:26) to a base that doesn't understand what's a good creative. And what ends up happening is I don't know if you have tried to produce creatives or like LinkedIn headshots, for example,

Ashley Stirrup (24:35) Sure.

Nishanth Kadiyala (24:36) you will end up in.

Nishanth Kadiyala (24:37) all these limbo states of hundreds of iterations trying to get something wrong, something else gets compromised and you don't know how to come out of it.

Ashley Stirrup (24:42) Yep. Yep.

Nishanth Kadiyala (24:45) And if the not stuff on my product was to reduce the amount of time it takes to produce a creative, this was clearly not helping. And

Ashley Stirrup (24:52) Right.

Nishanth Kadiyala (24:52) that's where I was like, okay, let's start with the version that's going to have some level of control, with a version where there's no direct access to the model, and then graduate towards giving some level of basic prompting.

Nishanth Kadiyala (25:06) that allows users to interact with the model to say, hey, you what? I'm going after back to school theme. I'm going after a Christmas theme. So give me something around that. So

Ashley Stirrup (25:14) Yeah.

Nishanth Kadiyala (25:15) opening the doorway slightly in a more controlled manner was something that was a critical decision that helped us come up with a version that everyone in the ecosystem was happy with. Right?

Ashley Stirrup (25:26) Yeah.

Nishanth Kadiyala (25:27) Netnet over a course of eighteen months, we were able to go from a zero to one state

Nishanth Kadiyala (25:33) All the way from alpha to an open beta to north of 250 plus advertisers trying the product. More importantly, the creative production time has reduced from multiple weeks to a matter of minutes. All it takes is like a click, and in within five minutes, you would see the final creative.

Nishanth Kadiyala (25:51) went from mid eighties to all the way till hundred percent in a couple of months, but north of ninety-eight

Ashley Stirrup (25:57) Wow.

Nishanth Kadiyala (25:57) percent for like multiple months in a row. So that

Ashley Stirrup (26:00) Yeah.

Nishanth Kadiyala (26:01) in itself tells the story of removing the friction from the engineer's perspective. Who's saying, I don't know what am I doing, just help me get started with the ad and all of a sudden all I have to say is what I know, what items I'm trying to sell. And then we produce

Ashley Stirrup (26:13) Yeah. Yeah.

Nishanth Kadiyala (26:16) a creative quick enough, but also highly

Nishanth Kadiyala (26:19) compliant with our rules, which makes it extremely simple for them to launch an ad.

Ashley Stirrup (26:24) Yeah. It's in so interesting. So it sounds like basically you kind of you had a process that you wanted to be able to follow. You knew you weren't gonna be able to solve everything right away and you just kind of kept tackling piece after piece over time, in order to get there.

Nishanth Kadiyala (26:37) Yeah, and also building

Nishanth Kadiyala (26:39) that data science muscle, right? That was the other important

Ashley Stirrup (26:41) Mm-hmm.

Nishanth Kadiyala (26:42) piece. Like we were early in our journey relative to other

Ashley Stirrup (26:46) Mm-hmm.

Nishanth Kadiyala (26:47) adtech first companies probably, and to be able to build a team that understands the nuance of stochasticity and build an AI Eval engine, a continuous pipeline that's able to make it easy for us to evaluate to begin with manually and then evolve towards LLM as a judge where it's

Nishanth Kadiyala (27:07) already segregating the content into high risk and low risk and we are able to just look at the high risk elements to begin with.

Nishanth Kadiyala (27:15) And that in

Ashley Stirrup (27:16) Yeah.

Nishanth Kadiyala (27:16) itself became like a momentum builder because now all of a sudden everyone is like, if the time it takes to fix something is quick enough, my appetite or risk ability to launch something is slightly more because I can fix it quickly enough.

Ashley Stirrup (27:30) Yeah. And you've only been at HelloFresh a relatively short period of time. How's your role changing as you've switched ups?

Nishanth Kadiyala (27:38) At HelloFresh, I think it's I have had the luxury of sitting on the retailer side and represent multiple advertisers and now I'm on the other side. So I'm now

Ashley Stirrup (27:46) Mm-hmm. Yeah.

Nishanth Kadiyala (27:48) with the brand, advertising. And here I take a foundationally different approach in the sense that

Nishanth Kadiyala (27:54) It doesn't have to be a built everything attitude.

Ashley Stirrup (27:58) Mm-hmm.

Nishanth Kadiyala (27:58) I care about time to market. I care about what I'm good at. So I'm building the teams around the same whole idea of optimizing the ad tech funnel. And

Ashley Stirrup (28:10) Mm-hmm.

Nishanth Kadiyala (28:10) when I think in terms of paid social, let's take the example of Meta, right? So Meta now became like a huge black box. All they say is, Hey, we know everything, or maybe like

Nishanth Kadiyala (28:20) We don't know much. The algo knows everything. It knows who to go after. You just give us the creatives.

Ashley Stirrup (28:27) Yeah.

Nishanth Kadiyala (28:27) So in this landscape

Nishanth Kadiyala (28:29) Creative velocity, creative diversity, and creative excellence. These three become extremely critical. And

Ashley Stirrup (28:37) Yeah.

Nishanth Kadiyala (28:38) at the scale of HelloFresh, that's operating between multiple brands and multiple entities or countries, how do I build a system that's able to account for the nuances of different brands, different countries?

Nishanth Kadiyala (28:55) different languages and how do I platformise it. So that's been

Ashley Stirrup (28:59) Yeah.

Nishanth Kadiyala (28:59) my focus area.

Nishanth Kadiyala (29:00) with the ability to build on top of my foundation that I have garnered at Walmart because at the end of the day the principle remains the same I have to automate creatives

Ashley Stirrup (29:06) Yeah. Yeah.

Nishanth Kadiyala (29:08) with the lens of I need a lot more diversity. So how

Ashley Stirrup (29:12) Yeah.

Nishanth Kadiyala (29:12) can I repurpose existing raw footage that's coming from so many different channels and

Ashley Stirrup (29:17) Mm-hmm.

Nishanth Kadiyala (29:18) make it exciting for Meta to give us better performance. So

Ashley Stirrup (29:23) Yeah.

Nishanth Kadiyala (29:24) yeah that's been like the

Nishanth Kadiyala (29:26) biggest shift in mind space. And also we are relatively a much smaller shop. So that

Ashley Stirrup (29:32) Sure.

Nishanth Kadiyala (29:33) gives me the luxury of saying, Hey, you know what? Let's fail fast.

Ashley Stirrup (29:37) Yeah.

Nishanth Kadiyala (29:38) With Walmart, it's beast, right? So

Ashley Stirrup (29:40) Right.

Nishanth Kadiyala (29:40) every move that you make, if you make a move, it gets to ten teams who go to thousands of advertisers, who go to

Ashley Stirrup (29:45) Yeah. Right.

Nishanth Kadiyala (29:48) ten thousand designers. And

Ashley Stirrup (29:50) Yeah.

Nishanth Kadiyala (29:50) all of a sudden I want to pivot back, it becomes an exercise of oops, wait, one second.

Nishanth Kadiyala (29:55) Let me redo this, it's very expensive. With HelloFresh, it's a close cohort of folks marching towards the same goal. So

Ashley Stirrup (30:05) Yeah.

Nishanth Kadiyala (30:06) they have the appetite to s come back and say, you know what, like give me things quick. And if things are not working, I'll give you a faster feedback loop. And that in itself becomes like a massive moat compared to like a bigger firm out there.

Ashley Stirrup (30:21) Yeah. Yeah.

Ashley Stirrup (30:23) so

Nishanth Kadiyala (30:23) Yeah, yeah.

Ashley Stirrup (30:24) are there any things in particular that you're especially excited about within the world of A/B testing that you know you'd like to see tried at HelloFresh or wherever you go next?

Nishanth Kadiyala (30:35) I would love for teams to start leveraging AI as a foundational tool because most of the experiences are moving away from the standard web experiences to agentic, voice,

Ashley Stirrup (30:48) Yeah.

Nishanth Kadiyala (30:49) et cetera. So to evolve our way of thinking in terms of how do I launch a product that's meaningful enough, that's reliable enough before I go into this whole landscape of A/B testing, right?

Nishanth Kadiyala (31:02) Because it should be seen as an extension is my way

Ashley Stirrup (31:02) Yeah. Yeah.

Nishanth Kadiyala (31:05) of operations for my team. Right. So it's like let's have a strong golden data set and a strong eval process and let's automate it at the right time so that we're able to build things at scale at a faster pace.

Ashley Stirrup (31:20) Yeah, makes a lot of sense. Well, Nishanth, thank you so much for coming on today's show. I feel like we learned a lot from you. The whole online advertising space is super fascinating, and the fact that you were able to make so much progress in just eighteen months is is pretty incredible. It it shows what the long term potential really is.

Nishanth Kadiyala (31:39) Yeah, absolutely. Again, I continue to be passionate about retail media, creative automation, AI automation in general. Right. So it's a pleasure to be here and exchange notes. If there are folks out there who are solving similar challenges, I would love to jump on a one on one and exchange notes.

Ashley Stirrup (31:56) Sounds great. We'll make sure to have your contact information, your LinkedIn profile in particular, in the podcast description. So thank you so much.

Nishanth Kadiyala (32:04) Sounds good.

Nishanth Kadiyala (32:05) Thanks for having me.

No items found.

Subscribe to the podcast

Takeaways from this conversation

A representative golden dataset and ongoing evaluations are as important as the generative model itself.

S2 | E11

The Walmart Connect effort reached 250+ advertisers and cut creative production from weeks to minutes.

S2 | E11

Start with the advertisers and product categories current technology can support, then expand with evidence from real users.

S2 | E11

AI creative must clear guardrails for product fidelity, context and brand compliance while models keep improving.

S2 | E11

An ad that looks better is not necessarily an ad that performs better; A/B testing turns aesthetic debates into learning.

S2 | E11

Top takeaways from other favorite conversations

All Takeaways

Fin A/B tests everything, even one-character prompt changes, and treats a 20 to 30 percent win rate as a sign of a healthy program.

Go to S1 | E28
Theme
Culture
Role
Data Scientist
Industry
Business Tech
Featured
false

More conversation history made Fin more helpful and more prone to fake promises, until a targeted prompt fix removed the hallucinations.

Go to S1 | E28
Theme
Testing AI
Role
Data Scientist
Industry
Business Tech
Featured
false

Institutionalize learning: align OKRs with “fail forward,” and be willing to kill low-performing features quickly.

Go to S1 | E6
Theme
Culture
Feature Flags
Role
Exec
Industry
Identity / Gov Tech
Featured
false

When customers ask for price, test value messaging such as benefits, ROI and time saved.

Go to S2 | E4
Theme
A/B Testing
Role
Product
Industry
Logistics
Featured
false

Accept that being wrong is the point—experimentation only works when leadership embraces humility

Go to S1 | E9
Theme
Culture
Role
Product
Industry
Media & Gaming
Featured
true

In low-volume B2B, read losing experiments for sub-segment signal; a "failed" Stripe form simplification revealed the form was blocking legitimate small-business buyers using Gmail.

Go to S1 | E22
Theme
A/B Testing
Role
Data Scientist
Industry
Business Tech
Featured
false

Use pricing experiments to trade volume for revenue quality; pair higher monthly prices with stronger annual discounts to grow annual attach.

Go to S1 | E1
Theme
ROI
Role
Exec
Industry
Business Tech
Featured
false

A tied overall result can reveal meaningful gains when you analyze individual markets or customer segments.

Go to S2 | E13
Theme
A/B Testing
Role
Product
Industry
Travel & Hospitality
Featured
false

Align on the North Star before building, whether it is revenue, engagement, NPS, or fewer support tickets, and set guardrails so an engagement feature can never quietly drag sales down.

Go to S1 | E42
Theme
ROI
Role
Product
Industry
Consumer Tech
Featured
false
The experimentation edge podcast logo with a picture of host Ashley Stirrup