Summary
Mohan Dasannacharya, Ex Albertsons, Shutterfly, eBay, joins host Ashley Stirrup to talk about what it takes to trust the numbers an experimentation platform gives you. Mohan's path runs from physical design to UX at eBay, product and experimentation at Shutterfly, and now the in-house experimentation and analytics platform at Albertsons, which moved off Adobe Target and cut results reporting from about 7 days to overnight.
Mohan explains how A/A tests that kept producing winners led Albertsons to split tests into A, A and B for almost 2 years, why a flat add-to-cart test turned out to be a false positive once the team looked at next-step conversion, and how a Shutterfly bug hid a 6 to 7% lift. He also covers exposure versus impression tracking, setting expectations before a test runs, keeping analysts independent from product owners, and how personalization and democratized testing will demand longer tests and better guardrails. For product, data and experimentation leaders who want results their whole company can believe.
Chapters
00:00 Welcome Mohan Dasannacharya of Albertsons
00:31 From physical design to eBay, Shutterfly and Albertsons
02:00 Experimentation through the dot-com bubble, 2008 and COVID
02:49 How Albertsons moved experimentation in-house
04:25 Why grocery shoppers add to cart from the grid
06:02 The biggest lesson: make sure you can trust the platform
07:44 Splitting tests into A, A and B
09:12 Why personalization raises the stakes for platform checks
10:11 How the platform team validates experiment results
11:20 The add-to-cart button that became a plus sign
13:13 Validating projected wins after launch
14:53 Track journey conversion and next-step conversion together
15:45 A novelty effect in reverse
16:12 The Shutterfly bug that hid a 6 to 7% lift
17:38 Balancing test volume with test quality
19:23 Exposure versus impression tracking
20:21 Set your expectations before the test runs
22:34 Keep analysts independent from product owners
23:35 When 80% of eBay's home page tests claimed victory
24:30 Personalization and pre-picked test audiences
26:15 Bake-in periods before you measure
27:20 Democratizing experimentation with gatekeepers
29:08 Running longer tests for smaller personalized audiences
30:30 Grocery as an omnichannel experience
Notable Quotes
“If you have the traffic, you can run a constant A/A test to make sure that the platform is working as expected.”
“I don't think it's one or the other. You have to actually have both.”
“Having a platform that can do experimentation doesn't mean that you use the platform as an answer.”
“It's always important to have a baseline of what you're expecting so that you're not cherry-picking the data.”
“You're not going to get a true result in three weeks.”
Transcript
The Experimentation Edge - Mohan Dasannacharya
Mohan Dasannacharya (00:00) one of the things being tested was instead of having a button on the grid which actually said add to cart, it was changed to a plus button.
Mohan Dasannacharya (00:09) And the A/B test the conversion remained very similar. It was flat. So it was like, okay, we're getting more real estate. This is something that we should do. So we made that change, then we realized that actually it is.
Mohan Dasannacharya (00:22) True that the conversion was the same, but over time it started seeming to decline. And we were wondering is this just a traffic thing? And then we started digging deeper into other metrics. What we realized was that the journey actually was changing. So instead of people adding it from the grid, there were more people going to the product detail page
Ashley Stirrup (01:13) Hello and welcome to today's episode. Today I'm excited to have Mohan Dasannacharya, Director of Digital Product Management at Albertsons Companies. Welcome to the show.
Mohan Dasannacharya (01:23) Glad to be here.
Ashley Stirrup (01:24) so Mohan, could you tell us a little bit about your your background?
Mohan Dasannacharya (01:28) Sure. I actually have a very windy path where to where I've come right now. I started in design and actually in physical design, but then I moved and did my master's and then did some consultancy before joining eBay in UX design.
Mohan Dasannacharya (01:44) and this was back in early 2000s when they just started experimentation there and there's some interesting experimentation stories. We can go back into that. And then I joined Shutterfly where I was handling the design part of it, moved into product, did a lot of experimentation there on multiple platforms, both on Adobe as well as OptimizeLe, and then moved to Albertsons, where I am now.
Mohan Dasannacharya (02:08) Where I started actually first with some of the other stuff and then was handling both the experimentation as well as the platform. And lately I have been focusing on the platform itself for analytics and we moved our experimentation platform in-house. so we took it off of Target and we moved it in-house and we set up the exposure data and all that tracking in-house itself. So
Mohan Dasannacharya (02:33) That's my journey in short,
Ashley Stirrup (02:35) you've had pretty amazing a career and I've seen it seen an awful lot. I would imagine one of the biggest shifts in that time was just the shift to mobile. But I'm sure there's
Mohan Dasannacharya (02:45) Yes.
Ashley Stirrup (02:45) many other big shifts you've seen during your career.
Mohan Dasannacharya (02:48) Totally, totally. joined the industry when we were going through the two thousands, early two thousand bubble, went through the two thousand eight and then like went through COVID. so all those things and
Ashley Stirrup (02:59) Yeah.
Mohan Dasannacharya (03:00) the experimentation has its has its own journey through that. And actually interestingly at Albertsons we are now also using APT or MasterCard test and learn, which is for physical stores. But
Ashley Stirrup (03:10) Mm.
Mohan Dasannacharya (03:10) not too much into that, but just a few years now. So that's also interesting.
Ashley Stirrup (03:14) Got it.
Ashley Stirrup (03:16) And what does experimentation look like at Albertsons?
Mohan Dasannacharya (03:19) So Albertsons has been like actually had a big boost during COVID towards the digital. And that's when I think they did a lot of hiring in the digital space. both prior to that it was a lot of coupon clipping that happened, now there's also pick up and go and delivery that happens. So they are trying to do a lot of experimentation mainly to see that
Mohan Dasannacharya (03:41) The personalization is kicking in both with the front end UX as well as with the models and all those things. So we have now created a internal experimentation platform where people can set up and that's constantly improving and developing. We have a dashboard that you can see all the experiments that are going on. which previously you'd have the data scientists have to run the numbers through and it would take them around
Mohan Dasannacharya (04:06) seven days for across the gamut of experiments that are running. Now that actually happens overnight a couple of hours in the morning and they can look at it. So that dashboard also was set up. So it's actually
Ashley Stirrup (04:16) Yeah.
Mohan Dasannacharya (04:17) helped increase the velocity of experimentation to make sure that things are running smoothly. But
Ashley Stirrup (04:24) Yeah.
Mohan Dasannacharya (04:24) we originally were on Adobe Target, we have exited that and we are now on our internal platform.
Ashley Stirrup (04:30) Got it. And I would bet that the digital channel has become just so much more important for Albertsons overtime.
Mohan Dasannacharya (04:38) it has. It's not a large chunk of the from a revenue standpoint because it's grocery people still like to go into the stores. but
Ashley Stirrup (04:45) Yeah.
Mohan Dasannacharya (04:45) yeah, it is there's two parts. One is the actual online purchaser and then how they use the tool itself or the digital properties itself to enhance their experience, whether it's coupon clipping or figuring out things within the store, etc., and driving loyalty.
Mohan Dasannacharya (05:02) how the company uses it to drive loyalty and how custom consumers use it to actually clip coupons and figure out what the best deal is, et cetera. So and then personalization of obviously in using prior orders to unlike a lot of other retail grocery is a very repetitive purchase cycle. So you it's
Ashley Stirrup (05:20) Yeah.
Mohan Dasannacharya (05:20) not like you're going to buy a different types of milk every time you go to the grocery store. unlike
Mohan Dasannacharya (05:25) Many companies where you have retail, you want to go to the product detail page and look at the product details. In grocery, you're actually doing a lot of the buy clicks on the grid itself. And there's an interesting A/B test story about that, which we can get into whenever you feel like. But that's a
Ashley Stirrup (05:43) Yeah.
Mohan Dasannacharya (05:44) another one that was a very like
Mohan Dasannacharya (05:46) false positive kind of story that is there. So it's a very interesting one there.
Ashley Stirrup (05:51) Awesome.
Ashley Stirrup (05:51) Well, I look forward to that. where would you say you've gotten the most value from experimentation at Albertsons? what types of learnings has been the most beneficial?
Mohan Dasannacharya (05:59) I think the the one learning that has been most beneficial actually in terms of experimentation is partially with the experiments itself, but that is something that you learn through years of experimentation that that you're going to get anyway. And there are huge benefits in some of the cases, some of them are flat. But I think the biggest learning that I had after a long time was
Mohan Dasannacharya (06:22) How do we make sure that we can trust the platform? Right. I think we got some anomalies a few years ago, and we were like, this can't be right. And that's when we started questioning why it is happening. We had a lot of discussions with the owners of the platform. And at that time, we then actually started doing A/A tests. for a short time we had to actually stop our tests. We did A/A tests.
Mohan Dasannacharya (06:46) where we were like looking at it and saying, okay, what's going on? And we had winners in an A/A test, which makes no sense. Like
Ashley Stirrup (06:53) Right.
Mohan Dasannacharya (06:53) we are running exact same experiment. We are running it run on volume and we we are get having the platform tell us that there's a winner. That can't be possible. So we had to go through and figure out what was going on. And then they we had a lot of discussions set up was what's the setup, where's the mistakes, what's going on.
Mohan Dasannacharya (07:10) Finally, the data science team along with the vendors were able to like piece together and we had a more balanced platform. But since then, for a long time, almost for two years, we broke our arms into instead of having an A and B, we had an A, A, and a B. So we would first look at the A/A to say, is this a valid test? And if
Ashley Stirrup (07:30) Right.
Mohan Dasannacharya (07:30) it is flat, that's when we started.
Mohan Dasannacharya (07:33) putting in it pitting it against the B. Because
Ashley Stirrup (07:36) Right.
Mohan Dasannacharya (07:36) otherwise everyone in the company had lost like confidence in it to say like this is actually winning or losing. So that's where I think the biggest learning was to periodically test your platform because there is nothing that is there to confidently QA the platform itself.
Mohan Dasannacharya (07:53) if you have the
Mohan Dasannacharya (07:54) Traffic, you can run a constant A/A test to make sure that the platform is working as as is expected. But if you don't have that maybe every six months, just run a A/A test to say, okay, yes, it's still a solid platform and it's going as expected. So I think that's
Ashley Stirrup (08:10) Yeah.
Mohan Dasannacharya (08:11) that's an important learning that I had is like make sure your platform is solid. So yeah.
Ashley Stirrup (08:14) Yeah. Yeah, it's a really important reminder 'cause there's so many ways that
Ashley Stirrup (08:19) your platform could break, right? you could have a bug in how you're doing assignment or maybe page load performance or s yeah.
Mohan Dasannacharya (08:24) Exactly. And and it becomes even
Mohan Dasannacharya (08:28) more important right now with the with personalization and things that are happening where you don't have everyone getting the the experience might be the same but the content might be different. Or the experience might be different because of the component itself having variations. Those kinds of things and it's very easy to
Mohan Dasannacharya (08:50) feel that one is going to win, but it's not winning because of how the demographic is and how the variations in the platform, what they see, etcetera. So it's important to do that, I think it's a key learning from that.
Ashley Stirrup (09:02) Yeah. Yeah, it makes a lot of sense. And you know, how do you build a culture of experimentation at Albertsons? Are you doing things like peer reviews on experiment designs? Are you doing readouts?
Mohan Dasannacharya (09:14) So the experimentation team actually does that. We are more, as I mentioned, we are the platform, we are more focused on making sure that the platform is solid and is delivering the results as expected. So we do have there are times when there will be like, okay, we are not looking at it right or something, where we would constantly look at and check how the setup is to say, okay, because at times when they're doing personalization with
Mohan Dasannacharya (09:40) algorithms they might pick a wrong table or something of that sort and then they use that table in their result check and those could mess things up. So we are constantly doing that and validating that the results are as expected or not as expected, but accurate to the point that they should be. Because there's no
Ashley Stirrup (09:59) Yeah.
Mohan Dasannacharya (09:59) way of saying that yes, they are as expected. That's the whole point of doing A/B testing. Otherwise like why would you do the test, right? So
Ashley Stirrup (10:04) Right. Yeah. Yeah.
Ashley Stirrup (10:06) Yeah. Yeah, makes sense. Makes sense.
Mohan Dasannacharya (10:06) So So
Ashley Stirrup (10:08) and so
Mohan Dasannacharya (10:09) Sure.
Ashley Stirrup (10:09) maybe we could talk a little bit more about that example experiment you were talking about, where you had a lot of learnings. Yeah.
Mohan Dasannacharya (10:13) sure, sure. Yes,
Mohan Dasannacharya (10:15) yes. That was a very interesting one where as I told mentioned earlier, in grocery people like to don't have to go to the product detail page, right? So
Ashley Stirrup (10:23) Right.
Mohan Dasannacharya (10:24) one of the things that was being tested at the time was instead of having a button on the grid which actually said add to cart, it was changed to a plus button. It was just a small circle with a plus button.
Mohan Dasannacharya (10:36) And the A/B test on looking at it, the conversion remained this very similar. It was flat. So it was like, okay, we're getting more real estate. This is something that we should do. So the initial feedback was let's go ahead and do this. So we made that change, then we started looking at other data around it. And we realized that actually it is.
Mohan Dasannacharya (10:57) True that the conversion was the same, but over time it started seeming to decline. And we were wondering is this just a traffic thing? And then we started digging deeper into other metrics. What we realized was that the journey actually was changing. So instead of people adding it from the grid, there were more people going to the product detail page and adding it from there. So the pogo sticking was increasing, which meant that their the length of their their visit.
Mohan Dasannacharya (11:22) To add the same number of items would be longer. So if they have a set amount of time and they're going back and forth, pogo sticking between the grid and the product page, they're going to add fewer items. So that's when we realized, okay, this is actually the product adding was happening more from the product details page than it was from the grid. And when we changed
Ashley Stirrup (11:39) Yeah.
Mohan Dasannacharya (11:40) it back, it didn't switch drastically or very quickly, but over time it has gone back to.
Mohan Dasannacharya (11:46) More ads from the grid and less from the product detail. So that was actually a interesting, false positive that we got, which we said, okay. But if you don't keep continuous track of it, which is another thing of experiments that we need to think of, is how do you make sure that it's actually delivering on the result that it that it said that it would deliver? Like it's saying, okay, we'll over a year we'll get a 2% increase, let's say. Now,
Ashley Stirrup (12:09) Yeah.
Mohan Dasannacharya (12:09) how do you make sure that you're validating that?
Mohan Dasannacharya (12:12) over time and looking at it against what you what the expected data was. which is could be a slight extrapolation or because you have it's not an A/B kind of thing. It's more like okay, this is our projected path and this is where we think we should be, but then if it was two percent increase then you'd go there and those kinds of things. Unless
Ashley Stirrup (12:29) Yeah.
Mohan Dasannacharya (12:30) you have the bandwidth to keep a five percent base, which you're always keeping on and then changing every six months or
Mohan Dasannacharya (12:36) one year or something like that, which is some of the
Ashley Stirrup (12:38) Yeah.
Mohan Dasannacharya (12:38) some companies could do it, but yeah.
Ashley Stirrup (12:52) I think it's such a great example because just highlights so many different things, right? So one is really truly understanding all the behavioral changes. So it sounds
Mohan Dasannacharya (13:02) Yeah.
Ashley Stirrup (13:02) like initially you're looking at conversion, like okay, conversion's, you know, flat and this gives us some more real estate, so it's let's do it. But then as you look more closely you realize, wait, we're seeing more people click in and out, and so it's
Ashley Stirrup (13:16) lengthening the the cart sessions and that's leading to lower total purchases and so having that range of metrics that you look at is just so important. Yeah.
Mohan Dasannacharya (13:25) Exactly. Exactly.
Mohan Dasannacharya (13:26) as you go through different platforms you'd have different recommendations from the platform saying, you should just look at the entire journey conversion, next step conversion. And I don't think it's one or the other. You have to actually have both. Like if we had kept track of both at that time properly, we would have realized that okay, next step conversion is actually like going down a little.
Mohan Dasannacharya (13:48) and long session conversion is flat, but next step conversion to actually from the grid to the cart is gone down and we would have we should have realized that at that time. But I don't think this
Ashley Stirrup (14:00) Yeah.
Mohan Dasannacharya (14:00) was like when it was we were just starting experimentation or I was just starting in the experimentation at Albertsons, so it wasn't kept track of that time. So
Ashley Stirrup (14:08) Yeah. The other interesting thing is once you changed it back, it took a while for people's behavior to change. So it's almost
Mohan Dasannacharya (14:13) Mm-hmm. Right.
Ashley Stirrup (14:15) like a a novelty effect effect in reverse in that case. Yeah.
Mohan Dasannacharya (14:17) Yes, yes, yes, yes, very much so.
Ashley Stirrup (14:20) And so yeah, it just shows just how much richness there is in user data and truly understanding the buyer's journey and how
Mohan Dasannacharya (14:28) Very much so.
Ashley Stirrup (14:29) you can influence it and influence it in ways you never even, considered initially. So yeah.
Mohan Dasannacharya (14:34) Yes, yes, very
Mohan Dasannacharya (14:35) much so. Yeah, I think
Ashley Stirrup (14:36) Yeah.
Mohan Dasannacharya (14:36) over time we have seen that at Shuttlefly we did an experiment where there was a small error in the flow where there was a bug and we said, Okay, this shouldn't change the data that much. So but let's fix it and continue testing. When we fixed that, it actually had a
Mohan Dasannacharya (14:52) I think six or seven percent increase in the test arm. And we were like, whoa, that actually rocketed it up. And we again, similar thing. You then try to analyze the entire flow and then you start figuring out in hindsight, we should have thought of that. But we started the test, we knew the bug was there, we said, Okay, yeah, that bug shouldn't impact it that much, let's just start the test. And we
Ashley Stirrup (15:14) Yeah.
Mohan Dasannacharya (15:15) realized later like, no, we
Mohan Dasannacharya (15:16) need to get all the bugs fixed because otherwise the ripple effect can be
Mohan Dasannacharya (15:20) really harsh So so yeah.
Ashley Stirrup (15:22) Yeah. That's that's another great example. And so
Ashley Stirrup (15:26) as you're designing new experiments, how do you think about the learning? Like how do you maximize the amount you're gonna learn from an experiment by how you design it? Or do you have any tips for the audience?
Mohan Dasannacharya (15:37) Yeah, I think one of the things that you need to keep in mind, and this is always a debate within the company, in any company, is how much experimentation should you do or how should you make sure you're doing effective experiments? and I'm sure with the audience here or the folks you've spoken to over time in your show, you'd have had people who've done a
Mohan Dasannacharya (16:00) okay, let's maximize the number of experiments. And I think we have oscillated between that at Albertsons. first we're like, okay, we need to do our power calculations, make sure everything is fixed, do only relevant experiments because of the amount of traffic we have. And then we switched to like, okay, we let's try to do as much experimentation as possible. So I think where you need to balance it is you need to build a platform that can do as much experimentation as possible.
Mohan Dasannacharya (16:24) But at the same time, be very cognizant of the fact of what you're experimenting on. Don't experiment because you have a platform that can do experimentation doesn't mean that you use the platform as as an answer. You need to do the hard work to actually say, okay, this is actually the right design to test before testing
Ashley Stirrup (16:44) Yes.
Mohan Dasannacharya (16:45) it. It's otherwise you're basically wasting your time on flat tests or
Mohan Dasannacharya (16:50) going backwards. So we have to understand how we want to go about that.
Ashley Stirrup (16:54) Yeah. Yeah. Really having a a thoughtful approach to
Mohan Dasannacharya (16:58) Yeah, so
Ashley Stirrup (16:59) what you test, how you test, what you're trying to learn, just so important.
Mohan Dasannacharya (17:03) Ex
Mohan Dasannacharya (17:03) exactly. So I think when we built our internal platform we use exposure as the metric and you also have what we call impression tracking. But if there is a component on a page, we use the exposure as the entry point for a customer into the experiment.
Mohan Dasannacharya (17:21) So a visitor, if they visit the page and the component is exposed to them as we call it exposure, but we can also look at if they actually saw it. So if it's below the fold and they scroll down, that's when they actually see it. And you can measure both. So you can actually say, okay, is it a if they see it, it's a valuable thing, but where we have placed it, it may not be a valuable thing, and those kinds of things. But those are details that you can get into. But one of things to
Mohan Dasannacharya (17:47) Check with when you're doing the experiment is what is your expectation of the experiment beforehand. Again, this is something that we were discussing with someone who was at Albertsons a while back, of how when we do the experiment, you need to make sure that you set your expectations beforehand so that you're not polluting the experiment, right? You can't be like, okay.
Mohan Dasannacharya (18:11) If you expect a 1% increase and later, it came back as like 0.1%, and you're like, this was success. No, it was not a success because you expected it to bring bring a 1% increase. It got you only a 0.1% increase. Now, if you had said it's going to be get a 0.1 and because sometimes you'll push with the numbers after and say, Okay, this is actually winning, we'll let's go with it. but
Mohan Dasannacharya (18:34) You need to understand that if it was one percent and you'd get to it in three months or three weeks, let's say, of experimentation, and you got only to point one percent, then you it's not yet the level of confidence may not be there. So those kinds of things and then where in your journey with the design you are matters. Like if you expect the entire journey or the entire design to bring a one percent and you have finished five percent.
Mohan Dasannacharya (19:00) ten percent of the design or fifteen percent of the design and you're going to do it in phases and you get one point one percent, it's fine, but then those are all details that you'll have to
Ashley Stirrup (19:08) Yeah.
Mohan Dasannacharya (19:08) get into with each experiment. So
Ashley Stirrup (19:10) Yeah, it's
Ashley Stirrup (19:11) funny how consistently people bring up this topic when I ask this question. It really just goes to show that yeah, you need to do the work before you run the experiment to really think through what's the expected behavior and then what are the metrics I expect that are gonna move, and then you run the experiment and
Ashley Stirrup (19:29) Yeah, you might see a segment that's a winner, but that might not be stat sig. You also might not see the behavior that you thought
Mohan Dasannacharya (19:37) Right.
Ashley Stirrup (19:37) was gonna deliver that result. And so then that raises a whole nother set of questions on what's the data really telling you? So yeah, yeah.
Mohan Dasannacharya (19:43) Yes, exactly. So you always get
Mohan Dasannacharya (19:45) a lot of learnings. It might not be the learnings you expected. so you can always change your direction and go from there. But it's always
Ashley Stirrup (19:52) Yeah. Yeah.
Mohan Dasannacharya (19:53) important to have a baseline of what you're expecting so that
Ashley Stirrup (19:57) Yeah.
Mohan Dasannacharya (19:57) then you're not not cherry picking the data and because and one of the things I think is important is to have an experimentation analyst team.
Mohan Dasannacharya (20:07) that is independent from the product person. Because
Ashley Stirrup (20:11) Yes.
Mohan Dasannacharya (20:11) very often if the product person has access to the results, they are always looking at, it's winning in this. I thought it that I thought it would lose. It's like, yeah, I yeah, we expected that. This but this one's winning. So we knew that this is this is what we want to win like. You should have said that earlier. It's just too tempting. So
Ashley Stirrup (20:28) Yeah. It's just too tempting. Yeah, it's too
Ashley Stirrup (20:33) tempting to say, No, my feature really is winning. Yes, yes.
Mohan Dasannacharya (20:36) Exactly. Actually
Mohan Dasannacharya (20:38) that was a very funny thing. It was happened like years ago with eBay. We were like eBay had a lot of traffic, so you could do a lot of tests. There were like
Ashley Stirrup (20:45) Yeah.
Mohan Dasannacharya (20:46) I think between 30 and 40 tests running on the home page at one time. And like it wasn't that all the features were exposed to all the users. And like 80% of all these tests were claming victory Like I was in the design team at the time, I'm like
Mohan Dasannacharya (20:59) Really? Are you saying that everything and win we'll just dump all this stuff onto the home page and everything is going to win? How's that going to work? you'll have a
Ashley Stirrup (21:05) Mm-hmm. Yeah.
Mohan Dasannacharya (21:07) big bunch of stuff on the home page and no one's gonna know what to do. So,
Ashley Stirrup (21:11) Right.
Mohan Dasannacharya (21:12) i I think that's where you have to like be little balanced about it and say, Okay. Yeah. So
Ashley Stirrup (21:16) Got it. So was there a learning
Ashley Stirrup (21:18) in that particular case where were the designs
Ashley Stirrup (21:20) not the experiment designs not done correctly?
Mohan Dasannacharya (21:23) No, I think the learning there actually at the time was how do we get to personalization and a level where we show the right people the right components
Ashley Stirrup (21:33) Yeah.
Mohan Dasannacharya (21:33) from a design perspective. But I don't think we were there yet at that time. So it was then at that point saying, Okay, let's figure out what's the best between all this and launch slowly and then figure out how to balance things.
Ashley Stirrup (21:42) Yeah. Yeah.
Mohan Dasannacharya (21:45) But I think now we have the capability to understand
Mohan Dasannacharya (21:48) what is a user expecting like the because of their history and things like that, you can figure out what should be exposed to them to help their journey, what is exploration that you want to expose to them, those kinds of things. I think you can do a lot more with that now, which is what I was talking about is that the components itself can change, the content can change.
Mohan Dasannacharya (22:11) So when personalization is happening, that's another thing that you have to be very cognizant of in terms of testing it, because
Ashley Stirrup (22:18) Yeah.
Mohan Dasannacharya (22:19) you can do a random pick test versus pre-pick, where you say, okay, this is the million users who are valid for this test. Now, if you're doing that, then you make sure that when you launch the feature.
Mohan Dasannacharya (22:32) that that is the same demographic that is set in your s configuration, that that's the only demographic that's going to see it. So those kinds of things. So
Ashley Stirrup (22:39) Yeah. Yeah. And so I guess
Ashley Stirrup (22:41) in that example we were just talking about for eBay, it was just so powerful being on the homepage that if you got on the homepage it was gonna be a winner.
Mohan Dasannacharya (22:49) Pretty much. I
Mohan Dasannacharya (22:50) think that that yes, but also it's not like it was a test against it was there and wasn't there. There were a lot of tests that they were modifying stuff, right?
Mohan Dasannacharya (22:59) some of the things were already on the home page and then they were just modifying it, but then they would say, Okay, I'll take this piece out and put this piece in.
Mohan Dasannacharya (23:07) we also had a learning there of how much you'd have to have a bake-in period before the test. you talked about novelty piece earlier, right? Or the reverse novelty. So you have
Ashley Stirrup (23:14) Yeah. Yeah.
Mohan Dasannacharya (23:16) to have that time for things to settle down before they actually start experiencing it. Because otherwise it's just,
Ashley Stirrup (23:22) Yeah.
Mohan Dasannacharya (23:23) this is new, what can I do with it kind of thing. So so that's always there. So
Ashley Stirrup (23:25) Right, right, right. Yeah, yeah, super interesting. So
Ashley Stirrup (23:29) as you think about the future of experimentation at Albertsons, what are what are some of the priorities?
Mohan Dasannacharya (23:34) I I think there is a lot of push towards trying to democratize it so that people can do their own experimentation. So I think that is where the challenge is going to come is how do you open it up to a broader audience to trigger their own experiments through the dashboard that they have to set up experiments, right? They can set up their own experiment.
Mohan Dasannacharya (23:54) And say, okay, this is what I want to test, but at the same time, if you open that up too much, then you have people thinking that, I can test anything, let me just test it. So you have to have that learning. and there have to be people who are gatekeeping that experimentation. So I think that's
Ashley Stirrup (24:12) Yeah.
Mohan Dasannacharya (24:12) going to become true with many places where experimentation becomes a lot easier to trigger.
Mohan Dasannacharya (24:17) and then you have to have platforms that are figuring out how to manage conflict, prevent new experiments if there is not enough data or not enough clarity, let's say. And those kinds of things will have to be how do the how do platforms handle it as we have more AI doing stuff and have agents and stuff like that that are handling some of these things.
Mohan Dasannacharya (24:39) How do you manage the balance between all those things is going to become a interesting
Ashley Stirrup (24:44) Yeah.
Mohan Dasannacharya (24:45) prospect in the future?
Ashley Stirrup (24:46) Yeah, it's gonna be really interesting to see how that evolves 'cause
Mohan Dasannacharya (24:49) Yeah.
Ashley Stirrup (24:49) you don't want people just running a whole bunch more tests but kind of losing that rigor that goes along
Mohan Dasannacharya (24:54) Yes.
Ashley Stirrup (24:55) with being thoughtful on experiment design.
Mohan Dasannacharya (24:57) and I think the other thing is how do you make sure that you're having tests run longer so that you're actually getting the output that it needs. especially when you have personalization happening with both the content and the components, then your audience to a particular test is going to reduce. So you have to increase the length of the experiment, right? Because
Ashley Stirrup (25:18) Yeah.
Mohan Dasannacharya (25:19) if there are
Mohan Dasannacharya (25:20) X variants of these these are the people who buy seafood, these are the people who buy
Ashley Stirrup (25:25) Yeah.
Mohan Dasannacharya (25:25) bake go to bakery. all those things, how do they overlap? How do you like
Ashley Stirrup (25:29) Yeah.
Mohan Dasannacharya (25:30) determine for your power calculations what is the traffic going to be beforehand so that you can make sure that the product and business understand that if you want to actually see the result of this, you'd have to run it for a few months, and you can't be like
Mohan Dasannacharya (25:44) three weeks and if they're saying, okay, we want to go to we want to hit hundred percent exposure to everyone in three weeks, then you might as well say that then just go ahead and do it without a test because you're not going
Ashley Stirrup (25:57) Yeah.
Mohan Dasannacharya (25:58) to get a true result in three weeks. So I think that's where
Ashley Stirrup (26:01) Yeah, right.
Mohan Dasannacharya (26:02) the education part will come in as we move forward.
Ashley Stirrup (26:05) Yeah, yeah, I know, I totally agree. so interesting thinking about a business like Albertsons your example on the people don't really needing to see the product detail page makes total sense 'cause they're doing repeat buying and once they know what kind of milk they want, they just want to add it to their cart. But also the digital channel it's not like e commerce where you're just trying to sell clothes through a new channel. You you know
Mohan Dasannacharya (26:24) Right. Yeah, it's a very omnichannel
Mohan Dasannacharya (26:27) it's one of the examples where omnichannel really work is going to be come to the forefront. Like
Ashley Stirrup (26:32) Yeah.
Mohan Dasannacharya (26:33) not just for like you can go and do returns in the shop. It's actually even while you're shopping for a lot of things it's going to be helpful how you do the shopping and those kinds of things.
Ashley Stirrup (26:41) Yeah. Yeah, how it
Ashley Stirrup (26:42) complements that in person experience.
Ashley Stirrup (26:45) Well, thank you
Mohan Dasannacharya (26:45) yeah.
Ashley Stirrup (26:45) so much for joining us today. This was a a fascinating episode. It was great having you on.
Mohan Dasannacharya (26:49) Thank you. Thank you. It was great being on the show.
Takeaways from this conversation

Personalization shrinks each test audience, so plan power calculations and run tests longer.

Keep the experimentation analyst team independent from the product owner to reduce cherry-picking.

A flat primary metric can hide a changed journey; track next-step conversion alongside end-to-end conversion.

Write down the expected lift before the test so a 0.1% result is not declared a win against a 1% goal.

Test the platform itself: run a constant A/A test with enough traffic, or one every 6 months without it.
Resources
Top takeaways from other favorite conversations
Before you test, ask whether the outcome is uncertain and whether you would ship it anyway

In a regulated industry, every customer must be accounted for. Even one to two percent of users missing an experience is unacceptable.

AI creative must clear guardrails for product fidelity, context and brand compliance while models keep improving.

Use AI to accelerate builds, detect incidents sooner, and evaluate models; watch MTTR and MTTD.

The experiments that fail deliver the most valuable learnings, especially when you expected a slam dunk.

Shift quality left with automated checks so developers catch issues early without human gatekeeping.

Never ship a big-bang redesign untested; a hidden loss can run for months


