Transcript

The Experimentation Edge - Vijay Lal ===

Vijay Lal: [00:00:00] you fail fast. Once you fail fast, you learn from it and get back to from where you started and make this product better, even better than what you have imagined before

Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Sturrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions.

This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest.

Ashley Stirrup: Hello, and welcome to today's show. I'm excited to have Vijay Lal head of experimentation with U.S. Bank. Vijay, welcome to the show.

Vijay Lal: Thank you, Ashley

Ashley Stirrup: Yeah, it's great to have you here. You've got an extensive experience in experimentation dating back to your time at Comcast.

Is that right?

Vijay Lal: My pleasure. Yeah, it is

Ashley Stirrup: [00:01:00] Yeah. So yeah maybe you could kick things off by just telling us a little bit about your career and your time in experimentation.

Vijay Lal: Sure. Digital experimentation and experiences is a modern as well as very vast area. And I started working in experimentation platform, enabling better customer experiences from content as well as experimentation standpoint in 2016 when I started working at Comcast. And my motivation behind that was I was working on a redesign project for MassMutual.

It was not a responsive webpage, so I redesigned their project, and at that time, I realized that customer experience is a area where I would like to pursue my career and see myself growing. Yeah

Ashley Stirrup: Got it. And Comcast is a huge company. Were you working up more on the getting people to sign up for internet and cable and things like that?

Vijay Lal: That's right. I initiated my career [00:02:00] there for customers and when I say customer, those are prospect. And for sales and marketing team, I was working very closely with our marketing partners and several product managers to enable customer experiences for new customers

Ashley Stirrup: Got it. Sounds like you were dealing with a ton of traffic and a, a ton of potential customers

Vijay Lal: ~Traffic was huge. ~Traffic was huge. And at that time, I don't think the platform was able to support that volume of traffic. But we upgraded our system and we signed up with Adobe, with Adobe Analytics and Adobe Test & Target and some part of audience segmentation also we have implemented there.

Yeah

Ashley Stirrup: Yeah. Got it. And so then at some point you decided to transition from there into the financial services industry

Vijay Lal: Yeah. It was a great working experience at Comcast for close to six and a half years, and then I found that amazing opportunity at US Bank, which was also in similar kind of role, [00:03:00] but it was a different industry. Financial industry is way different than telecom and retail. Yeah

Ashley Stirrup: Yeah. Yeah, I'll bet. Yeah, for sure. Anytime you're... It's obviously more regulated, and anytime you're dealing with money that's something you don't want to mess up.

Vijay Lal: It is. It is, yeah

Ashley Stirrup: So how does e-experimentation at, at U.S. Bank work? Is it a centralized function that supports other teams or are data scientists embedded in the teams?

How does that work?

Vijay Lal: So ~it is it is~ combination of enabling product and ensuring marketing team are self-serve. That means there may be several marketing partners are there who want to run those experiments, and I and my team might not be able to cater their needs because the volume of experimentation may be huge.

So came with an idea that why not make this platform self-serve that anyone who does not know a thing, anything about technology, they can start using those platforms and run those experiments for customers

Ashley Stirrup: [00:04:00] Got it. And what were some of the challenges you had to overcome with empowering a, you know, broad team, people who aren't, probably aren't doing experimentation every day?

Vijay Lal: Simplifying complex problem. That means experimentation platform is is not a simple platform. Like you say that Adobe has a experimentation platform tool which is Adobe Test & Target. UI is friendly, but when it comes to executing experiments using those platform and tool, it might become a little complex for people who don't know or who are not proficient with the technology.

So making things easier for them to understand and also training them because technology evolves. Every month you see some new technologies there, some of the upgrades to the platform, upgrades to the tools are there. Keeping them aware on the changes and how to navigate in those ambiguity

Ashley Stirrup: Yeah. And I would imagine with different teams, just making sure everybody's technology all worked together and people were capturing data [00:05:00] correctly and all that, there's kind of a technical side to it as well as more of a kind of experimentation data science side to it.

Vijay Lal: ~Correct. ~Correct. That's right. Yeah

Ashley Stirrup: Yeah. Is there an example of an experiment you ran where you had a lot of learnings?

Vijay Lal: Learnings. Every experiment gives some learning to the platform team, engineering team or marketing team. So I can take an example where I was working on login widget, experimentation on login widget. So US Bank had a login widget which loads on the page after the page is fully loaded. And we came up with a use case where we want this login widget to be embedded on the page rather than waiting for page to fully load and after that show up the login widget we wanted it should load along with the page.

And when we were running that experiment or designing those experiments with our developers, engineers, that particular component is is very secured component because you are letting customer to enter their user ID and password. So [00:06:00] it is a separate component. And when we designed that component as a experience fragment which was coming from AEM we came across several challenges.

So front-end wise it was loading properly, it was displaying properly for users. But we wanted to ensure these experiences are delivered correctly to all users. Every customer matters. It is even if five percent of the customers are not able to see those experiences, that is a big deal. We started looking into all of the metric in our lower environment, and we found that some of the customers who have who are coming from slower internet speed or network challenges may or might be there and for those customers, it might not load properly.

So for that challenge, we came up with an innovative solution. It was a simple solution, but the implementation of that solution again takes a leap and came up with an idea that we will put those customers for whom these pages does not load within two seconds to a fallback. [00:07:00] Fallback means the front-end will look as same as the experimentation front-end, but the login widget is going to be displayed a little differently.

That means each and every customer has been accounted for that. So this was one of the example where working with multiple technology partners, product managers, marketers, and making them aware that what is this experiment is about, what is this fallback scenario, how those customers are going to be part of control or challenger.

Those are the part of the experimentation there.

Ashley Stirrup: Yeah. It's a good reminder that when you're running an experiment, you've got a wide range of users and some might be on mobile, some might be on desktop, some might have really fast internet, and others really slow. And often a best practice is not just looking at your overall performance, but breaking things down into deciles and seeing, what is the user behavior?

Am I getting the desired user behavior, in each of those deciles, and how does it deteriorate or not as you, [00:08:00] whether it's you're looking at performance or device type or things like that. ~S-~

Vijay Lal: Got it. Yeah

Ashley Stirrup: is that the kind of thing that you would try to make sure that the different teams were looking at as they ran each experiment?

Vijay Lal: Yeah, as well as security. We wanted to make sure if a page is loading and login widget is loading, that it has not been manipulated by a third party which might not be authorized to make those modifications. So security was one of the major concern, and we wanted to make sure every customers they are being accounted for.

And even if there are slippage of one percent, two percent, those are not acceptable. And it was high visible project where leadership and customers and customer satisfaction, those were important factors for that experiment

Ashley Stirrup: Yeah. Yeah, it's so interesting when you're dealing with a challenge like that 'cause obviously in banking you need the absolute highest level of security, but you also still wanna deliver a great user experience, and so how do you balance those trade-offs?

Vijay Lal: That's right.

Ashley Stirrup: that'd be pretty challenging.

Vijay Lal: Yeah

Ashley Stirrup: [00:09:00] Yeah. And in general, like how did you help empower the teams to share best practices, learn, get the most learning out of an individual experiment, things like that?

Vijay Lal: So documentation on the retrospective, it is one of the critical factor which is been ignored in most of the organization. But I think these days companies and their people, their product team, they understand what is the significance of those documentation part. Because once you learn it and until unless it is not documented, it was never encountered, you can say that.

So documenting each and every learning from experiment and implementing those learnings on the future experiment, that was one of the important key factor. And making sure technology understands that we encountered this first time when we were new to the navigation, but now we have been navigated to that village before, and we want to make sure that if we are stepping there again, we [00:10:00] don't come across that problem

Ashley Stirrup: Yeah. I think it's such an interesting topic. Like there's, how do you store learnings? How do you share them, get people to benefit from them both in the moment oh, if that worked for this product, how do we apply it to other products? But also, how do you remember it three years later when you're gonna do something similar and you may have, maybe nobody who worked on that project's, part of this new project.

So how do you make sure everybody accesses that information and leverages it in the best ways possible? I think it's an exciting opportunity for AI to help, find new ways to unlock that kind of value.

Vijay Lal: That's right. That's right

Ashley Stirrup: Yeah. ~What do you, what do you-- ~where do you see people investing in AI today?

Where do you think are some of the biggest opportunities to apply AI to experimentation?

Vijay Lal: I think when I use AI, I feel it has made life easier in terms of saving time. It does not do anything rocket science for the people when it comes to experimentation, but it saves a lot of time. [00:11:00] Because when we run experiment, it is test and learn, we call it. And when we test it, it takes its own time to reach those significance because customers, they are landing on those pages as per their need.

But once this experiment is concluded, we got the statistical significance. After that, the analysis of experiment, it might be time-consuming. So when we know there are winners or there is no significant change in the experiences it is easier for us to understand using the AI features that which experience is a winner, whether experience was performing really well or not.

And all those calculations has been made easier by utilizing AI capability

Ashley Stirrup: Yeah. Yeah. I really see there's kind of multiple paths of value for AI, and I think that when you can add AI to automation, you can really remove a lot of friction from the experimentation process that might have previously required a human in the loop. And now, you can automate those kind of lower [00:12:00] value, tedious things.

And then I think there's just so many exciting opportunities to unlock more insight from experiments or maybe combine data from research and surveys and things like that to ~un- ~unlock new insights. So super exciting time for experimentation that way.

Vijay Lal: Right. And there is another factor as well, because AI itself declares that AI can make mistake. You need to be mindful that, okay, whatever AI is saying, is it correct or not? And when you use AI to generate any content for experimentation before experimentation goes live, that also needs to be validated and human-in-loop should be always accounted for when using AI

Ashley Stirrup: Yeah. It's so interesting as AI is evolving, 'cause, a year ago I'd say it hallucinated a lot. And it still hallucinates today, but it's a lot more trustworthy. But that risk of, ~the, a, a hallucination~ sneaking in and then causing you to come to the wrong conclusion, it's definitely something people have to guard against. [00:13:00] Yeah. As you start working with a new product manager, maybe somebody who's, hasn't had as much experience with experimentation, what kind of guidance do you give them in terms of, how to get the most out of an experimentation program?

Vijay Lal: I think product managers should understand they are designing product for customers, not for themselves. And these customers should be internal customer or it could be external customers. So when any new product manager designs any product, they need to keep that thing in mind that they are developing it for someone, they're building it for someone, how this other customers is going to use it.

If it is internal customer, whether this customer might be able to like it or not. A product manager might like its product for sure. It is his baby. They have spent so much time in developing this product. But they need to think from end user perspective, making sure the product which they have been developing they are not building this product as a full product in the beginning.

Make [00:14:00] sure it is going as a release for MVP. And once MVP is accepted, they know that what changes needs to be done in post MVP. And post MVP product should be way much better than your MVP product. Yeah

Ashley Stirrup: Yeah, I think it's really interesting that a ~l- a ~lot of product managers end up being too conservative and maybe if they're in an organization where the culture doesn't recognize the value of losing and that, your win rate isn't gonna be, anywhere near 100%, and so you should be trying a lot of things and doing creative things.

And so you wanna do creative things, but then you find a way to test them with the least amount of effort as possible. How can you use a painted door or, a partial feature to test whether there's really some interest in that? And so that's a, a really interesting balance that I think a lot of product managers have to think through is how do I think big and test small?

Vijay Lal: And be fearless

Ashley Stirrup: Yeah. Yeah, absolutely. That, that comes back to the [00:15:00] risk-taking side of it.

Vijay Lal: Right, you fail fast. Once you fail fast, you learn from it and get back to from where you started and make this product better, even better than what you have imagined before

Ashley Stirrup: Yeah. Yeah. It's a combination of being fearless so you can try lots of things 'cause you never know what's gonna work, as well as being willing to iterate on a loser. Maybe lose a few times in a row, but you keep extracting more learnings as you go.

Vijay Lal: Yeah

Ashley Stirrup: Yeah. Yeah, absolutely. At U.S. Bank how do you think about getting alignment across teams and kind of north star metrics?

Does every team have a, a couple of metrics that they're aligned on, or does it really vary from team to team?

Vijay Lal: In my opinion, metrics should not be considered in a silo. every experiment or every idea should have an hypothesis, and based upon those hypothesis, product managers or marketing team, they should develop those metric. And it should not be just one metric. It can be one primary metric will be there that you want the customers to engage on the page on [00:16:00] the first visit, but there might be secondary KPI that when this customer is coming back to this page or this website or this product at later time, they are still engaged.

So it is combination of primary KPI, secondary KPI, but it should be driven by hypothesis, not by a leader wants to run some experiment or develop a product and this is the KPI. That would be traditional approach. Modern approach would be based on the hypothesis of the experiment

Ashley Stirrup: Yeah, and one of the things I hear a lot is that you might have a ~l- a ~North Star metric of, say, growing revenue, but~ the-- you also wanna test~ the thing that is closest to the feature so that, you're actually measuring the impact of the feature as opposed to, the further away you get, the more noise you introduce, more other things might be affecting the actual performance.

And so thinking through that connection between what is this feature doing, what behavior, immediate behavior do I expect it to result in, and do I see that? And [00:17:00] then what long-term behavior, which is probably the North Star metric, and how do I see if, okay, if I increase clicks or time on page or engagement, does that actually lead to more revenue down the road?

And does this new feature actually increase engagement? So

Vijay Lal: Absolutely. And it's good thing that you brought up this KPI point because data analysts, they know that why it is happening, but they might not know that why these the reasons behind those incidents. Like customers are rage-clicking on the page. There may be reason behind it, but customer is expecting something.

That button is disabled, and customer wants to see that why it is not been clickable. But data analyst only knows that they are getting loss in the revenue by customers not being engaged with that button. Yeah.

Ashley Stirrup: Yeah. Yeah, especially when you've got, different customers, different platforms, all that, that maybe the product's working great on most platforms, but not so great on an Android phone or something [00:18:00] like that. And so how can you identify that maybe the experience is degraded there? So yeah, that just really reinforces the whole point of, as you run an experiment, you want to identify your North Star metric, maybe your, the key goal metric for that experiment, but also have a lot of guardrails so that you can understand the impact it's having on a variety of different segments.

Vijay Lal: Absolutely

Ashley Stirrup: Yeah. And so how do you see experimentation evolving over time?

Vijay Lal: Yeah, there are good opportunities the way people or company run experiment today. Because when I see experimentation, they are being running traditionally as rule-based. That means if certain criterias are met based on customer's past experience do something. And it changes, it evolves.

Customer might be thinking differently on day one, but on day twentieth, customer wants to do something else. So real-time personalization is one of [00:19:00] the evolving area on the experiment. And when it comes to personalization, it should be real-time personalization. That means we know about this customer for last thirty days, but in la-- next ten days, customer might change their behavior.

And based on those behavior, the profile which platform has built, they should update those profile and start utilizing real-time customer behavior. So I think personalization with real-time customer profile, that is going to be evolving and also some of the agentic AI flow, which is which is been done manually today.

That might also change the ecosystem

Ashley Stirrup: Yeah. Yeah, it's really interesting to think about the intersection of personalization and LLM-powered features, a chatbot or what have you, and that, how do you successfully combine those two, and then how do you test those two? Definitely requires, a lot of data, a lot of users, and the right metrics. Yeah. Yeah, and how do you see things [00:20:00] evolving? A, a common topic is that a lot of people are looking for ways to even further democratize experimentation, make it available to more people in more ways. How do you see that evolving?

Vijay Lal: Yeah. So when you say democratizing experimentation, it could be again your internal team technology partners as well as the marketing partners. So make it available to them, make it a framework and the platform self-serve so that people and the technologists or marketers, they understand this product, and they should be able to use it fearlessly.

Also when these capabilities are given, it comes with responsibility because you got some power to put anything in production environment, which is customer-facing. But those also should be guided by your guardrail metric, making sure whatever is going in production, even if someone does not know about technology, they should know that what are the implications of these changes.

So I think that will be the one of the area [00:21:00] where self-serve along with acquaintance and knowledge that what will the implication will be biggest factor.

Ashley Stirrup: Yeah. Yeah, I definitely whenever you talk about things like that, ~y- I can't help but think~ about just how you create that learning culture across the organization 'cause ideally something gets learned in one place, other people, see that information. Maybe they have a different perspective on the user and they help you uncover additional insights.

But also just making sure you're bringing the whole organization along with the right ways to test 'cause we recently had Ronny Kohavi on a webinar at GrowthBook and he was talking about running trustworthy experiments and it's just, it's amazing how many ways you can run a test poorly and think you've got data that causes you to go in a certain direction when it's actually not trustworthy data.

And so the, just I really took away from that just how important it is to be, making sure you've got the right rigor with your experimentation program.

Vijay Lal: [00:22:00] Yeah, I totally agree with that

Ashley Stirrup: Yeah. Yeah, he was talking about at his time at Bing where they grew revenues significantly, but it was just by stacking small win after small win.

And if you're lacking that rigor, that makes it impossible to stack wins over time. Yeah. Vijay thank you so much for being on today's show. ~I-- You were a terrific guest,~ and I feel like we learned a lot, so thank you for everything you shared.

Vijay Lal: My pleasure, Ashley. It was nice talking to you sharing my thoughts and also getting to know your perspective gives a little broader area to discover in future.

Ashley Stirrup: Terrific. Thank you so much

Vijay Lal: Thank you. Nice talking to you.