Transcript

The Experimentation Edge - Sapna Kashyap

Sapna Kashyap: We did have a test where we all were just watching that it's going to be win because we are going to test a localized payment method for our countries and we all as a product team, business people and we as a CRO itself as quite sure that it would work because that's all the historical data. That's what it says. That's what the learning for all of us has been that, that if we localize something per country wise it did work, but the experiment went totally on the different direction, that is the negative direction.

Ashley Stirrup: Hello and welcome to today's episode. Today I'm excited to have Sapna Kashyap, CRO Lead at VidaXL. Welcome to the show, Sapna.

Sapna Kashyap: Thank you so much, Ashley. It's really great to be on the show.

Ashley Stirrup: Yeah. And for our audience, Sapna's good enough to get up either stayed up very late or got up very early, 'cause she's calling in from India. And so I appreciate that. Thank you. Why don't we start things off by you telling us a little bit about VidaXL as a business?

Sapna Kashyap: Yeah sure, so VidaXL is an e-commerce company that is operating on 30 plus countries and the experimentation program is really data driven over here and we are all data driven mindset for the experiments. So we really take experiments seriously and we are an e-commerce company but we also have B2B segment as well.

Ashley Stirrup: Got it. And like how many different teams are running experiments there?

Sapna Kashyap: Most of the team, the product team, the development teams, if they wanted to test any features, the business people, if they have some kind of idea, any different business decision they wanted to take, they just first want to validate from the experiment and the A/B test. So all the main business stakeholders from the product team to the development team, everyone is involved in the experiments here.

Ashley Stirrup: Got it. And can you tell us a little bit about your role?

Sapna Kashyap: Yeah, sure. So I am working as a CRO Lead handling the experimentation program at VidaXL. I have handled 155 plus experiment, annual experimentation roadmap. And other than that, I take care of the quality and the governance part of all the experiments that goes live. Also we initiated the experimentation maturity program where I am working on the different projects for the maturity part of the experiments on our organization. So that's a quick bit about me.

Ashley Stirrup: Got it. And can you say a little bit more about the maturity side? Is that like infrastructure or is that like teaching people how to run more experiments better?

Sapna Kashyap: Yeah, so it's overall on the all segment. It's also about teaching people. It's also about improving the communication with all the business stakeholders. It's also about the mindset shift for the experimentation so that every business people can understand it better, can understand the importance. All the people who is coming new to us, all the people who is sitting over there and wanted to understand better. So there are different kind of projects that we are leading that make them why we should A/B test everything, we should validate before implementing it. So it's a different kind of, it's about the mindset, it's about the process, it's about the tools, so everything overall we are covering on the experimentation maturity program.

Ashley Stirrup: Got it. So that's sounds like quite an investment by VidaXL. Is there a particular executive that's kind of really helping to drive this throughout the organization?

Sapna Kashyap: I believe our business, webshop manager itself is quite data driven so he really wanted to validate everything so he's really invested in the experimentation program and also wanted to see that we as a company are growing towards the maturity level of the experimentation program because we are testing and validating everything so that is the reason that the business people are really invested in this.

Ashley Stirrup: Got it. And have you had any particularly big winners or losers?

Sapna Kashyap: We did have quite a few. I would say I can name few of them. When it comes to winner, I would just like to name one of the winner that I had on the B2B segment. It was really a higher uplift than any of the experiment I have ever led. So it was regarding one of the tests on the B2B. The data has been gathered from the analytics later on to I did a survey on that to understand the pain point from the user that why user are leaving the registration page. So the objective was to increase the account creation. And after gathering all the responses, one of the key concern was that user wanted to first understand before doing the final sign up, they wanted to understand the USP, they wanted to understand the benefit, the service, the technicalities of the business. That's what we did the test on. The hypothesis was that if we feature more subscription information on the flow on the registration, then more account creation will increase. It led to 38.7% of the uplift, that was really huge for the account creation side, and the learning was that user wanted to be confident for the last time. They wanted to be confident about what they are signing up before sign up on the registration page. It was a huge winner and yeah, the key learning was that user wanted to understand for the last time the technicalities of the business, the services better way before they go for that business.

Ashley Stirrup: And did you say 38.7%? Yeah. That is stunning.

Sapna Kashyap: Yeah, it did, it did. Yes.

Ashley Stirrup: So I would imagine that had a huge impact in terms of revenue for the business.

Sapna Kashyap: Yes it did. It's a very extremely higher impact test.

Ashley Stirrup: Yeah, can you describe the buyer's journey there a little bit? Like I assume they come to the site, they browse your products, and then you're trying to get them to sign up for a subscription. Is that right?

Sapna Kashyap: Yeah, so it is like a B2B website, the flow. It's like the user come on the landing page, then they check the pricing section for all the service details, what they would be getting in that subscription, and then the flow comes to the registration page. So that's what the user journey used to look like over there.

Ashley Stirrup: Got it. And so did they have certain concerns in particular that needed to get addressed?

Sapna Kashyap: Yes, so a lot of users what they were doing is that they were directly coming onto the registration page also a lot of times that we checked from the analytics and those users were not able to see the subscription information at the same time all the service details that we were providing on the pricing page, reason being there was a huge drop off. So when we provided the service details, when we provided the benefits, the technical details for the service, did a huge jump on the creation.

Ashley Stirrup: Got it. And so it's a great example where, on the surface you might think, I'm adding more friction to this process. But yeah.

Sapna Kashyap: Yes, yeah. I think the assumption itself was that it would be adding the cognitive overload if we just overload the page with so much of information, but we added in a way that it's not too much of information, but still it served the purpose.

Ashley Stirrup: Yeah. And that sounds like people had this like what am I really signing up for here? Like, hold on, I need to know a little bit more before I move forward. Yeah. Yeah. Yeah, I love that. Like I just think it's just so important that we're always trying to better understand both our customer and the journey they're on. And then like what are the things that will improve the journey and what are the things that might add you know unnecessary friction to the journey? In this case, it was necessary. So yeah. And so let's say you're working with a new product manager and they have a feature they're all excited about and they just can't wait to test it. But maybe they're not thinking about them, well, what if this loses? And if I haven't implemented this feature quite right, what data will I want to know in order to know how I should be iterating? Like how do you help them design an experiment to get the right information?

Sapna Kashyap: Yeah, so we always start with having the research. If there are appropriate research that has been done, even if it's coming from the product manager, we are very much research oriented. The data should be there. And if it's the data and the evidence that is saying that we should go for the, the if there is any guardrail metric that they wanted to keep an eye additionally as well that if this conversion rate will increase there is a chance that return rate can also increase. Just to keep an eye we are very much focused on the guardrail as well that we don't want to lower the value for the guardrails that we don't want to see any kind of negative change in that area. So we sit, we make them understand and if needed that these are the metrics, that these are the key KPIs that can get negative impact as a business, or we should proceed with it, or we should target the primary metric that it doesn't hamper the guardrail metric as well. So we sit with them, we create, we understand the objective, we understand the problem statement, we understand why the intent behind the hypothesis and then we go for testing it and as what the conclusion is of the result, if it's good then it is something we go for the recommendation of rollout, if it's not good then we go to understand what's the learning that we have gathered, why the experiment has not worked, we see more additional data points, the behavioral analysis as well, why the user has not behaved in the way that it is expected, to understand better the why, and then in a lot of cases we do the iteration, we do the test in a different way and then it comes to the success of the same experiment.

Ashley Stirrup: Yeah. And are there certain personas that you'll kinda look at every time, men versus women, power users versus new users, things like that.

Sapna Kashyap: Right now the main focus for us is mostly to see on the device level, sometime on the traffic level, that from where the traffic are coming up. Is it something coming up from the Google Ads side or is it something coming up directly, direct traffic. So that helps us understand the intent of the user. Also, device-wise we do a lot to... We always try to segment the experiment into the device to understand the user better, which feature or which element or UI or any changes we wanted to implement. Is it same for the mobile or desktop? Not behaving, we are not going to implement for both. Just for one or for none if it doesn't work. So mostly it's for traffic-wise we see and device-wise we see.

Ashley Stirrup: Yeah, that's such a good reminder that not only do you have to do all this work, but you really have to think about it based on the experience, right? Like a desktop experience is so different from a mobile experience.

Sapna Kashyap: Yes, indeed.

Ashley Stirrup: And speaking of failed experiments, you had an interesting run around payment methods, if I remember correctly. Yeah, can you tell us a little about that?

Sapna Kashyap: Yes, indeed I can. So we did have a test where we all were just watching that it's going to be win because we are going to test a localized payment method for our countries and we all as a product team, business people and we as a CRO itself as quite sure that it would work because that's all the historical data. That's what it says. That's what the learning for all of us has been that, that if we localize something per country wise it did work, but the experiment went totally on the different direction, that is the negative direction, and again it's quite challenging for us as a CRO to prove the point from the evidence that we gathered that you know it is not something that this payment method had worked. User is still opting for the credit card, user is still opting for the PayPal more, but not this one, they are not choosing for this one. That's when the bottleneck there was to prove the point that it went negative, but we should not ship it as well, because there was so much trust that it would be winning and it didn't. So helping everyone understand, the business stakeholders, that we should not ship with it, it's really difficult nut to crack. And then it comes to that we did cross verify different sources of the data, Stripe data, different sources of truth. And then share the data to make them understand that it is right, it's not the data that we gathered is wrong, it's just that this payment method has not worked for us and whatever we have learned is that not every time something if we localize is going to win. It's not true.

Ashley Stirrup: Yeah. Yeah. I mean it's just so intuitive that okay. And you're selling into what, twenty or thirty countries, I think?

Sapna Kashyap: Yeah, 30 plus countries.

Ashley Stirrup: 30 plus countries. And so just naturally, yeah, the more we can localize, the better the performance should be. But in this case, people prefer the more international payment standards like PayPal and and credit card to the local payment method options. Yeah.

Sapna Kashyap: Yeah, indeed.

Ashley Stirrup: And so it actually, you know, it's a good thing you did test it, otherwise it would have cost you a lot of business. Yeah.

Sapna Kashyap: Yes, and it just, yeah, I think for business people itself it's just more like to implement, but still as the culture that we have to validate first, that's why it came for validation and then the result was like this.

Ashley Stirrup: Yeah. Yes. Yes. Yeah. So interesting. And in this case you had a lot of trust that you had to kind of build up because people were so sure that it was a winner that like the data's gotta be wrong, right? And so you really had to do a lot of analysis around it. Yeah, yeah.

Sapna Kashyap: Yes, we really have to dive a lot in this case, but at the end we prove the point from the data.

Ashley Stirrup: Yeah. Yeah. I would imagine that's like a kind of watershed moment, because when you get the most scrutiny, that's your opportunity to build the most trust. And then when you find something as surprising as that and people eventually accept the fact that, you know, it is actually right, then it helps them realize, wow, you know, there's a lot of money at stake with some of the things we're doing. And if we're not testing, we're risking more than we thought. Yeah. Yeah.

Sapna Kashyap: Yeah, yes.

Ashley Stirrup: Yeah, super interesting.

Sapna Kashyap: I think it's an awakening moment for all of us. It you know, just revised everyone that why we test. At the end of the day, it just reminded everyone again that why we should validate.

Ashley Stirrup: Yeah. Yeah. It's so interesting that probably the biggest challenge is to the experimentation program or also the biggest opportunities to like build the culture. So yeah, it's a great story.

Sapna Kashyap: Yes. Yeah.

Ashley Stirrup: Yeah. And so how do you see experimentation evolving at VidaXL?

Sapna Kashyap: I think it has been evolved. It's been five, seven years that VidaXL has started testing, started the A/B testing culture itself. And now it is at a point that we are maturing as a team. We are maturing as an organization in the experimentation part. And we did evolve a lot. We learned a lot from our own experiments. We learned from our own process and then also the mindset shift is something I would say it has evolved tremendously coming up from all the experiment result. It just like there's a saying that most of the experiment is not going to win, but it is a learning, that's why we are going to test it. So that's a biggest learning for all of us on our organization levels, all the people sitting at the business level as well.

Ashley Stirrup: Yeah. And when you say that the team has learned a lot over time, like what kinds of learnings are those? Are those more like how to design a good experiment or has it been more about like how to run experiments or more about just your users in general?

Sapna Kashyap: I would say it's more about how to design an experiment first thing, and again second part is how to gather better data, how to also ensure the quality is up to the mark, because that is really, really important to get the correct analysis for an experiment. So yeah, I think it's more for experimental design. We did our tweaked a lot. Now we are at a level we are still learning, we are still evolving, but still we are, we have found our ground. So it is really good that for us it is working now. We learned that it is very much important for an A/B test, for an experiment, that it is everything is as clear as possible before we run an experiment, otherwise we won't be able to gather the data that we are intended to. So it's like 70% of the work for an experiment is going to be done before it ship, before it goes live, and then there's a part of the analysis part that is just left. Yeah, most of the work we need to do before the experiment goes live.

Ashley Stirrup: Yeah. Yeah, it makes a lot of sense. That the better you design the experiment, you make sure you're capturing the right information, make sure you understand the journey ahead of time so that you've actually designed the feature correctly. Means you're gonna have to, you have a lot less work to do on the back end, less guessing about well it didn't work, why didn't it work and things like that. So yeah.

Sapna Kashyap: Yes, indeed.

Ashley Stirrup: Well terrific. Well, so it's been so great to have you on the show. I feel like you gave us some excellent lessons that can be applied to any experimentation program. So thank you so much.

Sapna Kashyap: It's great to be in this show and it's really nice to have the conversation with you as well.

Ashley Stirrup: Yes, thank you so much.