Transcript

The Experimentation Edge - Vipul Tyagi Recorded September 15, 2026

Vipul Tyagi: What we had observed was that people who engaged with the loyalty program converted more. They had higher order values and their retention was higher. So, with that premise, we went ahead and looked at the entire funnel from the home page to the checkout. We ran a bunch of experiments. We figured out that bottom of the funnel was where we were getting a ton of engagement with the loyalty program. But we also had a hypothesis that if we want to focus on improving the awareness of the program, we should do something on high traffic pages like the homepage.

Ashley Stirrup: Hello and welcome to today's episode. Today I'm excited to have Vipul Tyagi, Senior Product Manager at Quill. Welcome to the show, Vipul.

Vipul Tyagi: Thank you, Ashley. Thank you for inviting me. I'm pretty excited to have a chat and learn more.

Ashley Stirrup: Yeah. Yeah, I think this is gonna be a fun episode. Why don't we kick things off with you telling us a little bit about Quill?

Vipul Tyagi: Sure. Quill is an e-commerce brand. We sell office supplies online. We are a part of the Staples ecosystem. Our primary target audience are small and medium sized businesses. And I would say we are older than Staples, and yeah, this is whenever I go out and talk to a bunch of merchants, they are pretty excited to learn about Quill and how much revenue it does, considering it's an old brand and it's just office supplies. People consider office supplies really boring. I have started to find it very interesting now, just because I do understand how businesses, or I should say smaller businesses, operate, what kind of problems these people actually face, what budget constraints they have and what are the things that they're looking for on a regular basis.

Ashley Stirrup: Yeah, and it gets quite a bit of traffic as well, the site, right?

Vipul Tyagi: Yeah, we do get about 100K per week. And yeah, these are not just new acquisitions where we are paying to get the audience. These are returning customers who have been with us for a very long period. They do understand the value of shopping with Quill. We do offer, I would say, a loyalty program which gets a lot of people excited. And we do make sure that we deliver on the promises we make.

Ashley Stirrup: Yeah. And it's kind of an interesting business model, 'cause you have a mixture of both B2B and B2C within your customer base.

Vipul Tyagi: Yes, it is kind of mixed, and this is not entirely intended, I would say. We primarily operate with three models. There is a managed service where you have an account manager. These are primarily government offices, schools, and larger organizations who need a sales manager or an account manager to take care of their needs. Then there are small and medium-sized businesses who we want to sign up as membership customers, because then they can get great pricing, which they might not find anywhere else in the industry. And we do provide them refreshers in terms of additional benefits throughout the year as part of the benefits. And then there is a larger segment which acts as top of the funnel for us, where we have a mix of small businesses and parents coming in to buy crayons. So that's the B2C side of things. It's not intentional, but yeah, it's mostly pricing driven that B2C customers end up on our website.

Ashley Stirrup: Got it. Yeah, that's interesting, 'cause I imagine a lot of those would normally go to Staples.com, but they end up at Quill as well, is what you're saying.

Vipul Tyagi: Yeah, we need to understand that B2C customers are more price sensitive. They are looking to get the best bang for their buck, and that is where sometimes we get undercut by competitors.

Ashley Stirrup: Got it. Got it. And tell us a little bit about your role.

Vipul Tyagi: I am primarily looking after the online experience, starting from the add to cart page to the checkout funnel. There is a specific focus on how do we get more people to engage with our loyalty modules within this journey, making sure that customers understand the value of the loyalty program and continue to engage with it so that they keep coming back to us.

Ashley Stirrup: Got it. And how many experiments are you running a quarter, and what's your win rate?

Vipul Tyagi: In terms of win rate, at an organization level, I think it kind of varies. But yeah, I would say I'm part of the e-commerce team. We do run about fifteen to twenty experiments pretty easily every quarter. Each month, I think we do run about six to ten.

Ashley Stirrup: Got it. And in terms of your win rate, do you have a ballpark win rate?

Vipul Tyagi: Okay. So I was just consolidating whatever work I've done in the loyalty work stream, and I was able to recollect eleven initiatives, out of which eight of them were winners.

Ashley Stirrup: That's high.

Vipul Tyagi: Yeah. So this is where I tend to follow a philosophy. So I met the founder of BillionToOne. That's a biotech firm based out of the Bay Area. The founder is a data scientist. So he knows numbers, he talks with numbers every day. During our first conversation, he said that, yeah, you should know the direction, or you should know the result of an A/B test, before you conduct it. An A/B test just tells you how much. So yeah, I think what he was trying to say was that with all the triangulation methodology and the information that you have from externally, or the data sources that you have internally, you should have very high confidence about the initiative that you plan to launch. And that is where I think a hypothesis can go wrong. But yeah, if you have a really strong hypothesis in one direction, then you get more confidence about whether it's a success or not. In terms of an A/B test, I think that an A/B test measures how much success or failure happened. It will tell you the direction of the success, but yeah, it's just the magnitude. As a product manager, you should know where the direction would be for an experiment before you even start putting requirements for it.

Ashley Stirrup: Yeah. Yeah, absolutely. Do you have an example of an experiment you've run where you had a lot of learnings? Maybe something that, you know, you had that confidence was gonna be a winner and it turned out to be a loser, and that taught you something about the buyer's journey.

Vipul Tyagi: Yes. I will try to make it as simple as possible. Imagine yourself on a website which offers a loyalty program. You get points, you can redeem those points in exchange for coupons or gifts. Now, what we had observed was that people who engaged with the loyalty program converted more. They had higher order values and their retention was higher. So, with that premise, we went ahead and looked at the entire funnel from the home page to the checkout. We ran a bunch of experiments. We figured out that bottom of the funnel was where we were getting a ton of engagement with the loyalty program. But we also had a hypothesis that if we want to focus on improving the awareness of the program, we should do something on high traffic pages like the homepage. So that's when we launched a very small, I would say, proof of concept that we wanted to conduct. We launched a widget. You might have seen those pop-ups that randomly come up on the homepage. So we launched something like that, offered rewards that the customer would be able to redeem based on the points that they had in the bank. But we did not see any engagement there. And we ran this for, I won't say that it was a very small test, we ran this for about sixty days or so. Our engagement was less than point five percent. So yeah, this is where we were wrong, that people were interested in rewards before adding any items to their cart. So that's what we realized, that a loyalty program is an icing on a cake. It's not the cake. So that was the biggest learning, that people talk about personalization, relevance and all of the good things in the world. But we need to understand what is the core objective of the user who's coming to the platform. Their objective was to find the best and cheapest paper that they could get, or the ink and toner, or the printer that they wanted to purchase. And after that comes the part: are there any additional pieces that the loyalty program can provide us? Are there any coupons that we can add, or can we order any cookies or have any Starbucks gift cards just to redeem those points? So that was the biggest learning, that visibility on the home page alone would not solve for the awareness problem. It doesn't solve the customer's problem as such. We can have tens of hypotheses around, let's put a coupon on the PDP page and just see how many people add items to cart. But yeah, till that point, these rewards are not even relevant to the customer. They become relevant once the customer has, I would say, added all the items that they wanted to purchase. And I think this goes across industries as well. You would see a lot of bundles, those modules that ask you to bundle things on the cart page and not on top of the funnel pages.

Ashley Stirrup: Yeah. I think it's such an important learning from the show. It's something I see as kind of a theme across episodes: it's so important to understand the buyer's journey and how a feature can actually hurt, or maybe just in this case have no impact, at one step in the buyer's journey. But once you move it to the right step in the process, then it can have a big impact. Is that right?

Vipul Tyagi: Yeah, I have another example here. I found out that there were about forty percent of our accounts that would come to our website regularly and would not engage with the loyalty program. These people had less than a hundred points in a steady state. And there were a lot of comments in our NPS feedback that, yeah, people wanted low point rewards. So we came up with a very simple solution. We were inspired by Lego, which offers you to convert your points into dollars on the cart page. So we replicated that experience, and suddenly we started seeing a one percent increase in conversion. So all who were having small order sizes, those people started converting way more. As we keep points on our books, we lose some of that money, because that amount is lying on the books; it becomes a problem for finance in the long term. As soon as we launched that module, we started seeing an increase in, I would say, the redemption of points. A lesser number of points were now getting accumulated on the finance books. So it was a win-win on both ends. More orders, lesser points accumulation on the books. So finance becomes less liable for that money that the customer has already paid.

Ashley Stirrup: So it's reducing the liability on your books, but also it sounds like it was increasing spend from those customers, because now they're able to access those points without having the hundred point threshold. So yeah. Yeah, it's super interesting.

Vipul Tyagi: Very simple things. It's not something that we did some rocket science. It was tying things together about what customers were saying, what data was saying, and then getting inspired by, I would say, someone who's doing it right in the industry.

Ashley Stirrup: Yeah. It's a great example of how you might, you know, it just reminds everyone to keep continuing to iterate even on things that you have pretty well established, 'cause there's always the potential to find a new unlock that will work better for customers. So let's say you were working with a newer product manager, and they had a feature they were all excited about, and you know, they thought for sure it was going to be a winner. But you know, a lot of experiments fail. And so how would you guide them to design the experiment to make sure they get the most learnings out of that experiment? Let's assume it does fail, and you want them to think about all the data they would want if it fails versus the data you want when it wins.

Vipul Tyagi: I think, yeah, the problem in here sometimes is tunnel vision. Someone who's new will have a ton of hypotheses. But when they are trying to run a hypothesis, we tend to focus on, okay, this is what we are seeing in numbers and this is what we should expect once we run the test. But when it fails, we end up being short on hypotheses for why something happened. So I think there are multiple things that I generally ask a new PM: okay, what is the customer actually saying? What kind of hypothesis are you coming up with with this specific test? And yeah, there is an order of hypotheses based on the importance of that pain point the customer is already highlighting. So I would ask them that there should be a primary hypothesis and there should be a bunch of secondary hypotheses that you should try to validate. The test will be relevant to just the primary hypothesis, but yeah, the secondary and the tertiary hypotheses do come into the picture when you're trying to question the failure or the success. And after that, I would ask them, okay, what are your guardrail metrics? Because that is something that makes people think more. Because when people are thinking about successes, they think about which metric will move in the right direction. But we end up losing sight of what can go wrong. I think that is the side of the world most product managers should be thinking about on a regular basis. If I do something, then how does that impact my overall customer experience?

Ashley Stirrup: Yeah, I think it's particularly important as you think about, okay, let's assume this experiment fails, but you still have good conviction that I believe in this feature, but this variant of it is not working. And so then you want to know, like, well, how should I iterate on it? And did people even see it? Did they engage with it but not, you know, actually convert like I expected, however the metrics are? And yeah, I had one guest talk about how he would do the opposite of, you know, like, okay, I think we need to have less text on this page, do a variant where you had more text. And even if that other variant performs worse, it tells you you can move this metric. 'Cause often a lot of things you try just don't move the metric at all. And so at least uncovering what are the things I can do that will move the metric is an important step on that journey of kind of discovery.

Vipul Tyagi: So this is something that we are running right now, and it kind of relates to what you just mentioned. You would have seen that Amazon takes credit for the shipping and handling savings that you get. What they do, they put down the $5.99, they put a strikethrough to it and add the word free to it at the end. And they try to upsell you the membership, the Prime membership. Now we were trying to replicate the same thing. And there are different variations in how you can actually showcase that $5.99. Either you can showcase that with a negative sign, you can add that to the savings, you can have a strikethrough and make it free right in front of it. Now, these are different variations of that one single line item that is part of a very important segment, which is your overall order checkout summary, when a customer is looking at, okay, this was the original price, this is the kind of savings I'm getting, and that's the subtotal, taxes, and then the net amount. Now, this is purely a mathematical play. We want to take credit for that $5.99 that is an additional value, which we weren't taking credit for in the past. So my hypothesis was that once the customer realizes that I'm getting an additional benefit, they will think about stretching their pocket. They will think about adding one more item. Now with that hypothesis, we started running the test. And right now I'm seeing mixed results. I'm not seeing an increase in order value, but I'm seeing an increase in conversions. So what it kind of tells me is that once I'm more transparent on the checkout page, the customer trusts us more.

Ashley Stirrup: Yeah.

Vipul Tyagi: Yeah. I think that's where the cart and checkout pages become really important, because if there are pricing issues on your cart and checkout, you would lose the customer's trust altogether.

Ashley Stirrup: Yeah. Yeah, it's always interesting to see which features are going to drive average order value, which ones are going to drive conversions, and kind of just understanding how the customer perceives what you're putting in front of them. So I'd imagine getting that higher conversion rate's incredibly important to Quill.

Vipul Tyagi: Yeah. More conversions without losing on order values is something that I generally aim for.

Ashley Stirrup: Yeah.

Vipul Tyagi: Because at the end of the day, if I'm able to get one more order on a hundred customers, then yeah, we've added millions right there.

Ashley Stirrup: Right. So how do you see experimentation evolving at Quill going forward?

Vipul Tyagi: I would say experimentation should be divided across different segments. There are different pieces like, okay, how do you improve a page? How do you start communicating better? Which is the copy testing experimentation, where I think a bunch of AI automation tools have come into the picture. What I have noticed is that, yeah, you give them a page, they will come up with multiple variations, run those tests automatically, and at the end of the day launch the winning variant as well. And you just end up getting that incremental revenue. I personally think it's an important piece, but it's not something that will move the needle for the business. There are strategic pieces, I would say, which come in from the customer pain points. And you definitely need to test once you're solving for a customer pain point with a new feature or a new change. That is more structured and more strategic. That's where I believe we see more step changes. And yeah, those needle mover initiatives, you end up doing just three to five in one year. But with copy testing you can do, I would say, twenty in just one month. Right, that's one piece. And another part of experimentation that has come into the picture is more on how your data is structured, how you're emitting the data for the AI models that have come into place. Right now we know ChatGPT, Gemini, Meta, all of these AI models do crawl our pages. One A/B test that we are running right now is we are adding a semantic layer to our product description pages, like FAQs, adding reviews. And yeah, we're just doing it for one category. What we plan to do is understand what is the incremental lift once we add these additional FAQs, all these new pieces, to certain pages. Does that lead to incremental conversions? So that is one part which I believe needs to be focused on more. It's moving really fast. There has to be a ton of adoption in there. Not many people have done it with focus or given it the required time.

Ashley Stirrup: Yeah, that can be a tricky one. You don't know how long it'll take before it starts to show up in the LLMs. And so are you kind of identifying two similar product lines and making the changes to one and using the other as your kind of control?

Vipul Tyagi: Yes. So I can tell you a lot more about this. We are focusing on one category, which is paper. I am first ruling out all the outliers, the about 200 SKUs that do the majority of the business, and then splitting the rest into two groups. One gets the enriched pieces and the other one does not. So we do have an organic kind of a setup where 2,000 odd SKUs have those question answers, product highlights, product details in there, but yeah, the other 2,000 do not. And I would say, even if you're not able to see a success right off the bat within 15 days, there are leading indicators that tell you if your experiment is moving in the right direction. For example, if I start getting more impressions on those 2,000 pages which were enriched, that's one leading indicator. If I start getting more clicks, that's another. After that comes the part of click-through rate. If I'm able to get more clicks from Google Shopping on those SKUs which were enriched, that is a proof point. And that tells me whether in the long run we should be following this path, or whether it's worth investing.

Ashley Stirrup: Yeah. And so have you gotten some early data on tests yet?

Vipul Tyagi: I have a little bit. I've seen impressions go up. I'm yet to see additional clicks, because these SKUs, the entire control and test, were not getting a bunch of clicks. So it's too early to suggest whether it's getting the clicks. But yeah, with impressions, I think over time we should get additional engagement from customers as well.

Ashley Stirrup: Yeah, super interesting. What about AI? Are there things that you're looking to do with AI as you move forward in experimentation?

Vipul Tyagi: Yes, so as I just mentioned, I am pushing my organization right now to invest in the copy testing tools that automate the entire process, so that product managers are not involved in, okay, do you change the CTA color, or do you change the location of one specific module from above the fold to below the fold? So those are the things that I'm asking the organization to invest in, so that we can move faster and collect all those wins, so that in one year we have a ton of money to showcase, clear ROI. That's one piece. And secondly, I'm trying to build that culture around AI adoption, so that even if you're editing PowerPoints, you should use AI just so that you learn about it. I generally go through a lot of customer feedback. Initially I was going through each and every comment. Now I'm using Claude and ChatGPT to understand all the pain points of customers and group them. And even for research, I ask Claude to go through all the websites, take screenshots of specific pages, and come back with an opinion as a conversion rate optimization specialist: tell me what are the ideas that we can try out. So it really helps in terms of doing all that research. I think before 2024, we would spend, what, two months researching a concept and then coming up with potential scenarios. Now I think all of that work can be done within seven days.

Ashley Stirrup: Yeah. There's no question you can move a lot faster. You do always have to be careful, 'cause AI can be so confident as it gives you a bad recommendation. Yeah. And that's the one thing I've found interesting in some of the work I've done with AI. It's just like you kind of run a bunch of data through it and then you realize it's lacking this context. It didn't categorize things the way I would have categorized it. And so that kind of interactive, like, how do you go back and forth with AI to get it to look at the data the way you want it to look at it, is still incredibly important.

Vipul Tyagi: Yeah, this reminds me of one of the articles that I was reading, which a Wharton professor wrote, that now the job market is actually shifting towards someone who can talk to AI, assess the output of the AI and pinpoint holes. I think that is where it is still about being specialized in what you do, but it's more about how you engage with the AI models and question their output.

Ashley Stirrup: Yeah. I really think AI can be a huge amplifier for good people who do good work. But you still need to understand the overall, you know, customer experience that you're trying to optimize, and that a feature that maybe didn't work in one place might work in another, and AI might not necessarily recognize that these are very different parts of the buyer's journey. So there are still really important roles for humans in the whole process. Well, Vipul, thank you so much for coming on the show today. I feel like we learned a lot. You have a very interesting customer journey with different types of customers. And so I really appreciate you sharing all of your learnings with us.

Vipul Tyagi: Thank you. Thanks a lot.

Ashley Stirrup: Awesome.