Transcript

The Experimentation Edge - Dorothy Crepin

Ashley Stirrup: Hello and welcome to today's episode. I'm excited to have Dorothy Crepin, Senior A/B Testing Analyst at MilliporeSigma. Dorothy, welcome to the show.

Dorothy Crepin: Thank you so much. Thanks for having me.

Ashley Stirrup: Yeah, excited to have you on. Maybe we could start with you telling us a little bit about MilliporeSigma.

Dorothy Crepin: Sure, so I've been with the company since 2022. We have three divisions of the company, healthcare, electronics, and life sciences. I'm in the life sciences division. The company is known as MilliporeSigma in North America, but we are a business of Merck KGaA in Darmstadt, Germany. So for the rest of the world, we're known as Merck. We in the life sciences division have a pretty wide portfolio in terms of products that we carry, everything from different workflows to systems, chemicals, kind of anything you'd need to do any type of experiment or scientific process.

Ashley Stirrup: And who are your customers?

Dorothy Crepin: It's largely a B2B business, which was really a pretty big shift for me. We have some larger companies and organizations. It's throughout the world, so it's a global company. We cater to some smaller startups as well. So a little bit of everything, but largely B2B.

Ashley Stirrup: Got it. And do you have a lot of traffic?

Dorothy Crepin: We do, quite a bit of traffic on the site, but kind of the interesting thing about the traffic is because these are B2B organizations, not everyone that's coming to our site has purchasing power. So we have some people come to our site that are just there for their procurement role and they're simply working off a list and purchasing the products that were given to them by the scientist. Then we have the scientists that are coming that are looking for specific instruments and materials for their experiments. And then we also have a pretty wide education population. So people who are from universities and working towards degrees and are looking for more information and learning materials.

Ashley Stirrup: Got it. Yeah, so that makes the persona particularly important here. I mean a lot of industries tailor to personas, but this is kind of next level in that regard.

Dorothy Crepin: Certainly, and I think especially in a very regulated industry, sometimes not even from role to role, but also from country to country. There's certain materials that we can sell in certain countries or can't sell in others, certain ways, even the way that we present our company name across the globe is different. So it's been a really unique experience.

Ashley Stirrup: Yeah, that is interesting. And can you tell us a little bit about your role there?

Dorothy Crepin: Sure, so I am the Senior A/B Testing Analyst, as you said. I'm sort of an air traffic coordinator for the testing program. So I work with a variety of teams, including our product managers, our analysts, which is the team that I sit in, and really coordinate the experiments from inception to execution and the analysis afterwards. Just managing all the processes.

Ashley Stirrup: Got it. And so do you work with a pretty wide number of teams?

Dorothy Crepin: I do. And I think that's one of the great things about my job. I really love project managing and really connecting the dots for different people. It's kind of fun to get to work with the developers who are very linear, to the design team that thinks about things as a whole design, to the product teams that are thinking from the point of view of the customer. It's really interesting trying to streamline all that communication so that it makes sense to all the groups. I think it's something we're still working on internally, but it's been really neat to find out.

Ashley Stirrup: Yeah, I think it's really an industry wide challenge. So that's why I think it's one of the more interesting topics for the podcast. And so, how many different teams would you say you work with?

Dorothy Crepin: So our company is kind of working to become more of a product organization, specifically in our life sciences division. And so with that we have these product pods. And on the pod, you'll have a product manager, owner, you'll have someone from our UX design team. You'll oftentimes have a front-end or back-end engineer. And so they're all focused on one single product on our site, but all from different teams within the company. So I would say probably three or four different teams pretty regularly, but we have our site kind of divided into pods to fit with that structure.

Ashley Stirrup: Makes sense. And how many experiments are you running a quarter?

Dorothy Crepin: Right now our goal is to get closer to 12. That number's been varying quarter by quarter because I think especially once you start up an experimentation program, you kind of start with some of that low-hanging fruit that's maybe a little bit quicker to get to. And then I think like any organization, as you start to get into the meat of the work, it takes a little bit longer. We also have kind of played around with how we have our resources sitting and what the focus is. I feel very supported in my organization. Our leadership is very passionate about making testing a big part of the process. So I'm loving the fact that we're starting to devote more attention to that and make it a part of the actual product development lifecycle, not just something that comes in at the end.

Ashley Stirrup: That's great. And what led your leadership team, like is this new leaders that came in that decided this was important, or was there a trigger?

Dorothy Crepin: You know, I don't know that there was necessarily a trigger. I would say when I came into the company, the A/B testing program had just started, but it was largely used for on-site advertising. And when I came in, that passion was just kind of already there. I think some of our leadership had come from prior companies where they had very robust testing programs. So that sort of methodology was already ingrained into them with how they saw the product development process flowing. But I think everyone's kinda got an ear to testing right now, especially with everything coming out with AI and agentic testing. So I think we're just fortunate that we have leadership that isn't necessarily sticking with what they've always done. They're watching the news and watching what's coming out in the industry and it's just natural progression.

Ashley Stirrup: That's great. You know, I think one of the big unlocks for most experimentation programs is when someone in the organization realizes, wow, we are a lot worse at knowing what is actually gonna work than we thought. And once you have that passionate leader who's showing humility and recognizing that we aren't always gonna ship winners and all those types of things, they set the tone for the whole organization. So that sounds like you've got a really important piece of the puzzle there.

Dorothy Crepin: Definitely. And I think it's something that we're still working towards. I think our program is pretty sophisticated in terms of structuring and managing the process flow and how we look at experimentation and measure. I think we're still trying to get that ingrained process as to bringing up experimentation as kind of a first line of, is this the right thing to do for the customer? That, and in combination with some of our user testing, I think can be such a powerful tool that we're still working on unlocking.

Ashley Stirrup: Yeah, I really agree with that. User testing is such a valuable window into really understanding the customer journey.

Dorothy Crepin: Exactly. And I think one thing that we have struggled with is, the A/B testing is going to tell you what happened. It's going to show you the data and the numbers, but that user testing is really what we need for some of that nuanced why of why did they make that choice? Why didn't they click that button? And I think our users are, I don't want to say it's a niche industry, but it's not a particular consumer. They have to have some sort of area knowledge or expertise. And so I think our user testing group has developed a really great kind of ambassador panel that they've been able to pose those questions to rather than just the general public, which has really helped us a lot.

Ashley Stirrup: Yeah, that's terrific. Do you have an example of an experiment you've run where you've had a lot of learnings?

Dorothy Crepin: I think so far we have had experiments that we've had learnings from, but maybe haven't produced the metrics that we thought they would. And then we've had some that have been very successful. So we recently ran a test that was an autocomplete on our search function on the site. We found that a lot of customers will utilize a Google search or some other external search to get to our site, but they weren't utilizing our on-site search to the fullest extent. So one change that we made was this type ahead function, which originally, when you would start typing, would recommend a product that matched your search query. The team altered that to match a search term rather than forcing the customer into a product and kind of risking getting that wrong. And we actually saw double-digit increases in feature usage and also add-to-cart from those searches. I think for me, I was surprised that such a small change that seemed to sort of stray away from direct e-commerce of this is the product that we sell for your solution was really interesting to see. And I think there's some room for iteration there as far as how quickly we show them options, how literal we take their search, and just all kinds of avenues that we could go with.

Ashley Stirrup: Yeah, that's really interesting. I used to work at a company called Algolia and they provide search tools for companies like yourself, and yeah, type ahead was a big feature that the customers loved. And in this case it sounds like you learned some things about the buyer's journey, that maybe people didn't always want to go to the product, that there were other areas of your site that had important content for them.

Dorothy Crepin: I think so, and I think it also got us started to think about personas as well, because we probably do have the users on there that are maybe from the procurement side and they have a very specific product, they just need to find it. They're not looking for other options, they're looking for the product that was given to them on this list. But then we have some people that may be at the beginning stages of their research and are looking for the right product for their workflow, and to really pigeonhole them into the product that we think they're talking about, I think was a little bit limiting. So I'm kind of wondering what we're going to see. Do we see an expansion of the breadth of product that are being discovered and chosen ultimately? I think it'll be interesting.

Ashley Stirrup: Yeah, that does sound super interesting. I think I'd have a lot of fun with all the analytics around that. So let's say you're working with a new product manager and they've got a feature they're all excited about and they're just sure it's gonna be a win and so they can't wait to test it. And maybe you wanna sit down, explain to them that not all tests are winners, and if it's a loser, let's make sure we design it so we get the most learning from it. How would you help them do that?

Dorothy Crepin: So I think the first thing starts at inception of the test. Really asking the product manager, what problem are you solving for the customer? Originally when I started working in the program, we looked at creating a hypothesis off the bat, but I think even that was too far in the funnel. So we've started just asking, what problem are you trying to solve, and what does the solution look like? What will it look like when it's complete? I think also setting expectations on how we are measuring success is big. So I listened to a webinar maybe 18 months ago where they were talking about revenue specifically, and everyone focuses on revenue and our organization does as well at the end of the day. It's what stakeholders are measured on, and it's obviously very important. But revenue is an outcome. And so what we really want to look at are the metrics that lead to the outcome of revenue. Ultimately, we do still focus on attributing revenue to experiments, but what is the next action that you're trying to get a user to take? Because not every page on the site is responsible for conversion. For example, if we're doing a test on a product detail page, the cart is really responsible for conversion at that point. The PDP is responsible for getting the user to add the product to cart. It's more product discovery at that point. So I think really setting clear intentions of what the metrics are and how we're going to measure. Then I think for us, it's also really important to understand interactions of other experiments. We have some teams that very quickly complete work, they're ready to roll experiments out, and we have four or five that are ready to go, but they're all interacting in the same area. So then it's kind of a matter of, how can we make those tests mutually exclusive? Do we have enough audience for that? And is it going to produce a meaningful change that we can actually measure?

Ashley Stirrup: Yeah, those are some great answers. One of the things that I particularly like is just making sure you're thinking through the, okay, what is this feature gonna do? What are the metrics that might move if this feature accomplishes what it's trying to accomplish? And then also doing multiple variations. And so I had one guest on who says they always look for, can we do the opposite of what we're doing here.

Dorothy Crepin: I love that.

Ashley Stirrup: So if we're making, trying to remove words from this page, let's also do one where we add words. And he said that a surprising amount of times the opposite of what they were doing won, versus what they thought was gonna be the winner. And so I think those are the kinds of things you can do that really help you get to what you were saying earlier, which is the why. Okay, this failed, but why did it fail and how can we, what are the elements? We had another guest that talked about doing the opposite for a different reason though, which is, can we at least move the metric? Like even if this moves it the wrong way, that tells us, okay, we can move it, because so many of the things they were testing just didn't move the metrics at all in either direction.

Dorothy Crepin: And I think we're working too, we have an incredible data team, data science and analytics, that is extremely bright. We have so many talented people on the team that are making dashboards and different self service tools for our product managers. So I think just as important as having the metrics that you're going to measure are coming in with metrics too. So can we ground this customer problem or this customer opportunity in data? Is there a reason or is this just a gut feeling? And not that a gut feeling isn't something bad to act on necessarily, but I love when we've got a product manager that's coming to me with data points specifically saying, this is a proven problem, not just something that I think.

Ashley Stirrup: Yeah. And I would imagine, given what you've told us already, that really breaking down the experiment results by these different personas that you have is super important for every experiment.

Dorothy Crepin: Yes, I don't think we're quite to that point yet, but I think we have a vision of what it's going to look like. We've started trying to make our experiments more targeted in terms of who we're letting into the test, whether that's geographically or persona. But I would love to get to a point where we are testing on specific personas and really starting to personalize the journey. There's obviously a big risk there of, if you get the persona wrong, do you risk not showing them product or showing them too much? But I think there's a lot of upside there too.

Ashley Stirrup: Right. Yeah, I actually think that's one of the biggest opportunities for AI. Obviously with AI, every development team can move faster. But in so many cases, even before AI, ninety percent of the features people were releasing nobody was using. And so it's less about delivering more features, but more, can you tailor the most important features to your different personas?

Dorothy Crepin: Agree.

Ashley Stirrup: So how do you see experimentation evolving?

Dorothy Crepin: I think you kind of hit the nail on the head. I think everyone's already talking about how AI is going to affect experimentation. I'm starting to look specifically into agentic testing, looking at can we build some of those personas into an agent and maybe not articulate exact effects, but can we get some sort of directional preview of an experiment? Particularly since we have so few people who are going through the purchase process, but we have a lot of people in the browsing process. How can we test on those users in a meaningful way and make those connections to revenue ultimately and to that outcome? So I'm really excited for that. I'm hoping that my team will start to focus on that here in the next year or two. I've also just seen a lot of conversation around the way that tests are evaluated. I know our team specifically has been looking at different models and methodologies. And does that need to be unique to every experiment depending on the metrics that we're using and where it is being tested on the site? It's kind of becoming less of a one size fits all where someone isn't just going with a Bayesian methodology across their entire program. It seems to be becoming bespoke, and I'm noticing that as well in experimentation providers. Our vendor in particular has started using multiple different evaluation models, which is kind of cool seeing that coming there too.

Ashley Stirrup: Yeah, that is interesting. I think there's so many opportunities for AI to help in the experimentation process. I kind of break it down into three categories. So one is just making it easier for more people to run experiments, the other is automating the process, because I think there's just so much friction in most companies' experimentation process today. And then the third one is doing deeper analytics on the back end. So you can get more learnings out of every experiment.

Dorothy Crepin: Yeah, and even, I think to your point on analysis of getting deeper learnings, but just more as well. I think we select a primary metric and then maybe a couple of secondary. But if you can develop something where they're picking up any anomalies that are just one standard deviation away, the areas that you didn't think would be affected by an experiment can start to be noticed. And I think that's gonna change some of the ways that we look at effects of experiments. I think we're gonna probably go beyond primary and secondary and have some other category.

Ashley Stirrup: Yeah, it's so interesting. We had a guest on a webinar, they're at the Philadelphia Inquirer, and they had different teams. So they had their content team, they had their marketing team, and then they had the team selling advertising. And as you can imagine, the team selling advertising wants to have lots and lots of ads on every page. The content team, not so much, right? They want to have a really great user experience. And so experimentation allowed them to kind of say, okay, these are going to be the set of metrics we always look at, and we're willing to make certain trade-offs between things. And it helped them create trust across the teams as you're looking at these different metrics and how do you make trade-offs across them. So I think I could see that being important in your business as well.

Dorothy Crepin: Sure. I definitely, and I think we have product line owners or even regional marketing teams that are all fighting for their space on the site to promote their project or make the site experience best for their region specifically. And it's very tough when you're thinking of the global experience. So I think being able to break that down, it's nice.

Ashley Stirrup: Yeah. And then being able to go to people with data and show them, well, I know you love this thing, but it's not performing so well here.

Dorothy Crepin: This is why. Yes.

Ashley Stirrup: Exactly. We're not just making a corporate executive decision. We've got real data behind it.

Dorothy Crepin: Yeah, there's a reason behind it.

Ashley Stirrup: Yeah, that's terrific. Well Dorothy, thank you so much for coming on the show today. I think you shared a lot of wonderful examples and such a different business than we typically have on. So it was great to have you.

Dorothy Crepin: Of course, thank you so much for having me. This has been really interesting and getting to know more about the podcast and your company as well. So thank you for having me.

Ashley Stirrup: Thank you.