The Experimentation Edge - Arie Polycarpou
===

Arie Polycarpou: [00:00:00] I think if you do a test like ~s-a~ straight up AB test, maybe it's a fifty/fifty chance, but you also have to consider, like as a company, you're doing a lot of things as a business and have built on previous knowledge and descriptive data and UX best practices that you might not always be...

To win and truly have a significantly better experience, like it's gonna take maybe a little bit extra work.

Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Sturrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions.

This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest

Ashley Stirrup: Hello, and welcome to today's episode. I'm excited to have Arie Polycarpou, Senior Manager Test and [00:01:00] Learn at Aspen Dental. Arie, welcome to the show.

Arie Polycarpou: Hi, thanks for having me. Very excited to be here.

Ashley Stirrup: Yeah, excited to have you on as well. You have a great background. Maybe we start with you just sharing a little bit about that. Like, when did you first get into experimentation and yeah, just tell us a little bit about your career.

Arie Polycarpou: Yeah. So for me, I have been in the digital space almost my entire career. ~I knew when I started my career, ~I've always been really interested in analytics. As a kid, I was, like, a very into sports and a data nerd. In college, I studied marketing, and marketing analytics was the field that really stuck out.

And then I kinda got placed into digital as part of my first internship program, and I immediately just fell in love with the field. I think a lot of the times in the data world, you don't have the richest data. You're trying to put together missing pieces. But with digital, you kinda really get to see a whole customer experience start to finish, and I think that always drew me.

And then I got placed into a standard digital analytics role, and then I was at Kohl's Department Stores at the time, and A/B testing was becoming a [00:02:00] growing field. And there was one person that was working on it, and then he started... it was a growing field, so he needed help, and I ~I got... I ~was really interested.

I volunteered, and yeah, 12 years later, here I am still working in not only digital analytics, but A/B testing and helping kinda lead programs and doing that from start to finish.

Ashley Stirrup: Terrific. And after Kohl's,, you were at Total Wine, is that right?

Arie Polycarpou: Yeah. So I started my career as an analyst at Kohl's. I was at Marriott for a little bit as well helping with their ~an- their~ measurement of A/B tests and really applying some rigor to that. But I quickly transitioned over to working at Total Wine, where Total Wine was a company that grew tremendously during COVID as, places shut down, the need for alcohol at home grew.

So with that became a, a growing need for building an A/B testing program in-house. So yeah, I joined there. They had done some testing one-off for a while, but really helped bring the program from just doing a couple big product tests a quarter to a kind of a full-fledged, hundred-test-a-year program at Total Wine.

And then yeah, ~I to- yeah. And then I... Oh, sorry. Yeah, I ~took a [00:03:00] small break to do graduate studies in, specifically in digital transformation, brought in to understand where the digital field's going. And then I am now in Aspen Dental doing a little bit of a similar role where they had A/B testing and working with an agency, but bringing it in-house and again, really trying to accelerate the program and the rigor of the analytics and the depth of what we're able to measure and do and work cross-functionally within the organization.

Ashley Stirrup: Terrific. And Aspen Dental's a little bit of an unusual business. Why don't you tell us about that?

Arie Polycarpou: Yeah. So traditionally my career has always been B2C, and this technically fits it, but Aspen Dental it's a private company, part of a larger company called TAG, which owns several healthcare organizations. And I think we specifically like to describe it as healthcare retail. So ~As- ~Aspen Dental is the largest of the companies in the holding group.

Has about 1,100 offices across the country. And yeah, our goal with the website is attract new customers, bring them in, help them learn about what we offer as a business, and get them on the books to get an [00:04:00] appointment.

Ashley Stirrup: Got it. And do you own the local branches as well?

Arie Polycarpou: No. So all of our ~o- our~ offices are owned by private dentists. We're... Dentists are recruited and it works in a model ~not too unsimilar- ~dissimilar from a Marriott, right? Where it's like we help provide a lot of resources and training and education, but ultimately, like they're the, the practice owners are the dentists, or dentists can own several and hire out other dentists within it

Ashley Stirrup: Got it. That makes total sense. And so can you tell a little bit about your team and the structure of the group?

Arie Polycarpou: Yeah. So I know when I joined both here and at Total Wine, it was a similar where I was hired as an individual contributor, really helped build some of the connections, put in the rigor, set up those processes and standards, and then eventually as we grew, build out a team to help support that. So right now I am a, a senior manager of test and learn, I think as you introduced me at the start of the interview, and I have a team of two a manager of analytics and then a, a coordinator of testing and [00:05:00] analytics to help I think our role focuses a lot on what I call descriptive analytics too, understanding how the website is performing. But really I think our bread and butter and what really I think moves the organization forward is how do we... Knowing what the opportunities are in our business, how do we turn that into tests and work with the right partners to build and set up?

Ashley Stirrup: And you're running a lot of tests

Arie Polycarpou: Yeah. Yeah. This is my first year at the company. I only joined in October of 2025. So we've been scaling up. I think I generally like to build a program that runs about 100 tests a year. And I think as we mature, that number I expect to grow and branch out as other members of the organization are like...

Right now I think as we start the organization, I know this might be a question later, is let's try to make sure we're attacking the low-hanging fruits. What are things we've wanted to do for a while? What are some of the bigger value opportunities? And then eventually, as you grow and mature as a program, you knocked out the low-hanging fruits, you have a process for the big things, but then you're also doing small iterative tests.

For us, it could be across [00:06:00] merchandising lines or looking towards localization and getting very specific and detailed and running iterative tests across different markets. So yes, we're, I think, to answer your question, aiming for about 100, and I hope to grow the program to full scale. Yeah

Ashley Stirrup: And do you work with a number of different product managers, or

Arie Polycarpou: Yeah. Yeah, so there's different product managers that own different sections of our site and operations. So we work really closely with them to help build and support our team. We work also with what we call the production group or merchandising to do they have the content management tool.

We can do tests within that. It's connected to our ~tested program-~ testing program doing that as well. And then we have a team of front-end developers as well, where, if it can't fit within the scope of the product team or, isn't something that our current capabilities can help build today, like we can write front-end code and help run those tests as well.

So a lot of different avenues to help us make sure we're testing and optimizing.

Ashley Stirrup: Yeah. And one of the things I think is just so important as a part of having a strong experimentation program is just building the [00:07:00] culture where you're helping everybody to think, with an AB test mindset. Think test it first and just learn from each other and make sure you're setting up experiments in a rigorous, appropriate way.

Like, how do you kinda tackle that side of it?

Arie Polycarpou: Yeah, that's a good question, and it's one where I think rigor is extremely important. 'Cause if you wanna run tests, you wanna feel really confident that what you're doing, you know what's happening. So that is important. But when you're talking and trying to preach that culture and get stakeholders involved, you're not always gonna get people on board and fully understanding of what's going on with statistical significance, T-tests, terminology.

So it's again, being able to, do the rigor in the back end, really understand, be able to connect the data sources, and then being able to communicate that. Keeping, tests simple, sharing wins. Those are a lot easier to share, right? This is a clear win. Cross-functional effort.

Everyone was involved. This is great. That makes people wanna do more, right? And seeing that win and play out and then eventually translate to your business. The [00:08:00] losses, again, usually when you're starting, you have enough wins to help carry that momentum, but the losses, you can learn from them too. I think it's one thing to be like, "This lost, let's move on," but then understanding why, I think especially with certain groups of people, is just really interesting.

It's like, "Wait we didn't think this," but then you explain why or see where maybe they did an intended behavior and made the click you wanted, but then they dropped off after that. Having that and then helping brainstorm, think through iterative things, and just get people again thinking in a test-and-learn mindset.

I think that's one of the first ways of maturity I see testing in a program is people, they wanna do things, and their first thought is how... is like not thinking, "Oh, do we need to measure this?" It's like, can we put this into the AB testing process and pipeline and then build on it and learn and see what happens?

Ashley Stirrup: Yeah. Yeah, it's interesting. We were recently running an A/B test internally at GrowthBook, and test went great, rolled it out. Not getting the benefits we hoped," and then we realized that step worked great, but then the next [00:09:00] step, there's three more things that we hadn't thought about that those frictions are still there.

And so you can improve one step of the process, but not improve the whole thing. And so that's, that to me is where the real power of experimentation is it starts to help you connect the dots on things like that and really understand where you're having impact and where you still have that room to improve.

Arie Polycarpou: Yeah. Yeah, and that's why it's important. And usually, I know for us, we actually have this conversation regularly, is we feel very confident that this test will drive the next action or not, but then everything that happens down funnel it usually does have some type of interaction. And sometimes you maybe see some noise, sometimes you don't see something during a test, but it's also important to try to understand and learn from that as well

Ashley Stirrup: Yeah. Yes. I feel so often the most important learnings come from those experiments that lose

Arie Polycarpou: Yes.

Ashley Stirrup: Do you have ~an e- ~an example of one where you had a lot of learnings?

Arie Polycarpou: Yes. I do. ~Let me... Yeah ~That's a good one. I think one that... I've [00:10:00] seen this in a few companies I've been at, and very recently too, is how we approach, especially for B2C, but many companies, you have what you call a global navigation. Like, how are people navigating across your site? And I think oftentimes you do redesigns and expect ~this this is 100% better, ~and you expect things are cleaner, it's a fresher color palette, it should be good.

And then you actually usually, the first test that you run, see a decent amount of friction, I think especially when it comes to navigation. From some companies that I've been in previously, it's because you have a lot of return customers and they just got accustomed to where things are, and then they get frustrated.

For ~e- ~us, even at Aspen, where maybe more of our customers are newer that come to our website, still, I think it's things that we maybe don't estimate or expect them to be looking at. They're just important, and they need to be front and center and prevalent. Like for instance, if someone is constantly looking for whether it's like, trying to learn about a specific product line, try to see about specific offices, just making sure that information, you don't [00:11:00] change that and reduce that behavior and trying to understand why they're doing it and making sure that they can still get to the places they need rather than thinking about maybe the front end experience first.

Ashley Stirrup: Yeah, it can be so hard when you've, you've got a single homepage and there's so many different customers with so many different use cases, and how do you optimize for them all? And so it sounds like you were doing a, a rebuild on the main nav and~ y- it, it~ didn't work as well at first.

Was there a kind of a, a key need that you uncovered as you were doing that?

Arie Polycarpou: Yeah. And honestly, it's not even necessarily that I just completely did horrible, but we fundamentally changed behavior in probably a way we didn't expect. There are certain things that maybe we didn't have before. Let's say for us is, understanding what location you're at. I think before we used to have a pretty clear this is the location you're set at front and center.

And then again, for us, it doesn't maybe seem horribly unintuitive. It's just behind a hamburger click. But honestly, just that [00:12:00] alone and a user's not expecting it or so... it's... A click is different than just what upon looking. So again, we changed behavior. What people were looking at on our website was focusing on different aspects, and I think that can be okay if you're trying to maybe change your perception of a brand or highlight something that's really important.

Like for us, if it's affordability or even promotions, like, yeah, maybe you wanna push that there, and that's forward, and this test was good. If for us, we wanna get users connected to their local office and get the information they need, like maybe it's something we need to reconsider, retweak, or bring a little bit more prominent to the, the front

Ashley Stirrup: Yeah, makes sense. And let's say you start working with a new product manager they're not that familiar with A/B testing, and you want to help them design an experiment such that if it loses, you can figure out why and extract as much learning as possible. How would you guide them to think about that experiment design?

Arie Polycarpou: Yeah. And I think this is a question I've had with even more senior product [00:13:00] managers that maybe understand and have been involved with testing. I think there's always the battle between do you build this full-fledged multi-component experience that we know like this is better, like it tested well in UX, this is where we wanna go, let's just push that out and learn?

Or do you find a way to test yourself into that better experience? And I usually, especially if you have time on your side, prefer that latter approach, 'cause then you can isolate elements, understand what's going on. You can set up an MVT test as well to accomplish that. But if it's a big change, like working to iterate your way off there and really be able to then put a financial value on the changes.

'Cause they're gonna be like, if something works... If it doesn't work, they're gonna be like, "Why didn't it work? And you did this whole thing." And I'm like, "We didn't isolate for that. There was a lot of things that changed." But if you're able to isolate it a little bit more, run a clean multivariate test, you can then isolate and be like, "This drives this amount.

This drives, 5% versus 2% versus this actually maybe hurt us a couple percent." Or honestly, what happens more often than [00:14:00] not is this didn't impact our bottom line business. So yes, I think trying to get them to do that approach, practice patience, and then really be able to quantify and then help them give their teams that like this is the work that you're doing does have this strong financial value to it, I think has been in some sense, my most...

The way I've made the most progress and success in helping take that culture is really showing like your efforts are paying off and worth valuable amount. And again, if it loses, like we know why, and we can... If you do it smart and you have enough data, you can ~under- you can~ tweak it and understand maybe why or go a different ~direct- the smart follow-up ~direction.

Ashley Stirrup: Yeah. Yeah. What's your typical win rate?

Arie Polycarpou: Yeah. So ~honest-- ~again, it depends the company I've been at and how many tests you're doing. I think I typically strive to do a win rate, a true win rate of about twenty-five percent. I think if you do a test like ~s-a~ straight up AB test, maybe it's a fifty/fifty chance, but you also have to consider, like as a company, you're doing a lot of things as a [00:15:00] business and have built on previous knowledge and descriptive data and UX best practices that you might not always be...

To win and truly have a significantly better experience, like it's gonna take maybe a little bit extra work. So I know for me, when I'm starting in an organization and we're a bit newer, we sometimes see about a forty percent win rate. But I actually expect with maturity that should go down a little bit.

If you have too low of a win rate, then you're wasting effort testing, not testing the right things. But I think usually a sweet spot of that twenty-five percent. But again, most companies I've been at as they're starting out, it's closer to forty percent.

Ashley Stirrup: Yeah, that, that makes a lot of sense. A lot of the major players in the space are quoted at, like Bing, I think they were at 20%, and Airbnb might have been even as low as 10%. So the more mature your product is, the harder it is to move the needle.

Yeah, one of the things I think is so powerful about experimentation is being able to stack wins, the compounding nature of it. 'Cause often each individual experiment might not move the needle that much. But if a program, if the team [00:16:00] doesn't really understand, "Oh, we got these two wins and we avoided these two huge losses," you add all that up they don't necessarily see the impact that they're having unless you're thinking about how to measure that correctly.

How do you communicate that internally?

Arie Polycarpou: Yeah. So I think for me, I love to be optimistic, and I love to promote wins and understanding better. Better is always my view with testing, right? But then the amount and the quantity that to expect, I ~re- I ~usually do try to be a little bit more conservative. Whether it's I'm baking into estimations, I expect a depreciation impact.

Yeah, we saw a five percent lift during this test, but I don't know if that's gonna hold across an entire year of this experience and market. Partially because, customers maybe get used to it, it gets baked in. Others is because you're building onto that with more things. And yes, like a five percent win that on a homepage now and then another five percent yeah, it doesn't...

I know the math should make it seem like it does work like that, but it doesn't. So again, just being a little bit more conservative. So when I do add up [00:17:00] tests like that we can just be already, have already like some deprecation in place and some haircuts as we maybe call them to be a little more conservative.

~The, Oh, and the other thing we do too is sometimes like holdouts long term, but those are usually more messy, and I don't want us to revert back to a control experience and yeah.~

Ashley Stirrup: Yeah, holdouts can be tough just to ends up forcing you to maintain two sets of code basically.

Arie Polycarpou: Yeah, and not ideal

Ashley Stirrup: you do it, the harder it gets. Yeah. ~Ro- ~Ronnie was saying that we had him on a, a webinar a little while ago, and he was talking about applying a 20% haircut to your wins, when you stack them all up together

Arie Polycarpou: Yeah.

Ashley Stirrup: curse.

Yeah.

Arie Polycarpou: I like the term, and honestly, that's a ~s- it's a ~similar approach to what I do, and yeah, usually 20 to... Depending, if his, if he's ran the test for a shorter period of time, the lift was really strong, but maybe the confidence wasn't, like I'll maybe even be more and try to play that into the

Ashley Stirrup: Yeah, for sure. Yeah, you've always gotta have enough power, otherwise then you start to exaggerate the win rate and yeah, it can lead to a lot of false positives, false negatives. For your business, I would guess that a key thing, key metric that you focus on is just the number of [00:18:00] appointment bookings you're getting per location.

Arie Polycarpou: Yeah. Yeah, I think when I look at our business, that's always our North Star. That's what we can control the most and measure all the way, through thoroughly. And yeah, that's definitely getting the appointments on the book to, as I think some people in organizations say, give the offices a chance to serve those users and customers.

Ashley Stirrup: And are there other kind of key metrics that you look at?

Arie Polycarpou: Yeah. So again, I think the-- if we call conversion rate the North Star, there's all the steps in between to get from a session to conversion rate. So especially for us, our scheduling funnel, where are people dropping off? Are people even getting to our scheduling portion of the site? Are people even getting to the next page?

What's our bounce rate? And sometimes minimizing that. And then we look my view of the website is give the offices a chance to serve a user. But we do wanna monitor and see what those users are doing at the office as well.

And I think the biggest reason for that is just the ~qual- ~quality is a rough word to use, but just the value of a patient, right? Are we changing the types of [00:19:00] users that are coming in? Or even for us is are we changing the types of offices that are getting more patients? like I know, I think a big conversation in the world of UX is ~s- like~ ratings and reviews, and we have eleven hundred offices, they each have different star ratings and levels of service that they're associated with it.

So if we're maybe, if we're promoting stronger offices and getting more users there, that's probably gonna be better for our business 'cause they have higher, show rates and higher value, return value. And then, if we're getting more users into maybe less strong offices that are as good at doing that or more ~s- ~certain markets and maybe more competitive world, yeah, can be...

Yeah, so important.

Ashley Stirrup: Yeah, super interesting 'cause you're-- it's not you've got one consistent customer if you think of each location as a customer, like where they're located, just the quality of the staff there, all those things can really drive it. So you- in some ways you almost have to get down to a location by location analysis to understand kinda like total lifetime value of [00:20:00] a customer and, how much impact did this experiment really have?

Arie Polycarpou: Yeah. Yeah. So that's we talked about metrics that we looked at. We also have like our segments and the types of users, whether it's how they came to our site or then where they were bucketed for us, whether it's like what's the reason that they're coming and booking an appointment versus yeah, where they ended up booking

Ashley Stirrup: As you look forward, how do you see experimentation evolving at the Aspen Group?

Arie Polycarpou: Yeah. I think the way I see it evolving in the Aspen Group is in a way that's similar to maybe how I see the testing world evolving longer term. ~I think we talked... I'll split it into two parts because first, within the Aspen Group, I think it's what I mentioned at the beginning of this interview where, ~I joined less than a year ago.

We're starting to build the muscle, build, the ~l- the~ ways we can build tests, the teams that are involved, the level of sophistication. So I think with that, I expect us to, continue to mature, get to a point where we're... have attacked a lot of our low-hanging fruits and are really looking to optimize within, and then branching out, whether it's localization, as we just talked about, knowing each offices could be served differently, have different impacts.

Whether it's then focusing even deeper on what we call our service lines, like the offerings that [00:21:00] we have and what we present forward. For instance, what might get someone to book an appointment to get their dentures fixed with us is probably a ~di- ~different treatment than someone coming in for an emergency.

And what we do there. And so really getting to work with some of those what we call product line owners and getting more specific or getting like the local owners. Yeah. So that's where I see it within Aspen. And then long term, technology is advancing a lot, and I think you, you hear the word AI a lot and what that means.

A lot of these testing tools are putting AI in their tools to help you, analyze tests quicker, look up tests quicker, even help potentially build tests. And I think that's a phase, it's interesting and I've looked into and I think Aspen is definitely looking into, but it's what's the right amount for us and how do we do that?

And then also in a way that keeps a clean code base and makes sure that we leverage like our top engineers to continue to have control of the site and not pushing what we call bespoke code or,

But also like for me, helping with analysis too is like, how do we, [00:22:00] train that to help serve as some extra additional analytics hands or even support and guidance rather than...

the work that we're already doing

Ashley Stirrup: Yeah. Yeah, I really think that AI has the potential to just streamline a lot of the experimentation process on the things that are very clearly defined. I think there's other areas that where they-- where it's ambiguous, you still need a human involved. It's just really hard to give all the context to AI to help it make good decisions.

Arie Polycarpou: Yes

Ashley Stirrup: so I think there's tremendous potential, but you have to do it in the right ways. And I really believe it's gonna, unlock people to do a lot more, be more creative, be more productive. and, we're seeing it in AI coding already today, and so it's a very logical next step to extend that to the, the experimentation space.

Arie Polycarpou: Yeah, no, and I think ~I, I think ~you described it best. It's the same way. It's meant to, again, enhance what we do, help support, streamline very, simple things that we can then use to help bounce... start thinking a little bit more strategically and [00:23:00] higher level and again, working more on maybe the people operations part or how do we get these teams involved, and being able to be better communicators.

And yeah, a lot of ~po- ~I think benefits that could be, if done right

Ashley Stirrup: Yeah. And for a business like yours, I could see if you could find ways of empowering each location to do its own A/B testing, but in very controlled, with guardrails and with the right rigor, that could be a real unlock.

Arie Polycarpou: Yes. Yeah

Ashley Stirrup: Yeah. It's a little further down the road though, but-

Arie Polycarpou: Yeah. But yeah, who knows? We'll see how much further is the

Ashley Stirrup: Yeah. You could get pretty busy that way

Arie Polycarpou: Yeah,

Ashley Stirrup: if you're supporting that many locations and they're all coming back to you with questions. So

Arie Polycarpou: Yeah

Ashley Stirrup: Yeah. Keep you fully employed for a long time.

Arie Polycarpou: Yes

Ashley Stirrup: thank you so much for joining the show, Arie. You've been a great guest. It's been fun to learn about your business and I feel like we learned a lot today.

Arie Polycarpou: Good. Thank you. No, thanks for having me.. Happy to be here.

Ashley Stirrup: Thank you

[00:24:00]