Notable Quotes

"It's not did we just get this live or not, it's how did this impact the business, and was it a net positive or not?"

"Just because somebody found your feature and enabled it does not mean they were delighted by it."

"The wins build our credibility and that we know what we're doing. The losses also create a safe culture where we can test."

"I call it the wine effect. You don't always buy the cheapest wine. You buy the second cheapest wine."

"It's this tightrope, and that's why testing's so important, because there is that diminishing return where, okay, we simplified, but oh, now we oversimplified."

Transcript

The Experimentation Edge - Danielle Olean === [00:00:00]

Ashley Stirrup HOST: Hello, and welcome to today's episode. I'm excited to have Dani Olean, Director of E-commerce at Box, with us. Dani.

Danielle Olean GUEST: Hi, Ashley. Thanks for having me

Ashley Stirrup HOST: Yeah, excited to hear your story. You've got a terrific story. Maybe you could kick things off by just telling us a little bit about yourself and about Box.

Danielle Olean GUEST: Sure. Yeah. My name's Dani. I have been in e-commerce in some shape or form for over the last 15 years. Started actually in the B2C realm with some companies such as Wayfair, Drizly, I'm sure you've heard of them. Those were really fun rides, and then over the last few years, I really morphed into the B2B world of e-commerce, which I think can learn a lot from B2C. And so spent almost six years at Zoom growing that business through the pandemic, and now here I am at Box. So really excited to be bringing kind of those testing expertise to Box, driving online revenue growth and helping us just reposition ourselves as a company In that transformation [00:01:00] from being just a cloud storage company to an AI platform to really manage all of your unstructured content and be that kind of intelligent content management platform.

Ashley Stirrup HOST: Yeah, I tend to think of Box as the SharePoint for kind of the rest of us. Is that kind of where the company started off?

Danielle Olean GUEST: It started out, it had a really interesting story. It started off definitely being more focused on cloud storage. There was even a big kinda push towards a personal use. But I think we really became known in the industry for the enterprise security grade cloud storage that we offered. But now we're so much more than that with helping you structure your unstructured data, doing metadata extraction, building workflows and automations to really manage all of your content and the data that lives within it

Ashley Stirrup HOST: Wow. Sounds like the company's come a long way. That's pretty exciting.

Danielle Olean GUEST: Yeah. It's really exciting

Ashley Stirrup HOST: And I can just imagine your time at Zoom it spanned pandemic, and then the wind down [00:02:00] afterwards, right?

Danielle Olean GUEST: Yes. That was quite an interesting ride, I will say, being the one person that ran online sales pre-pandemic, and we thought it was a cool little thing we could grow for the company. And then all of a sudden it grew way bigger than we ever thought in such a short amount of time, and my one-person team became a whole 100-person department to stand up the growth of online. And then we were tasked with continuing to stabilize that post-pandemic, w- was an interesting challenge in and of itself

Ashley Stirrup HOST: Yeah, I'll bet. And what were some of the things that you focused on there?

Danielle Olean GUEST: So there I also drove the online sales, the whole end-to-end customer life cycle. And I think I'm called an e-commerce product manager, but really I represent a lot more than that. I believe I represent self-service for the entire customer life cycle. You hear a lot about AI right now. I've been doing those automations and those self-service, not always with AI, sometimes with machine learning but for a long time. And really that is what we try to promote as an e-commerce organization is how do we [00:03:00] allow automation, self-service, people to really do all of this with a touchless experience? And that's where this digital optimization becomes critical to making sure we're making the right decisions.

Ashley Stirrup HOST: Yeah. And so you're really focused on that kind of whole sign-up flow and then finding ways to reduce churn and things like that, yeah?

Danielle Olean GUEST: Yep. Starts at actually even the homepage to how are they getting to the pricing page? How are we getting the traffic even to the website? Through the checkout flow, I also own the in-app purchasing experiences, so a couple different avenues there. And then it even extends to post-purchase because if I sell you something online, you perceive that we have value. But if then you're stuck after that purchase and we don't actually allow you to activate and create a habit you're never gonna realize that value, and then you're gonna ultimately churn. So that's where we're really looking at that entire end-to-end customer life cycle from awareness to purchase to conversion and retention and upsell.

Ashley Stirrup HOST: Yeah that's pretty comprehensive. And how did you think about experimentation within that?

Danielle Olean GUEST: Yeah, I [00:04:00] think what's very unique about my role as a product manager is I'm not held to just launching things. It's not "Great, you've launched this, you've met this deadline. You're good. You're doing your job." I'm held to a revenue number, and inherently being held to a revenue and a growth number, I need to be able to measure if I am growing the revenue, if I am driving incremental revenue, and A/B testing is one of those kind of critical foundation frameworks that we use to make sure we are making the right decisions. Because at the end of the day, it's not did we just get this live or not, it's how did this impact the business, and was it a net positive or not?

Ashley Stirrup HOST: Yeah. It's interesting if you look at the evolution of your career, you went from pure B2C to a little bit hybrid B2B, B2C to truly B2B. Is that fair to say?

Danielle Olean GUEST: It definitely is an interesting evolution of- Yeah. And I think what's also really interesting is the e-commerce world in B2C, you're used to things like a product list page, PLP, product detail page, PDP, these filters, this [00:05:00] merchandising, and traditionally you never needed that for B2B, right? You just had a couple plans. They were clear, they were easy. But now when we're getting into this world of value-based selling and use cases and the solutions and what AI can actually do and not just a list of features, those types of things that are traditional in B2C are now making some headway into B2B

Ashley Stirrup HOST: Yeah. And I think one thing that's different about the roles you've taken on is they have had that revenue component directly to it. Whereas I think a lot of people that are more in traditional B2B product management, they're measured by how much their product evolves over time, but they're not really held to a revenue number.

Danielle Olean GUEST: Yes, that is correct. And I have a unique role in that I actually sit in the middle of product marketing and sales. So like I go to sales calls and report my numbers regularly, but then I go to the product organization for planning and have our triad there between product engineer and design.

Ashley Stirrup HOST: Yeah. And so for somebody that was more your traditional product [00:06:00] manager and, they don't necessarily own a sign-up flow they're mostly either, building new features or they're enhancing things based on individual customer requests. How would you recommend they think about experimentation?

Danielle Olean GUEST: I think experimentation is critical to everybody. At the end of the day, no matter what type of product organization you are, even if you're using AI, time is finite and resources are finite. You can only build so much. You can only do so much, maintain so much without investing more. And everything you build also becomes an opportunity cost for something else. So I think a testing approach helps in two ways. One, when you're talking about that opportunity cost, how do you get the minimal viable product out there so you're not spending a lot of time in tests, that it actually makes sense before you're investing in that further? So I think that becomes really critical. And then the other thing is too, is like when you're building this, is it actually adding value to that overall product? So you may not have a checkout flow, but you certainly have some [00:07:00] flow. Even an AI chatbot has... You're getting a response, you're having a chat, there's an answer. I'm either abandoning the flow or I'm coming back as a returning user. Say I wanna create a document. I need to find that place to create a document. I need to create that document, and hopefully it has the formatting needs I want. And so those are all things you can measure in a funnel, is how are they getting to there? Are they getting there? Are they starting the action and completing the action? And then you can identify also based on that test where there may be a problem, because you may have people that aren't really getting there and finding it, but of the small people that are finding it, they're actually using it at a high rate. So what that's gonna inform you is that this is actually a useful feature, but the placement is wrong. And so those learnings help you early on get some of those things out of the way and get those understanding that don't then become huge customer complaints or tech debt later on

Ashley Stirrup HOST: Yeah. I think that's such an important point that's really been a little bit of a theme across different guests on the show is that they're using experimentation [00:08:00] to understand typically where a customer is in a buyer's journey, and then if they introduce a new capability, is that aiding the buyer's journey or is it creating friction that maybe it's still a good idea but it's in the wrong spot in the flow? And I think that same mentality can be applied to B2B product management to say, "Okay, great, you rolled out a powerful new feature, but is your audience actually finding it? Are they using it? Have you highlighted it in the right way?" That type of stuff can be really powerful.

Danielle Olean GUEST: Yeah, absolutely. And I think even, I was actually talking about this today, that just because somebody found your feature and enabled it does not mean they were delighted by it. Did they have continued usage? And those are things you should also be measuring in the test to see if it was a good decision

Ashley Stirrup HOST: Yeah. Which I think is super important second point that it's like anything that's significant enough that you've put a decent amount of engineering effort into, you should understand how do I expect my user behavior to change, and then how am I gonna measure that to see if I actually delivered on that or not? ' [00:09:00] Cause, what we've learned in traditional experimentation, maybe less so in B2B, is that only two or three out of 10 actually, land like you thought. Doesn't mean the other ideas aren't good ideas, it just means maybe you missed a piece of put it in the wrong spot, used the wrong words, maybe misunderstood something about what customers really wanted. And so it's such a huge opportunity for a B2B product manager to like basically double the impact that they're having 'cause they're catching all those learnings and then building the product that the customers actually want to use.

Danielle Olean GUEST: They're evolving the product based on customer feedback rather than just based on the assumptions in their own hypotheses. And I think it's really important, too, that as we're also getting to this world where you see a lot of people adding new features, right? There is going to be a breaking point where those new features actually potentially could detract from the core activity of what they're trying to accomplish from your product and create frustration. And if you have that blind spot where I've launched so many new things that I'm [00:10:00] actually making people churn out because they can no longer use the core things that they needed to every single day, you're gonna be damaging the business in a slow and an over-the-long-term way. So that's something you definitely wanna avoid, and that's also why testing can be so beneficial.

Ashley Stirrup HOST: Yeah that's such a great point. It's funny I vibe coded this little app for myself and a couple of the other folks on the team, and then I went to show it to people. I hadn't looked at it in a few days and came back "Dang, I put too many features in this." "Where's the thing I really care about? Oh, that's hidden at the bottom of page three," type of thing.

Danielle Olean GUEST: And it happens with builders, right? We get obsessed with what we're building. We wanna add more. We keep, we wanna innovate. We wanna be on top of competitors. But you just cannot lose sight of the customers and how they're interacting. And by the time that they're complaining to you, it's too late. They've already lost a little bit of trust

Ashley Stirrup HOST: Yeah, for sure. Another thing I think you've had a lot of success with in your career is creating visibility for the things you've worked on and getting executive buy-in. Could you tell us a little bit about that?

Danielle Olean GUEST: Yeah, sure. And so I can give an example at my prior role where I had no funding for [00:11:00] any A/B testing tool. And so in order to prove that I should be funded for an A/B testing tool, which are not cheap, I had to run basically a manual scrappy A/B test and show that we had a ROI positive. So we did that. We had to split the audience ourselves using our own home-built tools, and we showed up to a 30% conversion rate. So we said, "Look at what we did with nothing. If you can invest just a little more, like we could do even more." So it's a little bit of that challenge, let me show you, now I showed you, do you wanna put your money there? I think that's really... It's a respectful challenge, obviously, but like some of it is educating that this is an opportunity and look what we can do. We can do even more. I also think what's really important is showcasing every single win and every single loss. Because even a loss we come, we present it. I actually have a template for an impact report, and on a biweekly cadence, we meet with the COO. On a monthly cadence, we meet with the CEO, and we share, "This is what won, this is what lost. Based on the loss, this is what we learned, and this is what we're going to do [00:12:00] next." Both the wins and losses are important because the wins build our credibility and that we know what we're doing. The losses also create a safe culture where we can test. And as long as we're not driving huge risk for the company, and that's where we always have different test criteria to mitigate that risk, then that should be encouraged that we're taking those chances because we're testing, and if it does hurt it, we're gonna roll it back before we launch it to 100%. Really building that culture, I think, in both ways, and then continuing to share that we are continuing to drive an impact shows the value of the overall program.

Ashley Stirrup HOST: Yeah. And for an organization that is more B2B that maybe isn't as familiar with experimentation, getting them to understand that maybe a 20% or 30% win rate is, industry normal can be so powerful 'cause for the people that get it, suddenly they realize that the importance of humility, right? That we all suck at guessing which features are really gonna land and which aren't.

Danielle Olean GUEST: Oh, we do. And it's great to hear that after, I think I'm one year in my [00:13:00] new role now, and I've really done a lot to promote AB testing, and we've had some wins that shows the value of it. And now I have executives that say, "Hey, somebody wants to put a new feature, but let's make sure we test that." And that was one of just the biggest wins in and of itself, just to create that culture and them understanding the value in that

Ashley Stirrup HOST: Yeah, that's terrific. And then it sounds like with the dashboards and quarterly look backs, you're trying to help a broader audience kind of learn from the experiments you've done. Is that right?

Danielle Olean GUEST: Yes, and we actually do it in a few different modalities. So I mentioned we do the executive forums. Today we actually just did a whole forum for marketing, and I now have four or five new meetings on my calendar where people wanna learn about AB testing, and they wanna figure out how to do it on their own. And next goal is to do it for the product organization and build more value in them understanding why they should do that. So it's this tops-down approach and then down-up making sure that both executives buy in and believe in it as a culture, and then we're also evangelizing it across the company as a culture

Ashley Stirrup HOST: Yeah. Boy, that sounds like a lot of fun, 'cause that's where you're really [00:14:00] changing how people do work.

Danielle Olean GUEST: Yeah, and it's exciting. And because we're revenue driven, like we can say we had this idea, we had this hypothesis, and look at what it did, and look at the metrics. And there's no really questioning that, and I think people have a desire to do that, right? Who doesn't wanna go to the COO and say, "Look at what, I had a hypothesis, I looked at this and I drove hundreds of thousands or millions of dollars in revenue." That makes you feel good. And then when you do have a loss, you're like, "I'm just looking for the next win," and you all get on board with that mentality

Ashley Stirrup HOST: Yeah. Love that. So yeah, could you tell us a little bit more about how experimentation works at Box? Like how much of it is centralized versus decentralized?

Danielle Olean GUEST: Sure. Right now we have two main teams that are doing the A/B testing. We've got the marketing team that does testing across the website where they're really focusing on generating pipeline and leads. And then I'm taking part of the website 'cause the pricing page does start there, and so that's where I'm A/B testing, but I'm A/B testing typically through checkout and then the post-purchase experience. Right now [00:15:00] it's centralized and decentralized in that we almost always work together when we do tests, but we also do have separate goals and ways that we can run tests without running into one another. I think as I continue to promote A/B testing across the org and we see more people wanting to get involved, we'll probably need to reconsider what that strategy looks like, 'cause you'll need to have a little more centralized governance if you have multiple different teams running tests, and we'll probably need to unify the tech stacks a little more. But right now we're able to fare pretty well today.

Ashley Stirrup HOST: Got it. And I think you said that, where you are in your journey, you're focused on the big rocks now and seeing the job evolving a little bit over time. Is that right?

Danielle Olean GUEST: Yeah, and so whenever I do A/B testing strategically I, first of all, I look at everything. I look at every experience I own, I look at all the plans, and I try to identify where is the biggest problem affecting the most customers. And so I think of my A/B testing strategy as a pyramid, where the bottom of the pyramid is the biggest part, and so that's where I'm figuring out [00:16:00] what's impacting everybody. So obviously, I'm starting with the pricing page, right? I have the most customers on the pricing page. Then I'm going to the checkout flow, and then I'm actually even assessing between web, the mobile app or the desktop app, where do I have the most customers, and where does it make sense to optimize those flows? And so we're doing complete overhauls. A lot of these flows have not been overhauled in about maybe 10 years, maybe five years. I don't know the exact timeline, but it's been quite some time, and they're not modernized. And so when I looked at the pricing page, I said, "Hey, there's a huge opportunity to simplify this and reduce cognitive overload." When I looked at the checkout flow, I said we don't even have Google Maps API integrated here. We're making them type in every single piece of their address." And God forbid, if they have a separate billing address and payment address, now they're typing in two full addresses. So we're looking at where are these experiences that are touching everybody, and we're doing large overhauls. But I then can't go and overhaul checkout again next year. That doesn't make sense if I've already optimized it. So that next level of the pyramid is starting to look at [00:17:00] sub-cohorts and looking at those cohorts' experiences and optimizing for that. And so an example may be I'm just doing checkout for desktop. That's where the most people are. Then maybe next year, it's more mobile web and in-app. Maybe it's the desktop app. Maybe I'm looking at our top three non-English-speaking countries, like Japan is a top country for us, and they do have different cultural needs when it comes to buying. They are more high context, and they do make decisions more as a group. So maybe we start to test and alter checkout and pricing flows to have downloadable PDFs that have a lot of information, so they can read it, they can bring it back to a group and come back and make a decision. So those are ways that we're starting broad, and we're starting with overhauls, and then we move up this pyramid to cohort out more and more the bigger cohorts we have, and then they move to smaller cohorts, and then it moves to high volume, smaller testing once we go through all of those big rocks, and we kinda hit that top of the pyramid.

Ashley Stirrup HOST: Yeah. That's pretty impressive that you've identified pretty wide spectrum of things to go do. That sounds [00:18:00] like a multi-year job right there.

Danielle Olean GUEST: Yeah, I've been told that a lot. You gotta keep yourself employed, right? And if you say, "I just own checkout," there's not a whole lot to do just with checkout. But when you can own the whole world of that end-to-end life cycle and then again, you're not only A/B testing things that drive gross revenue, I also can now drive things that mitigate churn, and so that's also exciting

Ashley Stirrup HOST: Yeah. So is there an example of an experiment where you had a lot of learnings?

Danielle Olean GUEST: Yeah, and so I can talk about, I think an experiment that we did that led to a series of hypotheses that showed that one hypothesis was true, but then as we went down that line it became untrue, which was interesting. So I mentioned when I had joined, we overhauled the pricing page because there was an opportunity to grow our conversion rate there. What we decided to overhaul was to simplify it. It is robust. There was a lot of colors, there was a lot of elements. We're trying to share security features, compliance features, AI features. There's eight self-service plans on our pricing page, and there was once three [00:19:00] tabs, now there's two tabs. And so I was like, "This is really difficult for a customer to understand. This is really overwhelming, and it's challenging." And so the hypothesis was that by simplifying the page, we would reduce cognitive overload and allow people to choose what plan they wanted and purchase. So we did a very large overhaul. There were some cases where a single feature in the plan card would wrap three lines, so we made a rule that in some cases, it can wrap two lines, but it really just needs to be one line, and let's use a tooltip if we need to say more. And so we did this overhaul, and it was a huge success. It drove a really great gain. We got more than what we thought we would get in the conversion rate through the hypothesis, so it was a big win. And whenever you get excited on a win, you're like, "Let's do more of that. This is the right hypothesis." So we decided let's test removing our slash-out pricing. Maybe that's also causing cognitive overload 'cause you see one price point, it's crossed out, but then you see the list price, and we're like, "Maybe that's another simplification." And so we tested that, and that was a failure. And one of the [00:20:00] learnings was that although we showed at the top of the page when you buy annually, there's 25% off, it was actually really heavily reinforced through that slash-out pricing. And so we thought that continuing to simplify and remove distracting elements would help, but in this case, that element was valuable to customers. We did do one more test to simplify 'cause I was like, "Ah, I think there's still more things we can do." And so we did another one where we decided from there to reduce the feature set even more. So rather than writing all of the features each plan had, we only wrote the differentiated features and wrote all features in prior plan and then just called out the differentiated. We were like, "Maybe this is gonna make it easier for them to understand the value propositions of each plan and simplify further." But again, we were showed that hypothesis was incorrect, it was a failure because we were not giving enough context for what was available in all of the plans, and our plans are complicated, and customers did want that level of detail. So [00:21:00] after one win of simplification and then two losses, then we were like, "Okay, let's slightly pivot the strategy a little bit. Maybe we have done a lot of simplification." And the funny thing that we're testing now is an addition. We're actually adding now a third tab to address a new persona on the pricing page. And so we believe that we've simplified it enough that now by serving this third persona, hopefully we won't go in the other direction and create too much confusion or cognitive overload. But of course, we're gonna test it to figure out what the results are.

Ashley Stirrup HOST: Yeah. But I think that's such a powerful example because you're literally learning about the buyer's journey almost at every step. And as they're going deeper, maybe you start off, you keep it nice and simple and help them find their way a bit. But then as they move, further along and they're getting closer to clicking that buy button, you need to make sure they have the right information at the right time.

Danielle Olean GUEST: It's this tightrope, and that's why testing's so [00:22:00] important because there is that diminishing return where, okay, we simplified, but oh, now we oversimplified. Okay, this is our happy medium

Ashley Stirrup HOST: Yeah. And there's literally no way you could know exactly what the right balance is unless you test it.

Danielle Olean GUEST: You would not know. Exactly. Yep

Ashley Stirrup HOST: Yeah. And in general, how do you try to approach a test that where you've lost? That's where the biggest learnings are, but you have to be smart about how you extract those learnings from a lost test. How do you do that?

Danielle Olean GUEST: Yeah, and I think you really wanna analyze... so we always have main KPIs, we have secondary KPIs, so we're trying to look like holistically what actually happened here and why this lost. And there are a couple s- scenarios where maybe something lost, but it was okay, or maybe something was flat and we still wanted to launch it. And so an example typically is if a test is flat, we mostly will roll it back because we don't want to invest the extra resources to code it, deploy it, and make sure we don't create any regressions. But we did have a flat test that fixed technical debt, so we were like, "You know what? This is actually a [00:23:00] net positive from our backend, so let's go ahead and do that." And that was an example where we went and launched it despite it not having a positive revenue gain, but it had a positive technical gain. We had other examples where my two key main levers for an e-commerce flow is conversion rate, how many people am I getting from the start of checkout to the completion of checkout, and average order value, how much money are they spending in checkout? And so you may have a test where you've actually increased conversion rate, but you've decreased average order value, and it's the relationship of those two metrics that create a revenue gain or loss. And so you can have a loss because you've increased conversion rate so much, but on the inverse, decreased average order value so that there is a net loss in revenue. And so we've had a couple of those tests, but one of the things we were trying to look at was we are changing the business to be less of an add-on business, more of a platform and suite business, so they are gonna be buying less in checkout, but we wanna get more people through checkout. And so we [00:24:00] did believe that it was the right thing to do to get more people from point A to point B, and then we decided to do a follow-up where let's do some sort of follow-up where we're either discounting additional licenses to increase average order value or doing a great upsell prompt so that we are making sure we're continuing to drive up that value. So we parsed it out as maybe this wasn't a full win, but we're gonna take a win on this and then continue to iterate to drive up those other secondary metrics.

Ashley Stirrup HOST: Yeah. Super interesting when you're in a business that's evolving, right? And so it's lifetime value of the customer that you're trying to optimize for as well as that initial sale.

Danielle Olean GUEST: Yeah, exactly. And we definitely had a test that it won, but it did not win in the way I thought it would win, and that's also an interesting one. This was in a prior role. We had launched a s- a plan, and we called it Basic Plus, and we had a churn problem. So we were trying... We didn't even ever show this plan on the pricing page. I think that's what some people also don't think about when it comes to A/B testing. Sometimes you can show plans at different part of the customer life cycle [00:25:00] to drive certain metrics. And so we never showed this on the pricing page. We only showed it when people canceled. And my hypothesis was it's cheaper, they think we're too expensive, they're gonna take the cheaper plan, and we'll at least have a partial churn instead of a full churn. Great. What actually happened with the test was, because it was a cheaper but we had 70% less features, it actually made their current plan look more valuable. I call it the wine effect. You don't always buy the cheapest wine. You buy the second cheapest wine. And so that test won, but it was because we made their current plan look more valuable. So rather than them switching, more people actually stayed with their current plan than switching to the discounted offer that we had. So that was an interesting learning too

Ashley Stirrup HOST: That is such a great example of, we think we know what we're doing and then we find out what, what's really happening. And another great example why you try to run lots of different tests on different things, 'cause you're gonna find surprising results that give you new insight into how people think about your products and services.

Danielle Olean GUEST: Yeah, exactly. And like we have slightly the inverse problem now. As [00:26:00] I mentioned, we've got eight self-service plans on the pricing page. So I'm going to probably start consolidating some of those. I'm not gonna end of life them completely. I might remove them from acquisition 'cause they're hurting acquisition, but show in things like the upgrade flows, the downgrade flows to help drive revenue in other areas of the customer life cycle

Ashley Stirrup HOST: Yeah. Makes a lot of sense. You could see landing people with one offering, helping them to get that initial aha moment of a great customer experience, and then you say, "Oh, now would you like the add-on AI features?" Or whatever it is. So yeah, it makes total sense. That's, I hadn't really thought about pricing in that way before, but that's a great example. And so as you look forward, how do you see experimentation at Box evolving?

Danielle Olean GUEST: Yeah. I would love to continue. I wanna get more people involved in it and get more people to test across the company and create it more as a cultural norm. I think that's really important. I also see that there's two types of A/B tests, and mostly what we've been talking about today are what I call optimization tests. How do I get somebody from point A to point B quicker, [00:27:00] faster, better? That's optimizing the pricing page, checkout flow. What we're going to continue to evolve into is monetization tests. So not only testing things like experiences, but testing plans and offerings and changing the features and the plans and the, testing the product market fit of the plan itself. So that's one way that we're gonna evolve in expanding the tests. Also I'd love to be able to test more experiences, especially onboarding and post-purchase. And I think continuing to do those personalized cohort tests once we get through those big rocks. So really excited to start seeing it from an international lens as well as a mobile first lens.

Ashley Stirrup HOST: Yeah. Just everything you said just shows just how much potential there is to do experimentation in a B2B world, right? There's literally so many different dimensions that you could be looking at the business at through the eyes of experimentation.

Danielle Olean GUEST: Yeah, exactly

Ashley Stirrup HOST: And then I think you're also doing work around AI, is that right, for experimentation?

Danielle Olean GUEST: yes. Yeah. So we have a little bit of a pet project right now where we're [00:28:00] trying to set up an AI agent to do a few things for us. One, to do like a competitive analysis, so we can see what other people are testing, just get a pulse on what's going on in our ecosystem out there. We also are trying to set up an AI agent to help us ideate on AB tests. So say, "Look at our checkout flow, look at our pricing page, look at the competitive research you've done, and help us to ideate." And one of the things we have built and is live today, but we're still evolving it, is this dashboard of all of the past AB tests we've run, so that anybody at the company can just look up, like what was done, what were the results of that. And again, that really starts to build out that AB testing data-driven culture at large. And so really excited to get that stood up. Right now that dashboard is mainly within our team, but we're working on making it more robust and then sharing it out further.

Ashley Stirrup HOST: Yeah. So much potential with AI around experimentation and I really that idea of having the kind of the competitive agent that's going and looking 'cause there, that's [00:29:00] such a rich opportunity that I think a lot of people overlook is what is everybody else doing? And, particularly, you hear it more with e-commerce and people watching Amazon or a Netflix or somebody like that. But I think every industry has the opportunity to try to learn maybe not directly, right? But at least they're getting ideas and input that they can leverage. So that, I think that's a fabulous example,

Danielle Olean GUEST: Yeah, and B2B should be looking at both their competitors and the people like Amazon. There are things that I can take from Amazon that I'm like, "Oh, that's a good idea. Let me test that too." So look at traditional B2C too, because they're a little more mature when it comes to at least checkout flows.

Ashley Stirrup HOST: Yeah, 100%. Dani, this episode has just been chock-full of wisdom and nuggets. I really enjoyed it. You brought great energy to the episode as well, so thank you so much for joining us today

Danielle Olean GUEST: Awesome. Thank you for having me. It was a pleasure

Ashley Stirrup HOST: Thank you