Transcript

The Experimentation Edge - Michael Mathews Recorded 2026-09-14

Ashley Stirrup: Hello everyone and welcome to today's episode. I'm excited to have Michael Mathews, Director of Data and AI product at FanDuel. Mike, welcome to the show.

Michael Mathews: Hey, thanks Ashley for having me.

Ashley Stirrup: Yeah, why don't we start by just talking a little bit about FanDuel as a business?

Michael Mathews: Yeah, so FanDuel is the leading online sportsbook in the US. They have a leading online casino, big in daily fantasy sports, and just released a prediction market almost a year ago. And a couple other, you know, small products, which we call flywheel new ventures that kind of allow us to build that flywheel between customers and cross selling them across some of the the big products that we have.

Ashley Stirrup: Yeah, it must be a pretty dynamic business these days with everything going on and massive volume of visitors as well.

Michael Mathews: Yeah, huge huge volume of of visitors, many millions of them active per month. A lot of competition from the the prediction markets that just you know came online about a year or two ago. So some really interesting time working with not only sportsbook products, but just all of the the full breadth of predictions that are now coming into the market, which has you know, gotten gotten us accustomed to different types of people and customers that were never sports fans. So it's interesting times.

Ashley Stirrup: That is interesting, widening your customer base. So could you tell us a little bit about your role and some of the things you're doing on the kind of experiment infrastructure side?

Michael Mathews: Yeah. So I manage a few different product teams at FanDuel. So my main focuses are data platform, machine learning platform, data science, and a team we call AI Solutions, which is is a team that deploys AI agents across the company for different purposes. My role within the experimentation process at FanDuel is we build platforms and services, components, ways of working, different tools that allow experimentation teams to run faster, more concurrent experiments, more intelligent experiments, and you know, helps to close that loop between experiment readout, all the way to proposing the strategy for the next experiment. So yeah, the the teams build tools across the entire company and you know part of that is definitely the experimentation platform.

Ashley Stirrup: It and you're doing some pretty cutting edge stuff when it comes to AI and experimentation.

Michael Mathews: Yeah, c I can go into the different autonomy tools that we've built. I think the f where I where I should start is something that we call an auto researcher. And this is something that is just catching on in the industry now, not just I I think within sportsbook casinos, but across like any B2C application, iOS, et cetera. What you're seeing is an auto researcher that does the search part of data science on its own. So you give it a problem and a way to score answers, and it spends a few hundred attempts writing models and testing them and keeping you know eventually what survives that process. And the output is the winning approach as code that you can read. So I can start with the input just to kind of give some detail into what this means. There's three things that are required. You need a data set or ideally many data sets. You need what's called an objective function. For example, this is some goal, like I want you to increase the prediction accuracy of customer churn. This is the scoring rule that defines what good looks like in this case. And then you need a starting strategy file, which is usually whatever logic runs in production today. Then it does this thing which we call hill climbing. It reads the objective, it proposes a strategy, and then it writes an actual ML model to implement that strategy. It scores it and then it goes again. It does this about you know 300 times. And a run takes about an hour or two and and cost us today about $300 in tokens. And it stops when the score plateaus or when it lands on the same number five or six times in a row. What comes out is just code that you can read. It's a telemetry table with all 300 strategies, what each one scored, and a Git diff of the winner. So a data scientist or a product manager or an analyst could open it, see the model's code, hyperparameters, the features it built, everything it did is outputted to a simple human readable code file. So with all that said, I think there's three really interesting things that surprised us as we are scaling this idea up. The first is that it builds its own features. So on one run, it started with 756 candidates and finished with 20. A feature here just might be whether this customer bet in the last seven days, whether they've ever placed a parlay, whether they follow the NFL. When a feature provides low signal to achieving the objective, the engine just simply throws it out and builds something else. Second is the holdout, which is enforced by the framework. So the worker that writes strategies has no access to the validation set. Only the evaluator sees it. And that matters enormously when something is optimizing this hard against the score. Because anything that can see the test set will eventually find an answer. It cheats essentially. So

Ashley Stirrup: Right.

Michael Mathews: I've gotten some pushback on the potential for data leakage, randomized training and test sets. So we made it structural to the agent's process. The third and maybe the most interesting one is it reframes the problem a lot of the time. We pointed it at a model that guesses which World Cup team a customer cares about or wants to bet on. Before it tried to improve our approach, it looked at how we were scoring success and noticed that a wrong guess cost us more than a right guess gained us. So the smart play was to stop guessing about the most people. And only make a call on the small group it felt most confident about. It threw out the approach, we handed it, and it built something that covers about 1% of customers and it gets those right roughly 70% of the time. And you know what's interesting is no one told it to do that. It just read our own scoring rules and just was able to understand and build something more carefully than we had thought of.

Ashley Stirrup: Did that lead you to go back and change your scoring rules? Yeah.

Michael Mathews: Yeah, absolutely. Yeah, we tested different approaches. We ran this specific test that I'm thinking of, I'll talk a bit about what that test was, but we

Ashley Stirrup: Hmm.

Michael Mathews: Run maybe three or four of them in a row and changed it based on the results and it it feeds back into the start of

Ashley Stirrup: Yeah. What

Michael Mathews: Proposing it.

Ashley Stirrup: And I should be more clear, what I meant was were you unintentionally giving too many negative points to guessing the wrong team?

Michael Mathews: Yeah, exactly. Yeah, the human design strategy was giving too many points to guessing the wrong team. Yep, that's right.

Ashley Stirrup: Got it. Got it. Super interesting. And is this a whole approach that you just described really best suited to algorithm tests versus say UI tests?

Michael Mathews: No, it's perfect for UI tests. I can maybe go into what some of those UI tests are, if you want to talk about about that.

Ashley Stirrup: Okay, sure. Yeah.

Michael Mathews: Yeah, so this autonomy that I'm talking about, what's interesting about it is you can run these over and over again and just continue to get new learnings and apply them back to the next strategy that's being proposed. So to just to make this tangibly you know, something that we've done, we have a lot of contents on the home page of our apps, right? If if you think about a sportsbook or a casino, these are very similar to any content app from a Netflix to a DoorDash, to an Amazon, where you potentially have the same UI/UX format for all of these applications, but you have different swim lanes of content, you have different categories of content. So for the sportsbook in particular, we have certain categories like popular parlays, which is one that we show to customers. Some background because I think it's the most contested surface we have. A parlay is just a multi-leg bet. So if you don't know betting, it's parlay is a bit controversial for the customer. The margin for the sportsbook is higher than a straight bet. And plenty of our customers distrust that we surface them that. We show popular parlays is one of the first things you see when you open the app. So there's a real question about what to do there. When do we show that module? When do we push it below the fold? When do we take it off the page entirely? Our instinct was that hiding it from the wrong people would be a clean win. Fewer people see a high margin product that they distrust than the people who like it still go find it and everyone's better off, hopefully. So we ran that. We did it twice. Two separate live tests, different sports, different surfaces. Both of them lost revenue with statistical significance. Hiding the module from customers who had already engaged with it cost us with no offsetting gain anywhere else. We assumed low engagement meant low intent, but it didn't. And some of those people who were about to come back, you know, didn't come back. So w what we did with it was we just wrote it back into the objective function. So what what you mentioned earlier around like did that strategy make its way back into yes, we did that. Showing the module to somebody who's going to engage became worth 10 points in the next round and hiding it from somebody who's going to engage cost us 25 points. So like a two and a half times the penalty. Because we paid for that asymmetry in in real dollars in the previous test. We also built a hard floor into it as a learning. So like inside the segment that engaged in the last 60 days, the model can hide the module from at most, you know, 10% of them, whatever it wants to do. And two losing tests put that constraint there for the next one.

Ashley Stirrup: Yeah. So you've rolled out some pretty amazing tools to the whole organization. Has that changed how you work with these different teams, how you create a culture of experimentation, things like that?

Michael Mathews: Yeah, definitely. I think we have a lot of radical ideas. And it's I think one of the hardest things to do, especially at the scale of FanDuel, is just kind of like get everyone on the same page and sort of being influential and convincing that you should move a certain direction over just the status quo of what we've been doing. So I would say that you know, AI in particular ha is is controversial when you should use it, when you shouldn't use it, what it's good for. What I'll say is that with product experimentation, agents and LLMs are uniquely really good at finding improvements to whatever you're trying to measure. Like whether it's a churn model where you're trying to predict will this customer come back and purchase something in the next seven days or will they not? If you have enough data, you have enough time, you have enough people, you can figure out, at least in an offline sense, can we improve this prediction? And that's never been you've never been able to to automate that part of the job before. It's always been a person that would be searching through data sets, doing exploratory data analysis, and finding different features and models and hyperparameters to apply to the problem to raise whatever that prediction is gunning for. So yeah, it's it's just long story short, I think people are starting to warm up to the idea and the way that you get them to make changes is you have to show evidence. So that's what we're trying to do.

Ashley Stirrup: Yeah. Well there's no question AI is uniquely good at a number of things. Has this are there certain things that you feel like humans are uniquely good at that you think that they'll they'll spend more time on as these kind of technologies come out?

Michael Mathews: Yeah, I think there's a lot we can do to speed up human processes and there's a lot of processes that we can hand over entirely to AI, but I think the number one process that won't change for a really long time is designing the like the hypothesis for an experiment and the goals. I think you can have AI plausibly come up with that and it w I I think it would do a decent job at it. But I think humans are uniquely in a good position to come up with what the abstract goal is for for a business, especially across like many different products and services and features. It's hard to give an agent the full picture of not only the business itself, but maybe like the macro or the micro markets that that surround it. So I can definitely like talk about where I think the autonomy should be put in place and where the human can add value, but very broadly it's like we need we need really good hypothesis from people who have been in the business for a long time, have gone through the different experimentation life cycles, and can uniquely point out like where there's opportunity in the business itself. And then from there, kind of like the micro aspect of finding the strategy to go after that hypothesis and prove it right or wrong, or building an experiment or like an and guardrails around an event around an experiment, I think a lot of that can be automated. And again, the reason why is there's some if as long as there's a goal that you can measure it against, it can just search through different solution spaces, different data sets, and it can, you know, quote unquote hill climb against a number that you give it, w which it's uniquely in great position to do.

Ashley Stirrup: Yeah, I would imagine it must be even more important to really think through the objectives like you talked about in terms of how many points, negative points does this get and positive points does that get. And I assume some of that only comes with like learning, you know, like doing it a few times and and starting to realize the impact that different assumptions have. So has this led you to be able to run more experiments a quarter?

Michael Mathews: Yeah, I think so. I think we have been I started at FanDuel in January and since then we've definitely increased the number of experiments. We're about maybe trailing twelve months around two hundred something experiments and expect that to potentially go up to five hundred or so. I think we're running experiments concurrently seven or eight, you know, with max being around like fifteen at the same time. And I I expect over time that number should also go up. And I also think you can start to use agents and intelligence more effectively to increase the concurrent number of experiments. A lot of the conflicts that you see with concurrent experiments are just because they were designed poorly and and maybe ran together in in such a way that doesn't make sense for the most throughput that you could possibly have.

Ashley Stirrup: Yeah.

Michael Mathews: So yeah we're we're we're I we're j identifying opportunities for throughput as well as like the total volume over time.

Ashley Stirrup: Got it. And are you able to measure the business impact you're having?

Michael Mathews: Yeah, absolutely. So maybe like how we measure things. There isn't one North Star that everyone agrees on and it kind of depends what product or person you're talking to. We have several metrics that different documents might call a North Star. I suspect that's more common than most people admit. Where I've landed and you know this is my personal position, is that I think you need two different things. You need a North Star, which is what the business is for at the end of the day, right? Mine is long run expected value per customer. Can debate how to define that, but I think that category of long run expected value per customer is the right one. Expected because realized revenue in a sportsbook business is luck contaminated. And I've watched like a five percent realized number change substantially under luck adjustment. That just means like if you have an expected margin of three percent for you know the NFL games running tonight, sometimes you'll see six percent margin, sometimes you'll see minus three percent margin, but it's inherently lucky. So

Ashley Stirrup: Yeah.

Michael Mathews: We have to adjust it for luck, which is interesting.

Ashley Stirrup: That is not too many businesses have that dimension to it. Like, well this

Michael Mathews: Yeah.

Ashley Stirrup: This might be a profitable product. Yeah.

Michael Mathews: Right, right. And over time it converges to whatever the expected margin is. It's just very spiky on a day to day or a week to week basis. So it's

Ashley Stirrup: Yeah.

Michael Mathews: Not something to freak out about if your margin dips to, you know, minus twenty percent last night. Like it's because the volume is low and over time we know it centers on whatever the expected margin is. Expected margin is just it's just an ML model that we run that predicts like what we should be earning and it tries to remove

Ashley Stirrup: Yeah.

Michael Mathews: The luck adjustment. Yeah, so like for long-term customer value, the horizon has to be long because you know, short-term revenue optimization in a sportsbook has a specific specific failure mode. The highest margin products are are the ones that customers generally trust least. And separately you need some OEC, which is you know what you grade an individual experiment on. And that always has to be short horizon, right? Sensitive enough that a a three-week test can move it and and measurable without depending on a model in the middle. A 12-month forward value isn't observable inside a three-week test. So to grade an experiment on it, you have to be able to predict it. And then you know, you're optimizing a model's opinion of an outcome. So every result you know could inherit the model's bias. So the work is kind of a link between the two, proving that the short-term metric you grade experiments on actually predicts the long-run value that you care about at the end of the day. And you do that by looking backward, right? Taking experiments

Ashley Stirrup: Yeah.

Michael Mathews: You've already run and checking whether the short-term movement predicted where those customers ended up a year later.

Ashley Stirrup: Yeah, I would imagine that's something you have to just be constantly monitoring and that a model that might be predictive today might not be as predictive in a year.

Michael Mathews: Yeah, absolutely. And there's a couple really interesting, unique metrics that only sportsbooks deal with. The first one is is the one I just mentioned. It's the the luck adjusted margin. The second is responsible gaming, which is a category most companies don't have. There are metrics that we optimize to stay inside of it. And it's unique because if you're optimizing too well, you can get into a situation where you've now lost the customer forever. So we have we have really strict guide strict guardrails in place to make sure the customer doesn't fall into the category of addiction and we take it really seriously. So a machine, it doesn't know that a number going up can be bad. So as we

Ashley Stirrup: Right.

Michael Mathews: Scale up optimization autonomy, we have to guide the optimization in such a way that doesn't result in addiction. It's a really interesting and difficult problem. We can't be too good at what we do.

Ashley Stirrup: Yeah, that is interesting. Switching gears a bit, do you have an example of a specific experiment you ran where you had a lot of learning?

Michael Mathews: Yeah, I mean I think the show/hide popular parlays module was the most interesting. Maybe maybe we could talk about a a churn model. So you know, I think what I've learned over the last few months is that I think the cleanest impacts that we're driving is on cost and time and like we've measured that in whenever I talk to leadership or we we have conversations around, you know, moving priorities up in you know, in the category of these these services that we're building for experimentation. I always say that like, you know, a a a data scientist improving a model today is a is a multi month project. And they get to a handful of those a year because they take a while to build and their calendar is full. So we pointed the auto researcher at this one as well. It was just one run, it took about an hour. In the case of this churn experiment and resulting strategy, it took about 750 candidate features down to 20 and increased the the F1 score by about six percent, which we think increases revenue over baseline by at least a million dollars. Yeah, and this is just one run of our auto researcher.

Ashley Stirrup: Yeah.

Michael Mathews: Then we tested it live. We had three arms, roughly 47,000 users each, about 10 weeks for the test duration. One metric came back significant, which was we called the 14-day lapse rate, which just means like a prediction if the customer will make a bet or not in the next 14 days. This resulted in, you know, down about you know, nine tenths of a percentage point. So in plain terms, it's just a about a couple hundred customers a year that we were able to keep who would otherwise had left. And we want to apply a lot of models to this idea. So that you know, this one was potentially results in maybe a million dollar gain. It's it's just one model.

Ashley Stirrup: Just so I understand, so you were testing out different models for predicting churn and then were you taking certain actions based on what the model told you?

Michael Mathews: Yeah, so this is where this is where this experiment gets difficult and we're trying to really control for different variables in the future. What sportsbook and casinos do is when they tr when they predict a churn for a customer, they obviously have to take an action. The action that we take is some sort of promotional campaign to get that customer to come back and make another wager or come back and play the casino or whatever it is. So we do something called business as usual, which is control. It just means give like everyone that they would have given a promotion to every Thursday. So it's like we have this one cohort or category of customers, which is most of the population, that we're giving a we call it a bonus, a a bonus campaign to that customer base every Thursday, same time. So they're conditioned. The treatment in this case is we're giving these customers you know specific promotions based on who they are, based on if the model predicts this customer is going to churn or not. And this can happen, whenever. This isn't like a strict timeline every Thursday afternoon. So the problem that I'm having is trying to understand. How to control for what we do about it to get them to not churn. And that's where I think, you know, it can go into like what the autonomy future looks like at FanDuel. But I think we need to start controlling experiment designs that marry the strategies that we're proposing through our auto research. Because today they're a little bit j disjointed where we're proposing strategies with, you know, the this auto researcher that we've built. And then the human is still making calls on the experiment design. In this case, how do you offer the customer you know, the right promotional or bonus package to get them back? And how do we know that it's not you know, kind of contaminating the results? So that's definitely like a next step for me is trying to figure out how to ca how to control for that.

Ashley Stirrup: Yeah. And just to be clear, the work you're doing, you're not actually like making UI changes, like you're not having the AI do that. You're just having the AI kind of come up with recommendations and algorithm changes and things like that, yeah?

Michael Mathews: Yeah, I mean I can I can talk about what we're doing more specifically. I know I I throw out this term autonomy without explaining

Ashley Stirrup: Huh.

Michael Mathews: It. So this is where I think most companies are going in the future and you know, I think FanDuel is still in the thick of this and we're working on becoming more confident with our autonomy. But from a high level, you know, experiment stages have like six six different components to them. So someone forms a strategy. Somebody designs an experiment for that strategy or or hypothesis. Who is the audience? What are the arms of that audience? What's the split between the arms? What's the duration of the experiment? Next someone builds what I call the envelope, which is the safety wrapper around it. This is the budget, the guardrails, the regression checks and the rollback logic. Somebody launches that experiment, somebody monitors, somebody analyzes it. It's very human intensive and it could take from hypothesis to experiment readout. It could take months. And it does take months. Our auto researcher at FanDuel does one of those six today. The other five are a person working by hand, you know, spending their time. So I want to just briefly talk about what these six levels are. Level zero is where I think most cutting edge companies are at today. It's just where the machine proposes a strategy and stops. So the human does the other five. Level one is the machine also designs the experiment. So it also picks the audience, the arms, the split, the duration. A human still builds the envelope, it still launches, it still watches. There's a rung in between that I'd argue matters most, which we call shadow mode. That's where the machine produces the complete package. It designs an envelope and all, it runs its own launch and monitor and rollback logic against results in production. And humans decide what actually goes live, but you record the machine's package, it's calls against every human edit, every kill, and every result. And nothing is shipping from shadow mode. What we're buying from all this is evidence. Right. The objection to all of this comes down to trust at the end of the day. And you don't talk anybody out of that without strong evidence. You show them six months of the machine's decisions beating the human-based decisions. And that's where you earn that trust.

Ashley Stirrup: Yeah.

Michael Mathews: Yeah. So level two is the machine building the envelope too. So after a human approves, it launches, it monitors, automatically rolls itself back if it breaches a guardrail. Human data scientist approves the full package and each launch and handles exceptions. This is the level I'm aiming for with my teams. Again, call it level two autonomy on just one surface or type of experiment. Huge unlock there, I think. I'll just briefly talk about what comes after that because this gets into further in the future. I think level three is where the approval moves. So the human stops approving individual experiments and starts approving just the envelope or the safety wrapper. Once an envelope is signed off, the machine runs experiments inside it on its own. And that's like the next big unlock because now it's traffic that becomes the cap on experimentation throughput. Today the cap is on you know how many launches a person has time to review, how many employees you

Ashley Stirrup: Right.

Michael Mathews: Have. Level four is a continuous stream on approved surfaces. So like a machine could allocate traffic, kill losers, scale winners. A human sets the objective and watches for exceptions. So in product experimentation language, this is where we start to trust autonomous deployment of a multi-armed bandit or multiple strategies at once that converge on a winner. And then finally, level five is the machine choosing which objectives and surfaces to go after. The line between four and five is just the objective itself. At four, it optimizes the one thing you told it to optimize. At five, you had a menu of approved objectives, each one ranked and guardrailed, and it decides which to chase where and how to weigh them against each other. Humans still define every objective on that menu. It's just the machine's job to allocate across that menu. I'll stop there. I think I usually get a lot of questions after going through the

Ashley Stirrup: You no, yeah. Yeah, no, you've you painted a great vision for just how the market could evolve and it it totally makes sense. So I'm just curious, like it it seems like what you're describing, like I'm I'm trying to see how it would fit. For example, let's say you've got an onboarding funnel and you're trying to understand what the user experiences at kind of each step of the process and you want to optimize each one of those steps. That feels a little more, I don't know, human intuition like, maybe we should have some more guidance here or what have you. Like can you see what you just laid out as applying to things like that, or is it better suited to kind of more algorithmic tests?

Michael Mathews: No, I think that's a perfect example. And it's something that you know we we want to be looking at as well. We have a very unique funnel for gambling. So the funnel just looks like this. It's registration is a first step. The second step is what we call KYC, which is Know Your Customer. You see that like in in financial and banking apps as well. And the the third step is depositing money into your account, and the fourth step is making a bet. So when when I apply this to the auto researcher and the autonomy ladder that I just spoke about, what I what I'm talking about is for each one of those steps, you have some verifiable number that allows you to say that we're doing a good job on registration or KYC or deposit, or we're doing a bad job. So just to simplify this, you could say did this person convert on this registration page or did they not? Right? Did this person convert on KYC or did they not? I it's it's more complicated than that because there's a lot of reasons why someone may or may not convert, but you could potentially give this to an auto researcher and say, here's all the data that I have on the registration page. I have every click that's ever been applied to a registration page. I have every like marketing source where we we got these customers which may impact registration for whatever reason we have all this data millions and millions of of customers now I want you to help me predict how we're supposed to get someone from registration to KYC right to deposit and a strategy in this case as long as long as a human can hypothesize, you know, what's going wrong on registration, you can give it a data set to predict how to improve it. So let's say let's say that we've tested registration or submit button size. Maybe the most simple experimentation example that I can give you. Let's say we have a history of changing the number of pixels on our registration submission button, and we have millions of these examples. We can potentially hypothesize that the larger a submission button gets, the more likely it is for someone to submit the page. You can potentially find this with a simple data analysis, but you can imagine like more complex problems where something very simple like just a l logistic regression algorithm or like whatever random forest or boosted trees, something that would take a very highly paid and highly skilled individual, like a data scientist, to build that sort of strategy to improve whatever you're hypothesizing against. You can do that a thousand times with an auto researcher versus like maybe you

Ashley Stirrup: Yeah.

Michael Mathews: Only have two or three data scientists. I think most most companies don't really leverage data science at all. Or maybe they have one or two. So, I do think that that a a registration funnel is a is a perfect application for all of this.

Ashley Stirrup: Got it. And one of the things you talked about was kind of scoring the I think you said experiment design. Like the like your confidence that okay, this is the right experiment to run. How how does that work?

Michael Mathews: So I think the experiment and the strategy are very tightly coupled. In the sense that you can't have a good one without the other. You can't build a strategy without an experiment design, can't have an experiment design without a strategy. So I think one of the challenges that that we face is these often become decoupled in large companies where you have maybe a product analyst or a product manager designing. A hypothesis or a strategy around an experiment. And then you have something like a commercial team or accountants finance team who come up with an experiment design because and not like the specific technical details of an experiment design, but this is our financial goal, this is what we want to learn from this experiment, and it gets converted into technical design, but they drive the design by what they want with the business, whether it's the marketing team. Whether it's like whatever throughout the company. And I think that's a big problem where you just you can't allow like outside influence from the business to influence experiment design. Of course they can influence whatever the North Star is, right? Like we want to we want more customers getting through registration. We see that that step of the funnel is really poor. So anything that you can do to get customers through registration is a good thing. Well, at the end of the day, you may build an experiment that gets everyone through registration, but for whatever reason, the changes that you applied on registration has them doing poorly on KYC or deposit or betting. Just a simple example, but that's why you just can't allow outside influence to design an experiment. They have to be very tightly coupled with the strategy.

Ashley Stirrup: Yeah. But y kind of going back to something you were saying earlier, you talked about it iterating on the experiment design and like just how does it know, this is a good experiment and how does it you

Michael Mathews: Yeah.

Ashley Stirrup: Know, versus the next one?

Michael Mathews: Right. So it finds a strategy first. That's the first step. And then what it does is it works to figure out how to build an experiment that's going to have the highest likelihood to either prove the hypothesis or disprove the hypothesis. So the entire purpose and the entire point of running an experiment is so that you run something that's statistically significant and gives you an answer. That this hypothesis was correct or it was wrong and why. Right. So like that's what it does. It it builds a strategy autonomously based on your data and the verifiable goal that you give it. And then everything around experiment design is only just geared towards you know proving or disproving that strategy.

Ashley Stirrup: Yeah. Do you give it the option of having multiple variants?

Michael Mathews: Yeah.

Ashley Stirrup: I would imagine it would pick multiple variants every time, yeah.

Michael Mathews: So it depends. Like we don't as of right now, we're not giving it autonomy for multiple variants. We have one, I guess it depends how you define multiple variants. The lat one of the most recent experiments we ran, we had essentially three different arms. We had a control, which is keep this content module for everyone. Everyone sees the same thing. The second arm was to remove it for everyone. Has doesn't have a propensity to want to bet on parlays. And the

Ashley Stirrup: Mm-hmm.

Michael Mathews: Third one was to push below the fold. So yes, I think in a lot of cases it will come up with with with multiple variants. It it also depends on what you're trying to achieve and the goal that you give it. I think in some cases especially in customer churn I've seen this it's predicting a a binary outcome. So it wants to propose a strategy like geared towards that binary outcome. So yeah I think it depends.

Ashley Stirrup: Yeah, yeah, super interesting. One thing I think's always extremely important when it comes to experiment design is thinking about okay, if you lose, how will like what's the information that you'll want in order to get the most learning that tells you, I should iterate or okay, you know, this is enough. I should go look at some, experiment somewhere else. And how does that all kind of factor into how you design experiments?

Michael Mathews: Yeah, I think that, you know, we always what's most important is that we always measure baseline. So like what if we just do nothing? Like what is the revenue that we're getting or whatever goal that we're setting for a particular experiment? Like let's just do nothing and see what happens and keep that at all times. And I think that like as long as we continue to see lift over baseline that's meaningful. It makes sense to continue to iterate until that lift becomes smaller and smaller. And you start

Ashley Stirrup: Yeah.

Michael Mathews: To see this as a preview when you're designing these strategies offline. So if you again going back to auto researcher, if it starts to develop and converge on a winning strategy that it finds, and it can't find a way to increase maybe the accuracy score that it's going after with churn prediction or whatever it is, you run it in production and you generally do see a lift over baseline in production if it works offline, like the it's highly correlated, right, from offline to online I've seen.

Ashley Stirrup: That's great. I mean that's a huge thing, right?

Michael Mathews: Yeah, it's huge. It doesn't always work, but I think the simpler experiment the better. Like hide this content or remove it or drop it below the fold. Something that simple, it generally does

Ashley Stirrup: Yeah.

Michael Mathews: Work offline. And when you have, you know, data sets with hundreds of billions of events, it becomes d like just the way to to find the correct answer that has correlation to to online results. So I I think like when you run experiments online and in you know, maybe you come to two or three in a row where you're just not able to find any lift over the previous experiment, I think that's a sign to slow it down and start to start to think about different hypotheses and different areas of the

Ashley Stirrup: Yeah.

Michael Mathews: Business that you can improve. So it's really just like a local maxima issue where you hit the top and more experiments isn't potentially worth the time or investment.

Ashley Stirrup: Yeah, interesting. Yeah, because like one guest had on, he talked about doing the opposite. So let's say you had a form and you wanted you you you wanted to test reducing the amount of text that's on that page. Also have a variant where you add more text on the page. And he said that they were surprised how often doing the opposite actually you know, won versus doing what they thought was gonna win. He also said it was very powerful just to identify, okay, reducing text didn't actually move the needle. Adding text actually hurt performance. And so finding that signal that, okay, can I actually move the metric, even if it's move it the wrong way, that tells you, okay, I found a lever that now I can go experiment some more on. And so I think exposing those kinds of learnings, even if you don't get a winner in a test, can be very powerful.

Michael Mathews: Yeah. Yeah, I mean overall I think losses are very valuable. The two losing parlay tests are worth more to me than some of the wins, because

Ashley Stirrup: Yeah.

Michael Mathews: They ended up as a a penalty weight in this case and a floor inside the envelope that we've developed. When you have a loss that becomes a coded constraint for the next experiment, that can pay dividends for a long time.

Ashley Stirrup: Yeah.

Michael Mathews: Yeah.

Ashley Stirrup: Yeah, it's interesting because it's a different dimension of learning. It's not just learning, my users like this, my users like that, but here's a better way to design the experiment itself. Yeah.

Michael Mathews: Yeah. Totally.

Ashley Stirrup: Well, Mike, thank you so much for coming on the show today. I feel like we got a a glimpse into the future. And there's clearly so much potential for AI around experimentation.

Michael Mathews: Yeah, absolutely. Thank you so much for having me. This was fun.