Transcript

Delina Ivanova (00:00) I think that a lot of product designers, product managers, and marketing folks have a really good intuition about what's going to work. So if something doesn't work, it's either going to be a messaging problem, a targeting problem, or an experience problem. And I think the goal of any experiment is to figure out, if you have high conviction something should work, which one of those 3 things do we need to fix?

Ashley Stirrup (00:52) Hello and welcome to today's episode. Today I'm excited to have Delina Ivanova, Director of Growth and Investing Data Science at Wealthsimple. Welcome to the show.

Delina Ivanova (01:02) Hi, thank you for having me.

Ashley Stirrup (01:03) Yeah, it's really fun to have you on the episode. Why don't we kick things off by talking a little bit about Wealthsimple? Tell us a little bit about the business.

Delina Ivanova (01:12) Yeah, for sure. So Wealthsimple is a financial services company here in Canada. We started off as a brokerage. We've recently launched our banking arm, so we now offer everything from checking, credit cards, personal loans, portfolio loans, and then of course all of our brokerage services. So trading, investing, both corporate and personal.

Ashley Stirrup (01:30) Sounds great. I love the name Wealthsimple. We all want wealth and we want it to be simple. So it's a great name. Love it. And so you just operate in Canada today, is that right?

Delina Ivanova (01:42) That's exactly it. We just operate in Canada today, yes. That's right.

Ashley Stirrup (01:45) Yeah, and you have 4 million customers. So that's a lot of customers. So you have a lot of traffic to do A/B tests on, yeah?

Delina Ivanova (01:56) That's right. Yeah, 4 million strong. 100%. We've got a lot of clients. We're always launching new features. I would say we're probably pretty fast with how quickly we launch features. So we do a lot of experimentation on everything from marketing, new features, both product side, communication side. So I would say it's a never-ending experimentation world where we are.

Ashley Stirrup (02:17) Love it. And is that your entire focus there, or do you do other things as well?

Delina Ivanova (02:22) So all of our data science teams do experimentation as part of our mandate, but we of course do other things like data modeling, of course, machine learning, predictive modeling, prescriptive modeling, analysis, deep dives, everything under the sun. I would say we're all kind of full-stack data scientists, so we don't really distinguish yet experimentation to be something separate. It's a core function of the data science offering.

Ashley Stirrup (02:46) Got it. And how long have you been doing A/B testing?

Delina Ivanova (02:49) I think as an organization, we've been doing some version of A/B testing for as long as we've been around. But I would say that we have more seriously invested in experimentation since last year. So it's been almost about 2 years now that we've been formalizing the process of experimentation, trying to bring people with expertise of how to design and run good experiments, trying to improve the culture of experimentation at the company. So educating our stakeholders, building the right processes, making decisions in the right way. And it's going to take a long time to get to a place where we're probably very mature in our experimentation practice. But at least for the last 2 years, it's greatly improved since I've been here at least, because the organization has invested a lot of time and effort and placed a lot of value on experiment results for decision making.

Ashley Stirrup (03:42) Yeah, that's exciting. You've been at a pretty wide variety of companies. Is there one you kind of look back on and think was particularly strong at A/B testing?

Delina Ivanova (03:51) I would say my last company, Misplay, was probably the strongest I've been at, because it was just such a good embedded practice in everything we do. One thing that I learned there, though, as we moved very quickly in doing a lot of tests for a lot of different product features and algorithms, like AI algorithms: when do you need an experiment versus when do you not? I think this is something that companies kind of come across. Experiments are valuable for a couple of things. The obvious thing is the learning and making a decision of what's next, but they're also very valuable to understand how much value a feature or a program brings. And so I would say at Misplay, we were very good about using them for both of those purposes. So we would always launch something with a test so that we could measure the lift, and then we could speak to, is this actually valuable or not. We probably were doing a lot of tests on features that we don't necessarily need to do tests on. So we eventually got to a point while I was there where we said, you know, certain features don't need an experiment. We have high conviction that this is going to work. We know from experience this is going to work, so we don't need to kind of spend the time setting it up. But then as we evolve the feature and change it, then of course we would test iterations so that we can learn incrementally what is working. I would say that other companies I've worked at, including Wealthsimple, the approach historically has been, let's kind of launch these things that we know are no-brainers. We need to have these features as part of our offering. These are core things that we need to have as a brokerage to attract the type of client that we're looking for. And so we don't really spend too much time here thinking about the lift necessarily. We wouldn't do a test just to measure how impactful is this, because the company's been growing so quickly, and usually when you're in a hypergrowth phase, you don't need to do that. The focus is more on just get as many features out and do the things that you need to do. But usually when companies stop growing as quickly, that's when you kind of start to need to optimize where you put your attention. And this is when typically experiments become more focused on not only is it working, but also is it adding value, and how do we pick the things to invest in.

Ashley Stirrup (06:00) Yeah. Do you have any good rules of thumb that you might offer listeners in terms of when would you test something and when would you say, nope, we don't need to test that?

Delina Ivanova (06:10) I think if it's something that is a complete no-brainer. So for example, let's say you have a very manual experience for your clients. And I'll use financial services as an example. In Canada, a lot of banks require a lot of documentation, including us. And so anytime we have an experience that's annoying for clients, like you have to upload forms, you have to upload PDFs, you need to go and scan something and put it in your app or email it to us. And if we find a way to make that experience better with some automation in the app, so it's no longer going to require people to do all of these annoying things, for sure we don't need an experiment, because essentially what we're doing is we're removing a blocker of somebody using a service. And so it should be accessible much easier. And so we don't need to test the lift. Like, this is a no-brainer. Anytime you have any situation like that, that's a no-brainer. I would say just go for it and don't bother testing. Now, from there, if it's a question of, our current funnel has 13 steps and we think a new funnel with 7 steps is going to get more people to complete the funnel, you should test that. Because sometimes if it's a very high-intent product, let's say a checking account, where everybody needs a checking account, if they like you and they've chosen you as a company, if your funnel is 10 steps versus 5 steps, people are probably still going to complete the funnel because they need a checking account. But if it's something optional, like let's say we want them to partake in a new feature, like Wealth Rank, for example, or some of our social features, a long funnel is going to be annoying. It's an optional product; people are not going to do it if your funnel's long. So here we can actually measure, in a case like that, what is the lift? Are we getting more people converting or signing up for a product? So that's kind of the rules of thumb I use. And then again, what I mentioned earlier about the growth phase. If you are, especially if you're working with PE companies, VC-backed companies, the companies that really manage performance tightly, and you need to be able to speak to, here's all the work we did and what is the value that that work added, then you probably should do experiments so that you can measure and speak to the value for your investors. But if you're not under that type of pressure, you can be a bit more lenient, I think.

Ashley Stirrup (08:21) Yeah, those are all great points. It kind of reminds me of an example from Ronny Kohavi, who's kind of one of the fathers of experimentation. He's written a great book and used to be at Microsoft. And one of the tests they did is they tested actually slowing down performance on Bing, just a slight bit, and then they measured the revenue impact. And it allowed them to understand what the value was of an additional millisecond. And I forget the stats, but it was something crazy. Like if you're a performance engineer and you could improve performance by 3 milliseconds, you just paid for your salary for the year. And so it led to Microsoft significantly increasing the team focused on performance once they realized what the true ROI was. So sometimes you might have a feature where, okay, this feature is a no-brainer, but there might be 7 other similar features that you're less sure of. And once you can put a value on that first one, then it helps you understand how to prioritize the other stuff. But yeah, love your examples. So for Wealthsimple, how is the team organized? Is experimentation kind of centralized? How many product teams are you working with? Things like that.

Delina Ivanova (09:34) So we don't have a centralized experimentation team per se, but we have a centralized group of people that are experimentation experts. And so they sit in different data science teams, but they are the ones that are driving our experimentation roadmap. So that includes things like best practices: how do we propagate the right practices across the teams? How do we educate people? How do we educate data scientists on creating, designing, and running good experiments? We have 3 data science teams that work directly with stakeholders. So my team focuses on growth and investing. So we work with our growth partners, marketing, and then investing product, so all of our investing features, brokerage features. We have a banking data science team, which also does credit risk and market risk. So this is our big modeling team. So all of our complex fraud models, credit models, market risk models, and all of our banking things like credit card, checking. And then we have an enablement data science team. So they work with all of our back-office functions. So client support, sales, brokerage operations, all of the different ops teams. So any kind of optimization or queuing problems, they would be handled there. Each team's data scientists work with specific product managers or business partners, and they are responsible for working with that partner to decide when should an experiment be done, how should we design it, what are we measuring, what are the metrics of success, how do we know that this worked, and then what is our go/no-go decision. Wealthsimple is growing very quickly. So we don't always want or can wait for an experiment result to be significant before we make a decision. There are problems that are more questionable than others. So again, some things are quick wins. We know they'll work, or they're very low risk. Like even if we launch this and it doesn't really work that well, it's fine. But there are other areas that we do need to test more. We need to understand, is this resonating with people? Are we messaging this product in the right way? Do we have the right structure? And so this is where, particularly on the data science team, we've been working with our data scientists to say, how do you design a good experiment? How do you know you're targeting the right people? How do you make sure that there's no bias in the groups that you're reaching? How long should we run this for? And what is the metric of success? How do we know that that's the right metric? How are we making a decision? And is the thing that we are testing aligned to the metric that we are measuring? And this is something we find when we design experiments very fast: this often is a gap, where we're testing something, but it actually isn't really connected to the thing we're trying to drive, and so we're not really measuring the right thing. And so this is something we're kind of working with our teams on now, how we elevate that practice.

Ashley Stirrup (12:15) Yeah, that's a great example. Do you find that in those situations, let's say you're trying to increase revenue, but this feature's actually an engagement feature, that it's still actually contributing to the thing you thought, but you just have to measure it differently because maybe it's an engagement feature, not a revenue feature?

Delina Ivanova (12:38) Yeah, I mean, 100%. I would say that whenever we launch new features, or when we're doing product marketing or trying to get people to adopt something, the goal is always going to be deposits or revenue. So those are the 2 things that we want to always drive up. And we have a lot of hypotheses on what does that, right? So for example, if we increase engagement with our app, is that going to drive revenue? Immediately, probably no. We have a lot of engagement features, but the thinking is that what it will do is get people to be familiar with the brand. They like the experience. While they're here, they maybe check out a couple of accounts, maybe they open accounts, and maybe once those accounts are open, they deposit, and maybe when they deposit, they drive some revenue. I would say that a good practice that we're trying to implement is, while these things are connected, of course we hope everything we do drives deposits and revenue, sometimes those metrics are lagging metrics. We're not going to see them until 60 days out, 90 days out, especially in financial services, because these are big decisions people make. You cannot run an experiment for that long. And so I think from a data scientist perspective, it's really important to understand the relationship of the behavior you're actually driving. Is that even related to the outcome that you want? And then what are the right in-between metrics that you should monitor for a period of time to know if that outcome is likely? And if you're not seeing that movement in engagement or account opening, for example, which might be the more immediate metrics, then you have to make a decision on early metrics, because you are not going to be able to see those lagging metrics for a while.

Ashley Stirrup (14:11) Yeah. A recurring theme on the show is that every business's customers are a little unique, and their buyer's journey in particular is unique. And especially with a product like this, versus say buying a pair of socks at a clothing store, it's going to be a much more considered purchase: multiple visits, checking out alternatives, doing research on third-party sites, all sorts of different things. So I would imagine a big part of your job is just continuously trying to further learn about the customer journey and all the different variations of it. So pretty interesting.

Delina Ivanova (14:49) Yeah, absolutely. And we talk a lot about, when we're designing tests, for example, I mentioned earlier, are we designing the tests in a way that's going to drive the thing that we're interested in? And this itself has been a learning journey. And I'll give you an example. One of the things that we do at Wealthsimple is we try to make tools available to clients that they cannot get at traditional banks. So for example, recently we've launched margin, the ability to borrow on margin. And this is not something that's widely available to the average person in Canada. And so it's a new product. It's newly available to Canadians through us and through a couple of other brokerages. And so the way that you grow this type of product is either you find the small group of people that are active users of this product at other companies and you try to build an experience and convince them to come over, or you try to educate people that it might be good for, that have not previously had access to this, and try to get them to try this sort of product. And so recently we ran a couple of experiments where we launched a couple of promotions. And one of the things we learned was, we've launched this promotion about this product, but we don't really have a lot of education. The people that we're targeting are not people that would typically be familiar with this. And so we're seeing these results where perhaps the behavior isn't what we expected. It's positive, but it's not what we expected. And we're thinking about, is it a targeting problem? Is it a messaging problem? Is it a product problem? And it's not really any of those things. It's more, in this case, an education problem. The people that we're targeting, they're not people that would know about what this is or why it's useful to them. And so now we have to go back and change our messaging strategy and think about how do we, we don't really need to test rates. We need to test messaging and how we communicate better, or explain this to more people. So that's kind of one small example of how you can run something as a version, you can learn from it and refine not only what you're testing, but you can realize that the thing that you're testing isn't actually going to drive the outcome that you want. And it's not that there's something wrong with the product or what you've built, but it's maybe some other gap that hasn't been addressed and hasn't really been considered in the design.

Ashley Stirrup (16:57) Yeah. Do you do survey work or other things like that to try to uncover those kinds of missing gaps?

Delina Ivanova (17:05) We do. We talk to a lot of our clients. We do survey work. My personal view on surveys and user research is that, in certain cases, it's useful, because people will tell you the thing that they would ideally do. For example, if you ask them, would you come over and trade on our platform, and what would be the things you need? And so people are going to tell you all these things, like, I really want this feature or that feature, and I need it to be this fast, and whatever. Sometimes that's true. When you're really talking to people that are high-intent people, maybe that's true. Is it true for somebody that maybe isn't a super active trader, for example? Maybe not. Maybe they're telling you the thing that they in theory would tell you if they were that type of trader, but maybe in their day-to-day they don't actually do that, and so it's not relevant. So I do think that a good survey or research strategy has to be combined with experimentation, because you need to hear the thing that people tell you, but then see the behavior that they exhibit, so you actually learn what people do. Otherwise you can invest a lot of time based on people's feedback that probably isn't well spent.

Ashley Stirrup (18:10) Yeah, I'm trying to remember where this guest was, but I think she was maybe Grubhub, but it was one of the delivery services. And they came up with a new app for the operations person at restaurants. And they went and interviewed their power users, and they told them all the great features they wanted. They went and built the app, and then the feedback from users was like, what is this? I only want to do these 3 things. So in the user research, they were like, here's the 10 things I want to do. In the real world, it's like, well, I'm actually really busy and I really only want these 3 things. So yeah, such a great learning. And so do you do any kind of experiment readouts where you get the team together to kind of both share best practices and demonstrate what good looks like?

Delina Ivanova (19:10) So I think this is an area we could be better at. We definitely share results. So we do run experiments, and then we share results at product reviews, or within the data science team at our own team meetings. We do share them broadly with the organization, so that everybody's kind of on the same page of what we're learning. Where I think our culture could improve right now is that we don't really have systemic learning. So for example, we do an experiment, we learn something valuable from that experiment, we take that learning, and then we adjust in the next round of experimentation, and then we do something that's a logical follow-on that would help us continue to learn on this journey of what works. Right now I would say, and part of this is connected to the stage of growth of the company, because again, we're growing pretty quickly. We're not really in a stage of growth where we need to incrementally optimize. So right now we're in a phase where we're just trying to find what is the thing that's going to drive significant growth, what is the feature or the messaging or the thing that we need to do that's going to drive fast, significant growth in a short period of time, whether that's deposit related or revenue related. And so because this is the stage of growth we're in, we want to test many, many, many different things and see which works, and then go and build that thing, and then kind of continue to find those nuggets. The thinking is that once we've kind of exhausted those nuggets, then you get into this sort of incremental learning path, where you test things over time and you have a proper experimentation roadmap. So this is an area of immaturity for us. We do definitely share results, we learn, it's useful. But what ends up happening sometimes is, as you have natural movements between teams, as people come and people go, we tend to kind of test similar ideas over time. So sometimes we've done the same test actually multiple times. Maybe somebody didn't know, so another team did it, and this team did something similar. And sometimes we do run into overlaps where 2 teams are doing the same thing. And that's kind of the hardest part right now with managing our experimentation practice. So I hope eventually we do get to a place where we can be a bit more methodical about what we're testing, what we're learning, and how this is informing what we need to test next.

Ashley Stirrup (21:16) Yeah. But it sounds like you're also making the right trade-offs, right? You're in a state of hypergrowth. There's only so many hours in the day. And so you're moving fast, and hopefully you're finding a few big winners that really move the needle, and so they're easy to see type of things. So then as you mature a bit more, then you start going for more of the 1 percenter improvements, which you can still drive tremendous growth off of, but it requires some more rigor in terms of how you measure and how you design experiments, all that. Do you have an example of an experiment you've run where you had some surprising results and it taught you something about your customers?

Delina Ivanova (21:56) Yeah, absolutely. I mean, the most recent one that I mentioned with our margin work, this one's been very interesting for us, because we kind of did a few different variations of this. So in one variation, we found people that we think are a very good target audience, and we targeted them very specifically with a specific offer, and we said, if you come over, we'll give you X rate. And we also at the same time had a deposit offer that's live in market. So this is a broad marketing offer. So if you move your deposits to us, you're going to get 1%. Now, this experiment was actually built on top of a small user research experiment, where one of our product managers very specifically contacted people and had a very high conversion rate. And so we were like, this is a cool opportunity to productize that outcome, because we saw such a great response. How do we put this in product and reach even more people? We didn't see the results that we thought we would. So we thought we were going to see a whole bunch of people with margin loans saying, great, I'm going to move my margin over, here it is. Instead, what we saw was a lot of people making deposits that were not necessarily in margin accounts and not necessarily margin account transfers, but all kinds of other accounts. And so at first we were like, this is still a great outcome. We're seeing people respond to this, which is really good. But what we're actually learning is that our messaging probably was off. Because we kind of combined 2 different promotions in one, we didn't really isolate the thing that we're trying to learn. People responded to the primary messaging that they were seeing, which is, you're going to get a 1% match if you deposit X amount of dollars. And that outcome was really great, but it was the wrong outcome of what we were actually trying to drive. What we didn't immediately see is the other outcome that we were actually after. So that then leads to how do we make decisions in the future. So now that we're going to go out and do more experiments, part of it is we need to clearly refine our messaging, because I don't think that what we are putting in the promo or the reach-out is resonating. It's not quickly landing, the thing that we're actually trying to incentivize. And so probably we want to decouple those 2 things. Probably we don't want to combine incentives in the future in one. Even if somebody might be eligible for more than one incentive, we don't want to combine that in our messaging, because what you're trying to drive gets lost in the message that you're sending. And then there's a question about our audience targeting as well. So a lot of companies, especially in growth phases, and especially like ours, we sort of make everything available to everybody at all times. And this is good. We have a lot of things going on. We make them available to everybody. But what I at least theorize eventually happens is when you have 30 or 40 offers out in the market, it gets lost, right? The person opens the app, the person has very limited attention on what they're going to pay attention to in that day. And so if the right offer isn't in front of them, the product that is best for them that you have, they're not really seeing that. So someone where margin might be a really great product for them, they're maybe seeing a different offer, like a deposit offer or something like that, which they probably don't really care about. It's not something that's relevant for them. And so that was another kind of big learning: traditionally we've taken this approach of, let's just target everybody with everything and see who bites on what, and then kind of learn from there. And now we're starting to shift gears more towards, well, actually, let's figure out what's actually good for this person. Do we have something for them that's going to save them money, make them money? And again, for a margin user, a discount on a margin loan is probably much better than, let's say, a match offer if they don't have that big of an asset pool. And so how do we surface that in front of them and just target them with that specific thing? So that's been a very cool experience. And I think it's something that I've used as an example across my teams and with our stakeholders to inform how do we design better experiments? How do we critically think about what is it we're after, and how do we select the audience that this will resonate with? How do we design the messaging? How are we going to be intentional about the thing that we're putting out, so that when we do measure the result, the result will be related to the thing that we're testing, but also the result will be true? It will be correct. Is this working or not? Or do we need to refine something else?

Ashley Stirrup (26:06) Yeah. To me, just listening to that, it's like, whoo, that feels overwhelming. So many different offers and all that. But it also reinforces the need for good data on your target customers, so that you can make the right offer to the right person. So super interesting.

Delina Ivanova (26:23) Yeah, absolutely. And experimentation is actually a great way to learn about people. And I think that's also something that many companies miss when they're thinking about experimentation roadmaps, because yes, experimentation is good to learn about which features work, and yes, it's good to make decisions about where to invest time. But actually what it's most valuable for, to me, is learning about people's responses. We don't do this today. I would like to do this eventually, but can we put in one place all of the experiments we've ever done? And how do we sort of do this mega learning of, let's say person A has had 10 offers thrown at them, these 3 are discounts, these 7 are match offers. And within that, we've built basically price testing, sensitivity testing. What did they respond to? What did they not respond to? And sure, there's other factors, like did they see it? Was it at the top of their app versus in their rewards center or something like that? These are things that we can respond to over time. But to me, that's the most valuable learning in any company. So as you're running all of these tests, that's really good. But can you set up the infrastructure to learn at a mega scale? And then basically you have a mega test built in, and you can assess pricing, messaging, responsiveness, like how is somebody responding to that? And then there's your personalization engine, right? But yes, it's a long-term vision.

Ashley Stirrup (27:39) Yeah, super interesting. Do you have any tips for the audience on how to design experiments so that if your feature loses, but you believe in it and you want to iterate, how can you get signals that tell you how you might iterate?

Delina Ivanova (27:55) Yeah, it's a good question. I think that a lot of product designers, product managers, and marketing folks have a really good intuition about what's going to work. So if something doesn't work, it's either going to be a messaging problem, a targeting problem, or an experience problem. And I think the goal of any experiment is to figure out, if you have high conviction something should work, which one of those 3 things do we need to fix? So you should design a testing roadmap. So if we're launching a new feature, how do you test targeting? Well, you create a few distinct groups of users that you're going to target with distinct profiles. And so if you're not sure, maybe you've seen this work at a different company, it should work here, you're not sure which group it's going to resonate with, you can do A, B, C variants and test different types of profiles and watch who converted or who engaged with this feature most. If you have an experience problem, like you're not sure if the experience is designed the right way, so again, it should work, but maybe we're not designing the experience well, maybe you can test a couple of different funnels. That's sometimes hard, because it requires an engineering team to build several different experiences, which is not always great. We tend to use a lot of Braze, so we use a lot of lower-lift tools to test experiences before we invest in a proper engineering build. So that's always an option. Can you test this with marketing cards or some other tool like Braze that can help you test the concept before you invest resources in it, and then test a couple of variants of it? The other thing you can do is you can monitor the results. So if you have a single funnel, where's the drop-off happening? If people are entering the funnel, then obviously there's interest there. You're doing something right, because they're interested. But if they're not converting, then you're doing something wrong somewhere in the funnel. And that should be a trigger that you should probably review the experience. And then the last piece is around messaging. So the example that I gave earlier, that's a great example, I think, of how messaging can impact the results you're seeing. So again, in theory, we have some of the best rates on the market. So if somebody has a margin loan, they should be coming to us, because we have some of the best rates in the market, and people who use margin care very much about rates. And so if it's not working the way we think it should be, then we have a messaging problem. So our funnel's good, people aren't falling through, it's obvious we have this product, our rates are really good, then probably we're not reaching people in the right way. So either they're not seeing it, or we're leading with the wrong message. And so again, the test there is a couple of different variants of deposit-first messaging, rate-first messaging, personalized messaging, and see what kind of lands with the person.

Ashley Stirrup (30:20) Yeah. I love that. That's such a great answer. Thank you for that. It's one of the best answers we've had on the show, so that's fun. As you look forward for Wealthsimple, what are some of your goals, and how you'd like to see experimentation evolve there?

Delina Ivanova (30:35) I think one of my biggest goals for probably Q4 and next year, that I've been talking to my team about at least, is basically this meta-learning structure. So how can we kind of take the things we've done in different domain areas? Can we yield some insights from that? Even if they're not perfect, can we start to pull some insights from that? So that's one. The second thing that I hope we can start doing soon is more uplift modeling. I think this is another thing that teams don't always think about, or companies don't always think about, is that sometimes when you reach out to people with marketing or a new product, they actually become detractors, right? So it's something that annoys them. They actually don't want to work with you because it's way too much marketing. And this is often missed, because we don't really look at results like that. We don't really look at an experiment and say, who did we turn off with this, and who doesn't like us anymore because we shipped this thing to them? But I would like to see our teams start to do more uplift modeling. And I think that's going to be really important for us as a company going forward, because then again, as we have way more offers, as we do way more of these engagement tactics, we don't want to annoy people. Certainly not our intent. We want to make sure that what's most relevant for someone and the best deal for them reaches that person. And the way we learn about that is through figuring out who's going to respond to this, who needs more, who needs less, who needs just information, who doesn't want any of these marketing tactics. So I'd like us to do a bit more of that. And then I would like our data science teams to themselves be more upskilled in how to design and run proper experiments, and then also to translate that knowledge to our stakeholders, because I think there needs to be learning on both sides. And so it's a bit of a goal for me of, how can we use the experiments we've done? How can we educate our stakeholders? How can we bring them along to, here's the possibilities of what we can learn and what we can do and why this is useful, and then hopefully get that buy-in to have a bit of patience when we launch things, so that we can learn and then go with a big bang once we have some good learnings.

Ashley Stirrup (32:28) Yeah. Your example on turning off users reminds me, I bought this luggage online, and the marketing was a little, let's just say, aspirational, right? As I got the product, I was like, mmm, I was expecting more. And so I ended up returning it, but the whole process took a while. In the meantime, I was just getting blitzkrieged with emails from the vendor. And I'm like, okay, not only did you lose me on this purchase, but all these emails, now you've lost me for good type of thing.

Delina Ivanova (33:01) Yeah, absolutely. And I think now, in a world where attention is limited and marketing has become infinitely easier, because anybody can now craft copy with AI, with all of these tools, you just have so much more of it. Companies have Claude, companies have these tools, they can just bombard. And a lot of the mindset of traditional marketing is, if you repeat the same thing over and over, then eventually people will catch on. But it can be a turnoff, right? So it can actually do the opposite of what you're trying to do, and that's a big part of learning who you shouldn't bother. And again, some people, they're high-intent people. If they need your product, they'll come and they'll get it. So let them discover you, right?

Ashley Stirrup (33:42) Yeah. And also just, some marketers kind of need to learn how much is too much type of thing. What about AI? Do you feel like there's some good opportunities for AI at Wealthsimple, particularly around A/B testing?

Delina Ivanova (33:55) Yeah, absolutely. One obvious application is messaging and copy, right? The speed at which we can test different messaging variants is unbelievable. Actually, our team has been experimenting with personalization as well, where we can use these large language models as predictive engines, not just copy or writing tasks, but even something like, what do we think this customer will do next, or what do we think is the right product for this customer, or how do we send them a message that's personalized? Like, hey, if you transfer this amount, you're going to save $200 a month. That's an example of personalization that's deterministic. It's not a probability or an estimation. And so AI allows us to move very quickly on these types of tests. It's of course not perfect, it makes a lot of mistakes, but we can test concepts very, very fast. And then if something does work well, actually, it's unlocked a lot for my team. In a traditional data science team, you may invest time building a machine learning model to predict something, then test the effectiveness of that prediction, and then hopefully if it's good, you kind of work on tuning the model, et cetera. With AI, we can test these things much faster. If they work, now we know, for example, it's time to invest in a proper model, and we can take the time to build it. So I think from an experimentation perspective, testing ideas very quickly has become much easier. You don't have to convince people that something might work and then invest a bunch of resources on a potential that hopefully works, and then be stressed out about whether you've made it work or not. So it's much cheaper to test ideas, which is exactly what we want with a good experimentation culture, and then much easier to make decisions on where do we invest the time.

Ashley Stirrup (35:30) Yeah, that's a great example of using AI to help you identify pockets of opportunity and then where you want to go and do more in-depth investments. So that's a unique look at the opportunities for AI. So I really appreciate that. Well, Delina, thank you so much for coming on the show. This has just been a wealth of great information. So I really enjoyed it. Thank you so much.

Delina Ivanova (35:53) Thank you so much for having me.