The Experimentation Edge - Kim Ting Li===Ashley Stirrup: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest.Welcome to today's episode. I'm excited to have Kim Li, senior manager of online experimentation at Home Depot with us. Welcome, Kim.Kim Ting Li: Hey, Ashley. Thank you for having me.Ashley Stirrup: So Kim, you bring a very interesting background. You started your career as a neuroscience researcher. Could you tell us a little bit about your career and your journey to Home Depot?Kim Ting Li: Yeah. So my background was in neuroscience and statistics. I started off as a neuroscience researcher running clinical trials and doing experiments on human subjects. And then I decided to pursue my career as a data professional in the industry. So I switched gears into the industry, did a bunch of years of consulting work until I landed at The Home Depot. I've been at Home Depot for over four years at this point. Now I'm leading the online experimentation team.Ashley Stirrup: And that's a pretty big business, right?Kim Ting Li: It is. Many people didn't know about this, but Home Depot, we have an annual revenue of $150 billion as the enterprise. And then for the online org that I'm working with, we just surpassed $25 billion.Ashley Stirrup: Those are both pretty massive numbers. Of course, Home Depot's a big business, but you don't realize just how big it is. I'm sure that makes experimentation that much more important there.Kim Ting Li: Very fertile ground for experimentation because we're serving millions of customers across the United States and online as well. It's just very good ground for doing large scale experimentation.Ashley Stirrup: Personally, I've been a very heavy shopper both online and in store, and used your app a lot. I think you've done a lot of great things there.Kim Ting Li: Glad to hear that.Ashley Stirrup: Could you tell us a little bit about the experimentation model at Home Depot? How much of it is centralized versus decentralized?Kim Ting Li: For us it's a very centralized model. Basically any major change you want to make to the website homedepot.com, whether it's a content swap or a feature update or a model change in search or recommendation algorithm, it will be tested by my team.Ashley Stirrup: And how many different product teams are there at Home Depot? I'd imagine there's a lot.Kim Ting Li: That is pretty broad. We serve probably 40, 50 different business teams. Not only product, but we also have customers from data science and UX. If you have an idea, we will support you to test it.Ashley Stirrup: That's great. And how many experiments are you running per year?Kim Ting Li: We're in the low hundreds right now.Ashley Stirrup: And how do you see the program evolving over time?Kim Ting Li: We're really pushing for this experimentation practice to be drastically scaled up this year and next year. I think leadership's really looking forward to us expanding.Ashley Stirrup: And you've got a pretty big team working on all this, right?Kim Ting Li: Yeah, so I have 10 associates doing the analytics, setting up experimentation, making sure it's got the proper hypothesis and doing proper analysis on it. And then we also have a sizable developer team and an automation function, because we're churning out all these reports for different experiments and a lot of it has to be automated so we're not draining everybody's hours doing this one by one. So overall probably about 40 people on the team.Ashley Stirrup: And with all those different product teams, I would imagine it's quite a challenge just identifying good learnings, figuring out how to share those across teams, getting visibility with leadership. How does that all work?Kim Ting Li: Each of my analysts has a very specialized area. They get to really learn the business, whether it's top of the funnel search, the data science function, or the bottom of the funnel. So they focus deeply into that. And for every test, they work closely with the product managers to understand what they need to get out of the test. We help them set a proper hypothesis and make sure we're evaluating it properly. And then at the end of the test, we'll have a readout session with all the stakeholders, because sometimes you have other parties involved. Marketing might have interest in this feature, or UXers have a particular say. So they will all be invited to this public readout for about 30 minutes. And at the end of it, we send out a blast email with the final results to share across the stakeholders. But those results are also all stored in a centralized place, so anybody can go back into tests that ran maybe in 2020 to get the learnings and see if something has been done already.Ashley Stirrup: And do you get much visibility with the executive team there?Kim Ting Li: Yes. And that's simply because all the executives and leaders love to learn about the tests. They really truly understand A/B testing is the golden rule to understand incrementality, the real impact. So a lot of support, but also a lot of visibility, and with it comes a responsibility to do good analytics. These leaders will jump into review sessions with you and ask questions directly. They check the library, and sometimes ping our associates directly to ask about a specific test.Ashley Stirrup: That's great. It sounds like it's a relatively humble culture where people are open to being wrong and wanting to learn. Is that safe to say?Kim Ting Li: We definitely try very hard to do that. We really cultivate this culture of test and learn. We test because we don't know for sure. I sometimes ask my stakeholders this question: are you going to do something differently if the test result comes out bad? If you're not, then let's not waste time. So it's really a collaborative effort to get to what's best for our customer and customer experience.Ashley Stirrup: And one thing I think is interesting about Home Depot, just thinking about it as a customer, is you must have very different personas you're trying to create a great experience for. The contractor versus the DIY person, somebody focusing on a bathroom versus building a new fence. Very different kinds of projects and skill sets. How do you think about that when it comes to creating great online experiences?Kim Ting Li: That is a great point, because we do serve both the DIYers and the pros, and they shop very differently. If you're a DIYer, you might be deciding what lights to use, and so on. But pros are very goal-oriented and project-oriented. So we do have a pro website as well. Depending on the feature, they could be tested separately as different audiences.Ashley Stirrup: It is interesting, because Home Depot has such an advantage with so many stores, and the inventory at those stores is really different. So in general, how do you help your teams learn from losing experiments? Particularly if you already had conviction that this should be a winner and it was a loser, how do you iterate on it and learn from it?Kim Ting Li: That's a good question. For learning, the really important thing is why this happened. Of course, the test could be set up wrong, so that's the first thing to check: do we have tagging in place to measure what we want to measure all the way to when the result came out? If everything else checks out, then it's really a deep dive into the customer journey. Where did people fall off? Where did the gap start to widen? When we narrow it down to the why, then we really get to how we improve it. So that's always the most important part when a solid test goes wrong: how do we get to why, and what's the next step?Ashley Stirrup: That makes a lot of sense. How does Home Depot think about North Star metrics and guardrails? Is it consistent from experiment to experiment, or does it vary a lot?Kim Ting Li: It varies quite a bit, especially as a centralized team serving so many different verticals. A test trying to drive add to cart, that will be their metric. Something like a banner that's supposed to attract more visitors, that could be impressions or click-through. So it really depends on which team is doing it and what the goal of that test is.Ashley Stirrup: And do you track total ROI from experimentation, losses avoided, and things like that?Kim Ting Li: Yes. We consider that part of our team's KPI as well, to say how much learning did we help and how much money we helped save.Ashley Stirrup: I know you mentioned you want to ramp up the number of experiments you're running. How do you think about that? Is AI a big factor? Is enabling more self-service a big factor?Kim Ting Li: I think AI is very helpful in terms of development speed. The IT org is definitely going through a lot of evolution in how they code and how fast they can get product features ready, so that would ultimately speed up this team as well. And in insights generation, AI can play a role there too. But for us it's a lot about unlocking capabilities. Right now we're still very much client-side testing, and we're developing server-side testing capabilities so everything moves faster, end to end. Once you have a learning that says this is a winner, you should be able to release it very quickly. All of this would ultimately contribute to a better testing experience, but also faster and more numbers.Ashley Stirrup: Terrific. You covered a lot of ground very quickly. This was a great episode. I feel like we got a great window into experimentation at Home Depot. I really appreciate you coming on the show.Kim Ting Li: Thank you.Ashley Stirrup: Thank you so much.