Transcript

The Experimentation Edge - Andrew Miller Recorded 2026-10-02

Ashley Stirrup: Hello and welcome to today's episode. I'm excited to have Andrew Miller, Director of Product Management, Growth and E-commerce at RealPage. Welcome to the show, Andrew.

Andrew Miller: Hey, thanks Ashley. Good to be here.

Ashley Stirrup: Yeah, I'm excited to talk about RealPage. It's a bit of a unique business. Why don't we kind of start off with that? Like if you could just tell us a little bit about the company and its customers and things like that.

Andrew Miller: Yeah, so RealPage is enterprise proptech software company. PropTech short for property tech. And so our customers primarily are property management companies. You can think of that as management companies that oversee big apartment complexes. And the product I work on is called Loft. RealPage has a lot of products. It's grown through acquisition over the decades. So there's a ton of different products in its ecosystem. And I work on a product called Loft, which is effectively an end-to-end renter journey, is covered through Loft. So you can apply for an apartment in Loft, you can sign your lease, you can move in to your apartment with the help of Loft. That's the area I work in. You can pay your rent, talk to your manager. And get points for engagement and loyalty rewards all through Loft. So it's a B to B to C structure. And yeah, we've been growing at a pretty good clip.

Ashley Stirrup: Got it. And so when it comes to like A/B testing, are you looking at like B to B customer acquisition or are you looking at renter acquisition?

Andrew Miller: Yeah, A/B testing in Loft is focused on the renter. I would say pretty much exclusively, yeah, focused on the renter. So the customer that pays for the software is the property management company. The end user is the renter in the product. And so all of our A/B tests are focused on the renter's experience, optimizing certain moments in Loft to drive the renter to perform some action that helps the PMC, helps the renter or helps RealPage, things like that.

Ashley Stirrup: Got it. And it's kind of an unusual model. So in terms of your monetization, like are you doing that 'cause you're trying to create a great renter experience or does doing this actually lead to more revenue for RealPage?

Andrew Miller: Yeah. Yeah, it depends on which business unit within Loft we're talking about. So Loft has probably seven business units. Some are SaaS based, SaaS subscription fees. And so the property management company is paying the SaaS fee. Other business units, like the one I'm in moving, is transactional. And so there's like very, very close ties to a winning A/B test and generating more revenue for RealPage. So that's mostly the Area that I work in, but there's a bunch of other teams across Loft kind of thinking about it slightly differently, and we can get into that too.

Ashley Stirrup: Right. And for your team, is it kind of, short term transactional type of stuff or is it longer like I do a test and then hopefully next year they renew kind of thing. Got it. Yep.

Andrew Miller: It's transactional, yep. Yeah, quick context on kind of like what my product looks like, the moving product. It's essentially a marketplace, is the way we think about it. Although, we try hard to make it not feel like a marketplace to the renter. That's important to us. So if you're a renter using Loft and you're moving into your new apartment, call it anywhere from two weeks to three weeks before your move in. You'll get invited into the moving experience. And that experience is essentially a list of tasks organized into a checklist where we're basically nudging you to perform. Certain actions. Some of those are good for you as a renter, like changing your address, or good for the property to have that information or whatever it is. Other actions are monetizable, revenue generating for us, like setting up your utilities and things like that. And in those contexts, you know, we have a partnership in place where we'll get a commission based on the sale and so on.

Ashley Stirrup: Got it, got it. Super interesting. Yeah, maybe we could talk a little about your role there.

Andrew Miller: Sure. Yeah, so my role I'm director of product. In loft moving and I oversee the electricity business and a handful of other revenue lines for moving. So moving is essentially structured as we have internet which is our flagship revenue and then we have everything else which is electricity and all resident services. So that's my area. It's essentially a portfolio of products and services that I plug into the marketplace. Usually through partnering with another vendor or a startup that's looking for the scale that RealPage has, and make those services and products available, drive people into my checkout funnels for each of those services and drive revenue that way.

Ashley Stirrup: Got it. And how long have you been doing A/B testing?

Andrew Miller: Ever since I started doing product, so I got started in my career as a Ruby on Rails dev. I've been in tech my whole career. So I did development for a while and then I transitioned into design and did that for a while. And then the startup I was at, which ultimately got acquired by RealPage, we were growing our headcount at such a rate that we needed product people. We didn't have any product people. And so I transitioned into product. And it was kind of right around that time. So this would have been like, I don't know, eleven, twelve years ago. That I got interested in experimentation, A/B testing, and helped stand up an experimentation program there for the first time. I didn't really know what I was doing in terms of tooling. It was super fun and it was really successful. What we ended up doing is we used Pendo primarily for A/B testing and experimentation, which is a pretty limited experimentation tool. But at the time, I'm sure they still have this, at the time there was a way to write your own custom JavaScript. And it would launch that JavaScript on a button click. And so we built all these tests out that way. It was pretty fun. But since then I've stood up experimentation programs at two other companies. So I've kind of gone through that process in three different contexts, different tools each time, different team, different industries each time. So it's been really fun.

Ashley Stirrup: Yeah, and you were saying that you know, pretty different environments to do A/B testing in.

Andrew Miller: Yeah, yeah. So the first environment was, series B startup. The second environment in the proptech space. The second environment was series C startup and fintech, which was I oversaw the direct-to-consumer business for essentially we're selling life insurance. So there's a lot of funnel optimization work in that context. And then the third environment I stood up experimentation was here at RealPage. So enterprise scale, again back in PropTech, but really different tooling, really different team structures, more organizational complexity to navigate for sure.

Ashley Stirrup: Yeah. I love the fact that you've actually done all kind of the three primary roles of product teams, you know, engineering, design, and product management. That's a pretty unique background. Yeah. Yeah.

Andrew Miller: It helps. Yeah, it helps, for sure.

Ashley Stirrup: What's also interesting because you know, I was talking with a couple of other guests about, they're definitely product managers who they've grown up in A/B testing. And it's in their blood. And there are other product managers that maybe they've come through a business where that wasn't as important or they just hadn't built that muscle yet. And it's kind of a foreign concept. But it seems like it's become pretty central to how you do your business.

Andrew Miller: It really is. And so, you know, I like the kind of T shaped model of well really applies to any role but for PMs kind of a T-shaped model. You know, it's good to have breadth as a PM, a variety of skill sets. But my depth I really think of in terms of data, analytics. I write my own SQL, I do my own analysis. I can work real closely with dev to get all the event tracking in place exactly how we need it. And A/B testing experimentation, that's kind of my deep skill set on the product side, right? So and that's really pushed me into a growth track. It just makes the most sense. So the past few roles I've had have been really growth oriented. And I like that too because it's so tightly coupled to moving the business forward, which motivates me. You know, that's just kind of part of my personality is I like to see the work that I do have a pretty quick immediate impact on the business. It's just so much more fun and exciting, I think.

Ashley Stirrup: Yeah, it's fun to have a scoreboard and actually be able to move the needle on the scoreboard. So yeah. Yeah.

Andrew Miller: Ex exactly. Yeah.

Ashley Stirrup: So how does experimentation work at RealPage? I mean, how much of it is centralized? You know, some companies have like a s centralized data science function and they're kind of trying to evangelize experimentation to a variety of different teams. How does that work there?

Andrew Miller: Yeah, it's centralized for the loft organization right now. And the team is pretty small. So I was on that team when we were standing it up and then it ended up transitioning over into this growth role. Kind of still have one foot in with that team, but really it's just my friend and colleague Ken holding the fort down. He's doing awesome. And it's a centralized service. So all of the business units across Loft will create their own backlog of A/B test concepts and submit them into the optimization team. And those get reviewed and prioritized by Revenue Impact and a couple other dimensions, as you can imagine, confidence and effort and things like that. And then from there they get built out and Optimizely is the tool that we're using to launch and to measure test results. And then there's a there's a readout process after tests conclude. And it really varies, right? 'Cause like there's some corners of the product that get fifty X more traffic than others. So you know, there's teams that are in I'm jealous sometimes, like the payments team, for example, just gets tons and tons of traffic, one but it's once a month. And so that's kind of funny too. And it's high risk because there's a lot of revenue tied to it for the business. So whereas where I'm at, my traffic is lower than theirs on a monthly basis, but it's more consistent throughout the month, and it's not as high risk to the overall business, RealPage business. And so I can kind of run a little bit more freely in that

Ashley Stirrup: Yeah.

Andrew Miller: Respect.

Ashley Stirrup: Yeah. So it sounds like what happens is the optimization team kind of builds out the feature, you run the test, and then your team, if it's a winner, kinda has to rebuild it inside the product. Is that right? Yeah. And out of curiosity. Yeah, I was just curious.

Andrew Miller: Yeah, we do that. Yeah, there's a couple of winners that I've Yep. Go ahead. Yeah.

Ashley Stirrup: Given your product and obviously tied with engineering, I'm surprised you don't do those actually is build them as a feature inside the product and test it that way. Is there a reason you've gone with this model?

Andrew Miller: Yeah, speed. It's mostly speed. Yeah. Building a test in Optimizely and getting that to market is significantly faster than going through engineering. Part of that, and it depends, you know. In some contexts, that's because the test actually is wrapped up in rather complex areas of the software and it really is a big engineering lift. And so kind of faking it, so to speak, and Optimizely is faster. In other contexts, it's really about release cadence. And so depending on you know which engineering team I'm working with at RealPage. Some are gonna be on a monthly release cadence, some on a quarterly, some on a twice a month. And the optimization team just isn't constrained in that way. And so we can we can move a lot faster.

Ashley Stirrup: Got it. Interesting. All your products are cloud based though, yeah? Yeah, yeah, yeah. Okay. Yeah, super interesting. And how is RealPage trying to build a culture of experimentation?

Andrew Miller: Yeah, I mean that's super important to me. Very passionate about that. I think, you know, one of the things I can kind of go through what I've seen work to build culture, and we're also doing this at RealPage right now. And the first thing is, you know, ensuring that the right telemetry is in place, the right analytics are in place. In some contexts, when I've stood up experimentation, those have not been in place. And so if you don't have that, you don't you don't really have much to work with. And so, like getting people on board with the importance of behavioral event tracking and kind of making sure everyone understands that and sees the value there is prereq one. And then from there, once that tracking is in place, just go build up, build out some funnels. That's usually what I found is the easiest way to get people all on the same page because funnel analysis is so intuitive. You see the drop-offs and it just it just makes sense. Everyone gets it right away as soon as they see the chart. Other types of tests can be a little bit more nuanced and harder to wrap your head around, but funnel analysis is straightforward. So I like to build out a few funnels, high-value funnels, get everyone to look at the data together, and then you just start hunting for drop-offs, right? And the less that I do this in a vacuum, and the more I bring some key stakeholders along in the journey, the better. Because if I can get them to arrive at the insight even before I say it, we're golden, you know? They're like, that's the screen that has the big drop-off. And so from there, I like to just take some screenshots of every step in the funnel, put it in a Miro board or whatever whiteboard. And then put the data next to it, right? Here's the drop-off here, drop off here. Get a get a room of people together and just look at the data and start talking about, you know, w where's the right place to point an A/B test? Where do we want to optimize? And try to create the right psychological safety so that people just start rattling off ideas, you know. That's a really exciting time in the kind of experimentation program launch phase. Which is that those first few sessions where you're ideating with a group. I've I found that I can usually work from that backlog for like a year. So there's like so many ideas come out of it a lot of times. And then I'd say the last couple things is getting people to share ideas is one thing, but being able to organize those ideas in a way that's coherent is equally important. There's a framework that I've used. I was trying to remember who came up with it. A really great product leader, I forget her name. Maybe we can put it in the show notes. But it's called the Opportunity Solution Tree. And you basically create these broad opportunities of how you can optimize your business. And then with within each one, there's a few different solutions, a few different tactics kind of that you could take. And then under each of those, there's a you know a handful of experiments that you could run. And that's worked really well to get people on board and understand, you know, how all of these ideas relate to each other. And the last piece of rigor that and we did this at RealPage when we stood up the program about a year and a half ago, is I created templates for the test ideas themselves. And so anytime someone, a product person wants to submit a test idea to the optimization team, they have to fill out what's your hypothesis, you know, what's the metric that's tied to it. How do you what does success look like? And these simple little steps really kind of reinforce the discipline and the culture and they get people thinking in terms of learning. Obviously, everyone wants an outcome, but learning is sort of the baseline what we want to accomplish for each test. So those are some of the things I found that's worked really well.

Ashley Stirrup: Yeah.

Ashley Stirrup: Yeah, d if you have any templates we can share with the listeners afterwards, that'd be great if we could include that in the show notes. That'd be great.

Andrew Miller: Yeah. Yeah, I'm happy to.

Ashley Stirrup: Okay, terrific. And out of curiosity from your experience, how good are you at like assessing the potential impact of the different features? You know, the proposed features.

Andrew Miller: That's a cool question. Huh, what's my track record here? It's you know. Every time I've launched a new experimentation program, I've been so one of the one of the nice things about that is there's usually some pretty obvious low-hanging fruit that are that I've been right about most every time because it's just so obvious and no one's run the test yet. And so I might have Some biased data that suggests I'm I have a better track record than I probably would have it, you know, if I just extended the timeline out. I've definitely been wrong, for sure. And you know, the times that I get most excited is when there's an idea that I didn't think of, someone else did that's maybe not even on product design or engineering. And it's a total killer. Like it just it just crushes. But yeah, I think I'm usually pretty good, better than fifty percent, in my is would be my track record.

Ashley Stirrup: Yeah. Yeah, 'cause I would guess that come into a new role, there's a few things that are obviously broken, right? And so it's like, well that's low hanging fruit. But that once you get past those, you know, based on what I've heard, like, you know, Ronny Kohavi, real, you know, leader in experimentation has so many examples of where people misjudged, like, that feature, that doesn't matter at all and it's worth a hundred million dollars, you know. Yeah. And so, you know, that's the tricky part of prioritizing is like Well, if you really don't know, how do you stack rank these things, right? Yeah. Yeah.

Andrew Miller: That's great. I'll have to go I'll have to go listen to that conversation.

Ashley Stirrup: Yeah. Yeah. No, there's a couple of webinars on the GrowthBook website with Ronnie that super popular. They're also on our YouTube channel as well. Do you have an example of an experiment you've run where you had a lot of learnings?

Andrew Miller: Yeah, I can I can talk about two real quick. So first one first one's a winner. So I mentioned that part of the business I oversee is electricity setup for apartment renters. So you're moving into a new apartment, you need to set up your el electricity to have power. And the way that UX works, well call it six months ago or so. Is you click on the task and the side drawer opens up and I present to you all of the electricity options I have in my inventory. I primarily operate in the state of Texas, where there's more options. And so on average I was presenting eight. I think it was eight point two on average options per person. And so the test was basically, or the hypothesis was If I reduce that from eight to three, I'm going to reduce cognitive load. People are going to be comparing fewer numbers against each other. And I'll get a higher conversion rate because people are just going to kind of breeze right through and not have to think about so much. And that would ended up being true. It was a nice, a nice lift, 12% lift on conversion. And I think that's an I wanted to share that one because I think it's broadly applicable in lots of contexts, not just electricity setup, but basically. Basically, just reducing the number of options that someone has to think about. You see this all the time on pricing pages. Can increase conversion.

Ashley Stirrup: Yeah, th I think there's a famous jam or jelly study in grocery stores where they had like twenty-seven options and like nobody picked any and reduced it down. It was a huge winner. So yeah, that's

Andrew Miller: I've heard of that one, that's great.

Ashley Stirrup: Yeah, that's such a good reminder though, that you don't necessarily have to test you know, you don't have to come up with all your ideas yourself. Like you can go out in the world and look for some of the best winners and then say which of these might apply to my business. So yeah.

Andrew Miller: Yeah, yeah, totally. Yep. And the other one I wanted to share ended up not being a winner. And so if you think about how electricity prices are displayed, tends to be in cents per kilowatt hour. 8.2 cents per kilowatt hour, nine point three per kilowatt hour. And you know the thought there is like, okay, that's actually a compliance requirement. You have to display that data during the checkout flow. Fine. But the thought there is how many consumers really know what that means? Like, what's a kilowatt hour? Are people really gonna think about that and try to understand what that means for their monthly cost? And so the test was a price display test where we took the data that we had and estimated for the consumer what their monthly cost was going to be for a particular plan. And what was interesting about that is I saw a nice lift in the number of people selecting a plan to move on to the next step, but that ended up dropping below when it came down to the checkout moment, the final conversion moment. And so cents per kilowatt hour did win out in that. And during our retro, the strongest kind of theory I have on that right now is was looking at how we were displaying dollars per month and I was like, we're at we actually we tacked on cents. So instead of like $102, we displayed $102 and 14 cents per month. What would be the estimated cost? And the idea is there might be some dissonance there. We're trying to present an estimate but it's presented with a lot of precision. And I think, you know, a savvy consumer is gonna maybe second guess that a little bit. It's like what you're estimating down to the cent? That doesn't make sense. My electricity bill changes every month. And so the thought is that might have just confused people a little bit. So we wanna rerun it where we're gonna take the cents off and probably just round to the nearest five dollar, honestly. Just make it look like a nice round estimated number, because that's really what it is intended to be.

Ashley Stirrup: Yeah. That's super interesting. Like you'd think it would help provide more information, you know, like a better understanding, but you wouldn't think it would hurt conversion rates. And so did the as a result they just like don't finish signing up for their electricity? Is that the

Andrew Miller: They would just shop elsewhere. They would probably leave the product and go to Google or something. Yeah, 'cause there isn't a requirement

Ashley Stirrup: Got it. Got it. Interesting.

Andrew Miller: To set up through my product. Obviously I'd like for you to, but yeah. Yeah, they can just sign up elsewhere.

Ashley Stirrup: Yeah, that's really interesting. Yeah. I have an electric car, so I certainly had to like build spreadsheets and like ask ChatGPT for help 'cause I'm like, I don't understand all these different PG&E is our local electricity provider. And like I don't understand all these different plans and w what these numbers mean. So I'm sure most people don't go to that effort. So yeah. Yeah, super

Andrew Miller: Ex exactly. Yeah. Totally.

Ashley Stirrup: Interesting. Got it. Yeah. And any other tips for like how to as you're building an experiment, how to think about where might I iterate and you know, how do I design an experiment to get the most learning?

Andrew Miller: Hmm. Yeah, I mean, minimizing variables is always a good practice. Ideally one variable is changing. You know, I like I like to try to enforce that rigor. I don't always win. It depends, you know, because In a perfect world, you have enough traffic to where you can change one variable in a very precise way and get your learning that week. And you just move on with your week and over and over and over again. That's just not how business works. And there's always computing priorities, there's always constraints, whether that's engineering constraint, traffic constraints, post-analysis, bandwidth, whatever it is. And if you don't have a lot of traffic, like eventually, and I've seen this happen multiple times, you'll end up bumping into pressure to just launch wildly different tests. Like let's just launch let's just test a completely different experience. Which I think is totally fine to do from time to time. You just don't know why it worked exactly. And so the more you can narrow in on a particular variable, the better. But I think there is there's a fair amount of like upfront stakeholder management and expectation management that needs to be done to maintain that discipline without it getting kind of

Ashley Stirrup: Yeah.

Andrew Miller: Thrashed around.

Ashley Stirrup: Yeah. And coming back to the whole your I loved your whole framework for coming up with ideas and doing the funnel. As the product manager, you know, I would think that in some ways you're like, Well, you can just kind of go and set that list. Obviously you wanna bring in other people to get more ideas, but it sounds like you were also doing that to like kind of you know, build you know, get enthusiasm across the team for you know, doing the experiments you wanted to do.

Andrew Miller: Yeah, it's both really. And you know, in a lot of contexts I'm fine just creating the list myself. I've done that plenty of times. But it's two things. One is not every good idea is gonna come from me. It's just not. And so diversity and thought is really, really important for a long running successful experimentation program. If you have the same three brains pumping out ideas for two straight years, like you're just gonna start to get diminishing returns. And the other piece is culture building and just getting people involved because This type of thinking about optimization and data and growing the business, the more that can permeate throughout the organization, the better. Even if you're in customer support or sales or wherever you are in the org, it's just a it's really good, it's a good muscle, it's a good skill set for everyone to have, and it just benefits the whole company, is my view.

Ashley Stirrup: Got it. Love that. Yeah, that's such a such a great way to kind of frame it all. So in terms of how experimentation might evolve at RealPage, could you share, you know, some of your goals for where you'd like it to go?

Andrew Miller: Yeah, So I mentioned that it's a centralized service at RealPage today. And the team is small, the team is spread thin. I just got back from a conference in SF Lenny's product summit, and there was a talk by the Atlassian CPO, I think Tamar is her name, and she talked a lot about how PMs at Atlassian are starting to build more. Rolling up their sleeves, shipping code, getting their hands dirty in the building process. And I think that's what I'd like to see on the experimentation side in RealPage is just kind of you know ease the bandwidth constraints for that, you know, central team and get more product people rolling up their sleeves, using Claude Code, using Codex, whatever tool their team uses, to build their own technology tests. And maybe instead of submitting hypotheses, we have PMs submitting PRs. And the centralized team can be reviewing that code, doing their own QA, making sure everything's safe and reliable before we flip the switch. But that's what I really like to see. And I'm starting to have some of those conversations internally with the right folks to get that moving because it's just gonna be really tough to scale the practice across the organization. Unless we hire a ton and that takes forever and you have to go, you know, get budget approvals and so on and so forth. So yeah.

Ashley Stirrup: Yeah. Yeah. We've been doing a little bit of that internally at GrowthBook as well. What we more do is use it for prototyping. So we actually just maybe a week or two ago launched a new version of our Slack app, you know, so that you have a chat bot inside of Slack. And a couple months ago, after being inspired by a couple of podcast guests, I was like, We're we do some stuff in Slack today, but we could be doing so much more. And so I built a little prototype and showed it at the next company All Hands and engineering loved it. And so it's a great way to build excitement and like, okay, we'll make room in the roadmap for that.

Andrew Miller: That's awesome. That's great.

Ashley Stirrup: So yeah, yeah. Although it was funny because then they actually went and looked at what you could do inside Slack and like, okay, Claude was being a little overly optimistic in its prototyping. So we did have to dial it down a little bit. But still ended up with a great product. But it's just funny how s sometimes that's right,

Andrew Miller: Claude, so optimistic.

Ashley Stirrup: That's right. Look at all these things you can do. I'm like, well, not totally. Yeah. But no, it's definitely accelerated our time to market, just from the prototyping perspective and helping people get their head around how should this really work. So yeah.

Andrew Miller: Yeah. I l I love hearing that. And that's something that we've had success with too is I was using v0 specifically, the Vercel product to prototype A/B test concepts and designs and my optimization dev built this really cool harness that's kind of optimized for consuming my prototypes and then spitting out the code needed to launch a test in Optimizely. That was pretty sweet. We went from it was taking like two weeks to get a test launch to like a day.

Ashley Stirrup: Wow. That is awesome.

Andrew Miller: But w yeah, we haven't been able to scale that out across the org yet. So working on it.

Ashley Stirrup: We've actually done something in GrowthBook. I'll have to get you to check this out after the show where it's AI Visual Editor. And so you can actually go into the tool, have a little sidebar, and prompt it. And this the CEO loves to like put puppies in this picture, you know. But you can really control the whole UI just through vibe. So you can vibe code the feature and vibe code the experiment basically all in at the same time. So yeah, yeah. We're pretty excited about it. So definitely any listener who wants to check it out, would encourage you too. We have a free version of the product and you can go play with it yourself. So terrific. All right. Well, thank you so much. This has been just a really fun episode. I've loved it. So thank you so much for coming on the show.

Andrew Miller: Hey, thanks for having me. It's good to be here.