How RealPage finds its biggest A/B test wins, one funnel at a time

RealPage is an enterprise proptech company whose customers are property management companies, the firms that run large apartment communities. One of its products, Loft, covers the full renter journey: applying for an apartment, signing the lease, moving in, paying rent, talking to the manager and earning loyalty rewards. It's a B2B2C model. Property managers pay for the software, but renters use it.

Andrew Miller, Director of Product Management, Growth and E-commerce at RealPage, joined Ashley Stirrup on The Experimentation Edge to explain how his team runs experiments in that model, how he builds a culture of testing, and what 2 electricity experiments taught him.

Every test is about the renter

All of Loft's A/B tests focus on the renter's experience: optimizing specific moments to drive an action that helps the property, the renter or RealPage. How that turns into money depends on the business unit. Loft has about 7 of them. Some are SaaS subscriptions paid by the property manager. Andrew's area, moving, is transactional, so a winning test ties very closely to revenue.

The moving product is essentially a marketplace, though the team works hard to make it not feel like one. Two to 3 weeks before move-in, renters are invited into a checklist of tasks. Some help the renter or the property, like changing an address. Others generate revenue, like setting up internet or electricity, where RealPage earns a commission through partners. Andrew owns electricity and a portfolio of resident services, usually built by partnering with vendors and startups that want RealPage's scale.

Three programs, three contexts

Andrew started his career as a Ruby on Rails developer, moved into design, and then into product when his startup needed product people. That's when he got hooked on experimentation. He has since stood up programs at 3 companies: a Series B proptech startup, a Series C fintech selling life insurance direct to consumers, and now RealPage at enterprise scale, with different tools, teams and much more organizational complexity each time.

He describes himself as a T-shaped PM whose depth is data and experimentation. He writes his own SQL, runs his own analysis and works closely with developers on event tracking. That depth pulled him into growth roles. "I like to see the work that I do have a pretty quick immediate impact on the business."

A small central team that moves fast

Experimentation for Loft is a centralized service run by a small optimization team. Business units submit their own backlog of A/B test ideas. The team prioritizes them by revenue impact, confidence and effort, builds and launches them in Optimizely, and runs a readout after each test. Traffic varies a lot across the product. The payments team gets enormous traffic once a month, with high revenue risk; Andrew's moving traffic is lower but steadier, which lets him test more freely.

Winning tests are later rebuilt in the product. Why not build them in the product first? Speed. "Building a test in Optimizely and getting that to market is significantly faster than going through engineering." Some tests touch complex parts of the software, and engineering teams run on monthly, quarterly or twice-monthly release cycles. The optimization team isn't tied to either.

How to build a culture of experimentation

Andrew has a repeatable playbook:

• Get the telemetry right. Without behavioral event tracking, there is nothing to work with.
• Build a few high-value funnels. "Funnel analysis is so intuitive. You see the drop-offs and it just makes sense."
• Bring stakeholders along. He puts screenshots of every funnel step on a Miro board with the data next to them and gets a room to look together. "If I can get them to arrive at the insight even before I say it, we're golden."
• Make it safe to share ideas. The first ideation sessions often produce a year of backlog.
• Organize ideas. He uses the Opportunity Solution Tree, a framework from product discovery coach Teresa Torres: broad opportunities, solutions under each, and experiments under each solution.
• Use templates. Every test idea submitted to the optimization team needs a hypothesis, a metric and a definition of success. "Learning is sort of the baseline what we want to accomplish for each test."

Win: fewer choices, more conversions

Andrew's electricity business mostly operates in Texas, where renters have many plans to choose from. When renters opened the electricity task, they saw an average of 8.2 options. His hypothesis: cutting that to 3 would reduce cognitive load and raise conversion. It did, with a 12% lift. He shares it because it applies broadly. Pricing pages show the same effect: fewer options to think about can increase conversion.

Loss: an estimate that looked too precise

Electricity prices are displayed in cents per kilowatt hour, a compliance requirement that few consumers understand. So the team tested showing an estimated monthly cost instead. More people selected a plan and moved to the next step, but final checkout conversion dropped, and cents per kilowatt hour won.

The strongest theory from the retro: the estimate was shown down to the cent, like $102.14 per month. "We're trying to present an estimate but it's presented with a lot of precision." A savvy consumer may doubt an estimate that precise, especially when their bill changes every month, and simply shop elsewhere. The team plans to rerun the test with prices rounded to the nearest $5.

Keep variables to a minimum

Ideally, one variable changes per test. In practice, traffic, engineering and analysis constraints create pressure to test completely different experiences. That's fine occasionally, Andrew says, but you won't know why it worked. "The more you can narrow in on a particular variable, the better." Protecting that discipline takes upfront stakeholder and expectation management.

Why more people should bring ideas

Andrew could write the backlog himself, but he doesn't want to. "Not every good idea is gonna come from me. It's just not." Diversity of thought keeps a long-running program productive: "If you have the same three brains pumping out ideas for two straight years, you're just gonna start to get diminishing returns." And involving people from support, sales and beyond spreads optimization thinking across the company.

Next: PMs who ship tests

The central team at RealPage is small and spread thin. Inspired by a talk from Atlassian's CPO about PMs building more, Andrew wants product managers to use AI coding tools to build their own tests. "Maybe instead of submitting hypotheses, we have PMs submitting PRs." The central team would review the code and run QA before launch, so the practice can scale without a big hiring push.

He has already seen what's possible. Using v0 from Vercel, he prototypes A/B test concepts, and his optimization developer built a harness that turns those prototypes into the code needed to launch a test in Optimizely. "We went from it was taking like two weeks to get a test launch to like a day."

The takeaway

Andrew's approach works for any team trying to grow an experimentation program: instrument first, let funnels make the case, invite more people to bring ideas, give every test a clear hypothesis and metric, and keep shrinking the time from idea to launch.

🎧 Listen to the full episode →