
How UPS's Experimentation Team Generated Half a Billion From 80+ Apps With A/B Testing
Chapters
[
00:45] – Guest intro: Raj Mehta, Moxie’s tech transformation, and 9,000+ daily calls[
01:20] – Starting point: 90% of ops in spreadsheets; why a data lake became the foundation[
02:45] – Automating time-to-lead: 25 minutes to [
04:27] – Roadmap design: MVPs, single-branch pilots, and scaling what works[
06:05] – Culture building: framing AI as growth and upskilling, not headcount cuts[
07:34] – Two lanes of automation: deterministic scripts vs. LLM-driven workflows[
08:45] – Call intelligence: scoring every sales/retention call and coaching at scale[
14:05] – Impact and advice: recurring revenue compounding and Raj’s playbook for getting started
Notable Quotes
"If there are defined steps, it's automation. When there's deviation, that's where you need an LLM."
"We broke down the lead process and automated it end-to-end—from 20–25 minutes to under 30 seconds—and lifted conversion by 5%."
"It's more important to focus on the data and tiny improvements you can scale—they compound into significant achievements."
Transcript
The Experimentation Edge - Raj Mehta ===
Ashley Stirrup: Today we're excited to talk to Raj Mehta. Raj runs product and technology for Moxie Pest Control which when you're thinking digital, you don't necessarily think pest control, but ⁓ Raj is doing a lot of exciting things to bring technology to a more traditional business. They have over 9,000 calls every day into their call centers across Arizona and California. So it's rife with opportunities to apply technology. So Raj, welcome to the show.
Raj Mehta: Thank you, it's great to be here and nice to meet you as well, Ashley.
Ashley Stirrup: Thank you. So ⁓ when you joined, you're really starting from square zero in terms of technology, really bringing all these things in for the first time for the business. What were some of the first things you did in order to lay the foundation so that you could understand your customers better and help create better customer experiences?
Raj Mehta: So when I first joined Moxie, I looked around and I was amazed with how the business was running. And 90 % of the operations were running just in spreadsheets. It was fascinating to see how everything was even working in the first place. So we were using a lot of different systems for phone systems, monitoring our field, using like fleet surveillance technology and things like that. A couple different marketing softwares and all of this data was just... spaggored and nothing was consolidated. It was almost impossible to get a holistic sense of what is even happening across the business. So the first initiative had to be to consolidate and bring all of this data in one place, which is exactly what I worked on. So we created a data lake. It took us some time to bring all of that data in, but that was one of the first things that we worked on. It also would serve as the foundation for anything, any other automation audit or AI flows that we wanted to build. And so that was the hypothesis there.
Ashley Stirrup: So I'd guess that just bringing all that data together for the first time was pretty transformational for the company. Were there any surprises that came out of that? Like ⁓ ways people wanted to use data that you hadn't anticipated?
Raj Mehta: Yeah, I think one of the biggest changes that I've seen across leaders at Moxie is that when I joined, understood how technology could even impact operations. And so that's been transformational. Recently, we've We've automated our entire flow of how much time it took for a lead from when a customer hit a sub made on the website to the time it takes to reach our sales rep. It was, it used to take 20 to 25 minutes because it was so manual and we went ahead, broke down that process and scoped it out where we started making small improvements and eventually we were able to automate that entire process. Now it takes less than
Ashley Stirrup: Wow.
Raj Mehta: 30 seconds. That's improved our conversion rate by 5 % just by automating a process and having all of that data in one click.
Ashley Stirrup: Wow. Wow, I'm actually shocked that it's not more than 5%. I mean, that's a huge difference. So when you talk about that amount of time, was that their first conversation with a person or the first conversation with a sales rep?
Raj Mehta: So what used to happen was ⁓ either somebody submitted a lead, their information on the website, and then it would come in as an email into our sales inbox. Somebody would go in, pull that information out, process it, put it on our CRM system, and then assign it to somebody based on certain logic. All of that could be automated. And that's what we did. Yeah.
Ashley Stirrup: Mm-hmm. Got it. Okay, initially I thought you meant they were sitting on hold for half an hour, which should be a really bad customer experience. So getting that down to less than a minute would be a huge win, but that makes a ton of sense. So that was pretty obvious low-hanging fruit for you in terms of something to go after. How did you build your overall roadmap of kind of what to invest in and when and all that?
Raj Mehta: That's, you know, like you said, this is one of... This is traditional business, right? The good thing about that is we have tons of opportunities. So many things that we could automate and make our operations more efficient. The challenge has been to prioritize which ones to do. The way that I've been thinking about it is any project that we take up or that we decide to invest in, we start off building a small MVP version of it. And the way we've been able to do that is we test it out in one branch in one location with a small team. Once things, once we learn, because none of the projects work completely 100 % foolproof systems on the first go. So we learn from that, we iterate on it, and then we start scaling it and rolling it out to the other branches.
Ashley Stirrup: And is there kind of one location that's a little more tech forward or enthusiastic to try these things out or you're kind of rotating from group to group?
Raj Mehta: you We have a dedicated team. There are guinea pigs. run all of our, any new stuff that we want to work on goes through that team once they give us. So they've been trained by now on how to provide feedback, what they look for, and ⁓ they're very comfortable to change.
Ashley Stirrup: Got it. And were there any surprises as you went through all that? Were people of leaning in or were they highly skeptical and, kind of how did you work through the first couple of projects?
Raj Mehta: Yeah, so I think... Initially, there's always this fear around AI and what that could mean for people's jobs and things like that. so one of my major goals was to build this culture within the company that it's looked upon as positively. The way we did that is we started automating processes that would show direct value to our teams in how it can make them more efficient and help with their jobs. And so that's the narrative that we've been constantly ⁓ communicating with everybody here.
Ashley Stirrup: So your pitch is you're investing in growth versus just efficiency and cutting jobs. Yeah.
Raj Mehta: Yes, that's right. And that's exactly what we're doing too. So we want to keep our highest value people and make them even more efficient.
Ashley Stirrup: That's terrific. Yeah, yeah, that's a good way to put it is, know, kind of the opportunity all this is you roll things out, you see who embraces it, and there'll be a certain amount of natural attrition to the business. And so the more you can retain the technology forward AI people, the more that culture will just self reinforce itself.
Raj Mehta: That's true.
Ashley Stirrup: Speaking of AI, how has your journey with AI gone? I'm sure you've tried out different things and some have worked and some haven't. I know you've gotten some great wins, but I'd love to hear about how you even started to explore it.
Raj Mehta: Yeah. So the way we started off was the way I look at it is there are two paths towards automation. So one is just traditional automation where you have scripts and things that just. run automatically and the other side is where you actually need to use a use an LLM or an AI brain so to say and the easiest way to distinguish between both of those things is if there are defined steps that happen within a process, then it's simply an automation. And if there's deviation in how things can go, you still need to break down those processes so that the LLM can be more accurate when it responds to things. But then that becomes more of an AI thing. And the way we've identified those opportunities is just doing more product discovery. So speaking with our teams and trying to really go deep and understand what their day looks like and how do they do their tasks.
Ashley Stirrup: Yeah, and how did you identify your first AI project?
Raj Mehta: Again, so while we were building this data lake, simultaneously we were trying to do these discovery sessions. The one that stood out the most was... there was actually an operational ⁓ question, which was how do we make our customer experience better? And so the North Star metric over there was our NPS scores, our ⁓ retention is the best way to know if your experience is well. And so how do we drive those levers? And the way to do that was A lot of our interaction happens on the phones with our customers. So if we could understand what we were doing well and what we were not doing well, then we'd be able to give a better experience. The operations leader on one hand ⁓ believed that All we needed to do was coach our people and train them. Their philosophy was we understand, we've grown and scaled this business. And so we understand what a good sales call looks like or what a good interaction with the customer is. And then how do you expect artificial intelligence to even understand something like that?
Ashley Stirrup: Yeah.
Raj Mehta: And yeah, so we had to take a more collaborative approach, work with them to really understand what they were looking for and how do we score somebody's conversation.
Ashley Stirrup: Yeah. And so it sounds like you've been able to use AI to provide coaching. Was that kind of one of your successful projects?
Raj Mehta: Yes, that's right. So within our sales and retention teams, we track all the conversations that happen, grade them on different things that we like to measure. And then we have a weekly coaching with all of our frontline people and we train them on those things.
Ashley Stirrup: got it. And so it's not like you're doing it kind of one at a time. You're doing it as a team. And so how has AI helped? Has it helped you be smarter about the feedback you're giving or just kind of scale how you do it? Or how's that worked?
Raj Mehta: I think both things. It's really hard to when you're managing 10 to 15 people per leader to have to go and listen into five to six phone calls. Each conversation is six to seven minutes long. That's just a lot of time that goes into that. And so with a tool like this, we're able to one, track. across all of the calls. can evaluate all of the phone conversations and give us a high level picture. And then we're also able to go deeper into specific two or three calls that were good, that were not good, and then find the common themes where a representative can improve on.
Ashley Stirrup: Yeah. And were there certain themes that kind of jumped out through the AI training? Like I could imagine that there would be some around product knowledge, some around kind of just having a good building, a good rapport with the customer, probably some on sales techniques. Like how did you break those out and what kind of jumped out?
Raj Mehta: Yeah, so our sales manager is very tech forward. He has a sales playbook, he, so he worked with us to make sure that we're tracking every single metric on his playbook. And so often before a tool like this, a lot of the leaders would just. listen into conversation and it's not even necessary that you might have hit all of your goals on that playbook or you would have hit or missed a few and you didn't get the whole picture and so something that came up often was even something as simple as did you take the customer's name three times in the conversation and did you actually build a meaningful relationship with this customer or potential customer and so those were some of the main things that came out.
Ashley Stirrup: Got it. So it sounds like it sales coaching was kind of one of the big areas of focus. I would imagine that, yeah, go ahead.
Raj Mehta: That's true. And then we had another one, even a similar one built for our attention. And so similar concept, but on the other side.
Ashley Stirrup: Yeah. Well, it sounds like an obvious win there is just the ability to score every single call, which, you know, given the kind of depth that sounds like they had in a playbook, like I would think it'd be very hard for a manager to actually score people on the total checklist and actually listen to the call and really be able to fight, provide good feedback. Yeah. And so then that would free up the the managers to focus more on the coaching, knowing that they could look at the score at the end and say, ⁓ and you missed step seven in the process too, or something like that.
Raj Mehta: Exactly, you got it exactly right. It's also because when you have so many people within your teams, it's how effectively can you even coach or listen into all of those conversations. Yeah.
Ashley Stirrup: Yeah. And has that... Sorry, go ahead. Yeah, right. You're not even getting to all the calls and even the ones you get to. It's hard to do as good a job as you could if you're AI assisted. Have you been able to see an improvement in terms of call conversion rates?
Raj Mehta: that's true. ⁓ So our conversion with a lot of our focus has been on the sales side, right? So with the, with the, with this call intelligence, as well as with our time to lead and things like that, we saw a 5 % increment within conversion. Yeah.
Ashley Stirrup: Right. Got it. So yeah, so all these things kind of added up together to a 5 % which a 5 % improvement in conversion rate. That's probably pretty significant in terms of total revenue for the business.
Raj Mehta: That's right. Absolutely, I mean, and these are recurring subscriptions, right? So the lifetime value of the customers is pretty high. We have about six to 700 per ticket. And so that kind of compounds every customer that we gain as well as that we don't, ⁓ that doesn't cancel their services with us. So that makes a big impact. Yeah.
Ashley Stirrup: Yeah, that you retain. Yeah. Well, that's terrific. Well, we're getting towards the end of our time. If you were to give ⁓ guidance to your younger self before you joined Moxie or a friend who was going to go start on a similar path where you're bringing technology and AI in, what are like three pieces of advice you'd give them on how to approach it?
Raj Mehta: So I would say that there's a lot of big flashy things out there ⁓ that seem very interesting. I think it's really important to, especially with something like an operations business, it's more important to focus in on the data, look at tiny improvements that can be scaled and that eventually compounds and starts becoming pretty big significant achievements.
Ashley Stirrup: Yeah, that makes a lot of sense. It sounds like you also have really approached it the right way in terms of getting kind of the cultural buy-in, finding an initial team to really optimize things before you scale so you don't throw out, create a process that doesn't work and deploy it to everybody and then undermine your own credibility. So congratulations to all the success you had.
Raj Mehta: Thank you, Ashley.
Ashley Stirrup: Yeah, so thank you so much everybody for joining today's call. This was with Raj Mehta, is the vice president of product and technology at Moxie Pest Control.
