A product team at Ford Credit had a feature they were sure about. Drop a vehicle selector at the start of the online prequalification form, let prospective buyers pick the car they were dreaming about, and more of them would finish the application. It felt obvious. They were ready to ship it.
Geoffrey Bell asked them to test it first.
Geoffrey is the experimentation product specialist at Ford Credit, the captive lender that finances Ford and Lincoln vehicles on behalf of Ford Motor Company. He came up through experimentation at Lowe's, then spent time at Microsoft running tests across Xbox, before bringing that discipline to an organization that was newer to the practice. He has seen what experimentation looks like when it's mature, and he has seen what it takes to build it from scratch.
The vehicle-selector test is a good place to start, because it lost. And the loss is the point.
The test everyone wanted to ship
The hypothesis was reasonable. Give a customer the choice of a vehicle early, and you create commitment. "The idea was, if we give a customer the choice of a vehicle, then there'll be higher likelihood that they will actually convert into a prequalified customer," Geoffrey said.
So they ran it properly. A control with no vehicle selector in the prequalification flow, a treatment with one. Within two to three weeks, the data was unambiguous. Completion rates fell. Customers presented with the vehicle selector were finishing the application less often, not more.
That's a result a lot of teams would quietly shelve. Nobody enjoys telling a product manager that their idea cost the business leads. But Geoffrey sees those moments differently.
"Isn't it always the case that it's the losing tests that wind up creating or finding the most value, the most insight?" he said.
In this case, the value was twofold. First, the team did not ship a change that would have reduced qualified applications across one of Ford Credit's most important flows. Second, and more durably, leadership saw what experimentation could do. A program that was still earning its place had just caught an expensive mistake in motion. The loss bought credibility that a marginal winner never would have.
Geoffrey is honest about the nuance. At the time, the team could not yet measure whether the customers who dropped off were actually weaker leads, which would have changed the read. They have since gotten more sophisticated about that throughput. But even without it, the headline was clear enough to stop a launch and start a conversation.
The lesson from the first beat is the one most programs skip: a losing test is not a failure to manage around. It's the cheapest insurance the business will ever buy.
Why the loss pointed to a fix, not a dead end
A losing result is only wasted if you stop there. Geoffrey's team treated the drop in completions as information about timing, not a verdict on the idea.
They moved the vehicle selector to after the prequalification step, once the customer already knew they qualified, and saw gains. Same feature, different point in the flow, opposite outcome.
It reminded Geoffrey of a joke from his Lowe's days, when the team worked in classic e-commerce metrics like add-to-cart rate and revenue per visitor. "It was rarely the case that just putting an add-to-cart button earlier in the customer journey, it seemingly never led to more orders," he said. The instinct to push the conversion moment earlier almost always backfired.
The pattern underneath both stories is about meeting customers where they actually are. A prospective buyer filling out a credit form is not in the same headspace as one who has just been told they qualify. Get the placement wrong, and a good feature reads as friction. Get it right and the same feature converts.
That's why a single test is rarely the end of the conversation. Conviction in an idea plus a willingness to iterate on where and how it shows up beats a graveyard of abandoned hypotheses.
The piggy bank nobody puts on the slide
If the vehicle-selector story is about what a loss teaches, the next idea is about how you count it.
Geoffrey picked it up at Microsoft, where the experimentation program ran at enormous volume. He called it the experimentation piggy bank, and it had two sides. "Here's the revenue that we were able to realize from positive experiments. But here is the revenue that we saved from shipping poor experiences," he said.
Most programs only report the first number. The wins go on the slide because they're easy to celebrate and easy to attribute. The saves, the launches that tested poorly and never shipped, stay invisible. There's no line of revenue for a bad experience a customer never had.
But that second number is real, and it's often the larger of the two. Every prevented regression, every feature pulled before it reached production, is money the business kept. A program graded only on lift is reporting half of what it actually delivers.
This is the discipline Geoffrey is rebuilding at Ford Credit: a way to stand in front of leadership and show both columns. Not just what experimentation earned, but what it prevented.
The two percent that finally moved receivables
For a long time, Ford Credit's online experiments could move clicks, page views, and bounce rates. What they could not do was prove that a test sold a car. The most important part of the business happens offline, inside one of more than 5,000 dealerships, long after the customer closed the browser tab.
The hard part was never the math. It was the plumbing. An online test and an offline purchase are separated by time, by a dealership visit, and by every other experience a customer has in between, which is why so few programs ever close the loop. Connecting those two worlds took years.
When the team finally built the throughput measurement, it changed how product managers saw experimentation. Geoffrey gave the example that lands the point. In a purchasing flow, a control closed at a 30% rate among customers who purchased and financed with Ford Credit. The treatment closed at 32%. "That two percent incremental revenue gain, when you kind of project it out and annualize it, it has a large impact," he said.
On a business that moves tens of thousands of vehicles at roughly $50,000 each, two points of close rate is not a rounding error. It's a number leadership recognizes. And once an experiment could be denominated in receivables rather than engagement, the questions from product teams changed. They stopped asking whether they had to test and started asking what else they could.
Geoffrey is careful not to overcorrect. Revenue should not be the only measure of an experimentation program. Engagement, scroll depth, bounce rate, and the dozens of other behavioral signals matter, especially on pages where a purchase is nowhere near. But revenue is the language the business runs on. "Experimentation is one of those unique places that marries customer behavior with business metrics," he said. Tying a test to that intersection is what makes it impossible to ignore.
The experimentation balance sheet
Put the three ideas together, and a single mental model falls out. Treat your experimentation program like a balance sheet.
On one side, the wins: the lifts you shipped, denominated in the metric the business cares about most. On the other, the saves: the expensive experiences you tested and chose not to ship. Both sides count. A program that only reports wins is handing leadership an income statement with the costs torn off.
Geoffrey's path from Lowe's to Microsoft to Ford Credit is really the story of learning to keep both columns honest. The moves are repeatable:
- Test the change you're most confident about, especially when the team wants to ship on instinct.
- Treat a losing result as a question about timing and placement, not a final no.
- Count the revenue you saved by not shipping, alongside the revenue you earned by shipping.
- Connect experiments to the downstream number the business actually runs on, so the value is undeniable.
- When you deliver a result, lead with the story of what the customer did, then bring the numbers.
None of it requires a bigger tool budget. It requires deciding that a prevented mistake is worth as much as a captured win, and then proving it.
Full conversation with Geoffrey Bell, Experimentation Product Specialist at Ford Credit, on The Experimentation Edge.
