Fabian Hans of Cogniteer on why deep dives beat mass produced tests

Running an experiment is the easy part. The harder part is knowing which experiment is worth running at all, and understanding why it wins or loses when it does. That distinction sits at the center of a recent episode of The Experimentation Edge, where host Ashley Stirrup spoke with Fabian Hans, founder and behavioral psychologist at Cogniteer, a consultancy that helps enterprises build in-house experimentation programs and raise both their testing velocity and their win rate.

Hans is unusual among the show's guests. Most are practitioners inside a single company. He works across many, which gives him a wide view of where experimentation programs go right and where they quietly go wrong. The pattern he keeps returning to is a warning to anyone scaling a testing program: it is easy to produce more tests, and much harder to produce more understanding.

🎧 Listen to the full episode →

Who is Fabian Hans?

Hans got into conversion rate optimization the way a lot of strong practitioners do, by accident and by being honest. Fifteen years ago he applied for an online marketing role and was asked, as a small test, what he thought of the company's website. He told them plainly that he did not like it, and gave specific reasons why. They hired him as a CRO manager before he had the vocabulary for the title.

From there he built an agency, then moved in-house for the depth that agency work rarely allows, and four years ago founded Cogniteer. Today the company helps enterprises stand up their own experimentation programs, increase test velocity, and improve win rate, drawing on Hans's background in behavioral psychology to understand not just what users do, but why.

The trap of mass-produced tests

The agency years taught Hans what not to do. With many clients and little time, his team fell into a familiar habit: if a test had won somewhere, they assumed it would win somewhere else, and rolled it out again. "It ended up that all of the clients got the same tests," he said. "It's a mass production of tests. And I did not like this approach."

The clearest example was a single line his team added to the basket: your items are not reserved. "It's one line. It's super easy to develop. It's psychology. It sells well to the clients. And we had a 50% win rate with that," Hans said. On paper, that is a great result. In practice, running the same idea for the hundredth time stopped teaching him anything. "You're losing the passion for it if you always do the same things over and over," he said.

That is the quiet cost of copy-paste experimentation. The win rate can look healthy while the learning goes to zero. Hans went in-house, and later built Cogniteer, precisely to trade breadth for depth: fewer clients, deeper understanding of each one's users, and tests designed to answer a specific question rather than repeat a familiar template. Mass-produced tests can raise velocity. Only deep understanding reliably raises win rate.

Most ecommerce drop-offs are structural

One reason the same tests keep reappearing is that the same problems keep reappearing. Across most ecommerce shops, Hans sees identical leaks: a high drop-off in the basket, and another on the product detail page. His counterintuitive point is that these are usually not your fault. "That is not because of the company or of the product," he said of basket abandonment. "It's just shopping." In roughly 80% of the shops he sees, the same issues show up simply because it is ecommerce.

The deeper cause is a mismatch between the product and the channel. Online shopping, as Hans notes, effectively started with Amazon selling books, and books are easy to sell on a screen because you can judge the content and the cover. Most other products are harder. A washing machine looks identical to every other washing machine in a photo, so the image barely influences the decision, and buyers care about features like capacity and water use instead. Fashion is the opposite, where style and design carry the decision and the picture is the point. And some products resist the screen entirely. "Perfume, you cannot sell online, because you need to smell it first," he said. Other products need to be touched.

For experimentation teams, the lesson is to diagnose before optimizing. Before testing a new basket layout, it is worth asking whether the drop-off is a genuine interface problem or the structural reality of selling something the screen cannot fully represent. That framing changes what you test and what you expect a test to fix.

Match the interface to how people actually buy

If the first job is understanding the product, the second is understanding the buyer, and Hans is emphatic that not all buyers are the same. One client asked Cogniteer to help win more new customers. The business was built on habit buying, where customers return regularly to restock, much like buying the same groceries every week. The page was perfect for that returning user, who wants to come in, press a button, and leave. It was almost useless for a first-time buyer, who needs information to decide whether the product is even right for them. "All the information is not accessible," Hans said. "New users don't convert because they are confronted with the transactional element way too early."

The fix was not a better page but a personalized one. New users are given the context they need before buying, while returning users keep the fast, transactional path they prefer. Same product, two interfaces, matched to two very different states of knowledge.

The same principle reshaped a B2B client selling water dispensers. In B2B, Hans explained, the buying behavior is different: someone is handed a budget and asked to find a fitting solution, then gather offers to bring to a manager. They do not want to browse a catalog and guess which product suits a thousand-person office or a factory. So Cogniteer removed the shop interface and replaced it with a finder, a short survey that ends by proposing a solution and inviting the buyer to request an offer. It reframed the page from a product catalog into a lead-generation tool, and conversion rose sharply. Sometimes the highest-impact change is deleting the thing everyone assumed the page needed.

Why this matters for experimentation teams

Underneath all three stories is the same argument. The biggest obstacle to a strong experimentation program is rarely tooling. Hans notes that AI is already dissolving the old developer-resource constraint through prompt-based experimentation. The stubborn obstacle is understanding: understanding the product, the buyer, and the specific problem worth solving. Ask five people what is wrong with a website and you get five answers, and in-house teams often lose the outside perspective that a first-time user brings.

That is why deep dives beat mass-produced tests. A program built on genuine understanding of user behavior does not just ship more experiments, it ships better questions, protects the metrics that actually drive decisions, and learns something whether a test wins or loses. Velocity without understanding produces motion. Understanding paired with velocity produces a program that compounds.

For product, data, and engineering teams building that kind of program, the tooling should get out of the way so the thinking can take center stage. That is the problem GrowthBook is built to handle, connecting experimentation to your existing data warehouse with a statistics engine your team can actually trust. Explore the open-source platform and start for free at growthbook.io.

Listen to the full conversation with Fabian Hans on The Experimentation Edge, and consider where repetition may have quietly replaced understanding in your own testing program.