Standing on the shoulders of giants: 13 pioneers who made A/B testing possible

Every A/B test you launch relies on ideas that took almost 300 years of hard-won progress.

Control groups, randomization, p-values, statistical power, sequential testing: all of it had to be invented, often while arguing with people who didn’t want to hear the answer.

Today, an A/B testing platform like GrowthBook puts those ideas within reach of a product team. But the methods themselves came from a ship’s surgeon, a Vienna obstetrician, a brewer, a few agricultural statisticians, an FDA reviewer, and a handful of engineers at Amazon, Microsoft, and Google.

While there are hundreds of people this article could have been about, I focused on 13 people whose work every experimenter stands on, grouped by era, with what each contributed and where it still shows up in the experiments you run today.

The early observers: controlled comparisons before statistics (1747–1865)

James Lind: An early controlled trial

In 1747, James Lind was the surgeon aboard HMS Salisbury during a scurvy outbreak. He took 12 sick sailors, kept them in the same quarters on the same diet, and split them into six pairs.

Each pair got a different supposed cure: cider, dilute sulfuric acid, vinegar, seawater, a spicy herbal paste, or two oranges and a lemon a day. Within six days, one of the citrus sailors was fit for duty. It was a tiny, unrandomized test, but it was an early, well-documented comparison of treatments under controlled conditions.

The sobering part is what happened next. The Royal Navy didn’t make lemon juice standard issue until 1795, nearly 50 years later. A clear result is only the start. Someone still has to act on it.

Ignaz Semmelweis: The data nobody wanted to believe

A century later, Ignaz Semmelweis was working in the maternity wards of Vienna General Hospital, which had two clinics. In the clinic staffed by doctors and medical students, roughly one in ten mothers died of childbed fever. In the clinic run by midwives, the death rate was a fraction of that.

Semmelweis ruled out explanations one by one until a colleague died after being cut during an autopsy, with the same symptoms as the mothers. The doctors were going straight from the morgue to the delivery room.

In 1847 he required handwashing in chlorinated lime, and by 1848 deaths in his clinic had fallen to around 1 to 2%. The medical establishment rejected his findings anyway, partly because germ theory was not yet established and he couldn’t explain why it worked. He died in an asylum in 1865, and the “Semmelweis reflex” now names the instinct to reject evidence that contradicts what we already believe. Anyone who has watched a stakeholder dismiss a losing test result has seen it in action.

Inventing the statistics of experiments (1908–1950)

William Sealy Gosset: The brewer behind the t-test

William Sealy Gosset joined Guinness in 1899 as a chemist, part of the brewery’s push to bring science to beer. His problem will sound familiar to anyone running experiments on a smaller product: he had to make decisions from small samples, like a handful of barley plots or batches, and the statistics of the day assumed large ones. So he worked out how much uncertainty a small sample really carries and derived what we now call the t-distribution.

Guinness was protective of its methods, so his 1908 paper appeared under the pen name “Student.” The t-tests and t-based confidence intervals used in many small-sample analyses trace back to a brewer trying to make better beer.

Ronald Fisher: The father of experimental design

If experimentation has a founder, it’s Ronald Fisher. Starting in 1919 at the Rothamsted agricultural research station in England, he dug into decades of crop data and concluded that much of it couldn’t answer the questions people were asking, because differences between plots could confound comparisons of crop yields. His books Statistical Methods for Research Workers (1925) and The Design of Experiments (1935) laid out rules we still follow: randomize assignment, replicate, block on known sources of variation, and use analysis of variance to separate signal from noise. He also popularized the 5% significance threshold.

The Design of Experiments opens with a story from Rothamsted. A colleague, the algae biologist Muriel Bristol, claimed she could taste whether the milk or the tea had gone into the cup first. Fisher used an eight-cup randomized design to show how that claim could be tested fairly.

Fisher’s legacy is complicated. He was a committed eugenicist, and late in life he publicly disputed the evidence that smoking causes lung cancer. But nearly every A/B test descends from his insistence that random assignment is what makes a causal claim trustworthy.

Jerzy Neyman and Egon Pearson: Hypothesis testing and statistical power

Fisher gave us significance tests. Jerzy Neyman and Egon Pearson gave us a decision framework still widely used in frequentist A/B testing. In a series of papers between 1928 and 1933, they framed testing as a choice between a null and an alternative hypothesis and named the two ways to get it wrong: a Type I error, or false positive, and a Type II error, or missed real effect. That led directly to the idea of statistical power, which is why you pick a minimum detectable effect and a sample size before you start.

Neyman also developed the theory of confidence intervals in the 1930s, then established the statistics department at UC Berkeley. Fisher hated their approach, and the feud lasted decades. It’s a useful reminder that methods we now treat as settled were once fiercely argued.

Gertrude Cox: Making experimental design practical

Gertrude Cox’s career in experimental design turned on a footnote. In 1940, her mentor at Iowa State, George Snedecor, was asked to recommend candidates to lead a new department of experimental statistics at North Carolina State. When he showed her his list, which had only men on it, she asked why her name wasn’t there.

He added a postscript saying that if the school would consider a woman, he’d recommend her. They hired her, and she became the first woman to head a department at the college.

She built it into one of the leading statistics programs in the country, and with William Cochran she wrote Experimental Designs (1950), the book that taught generations of researchers how to set up an experiment properly. In 1949 she became the first woman elected to the International Statistical Institute. Where Fisher supplied the theory, Cox made it usable, pushing randomization, replication, and proper controls in applied research.

Abraham Wald: Sequential testing and survivorship bias

Abraham Wald was a Hungarian-born mathematician who left Austria after the Nazi annexation in 1938 and ended up in Columbia University’s Statistical Research Group during World War II. His wartime work on aircraft survivability showed why damage on returning planes cannot tell the whole story: planes hit in more vulnerable places may never return. It’s a lesson in survivorship bias.

His bigger contribution to experimentation was sequential analysis. Testing munitions was expensive, so Wald worked out how to evaluate results as they came in and stop as soon as the evidence was strong enough, without inflating false positives. The method was considered so valuable that it stayed classified until after the war. Every time you use sequential testing to safely check an experiment before it finishes, you’re building on a principle Wald helped establish.

Making the randomized controlled trial the standard (1948–1962)

Austin Bradford Hill: Making randomized trials a model for medicine

Austin Bradford Hill planned to become a doctor, but the tuberculosis he caught while serving as a pilot in World War I ended that path, so he turned to statistics instead. He helped design the Medical Research Council’s streptomycin trial for tuberculosis, published in 1948. The drug was scarce, so there wasn’t enough for everyone anyway, which made a control group ethically defensible. The trial assigned 107 patients using a random-number sequence kept in sealed envelopes, so no doctor could choose who got the treatment.

The trial became a landmark in clinical research and helped establish the randomized controlled trial as a standard in medicine. With Richard Doll, he went on to show that smoking causes lung cancer, and his “Bradford Hill criteria” are still used to judge when a correlation is likely to be causal.

Frances Oldham Kelsey: Making controlled studies a requirement

Frances Oldham Kelsey joined the FDA in 1960, and one of her first assignments was an application to sell thalidomide, a sedative already widely used in Europe, including for morning sickness. She found the safety data thin and kept asking for more, despite heavy pressure from the manufacturer.

Over the next year, reports emerged linking thalidomide to severe birth defects. Thousands of children were affected worldwide. Her refusal kept thalidomide from being approved for sale in the US.

President Kennedy gave Kelsey the President’s Award for Distinguished Federal Civilian Service in 1962, and the scandal pushed Congress to pass the Kefauver-Harris Amendments that same year, requiring drugmakers to prove efficacy with adequate, well-controlled studies. Kelsey didn’t invent a statistical method, but her insistence on reliable evidence helped build support for making rigorous studies a regulatory requirement.

Experimentation at internet scale (1975–today)

Tony Twyman: Be most skeptical of your best results

Tony Twyman spent his career measuring radio and television audiences in the UK. The rule named after him reached print through his colleague Andrew Ehrenberg in 1975: any figure that looks interesting or different is usually wrong. Twyman’s law explains why the most exciting experiment results deserve the most scrutiny.

An unexpectedly large lift should send you looking for a logging bug, a sample ratio mismatch, or a broken metric before you celebrate. Ronny Kohavi adopted it as a core rule of online experimentation, and it’s the reason good platforms check for problems like sample ratio mismatch automatically instead of waiting for someone to notice.

Ronny Kohavi: Turning A/B testing into an engineering discipline

Ronny Kohavi did more than anyone to turn online A/B testing into a discipline. He led data mining and personalization at Amazon, then joined Microsoft in 2005 and founded its Experimentation Platform team the following year, which scaled experimentation across Bing, Office, Windows, and more. His teams documented one of the field’s most humbling findings: only about a third of the ideas tested at Microsoft improved the metrics they were designed to improve, and at Bing the success rate was even lower.

His best-known example is a small change to how Bing displayed ad headlines. It sat in the backlog for months, and when someone finally tested it, revenue went up about 12%, worth more than $100 million a year in the US.

He later led experimentation at Airbnb, and Trustworthy Online Controlled Experiments (2020), written with Diane Tang and Ya Xu, is the closest thing the field has to a standard textbook.

Diane Tang: Running many experiments at once

Diane Tang solved the problem every company hits once experimentation takes off: there isn’t enough traffic to run every test one at a time. At Google, she was lead author on the 2010 paper “Overlapping Experiment Infrastructure: More, Better, Faster Experimentation,” which described a system of layers that lets many experiments run on the same users at once while keeping conflicting parameter changes apart. Teams still need to watch for interactions between experiments. That design became a template for large experimentation platforms across the industry.

She co-wrote Trustworthy Online Controlled Experiments with Kohavi and Ya Xu, who built and led LinkedIn’s experimentation team. Between them, the three authors codified much of what modern teams treat as standard practice.

What these experimentation pioneers still teach us

Strip away the history and the lessons are the same ones good experimentation teams relearn every year:

• Randomize before you compare. Fisher and Bradford Hill showed why random assignment helps separate treatment effects from pre-existing differences.
• Small samples are noisier than they look. Gosset’s t-distribution exists because intuition underestimates uncertainty.
• Plan for power before you start. Neyman and Pearson’s framework is why sample size calculations matter.
• You can peek, with the right math. Wald’s sequential analysis lets you stop early without fooling yourself.
• Be most skeptical of your best results. Twyman’s law is still the best first step in debugging an experiment.
• Most ideas don’t work. Kohavi’s data is the strongest argument for testing everything you can.
• Data doesn’t win arguments on its own. Lind’s trial preceded routine naval citrus rations by nearly 50 years and Semmelweis didn’t live to see his vindication.

There are many more names that could be on this list, including Charles Sanders Peirce and Joseph Jastrow, who conducted an early randomized, blinded experiment in 1884; Donald Rubin, who helped establish the potential outcomes framework for modern causal inference; and Esther Duflo, who helped bring randomized trials into anti-poverty policy research and shared the 2019 Nobel Prize in economics.

Modern platforms carry that work forward. In GrowthBook, it shows up in Bayesian and frequentist analysis, sequential testing, and automatic sample ratio mismatch checks, alongside newer methods like CUPED, which uses pre-experiment data to reduce variance.

That’s what standing on the shoulders of giants looks like in practice. If you want to see how it comes together, you can get started with GrowthBook for free and put these principles to work in your next experiment.

Design your next experiment

Use your own metrics, check data quality, and compare results with transparent statistics.

Start for Free