The Philadelphia Inquirer scales experimentation across the organization

Executive summary

A few years ago, experimentation at The Philadelphia Inquirer happened across marketing, product, the newsroom, advertising, and data, but those teams were often learning independently.

Experiments were difficult to discover across the organization. Teams used different processes, definitions, and success metrics. Because each department had different business goals, there was no shared framework for deciding which ideas to test, how to evaluate risk, or what the results should mean for the business.

The Inquirer changed that by building a centralized and democratized experimentation program.

Anyone in the organization can propose an experiment. A small cross-functional group provides the process, statistical rigor, QA, and governance to turn those ideas into trustworthy decisions. Every experiment uses predefined success and guardrail metrics, and becomes part of a shared research repository.

GrowthBook provides the experimentation infrastructure underneath that process. Today, nearly all of The Inquirer’s A/B experiments run through GrowthBook, giving its data, product, research, and business teams a common place to design tests, evaluate results, and communicate findings.

The program has run nearly 100 experiments with millions of readers, even though no one on the team has experimentation as their full-time job.

The result is a better way to make decisions and a shared view into business performance that really improves trust across teams. Plus, the experiments are having measurable business impact. One subscription-page test increased total subscriptions 11.8% while increasing premium subscriptions 126%. Another seemingly counterintuitive test increased total subscriptions 12.3% while simultaneously eliminating the cost of an underperforming subscriber benefit.

“Scaling our experimentation program has had a massive impact on the business, not just by helping us grow revenue, but by helping us identify opportunities to prevent losses and make better decisions with more confidence.”
— Alexandra Williams, Senior Data Analyst, The Philadelphia Inquirer

The challenge: Teams were learning in silos, often with conflicting goals

The Inquirer already had teams interested in testing. The problem was that there was no common experimentation system connecting them.

While Marketing had eyes on subscriber acquisition and retention, Advertising was prioritizing the on-site advertising experience and revenue. Product and newsroom teams had additional own goals for reader engagement and UX. 

That meant seemingly independent decisions could affect the same reader. A marketing experiment could influence a product experience. An advertising change could influence subscriptions. A newsroom UX decision could affect engagement somewhere else in the journey.

Yet experiments were not consistently visible across those teams, and each group could approach testing with different definitions, processes, and measures of success.

“We had the need for centralized knowledge, but didn't know how to create the bridge between the different departments to share, iterate, and make decisions.”
— Nicole Henninger, Ph.D., Director of Research, The Philadelphia Inquirer

The Inquirer wanted the benefits of democratized experimentation without creating an experimentation free-for-all. Its answer was to centralize the process, not the ideas.

A four-person operating model instead of a dedicated experimentation team

The Inquirer’s experimentation program is unusual because it does not rely on a standalone experimentation department. Four core contributors cover the experiment lifecycle alongside their primary roles:

Intake & facilitation
A Senior Agile Practitioner coordinates experiment intake, stakeholders, rituals, and timelines.

Experiment design
Research helps teams turn ideas into clear hypotheses and define the experimental design.

Development, launch & monitoring
Product and data work together on implementation, setup, data collection, and monitoring.

Analysis & interpretation
Data leads the analysis, while the broader group and experiment stakeholders interpret the results together.

That collaborative model adds QA naturally into the workflow. Multiple people see the experiment before launch, monitor the implementation, review the data, and participate in interpretation. 

That has allowed the team to run nearly 100 A/B tests in only two years while experimentation remains just one part of each team member's job.

Open to everyone, governed by the same rules

For The Inquirer, “centralized” does not mean putting experimentation behind a gatekeeper. Anyone from any department can propose a test, attend experimentation rituals, and see how decisions are made. Not every idea gets launched, but every idea enters the same decision framework.

The team evaluates potential risk, expected impact, UX value, and business value together. Experiments then move through the same intake, prioritization, development, launch, and analysis process. Before launch, the team agrees on the question the experiment needs to answer and defines its metrics. Sometimes they know they're going to make trade-offs where one number will go down, and they agree ahead of time on how much the other number has to go up in order to justify the change. 

Every experiment includes roughly three guardrail metrics designed to protect critical business outcomes. That discipline is particularly important at a news organization. The team is protecting two kinds of trust at once: internal confidence in the analysis and reader confidence in The Inquirer itself. Experiments cannot create an experience that could undermine journalistic credibility or reader trust, even if the treatment might improve another metric.

Nearly every experiment now runs through GrowthBook

The Inquirer has used different experimentation approaches over time, including A/B testing capabilities embedded in other vendor tools.

Today, the team has deliberately consolidated almost all of its experiments in GrowthBook. That gives the experimentation group greater visibility into how tests are configured, launched, and analyzed rather than scattering those activities across independent tools.

“Almost all of our experiments now are running on GrowthBook. That is intentional, so that we can be involved in every part of that process.”
— Nicole Henninger, Ph.D., Director of Research, The Philadelphia Inquirer

The Inquirer combines GrowthBook with its existing analytics infrastructure. The team instruments product behavior in GA4 and structures that data so newly captured events can flow into tables connected to GrowthBook.

For the data team, that makes new experiment metrics relatively quick to define. Because the underlying event data already exists, analysts can add metrics after an experiment has run when an initial result raises a new question. That capability matters because experiment averages often hide the most useful insight.

The team frequently digs into segments such as:

• Mobile vs. desktop
• Subscribers vs. non-subscribers
• Different reader and product audiences
• Different site sections, such as sports

In some cases, those segments behave dramatically differently even when the overall result looks modest.

The team currently uses a Bayesian statistical approach, although it has used both Bayesian and frequentist methods. GrowthBook supports the team’s workflow with experiment sizing and timing guidance, metric analysis, and the ability to examine additional dimensions as questions emerge.

Experiment 1: A “cleaner” UX caused sharing to fall 21%

The team wanted to declutter its article pages. One idea seemed straightforward: move article-sharing tools from the upper-right portion of the page to a more explicit position below the byline.

The hypothesis was that making the tools more visible and labeling them more clearly would increase usage. It did the opposite.

-21% share-tool usage overall

The effect was even stronger on mobile:

-31% on mobile

That result stopped the proposed design from being rolled out.

The test also revealed that mobile and desktop readers interact with sharing tools differently and gave the team a useful signal for future personalization. It also protected a behavior with downstream business value. Sharing exposes Inquirer journalism to new readers who may ultimately become subscribers.

A failed experiment had prevented an assumption from becoming a potentially damaging product decision.

“While we can safely say that this variant wasn't a winner, we now have the knowledge in our back pocket that, as we further segment our audience and learn who likes what, we can start customizing the experience for those users.”
— Alexandra Williams, Senior Data Analyst, The Philadelphia Inquirer

Experiment 2: Showing both subscription options drove a 126% increase in premium subscriptions

The Inquirer’s marketing team wanted more readers to consider its premium digital subscription. On the existing offers page, readers initially saw the basic digital subscription. The premium offer required scrolling farther down the page. The experiment put both offers side by side.

The results were dramatic:

+11.8% total subscriptions

+126% premium subscriptions

-28% basic digital subscriptions

The decrease in basic subscriptions was more than offset by growth in premium conversions, creating an overall win. The redesigned page also more than doubled the number of users entering the checkout flow.

For the team, that created another question worth testing. Getting readers into checkout is difficult, so the result pointed toward additional iterations that could improve the funnel even further.

“This experiment was a massive win for both marketing and experimentation.”
— Alexandra Williams, Senior Data Analyst, The Philadelphia Inquirer

Experiment 3: Removing a paid benefit increased subscriptions

The product team had a hunch that one benefit included with the premium subscription may not be providing added value to subscribers. This was a paid benefit the marketing team had been offering for some time, and it was up for renewal. They decided to run a quick test, and the results surprised them.

Using GrowthBook's Visual Editor, they created a variant that simply removed the benefit from the offer page. The result:

+12.3% total subscriptions

+15.5% basic digital subscriptions

-1.9% premium subscriptions

The modest decrease in premium conversion was overwhelmed by the increase in basic subscriptions. So removing something The Inquirer was paying to offer produced more subscriptions while eliminating the associated benefit cost.

The team cannot know with certainty why. Its leading theory is that the additional benefit created confusion for prospective subscribers. But it did not need to rely on that theory to make the business decision. The experiment supplied the evidence.

Negative results became organizational knowledge

The program’s impact cannot be measured only by winning variants. Much of The Inquirer’s experimentation strategy is deliberately focused on managing risk. That changes the meaning of a “failed” test.

When the share-tool redesign reduced usage, the team did not simply record a loss. It avoided shipping a weaker experience and learned something about differences between mobile and desktop readers.

When an experiment produces unexpected behavior in a critical business metric, the team may stop it even before conventional statistical significance. Historical knowledge about the product and its audiences acts as another guardrail.

For lower-risk decisions, the team may also make a call before reaching its maximum confidence threshold, balancing statistical certainty against the opportunity cost of leaving another experiment waiting in the queue.

The objective is not to maximize the number of statistically significant winners. It is to make better decisions with the appropriate level of evidence.

“A big part of [building trust] is building a culture where both wins and losses provide valuable information.”
— Alexandra Williams, Senior Data Analyst, The Philadelphia Inquirer

A shared repository makes every test compound

Running more experiments does not automatically create a learning organization. Remembering them does. Every Inquirer experiment is documented in a common research repository, including what was tested, what happened, and what business decision followed. 

The repository is accessible across the organization so that someone proposing a new idea can see whether a similar concept has already been tested. Product teams can use earlier reader behavior to inform new designs. Analysts can connect new findings with prior results rather than repeatedly starting from zero.

The repository turns individual experiments into organizational memory. Because experimentation is open to teams beyond data and engineering, those learnings reach the people making business and editorial decisions as well.

What The Inquirer learned after two years

The team distilled its experience into five principles:

1. Start with a shared process, not just a tool

Technology makes experimentation possible. A consistent operating model makes it repeatable.

2. Make every experiment answer a clear question

The team starts with the decision the stakeholder needs to make, then works backward to the hypothesis and metrics.

3. Define success and guardrails before launch

Agreeing on the metrics first prevents teams from hunting for a favorable result after the data arrives.

4. Create visibility across teams

Reader experiences cross departmental boundaries. Experimentation should too.

5. Treat negative results as valuable knowledge

Stopping a harmful change can be every bit as valuable as discovering a winning one.

From experimentation projects to an experimentation culture

For The Philadelphia Inquirer, the most important result is not one conversion lift. It is that experimentation has become a shared way of making decisions.

Marketing can bring an idea to the same process as Product. Researchers can help stakeholders turn assumptions into testable hypotheses. Data Analysts can examine results in the context of the business. Teams across the organization can learn from experiments they did not run themselves.

GrowthBook supplies the common experimentation layer that lets that process scale. The technology makes testing possible. The operating model makes it repeatable. Shared standards make the results trustworthy. Cross-functional participation makes those results useful across the organization.

All with a small team that does not have a single person dedicated full-time to experimentation.

“Ultimately, we've been able to significantly increase both our impact and our ability as a small team functioning across a large and ever-changing organization.”
— Alexandra Williams, Senior Data Analyst, The Philadelphia Inquirer