Why product and marketing should experiment together to protect the user experience

Product users don't know (or care) which team owns the page they're looking at. They click an ad, read a landing page, create an account, and start using the product. Users experience each of those steps as one journey.

In many companies, marketing and product teams test that journey separately. Typically, marketing tests the ads, campaigns, and pages that lead to a signup, such as landing pages and pricing pages, and product tests the experience that follows, such as onboarding and in-product features. Often, each runs its tests in its own tool, on its own calendar, and judges them by its own metrics.

The problem is, because each team measures only its own metrics, a test can improve one team's metric while lowering the other's. For example, a signup test that removes setup questions can raise signups but lower the number of new users who activate, while a product test that simplifies a page can improve engagement but lower the referrals that bring in new signups.

At The Philadelphia Inquirer, the marketing, product, newsroom, and advertising teams used to run tests separately, and each team had its own goals and success metrics. The Inquirer now runs almost all of its experiments on one platform through one shared process, where everyone involved agrees on each experiment's metrics before launch. The Inquirer has since run nearly 100 experiments through that process, and Alexandra Williams, a senior data analyst there, said the program has helped The Inquirer grow revenue and prevent losses. For example, one test of the subscription offers page, designed with the marketing team, raised total subscriptions by 11.8%.

This guide explains how separate product and marketing experimentation programs hurt the user experience and how marketing and product teams can share one program that's better for users and the business.

How separate marketing and product experiments hurt the user experience

Separate experimentation programs can hurt the user experience in 4 ways: each team can ship a change that improves its own metric while degrading another part of the user's journey, marketing experiments can advertise an offer or a feature the product doesn't deliver, client-side testing tools can slow pages down, and two tests can conflict on the same user.

1. Conflicting definitions of success

When product and marketing test separately, each team judges its tests by its own metrics. Marketing teams usually measure how many people sign up, subscribe, or buy, and product teams usually measure how people use the product and whether they keep using it. When a product test tracks only product metrics, the results won't show whether the winning variation negatively affects a marketing metric.

For example, a product test that adds advanced features to the app's main screen could raise feature adoption among existing customers while overwhelming new users on a free trial, so fewer of them become paying customers.

Neither team should decide on its own which tradeoffs between metrics are acceptable, such as how many more signups justify a drop in engagement, because that decision affects both teams’ goals. The two teams have to make that decision together and agree on one definition of success for each test.

The two teams can also use the same word for different events. For example, a "conversion" can be a click on a campaign's call to action in the marketing team's tool and a completed purchase or subscription in the product team's data.

Separate programs also keep each team’s results from reaching the other team. Marketing and product test different stages of the same journey, so a result from one stage can improve tests at another. For example, if a marketing test finds that ads promoting shared dashboards bring in more signups than ads promoting custom reports, the product team could show new users shared dashboards first. When only the marketing team can see that result, the product team misses an opportunity.

The Inquirer had this problem before it built a shared program. Its marketing team focused on acquiring and retaining subscribers, its advertising team focused on the on-site ad experience and ad revenue, and its product and newsroom teams focused on reader engagement and the user experience. Experiments weren't consistently visible across those teams, and each team used its own processes, definitions, and success metrics. Nicole Henninger, Ph.D., director of research at The Inquirer, said in a GrowthBook webinar: "Our main challenge is that teams were learning in silos. A few years ago, we had the need for centralized knowledge, but we didn’t know how to create the bridge between the different departments to share, iterate, make decisions."

2. Inconsistent promises across the funnel

If the two teams plan their tests separately, marketing can change a landing page's message without product changing the onboarding behind it. For example, a product analytics company might test a landing page that focuses on the session replays feature, while onboarding still walks every new user through building a funnel. The new landing page could bring in more signups, but those users would arrive looking for replays and get a setup flow for something else. Some of them might never activate, and others might cancel.

In this example, the landing page test would report more signups without measuring activation or cancellations, because the product team tracks those in a different tool. Marketing could ship the new landing page as a winner, and the product team might then see cancellations rise with no product change to explain it.

3. Flicker and slow pages from client-side testing tools

Marketing teams often use client-side testing tools because they let them change a live page without help from engineering. The tradeoff is that the browser loads the original page first, and a script swaps in the test variation afterward. If the script is slow, users briefly see the original page before it changes, which is called flicker, or a flash of original content (FOOC).

To prevent flicker, some tools add an anti-flicker snippet that hides the whole page until the script loads. Every visitor in the test then waits longer, including the control group. If the script is slow, the page stays blank until a time limit runs out, typically after 1 to 4 seconds. Slower pages cost revenue. In a Bing experiment, every 100 milliseconds of speedup improved revenue by 0.6%.

When the marketing team uses its own testing tool, the product team may not know the script is on the page. The delay appears in site speed data, but neither team's test results reveal it, because the control group waits as long as the treatment group. When both teams use the same tool, the product team can see which tests are running on a page when site speed drops. Both teams can avoid flicker and the delay by applying changes on the content delivery network (CDN) that serves the page, before the page reaches the browser.

4. Colliding tests on the same users

When marketing and product run tests in separate tools, one user can be assigned to a marketing test and a product test at the same time without either team realizing. Assigning one user to two tests isn't necessarily an issue on its own, because each test randomizes its assignment independently. However, two tests can conflict when a user in both treatments sees both changes at once and the two changes contradict each other, change the same element, or create a bug when combined.

For example, a marketing team could test a pricing page that starts every new user on a 14-day trial of the paid plan. At the same time, a product team could test an onboarding flow that prompts new users to upgrade from the free plan. A user assigned to both treatments would be asked to upgrade to a plan they are already trialing. Because neither team would know the other's test was running, neither would catch the conflict before users saw it.

How often do A/B test interactions actually happen?

A/B test interactions are rare. Two tests interact when the effect of one test's treatment depends on which variation of the other test a user sees. Microsoft checked for interactions in 4 major products, each running hundreds of A/B tests per day on millions of users. It found no evidence of interactions in 3 of the products. In the fourth, only 1 in 50,000 checks (0.002%) found evidence of an interaction. In the few interactions Microsoft found, one test made another test's effect larger or smaller, sometimes too small to measure, but never turned a clear positive effect into a clear negative one.

Because interactions are this rare, making every marketing test mutually exclusive with every product test (so that no user is in both) would guard against a problem that almost never happens. Mutual exclusion would also split users between the tests, which lowers each test's statistical power. A test with lower power has to run longer to detect the same effect.

Interactions are most likely to happen when two tests change the same element, or when one test's change depends on an element the other test changes. For example, if one test changes an ad's text from black to red and another test changes the ad's background from gray to red, a user in both treatments can see red text on a red background, which would make the text unreadable. For marketing and product teams, these tests are most likely on pages both teams change, such as a news article page where marketing tests subscription offers and product optimizes the reading experience.

Because most tests don't change the same element, the two teams can let most tests overlap and use mutual exclusion only for tests that could interact. Teams can spot tests that could interact when they run experiments on one shared platform and review planned tests together in a shared ritual.

What changes when product and marketing share one experimentation platform

On a shared experimentation platform, product and marketing get 4 things that separate tools can't give them:

1. one view of live tests
2. one set of metric definitions
3. consistent user assignment
4. shared guardrails

With one view of live tests, both teams can see which tests are live on each page and at each stage of the journey. They can catch two tests that change the same element, or a landing page that promises something the product doesn't deliver, before users experience conflict. When a test ends, both teams can also see its results and decisions, so a result from one stage of the journey can inform the other team’s next test.

With one set of metric definitions, both teams compute each shared metric from the same data, so a metric means the same thing in every test's results. The teams can then compare a marketing test with a product test and agree on what each change did for users.

With consistent assignment, a user who has the same identifier on the marketing site and in the product sees the same variation in both. A pricing page test can then report activation alongside signups, and an onboarding test can report its effect on paid conversion, so each test shows its effect across the whole journey.

With shared guardrails, every test tracks the metrics that protect the user experience, such as page load time and error rate, along with the other team's key metric. If a change improves one team's metric but harms a guardrail metric, both teams can see it in the test results and decide together whether to ship the change.

The Inquirer manages experiments across all departments with one process and runs almost all of them on one platform. Anyone at The Inquirer can propose an experiment to a core group of 4 people: an agile practitioner, a researcher, a product manager, and a data analyst. None of the 4 works on experimentation full time, and the program has run nearly 100 experiments. The core group began by making go or no-go decisions on individual tests and now aims to evaluate the entire user journey. All of The Inquirer’s experiments live in a research repository, so anyone at the organization can see which experiments ran, how they performed, and what the team decided.

A shared experimentation playbook for product and marketing teams

Marketing and product teams can use 5 practices to run experiments on the same users without hurting the user experience:

1. Sharing an experiment calendar
2. Making conflicting tests mutually exclusive
3. Agreeing on shared metrics and guardrails
4. Measuring the combined effect of both teams' changes with a holdout
5. Reviewing no-code tests before they launch.

Together, these practices give both teams a shared experimentation strategy.

Keep one experiment calendar by surface and journey

A single calendar of planned and live tests shows both teams which tests affect the same part of the journey in the same weeks. The calendar lists each test with the surface it changes (such as a page, an in-app screen, or an email), the stage of the journey it affects (such as landing page, pricing, signup, onboarding, or in-product), its audience, its dates, and its owner. Both teams can check the calendar before launching a test and talk through any test planned for the same surface or audience in the same weeks.

The Inquirer tracks every experiment on one board with a column for each step of the experiment process, from intake through development, launch, analysis, and interpretation. The core group meets for 1 hour each week, and any stakeholder with an experiment on the board joins the meeting.

Use mutually exclusive experiments when tests touch the same experience

Although interactions are rare, some marketing and product experiments do conflict, such as two tests that change the same element, or a pricing page test and an onboarding test that make different offers to the same new users. Mutual exclusion assigns each user to only one conflicting test. In GrowthBook, you can set this up with namespaces, which split users between those tests. Experiments outside the namespace are unaffected and can still overlap.

When the calendar shows two tests that could conflict, put both in the same namespace and give each one its own, non-overlapping share of the namespace’s range (both tests must use the same assignment attribute). Because mutual exclusion lowers each test’s statistical power and queues tests behind one another, use it only for conflicting tests and let everything else overlap.

Agree on shared metrics and guardrails

Product and marketing should agree on a set of metrics that covers both teams' goals, with one definition for each metric. For example, if both teams report paid conversion, both should compute it from the same definition, such as a first payment within 30 days of signup. Before each experiment launches, the two teams should also agree on the criteria for shipping it.

Once both teams approve the criteria, neither team can switch to a more favorable metric after the results come in, so both teams can trust the outcome. Alexandra Williams, senior data analyst at The Inquirer, said in a GrowthBook webinar: "One of the biggest lessons we've learned is that you need to build trust between teams to create a successful experimentation program."

Because a team can't always predict how its tests will affect the other team's metrics, every test should include those metrics as guardrails. For example, a marketing test on a landing page may include the number of activated users as a guardrail, and a product test on onboarding may include the number of new users who opt in to marketing emails. Both teams should also agree on shared guardrails that protect the user experience in every test, such as page load time and error rate. More metrics aren't always better, though, because every added metric is another chance of a false positive.

Both teams should also write down the tradeoff they will accept. For example, teams can agree that a new pricing page ships if paid conversion rises at least 5% and 30-day activation falls no more than 1%.

Once both teams agree on guardrail metrics, a marketing or product owner can propose a riskier change, such as removing a feature or redesigning a pricing page, because either team can stop the test if a guardrail metric moves in a harmful direction. Every Inquirer experiment includes about 3 guardrail metrics, and the core group may stop an experiment before the result reaches statistical significance if a critical business metric moves in an unexpected direction. Henninger said that at The Inquirer, "Experimentation has increased the creativity and confidence of the key stakeholders and product managers that we have."

Not every risk can be measured in a test. The Inquirer doesn't run an experiment that could undermine journalistic credibility or reader trust, even if the treatment could improve a business metric. Henninger said: "We have two levels of trust that we must maintain all the time." Internal stakeholders need confidence in the analysis they receive, and readers need to trust The Inquirer itself.

At The Inquirer, the person who proposes an experiment writes the hypothesis with the core group, agrees with the group on the experiment's metrics before launch, and takes part in interpreting the result.

Henninger said that when departments' goals differ, the core group sometimes uses prioritization matrices and scales, but "most of it is just open conversation that we're having where we come to an agreement together before anything starts."

Measure cumulative impact with holdouts

The sum of the measured effects of each team's winning tests tends to overstate the two programs' combined effect on a metric. Every measured effect is the true effect plus random noise. Tests reach statistical significance more often when the noise favors the treatment, so the measured effects of winning tests are larger, on average, than their true effects (a form of selection bias often called the winner's curse). A test's effect can also shrink after launch, because the product, its users, and the market keep changing.

When marketing and product each report a sum of their own winners, the company also has no single measurement of both programs' combined effect on a shared metric such as paid conversion.

A holdout measures the combined effect of both programs' shipped changes directly. Teams withhold all new changes from a small share of users for a set period, often a quarter, and then compare that group with everyone else on the shared metrics, which gives one estimate of the two programs' combined effect. At Airbnb, the measured effects of 6 winning experiments on a target metric summed to 7.2%, and a holdout measured a combined effect of 4%. Correcting the sum for the winner's curse brought it down to 5.3%, closer to the holdout's 4%. Airbnb notes that both estimates are noisy, so even the 7.2% and the 4% may not be significantly different.

Set review rules for no-code experiments

When a marketing team builds tests in a no-code visual editor, engineers usually don't review the code for each variation. Instead, a second person should review each test before it launches. The reviewer should check that the variation renders correctly on desktop and mobile, that the test targets the intended pages and audience, that events are being recorded for each metric, and that the test doesn't conflict with another test on the shared calendar.

At a minimum, teams should require the review for changes to high-traffic surfaces and to surfaces both teams change, such as the homepage and the signup form. The same review should cover tests that an AI agent creates.

At The Inquirer, a product manager and a data analyst oversee development, launch, and monitoring together. Henninger said the pairing "organically creates an additional layer of QA on every experiment because you have two people looking at setup, data collection, and how things are looking for any experiment."

How GrowthBook helps product and marketing teams experiment together

GrowthBook is an open-source, warehouse-native experimentation platform that growth and marketing teams and product teams can use together. The Inquirer previously ran tests with the A/B testing features in other vendors' tools. Now, Henninger said in a case study, "Almost all of our experiments now are running on GrowthBook. That is intentional, so that we can be involved in every part of that process."

These GrowthBook features help product and marketing teams follow the playbook above:

• AI Visual Editor: Once engineers install the GrowthBook SDK, marketers can build a variation on a live page by describing the change or clicking an element (Available on Pro and Enterprise). Visual experiments run on the same SDK as the rest of GrowthBook and use the same metrics. Visual experiments can also run through GrowthBook's Edge SDK, which applies the change before the page reaches the browser to prevent flicker.
• Agent tooling: Marketers and product managers can create and launch experiments through an AI agent using GrowthBook's MCP Server and CLI, and ask the in-app AI Assistant about results. Agent-driven changes are saved as drafts, and a person reviews each draft before publishing it.
• One set of metric definitions: Metrics are defined in SQL on the company's data warehouse and can be added to any experiment as a goal or a guardrail, so when a marketing test and a product test both report activation, they calculate it the same way. Product Analytics dashboards use the same definitions.
• Consistent assignment: GrowthBook assigns each user to a variation by hashing an identifier, such as a user ID, with the experiment's seed. As long as the marketing site and the product use the same identifier for a user, that user sees the same variation of an experiment in both places.
• Experiment Decision Framework: Teams choose decision criteria as an organization default and can change them for each experiment (Available on Pro and Enterprise). The "Clear Signals" criteria require every goal metric to succeed with no guardrail failures, and "Do No Harm" requires only that no metric is statistically significant in the harmful direction.
• Namespaces: Namespaces make conflicting experiments mutually exclusive.
• Holdouts: Holdouts withhold new features and experiments from a share of users (5% by default) and compare those users with the general population (Available on Enterprise).
• Insights: The Timeline shows experiments that match the selected filters with their start and stop dates; Learnings lists stopped experiments with their decisions and screenshots; and Metric Correlations plots how experiments affected two metrics together, such as signups and activation.
• Slack integration: The Slack integration posts alerts to shared channels when an experiment starts, stops, or fails a guardrail, so both teams hear about each other's tests as they happen. The alert for a stopped experiment includes its results. Anyone with a GrowthBook account can also message the GrowthBook bot or mention it in a channel to ask about an experiment.

Join a live demo or try GrowthBook for free to see how product and marketing teams can run experiments on one platform.