The Uplift Blog

DORA’s research identifies trunk-based development as a capability that drives software delivery performance. It’s not just a habit that high performers happen to share. Here, they work in small batches and merge into a shared “trunk” at least once a day to deploy continuously.

This only works if you’re building something that can be done in a day.

For instance, if you’re replacing a rules-based fraud detection system with an AI-powered risk scorer, it can take weeks. And trunk-based development becomes complicated as you see more merge conflicts.

That’s the exact problem feature flags solve by decoupling deployment from release.

In this guide, we’ll walk you through how trunk-based development works—and how feature flags make it possible for complex development pipelines.

What is trunk-based development, and why do you need to pair feature flags with it?

In trunk-based development, you maintain a single main branch and merge your work into it frequently—maybe daily, or even multiple times a day. The trunk stays deployable at all times.

If you’ve worked in a Gitflow model, you know what the alternative looks like. You typically have branches for development, and release can run for weeks. When the branches finally converge, the process of merging them takes longer than building the feature itself.

A 2025 analysis of 21 large-scale repositories found that merge-conflict potential grows with both the number and age of concurrent branches. So the longer your branch lives, the worse the merge gets.

That’s why even enterprises like Spotify run trunk-based development processes with feature flags. This lets their developers collaborate on the same codebase without needing to isolate themselves. They commit to main every day and catch any issues in hours instead of finding them during deployment.

That said, trunk-based development does require your main branch to be releasable. As a result, you have to find a way to hide unfinished code from users while you’re still building it.

Feature flags give you that capability.

You wrap new functionality in a flag from the very first commit and deploy it to production in a dormant state. But you only flip it on when you’re ready. So, you can keep merging without exposing the functionality to users. This makes feature flags a prerequisite for trunk-based development and without them, the model breaks the minute a feature takes more than a day to build.

How does flag-first trunk-based development work?

Let’s assume you’re building a new risk scoring model for your AI platform. Here’s how trunk-based development would work with feature flags within GrowthBook:

Step 1: Define the flag before you write any code

Before your team writes a single line of code for the new risk scorer, create a feature flag named ai-risk-scorer. So, every commit after that, such as the extraction logic or the model inference call, lands behind that flag.

In GrowthBook, it can look like this:

import { GrowthBookClient, UserContext } from "@growthbook/growthbook";

// Module scope -- initialized once at startup, shared across all requests
const gbClient = new GrowthBookClient({
  clientKey: process.env.GROWTHBOOK_CLIENT_KEY,
});

// Waits up to 3s for the flag payload; on timeout or network failure it
// resolves anyway and every flag evaluates to its fallback (off)
await gbClient.init({ timeout: 3000 });

// Per request -- user identity flows into every flag check
async function scoreTransaction(transaction: Transaction, user: User) {
  const userContext: UserContext = {
    attributes: {
      id: user.id,
      email: user.email,
      role: user.role,
    },
  };

  if (gbClient.isOn("ai-risk-scorer", userContext)) {
    return await mlRiskScorer.evaluate(transaction);
  }

  // Legacy rules engine continues running
  return rulesEngine.evaluate(transaction);
}

The isOn() call is synchronous. The SDK evaluates flags locally from a cached payload, so there’s no network call per check. You can wrap every commit in a flag without adding latency.

This changes what it means for code to be “in production.” The new scorer deploys and passes CI, but it doesn’t execute for any user. The feature flags keep it in an “off” state by default until you toggle it on. And your production environment will just carry the code without running it, so the main stays releasable even with half-built features inside it.

As a result of this process, your code reviews have changed. Instead of asking, “Is this ready to ship?” reviewers will start asking, “Is this safe to merge behind the flag?”

You’ll have a lower bar for review but a faster feedback loop during development.

Step 2: Commit daily and merge to main

Because the flag hides incomplete work from users, you can merge partial implementations into main every day.

For instance, if you’re conducting a 6-week migration, you’ll ship the risk scorer incrementally:

• Weeks 1–2: The feature extraction pipeline and model inference layer are launched behind the flag.
• Weeks 3–4: Response formatting and normalization of scores follow.
• Weeks 5–6: Shadow comparison against the legacy engine, plus performance tuning under production load.

Within GrowthBook, you can define how the flag behaves in different environments. So the flag stays off in production while you can test it in staging. There’s no “big bang” merge at the end of it. You just review and integrate each commit immediately, catching issues before they happen.

One thing you need to note is that when you’re merging code daily, it also changes how your CI/CD pipeline works. Since every commit runs the full build and test suite, the build time itself becomes a constraint. You’re adopting trunk-based development to avoid batching commits but if you use the same CI/CD process, you’re more likely to fall back into old processes. Avoid this by keeping pre-merge checks under 10 minutes and move slower ones to a post-merge state.

Also, your tests need to cover both flag states because main runs with ai-risk-scorer off, so the new code path merges and passes CI without executing. Make sure you force the flag to be on in your test setup to see how it works before deployment. In GrowthBook, SDKs accept feature values directly so CI evaluates flags from a local object instead of calling the API.

Step 3: Dark launch in production before you release

Once you’ve tested the feature and want to roll it out, you can conduct a dark launch. Here, you enable it for your risk operations or internal testing team first.

In GrowthBook, you can set this up with targeting rules and enable the flag for a specific segment. Define this segment based on attributes such as the internal email domain or member role. The Forced Value rule locks the flag for those users without changing the rollout state for anyone else.

Now the new model runs in production against real transactions at real traffic volumes—but only for the segment you chose. Your team then compares its output against the legacy engine on that live traffic and decides whether to widen the rollout.

The goal is to test the feature/model with real users and traffic volumes to see if the new change makes a meaningful (and positive) difference.

Best practices for feature flags in trunk-based development

Here are a few things you should keep in mind while using feature flags for trunk-based development:

1. Use branch by abstraction for large-scale refactors

There might be cases where your changes don’t fit in a simple on/off flag. For instance, if you’re replacing a fraud detection system, you’re not adding a new feature. Instead, you’re replacing the entire code path for another and both of these paths need to live in main while you continue to build.

Branching by abstraction solves this issue. Here, you define a single contract that says what the component does but without explicitly saying how it does it. 

Let’s say you’re migrating a risk scorer. The contract will be “take a transaction and return a risk score.” Both the versions of this risk scorer can do this but every part of your application calls the contract instead of the implementation directly. The factory (function) decides which ones to hand back at runtime based on the flag’s value. So, you can swap systems through a simple flag change instead of changing the entire code. 

Here’s what that looks like for our risk scorer migration:

interface RiskEvaluator {
  evaluate(transaction: Transaction): Promise<RiskScore>;
}

class RulesBasedEvaluator implements RiskEvaluator {
  async evaluate(transaction: Transaction): Promise<RiskScore> {
    // Legacy rules-based scoring
    // ...
  }
}

class MLRiskScorer implements RiskEvaluator {
  async evaluate(transaction: Transaction): Promise<RiskScore> {
    // New AI model inference
    // ...
  }
}

// gbClient is the initialized GrowthBookClient from Step 1.
// Call this per request -- not once at startup -- so flag changes,
// targeting rules, and percentage ramps take effect immediately.
function createRiskEvaluator(userContext: UserContext): RiskEvaluator {
  if (gbClient.isOn("ai-risk-scorer", userContext)) {
    return new MLRiskScorer();
  }
  return new RulesBasedEvaluator();
}

isOn() is synchronous. The SDK evaluates locally from a cached payload, so the abstraction doesn’t cost anything at runtime. You can wire every factory function through a flag check without worrying about performance.

However, there’s also a degradation benefit. Until init() resolves, and if GrowthBook is unreachable entirely, the SDK has no flag definitions to read. isOn() returns false and your fallback path runs.

For the risk scorer, it could mean the legacy RulesBasedEvaluator will continue to process transactions.

2. Manage flag states across environments

In trunk-based development, the same code deploys everywhere. Your feature flag’s states need to respect the environment’s boundaries, or the new code could activate in production while you’re still validating in staging.

Avoid this by defining behavior at the environment level.

For instance, GrowthBook handles this with one SDK connection per environment and each SDK carries its own key. GrowthBook generates the key so you can point the same variable at a different key per deploy:

# Staging
GROWTHBOOK_CLIENT_KEY='sdk-K2p9mXvQ4wRt'
# Production
GROWTHBOOK_CLIENT_KEY='sdk-8Hn3bZfL6yTc'

If you’re debugging unexpected behavior, disabling the flag won’t return false. The platform completely removes the flag from the payload, so the SDK finds nothing there and isOn() returns false. Your fallback path runs. 

3. Plan for the full flag lifecycle to avoid debt

The typical feature flag lifecycle looks like this:

1. You create a feature flag.
2. Ship the feature wrapped in the flag.
3. Release the feature by enabling the flag.
4. Monitor guardrail metrics while traffic increases, and roll back if they degrade.
5. Delete or archive the flag after releasing the feature and it holds up in production.

The problem is that it’s easy to forget about the cleanup step when you’re moving onto your next sprint. In fact, a 2026 study across Kubernetes and GitLab found that toggle removals lag behind additions by 35% and 13%, respectively. And this creates technical debt.

Let’s say you’re releasing the new AI risk scoring model. You create a flag called ai-risk-scorer with a clear owner and removal date (8 weeks from now). When the feature is ready, you can use GrowthBook’s Ramp Schedules to ramp traffic from 10% to 100% in increments instead of flipping it all at once. You can use Safe Rollouts which is a targeting rule with a Ramp Schedule and monitoring attached to it so that the platform runs sequential tests against the guardrail metrics you’ve defined. If the metrics look fine, you can keep proceeding to the next step until you’ve rolled the feature out completely.

But you have to remember to remove the feature flag after the rollout is complete. If you forget to remove the ai-risk-scorer flag, three months later, you can use the Stale Detection tool to surface it and pinpoint where it is using Code References.  From there the cleanup is two steps: delete the conditional in your codebase, then archive the flag in GrowthBook. 

You can also use GrowthBook’s GitHub skills to walk an AI agent through this process. Once you do, archive it first to confirm nothing broke and then remove it from your codebase.

That’s how you can keep technical debt low while taking advantage of feature flags.

3 mistakes that break trunk-based development with feature flags

Here are a few concerns you should keep in mind while working with feature flags:

1. Adding the feature flag after the code: If a developer merges the code to main and creates a flag later, you’re essentially rewriting live logic under pressure. A simple misstep can expose unfinished features/code to real users. That’s why it’s best to create the flag before you write any new code and wrap everything inside it from the first commit.
2. Long-lived branches with a flag bolted on at the end: This issue usually happens when a team adopts trunk-based development in name but still isolates work on a branch for weeks. It’s the inability to let go of old habits. But the problem is that as the branch accumulates more merge conflicts, the flag shows up as a release mechanism. It’ll mask a failed branching strategy instead of enabling trunk-based development. So, use the flag as a development mechanism from the beginning to avoid any kind of drift.
3. Flags that never get cleaned up: Every new flag you add to your codebase adds a conditional branch. For example, a single flag can create two possible code paths. But 10 feature flags create 1,024 code paths. When you don’t remove these stale flags from your codebase, they’ll accumulate into technical debt that slows you down and causes outages. So, assign an owner to each flag and target a removal date for temporary flags. Some platforms, like GrowthBook, automatically surface stale flags after two weeks of inactivity with no active targeting rules to help with cleanup—so that’s an option too.

How GrowthBook supports trunk-based development

Every new feature or model launch starts with a constraint we know too well. It takes weeks to build a feature, but the main still needs to stay releasable. Even though trunk-based development makes the build easy, you can only keep the main releasable without risk when you couple it with feature flags.

For example, in the ai-risk-scorer example we used, the code shipped daily and the internal team validated it against live transactions. But it was only possible using feature flags as it prevented us from adding too many merge conflicts as we added more code.

The feature flag acts as the release interface and the delivery mechanism to make this possible.

Platforms like GrowthBook give you the exact infrastructure to run this workflow:

• SDKs evaluate flags locally from a cached payload. The isOn() path is synchronous, so when you wrap every commit in a flag, there’s no performance overhead.
• You get unlimited environments with every plan, each with one SDK key.
• You can roll out features using Ramp Schedules, and opt into auto-rollback if any key metric degrades during the rollout.
• The revision system handles merge conflicts similar to Git. So you’re using the same mental model your team already uses for code.
• Features like Stale detection and Code References surface inactive flags and map them to your codebase. As a result, flag cleanup is the norm.

If you’re ready to explore how feature flags support trunk-based development, try GrowthBook for free today.

When it comes to feature flagging, it’s easy to build a simple server-side, on/off feature flagging system, but that simple project can quickly grow into a full platform as your requirement list grows. Use this checklist to assess your mid-term needs.  

What will you need in the next two years?

• Number of developers
• Number of features requiring flags per release
• Number of SDKs required
• Flag controls: simple on/off or the ability to control feature behavior through configs
• Ability to target users by browser, device, location, plan, etc
• Gradual feature roll-outs
• Automatic roll-backs if key metrics dip
• Audit logs - know who changed a feature and when
• Governance - ensure all new features/updates are reviewed and approved
• Debugging tools
• Integrations with other tools
• Flag clean-up capabilities
• Support for A/B testing
• Uptime/reliability requirements
• Performance requirements
• Time to market
• Better uses of your dev team’s time

Engineers are builders by nature. In fact, a DORA report found that 89% of organizations use an internal developer tool. So it’s not surprising that when you’re faced with the decision to build or buy a feature flag tool, you’ll err on the side of building one.

For some teams, it makes sense. But for others, it might not be the right choice.

For instance, Dropbox’s engineering team did the exact thing you’re considering right now. They built an in-house feature flagging tool, and for years, it worked. But as the product grew, so did the tools, and eventually they had six separate systems with overlapping roles. Even simple experimental analysis took days because of the time it took to wrangle the system.

If you’ve ever added a config toggle or database flag and watched it become the infrastructure your team relies on, you’ll recognize the problem here.

In this article, we’ll dig into why engineering teams build or buy a feature flagging tool so that you can decide which direction is the right one for you.

Why it might make sense to build your own feature flagging tool

Let’s face it. It’s a perfectly reasonable decision to build your own feature flagging system. DORA found that teams that use their own internal platforms tend to show 8% higher individual productivity and 10% higher performance. It makes sense because you’ve literally built it for yourself.

But beyond that, there are other reasons why it’s the right choice:

You get full control over your architecture

When you build your own flagging tool, you own every decision related to it. How it’s built, who can access and use it, and even how you set the governance system up. Every layer in the tool maps to what you need, not to what a vendor recommends.

For example, if your deployment pipeline manages environment changes/promotions, then your flagging system can hook into it. You’re not forced to work around a tool you’ve bought because a home grown system is purpose-built for your infrastructure.

You can integrate it with your existing stack

Even in our experience, we’ve seen that many teams have a very specific tech stack. But if it doesn't integrate with your stack, it takes significant time to build a workaround.

For example, think about the difference between importing a library versus using an external API. The former lives within your existing process, while the latter adds a dependency you can’t always control.

A homegrown flag system works like the library. It deploys alongside your services but also shares the same points of failure as your application code. If something goes wrong, you’re debugging one system instead of two.

Your use cases are fairly simple

Not every engineering team needs advanced targeting or multi-environment management. They’re just looking for a simple Boolean toggle to flip features on or off. In fact, a Reddit user built one using AWS Lambda to get and set feature flag states and flip flags.

Here are a few use cases where building a tool might be the right call:

• Fewer than 20 flags across a single service
• Boolean on/off toggles, which are server-side only
• One language with one deployment target only
• Engineers are the only ones changing the flag state

In these cases, it doesn’t make sense to buy a full-blown platform as it adds more complexity.

You don’t have to depend on another vendor

Vendor lock-in is one of the biggest problems in the SaaS industry. And it’s not any different in the feature flagging space either. Platforms that cater to mid-market and enterprise companies are known to be opaque with their pricing policies. In fact, several Reddit threads document cases where vendors changed their pricing models with little warning. In one example, a feature flagging vendor moved from user-based to per-service-connection pricing, and customers saw costs jump 5X because of their Kubernetes pod count rather than any change in actual usage.

When you build your own platform, you control its uptime and the costs of building and maintaining it. The costs increase only when your team decides it needs to.

You might prefer building in general

Sometimes you might just prefer building something for reasons beyond “investing” in your growth. Too often, buyers have a mental block when it comes to shopping for a new tool, especially with the number of options with limited differentiation these days and the ability to use AI coding assistants.

If it’s easier and faster just to build the tool, it makes sense to do so. In fact, even behemoths like Google and Meta built their own feature flagging systems. But it comes with a caveat. These companies also dedicated hundreds of engineers to the task and treat the tool internally as another product rather than another project. It’s part of the reason it continues to work for them.

In other cases, if the vendor’s not responsive or doesn’t improve the platform regularly, you’d be right to switch or build your own tool.

Where do homegrown feature flagging systems start breaking down?

Here are a few reasons why it might make sense to buy a feature flagging platform instead:

1. Ballooning costs of building a feature flag tool

The more you use feature flags, the more likely you are to be running your app on it. When your feature flag system goes down, or something breaks, your team is responsible for carrying the SLAs and on-call rotations associated with it. And many teams realize this before building their own tools.

For instance, Upstart, a lending marketplace, had evaluated this option. It realized that it would’ve required four engineers in the first year and at least two after that. Those engineers “would be building tests instead of building a product.”

The math looks like this, if we assume the salary of a junior developer in the United States:

• Year 1: 4 engineers × $87,519 = $350,076
• Year 2+: 2 engineers × $87,519 = $175,038 per year

If you’re going with a more experienced team, the costs can easily reach more than $500,000 just in salaries. And this doesn’t account for additional costs like:

• Server costs and maintenance
• Costs associated with an outage
• Time lost building a platform
• Technical debt and flag debt
• Security and compliance incidents
• Building advanced features (audit logs, support for multiple languages)

Every engineer you dedicate to this project is an engineer who could’ve been improving your core product or service. The long-term costs may not be worth the hassle, especially with the options available today. 

Even developers know this. In a Hacker News thread, one user estimated that a $100K engineering salary balloons to $400K fully loaded, meaning each engineer needs to generate at least $600K in annual value to justify the investment. At the same time, others pointed out that maintenance costs for internal tooling almost always exceed the initial build. And this is especially true once you factor in on-call rotations, documentation, and onboarding new team members onto a system that they’re not familiar with.

2. Targeting and rollout complexity

It might take 30 minutes to build a Boolean flag in a config file. But if your product manager asks you to roll out a new feature to 10% of enterprise customers, the config file can’t do that.

You need the ability to run percentage rollouts using deterministic hashing so that your users see the same variant every time they visit the app. This is usually the moment engineering teams realize that their config system won’t cut it for this purpose. In fact, a 2020 study confirmed that although feature flags and configurations are similar on the surface, they are used for different purposes.

A config value controls how your app behaves. A feature flag controls who sees what, and when. Once you need targeting, rollout logic, and per-user evaluation, you’ve outgrown your config system and you’re building something new whether you planned to or not. And the config toggle becomes an evaluation engine which needs ongoing maintenance in the long run.

Note: GrowthBook includes advanced targeting and rollout capabilities that enable granular targeting and deterministic hashing. It’s available on every plan, including the free tier.

3. SDK maintenance across multiple languages

Your first flag evaluation is probably in one language. But products grow, and within months, you’ll add a few more languages, especially if you’re launching mobile apps with it.

You need to ensure users get a consistent experience regardless of the device they use. If your Python SDK evaluates a targeting rule differently from the React SDK, users see different behavior depending on the device they’re using. It’ll be hard to debug in those cases and pull your team away from working on your actual product.

And engineering teams like Treatwell’s marketplace have already faced this. Because they integrate through Java, React, Swift, and Kotlin SDKs, building a homegrown system would mean building and maintaining four separate SDKs with four separate test suites, all while building their own product. On the flipside, GrowthBook ships and maintains 24+ SDKs across every major runtime, so SDK maintenance was moot once they switched to it.

So, if you’re using multiple languages and a feature flagging platform supports it, consider paying for one.

4. Governance, compliance, and access control

These days engineering teams are shipping features a lot faster and updating features even faster than that. When you’re running a development team that’s responsible for all these changes, you need new levels of governance to keep up with their pace. For instance, you need review processes before changes go live, especially when you’re modifying features running behind a feature flag. And you need a log of what changed and when to maintain quality and to pinpoint when new bugs may have been introduced. As your flag tool matures and becomes an important part of your infrastructure, your team will have to keep building more. For example:

• If a product manager needs to toggle a flag, you need a self-service UI.
• If something breaks in production, you need an audit log.
• If a regulated customer asks about change management, you need approval workflows in place.
• If your security team asks who can change the production state, you need role-based access control.

Also, if you’re handling sensitive data, you also need to ensure you don’t send data to a third-party tool. If you’re using a platform that comes with a self-hosting option, you can avoid this hassle. 

“The fact that we could retain ownership of our data [with Growthbook] was very, very important. We have data from children stored in our servers, and that's something that we have to really protect.”
— John Resig, Chief Software Architect, Khan Academy

The question is: is it worth the hassle to continue building the tool? In most cases, the answer becomes clear once you add up the features you’d need to build and maintain just to keep pace with your team's growth.

5. Stale flag and technical debt

Stale flags clutter your codebase and make it dangerous to modify. That’s because every flag that stays beyond its expiry date is a conditional branch your team might see in a remediation plan one day. If you remove the wrong one, you could also break production.

In short: it’s too many what-ifs to leave to chance. It’s such a huge problem that companies like Uber had to build their own automated refactoring tool, Piranha, to clean up 1,381 flags and remove 71,000 lines of code.

6. Risk reduction can become risk accumulation

Feature flags exist to make releases safer by decoupling deployments from releases. That’s a fact. But without proper lifecycle management, they can also become the very tool that increases risk over time. 

Every stale flag is a conditional branch your team has to reason about during code reviews and incident response. If you remove the wrong one, you may break production but if you leave it in, you’re carrying dead code that makes debugging harder and in turn, makes every release riskier. That’s why you need to also build an automated stale flag management system within your homegrown tool.

If you don’t have the time or resources to build an accompanying tool, you should consider buying a feature flag tool.

7. AI coding doesn’t change the math of building a tool

The dawn of AI coding assistants has also brought another question into play: “I can generate a feature flag service in an afternoon with Cursor or Claude Code. Why would I buy one?”

At the outset, it sounds great. You use a $20 tool to build a functional toggle platform in a day. But considering the hidden costs and regulatory environment you’re in, the answer’s different.

A 2026 study found that the “SaaScopalyse” is wildly overstated. AI lets you build tools for a nominal cost if you know exactly what you want and the problem is simple to solve.

But if you’re in a regulated industry or are considering mission-critical systems (like a full-blown feature flagging platform), that’s not the case. These tools don’t take responsibility for when the systems malfunction or go down. Long-standing vendors in the market have already spent years solving your problem and building the reps (and certifications) to remain a contender in the space.

How do you decide whether to build or buy a feature flagging tool?

A 2026 study found that a majority of software build vs. buy decisions are made through fragmented expertise and informal reasoning rather than a proper systematic evaluation. That’s why you need a proper framework to decide what the right choice is here.

Use this framework to do so:

If most of your answers lean on the left side of this table, you’re better off building a tool internally. If not, you’ve either outgrown the homegrown system or should just buy a feature flagging tool.

Note: Just because you’re buying a platform doesn’t mean you have to give up transparency or get locked into a single platform for years. There are open-source options you can self-host on your own infrastructure.

Plus, many feature flagging platforms (including GrowthBook) support the OpenFeature standard, a CNCF specification that offers vendor-agnostic APIs. It’ll help you use any supported platform and switch without a painful migration.

What should you look for in a feature flag platform?

If you’ve decided to buy a feature flagging platform, evaluate it based on these factors:

• SDK coverage for every language in your stack: You need to check two things here: how many SDKs does the platform support, and how many of those include the ones you actually need or might in the future? For instance, GrowthBook ships 24+ SDKs covering server-side (Go, Python, Java, Ruby, PHP, .NET, Elixir, Rust), client-side (React, Vue, Angular, vanilla JS), mobile (iOS, Android, React Native, Flutter), and edge runtimes. Other platforms may have a smaller support system.
• Local evaluation with no per-request latency: Your flag checks shouldn’t depend on a network call. GrowthBook SDKs download flag rules as a cached JSON payload and evaluate every check-in-process. So each call happens in sub-milliseconds and doesn’t have an external API in your critical path. If the platform’s servers go down, your application keeps working.
• Governance and access control: You shouldn’t have to build approval workflows or audit logging yourself. Make sure the platform has configurable approval workflows that require reviewers before any changes hit production. And it should be controlled based on specific roles in the organization.
• Stale flag detection and lifecycle management: Feature flags clean technical debt but can also accumulate technical debt over time. That’s why you need a way to automatically identify flags that aren’t active anymore or are not needed. Platforms like GrowthBook offer a Stale Detection feature to help you with this, and Code References tells you where it sits in your codebase.
• Ramp Schedules with automated guardrail monitoring: Even though gradual or percentage rollouts are relatively easy to set up in a homegrown system, it’s hard to build monitoring or observability alongside them. You need the ability to attach guardrail metrics to a rollout so you can detect when it degrades performance and roll it back immediately.
• Feature Evaluation Diagnostics: When a flag evaluates unexpectedly in production, you need to know what went wrong and where to fix the issue. Platforms like GrowthBook give you the capability to see a rule-by-rule trace with attribute values. So, you know exactly why a flag returned what it did for a given user.
• Prerequisite flags and typed payloads: Beyond boolean on/off, look for JSON flag values with schema validation and flag dependencies that prevent invalid state combinations. GrowthBook Enterprise supports prerequisite flags and JSON Schema validation that auto-generates a form UI so non-technical users can safely edit complex configuration values.
• Warehouse-native analytics: Your feature flag data should live alongside the metrics your team already uses for decision-making instead of within multiple analytics platforms. Look for a platform that integrates directly with your existing data warehouse (Snowflake, BigQuery, Redshift, Databricks) so you can analyze flag performance against your business data without building custom pipelines. Platforms like GrowthBook are warehouse-native by design so it queries your data from existing data sources directly and nothing leaves your infrastructure.
• Self-hosted option with real data residency: If you work in an industry with strict data residency requirements, make sure the feature flagging platform offers self-hosting. GrowthBook self-hosted runs via Docker or Kubernetes with no end-user PII, leaving your environment.
• Open-source codebase: If transparency and auditability are a concern, especially if you work in a regulated industry, you should be able to read every line of code in the platform you depend on. GrowthBook is open source under the MIT license.

Should you build or buy a feature flagging tool?

The answer comes down to three things:

1. Is your use case fairly simple?
2. Is the workflow recurring and painful enough to solve?
3. Is it worth redirecting resources to a product that’s not your main source of revenue?

Even Dropbox’s engineering team built its feature flagging systems in-house, and it worked for years. But when the maintenance cost outgrew the value they were getting, switching to GrowthBook was an easy decision. Now, they’ve consolidated six tools and run 3 billion feature evaluations every single day on their own infrastructure.

You might be biased towards building a solution as a virtue of being an engineer, but you might be better off buying one instead.

If you’d like to test-drive a feature flagging solution to make a better decision, why not try GrowthBook for free? Or book a demo to let us show you the ropes.

Running an experiment is the easy part. The hard part is knowing whether it was worth running at all.

Priya Singhee has spent a decade finding out where that line sits. As global head of storefront product analytics at Wayfair, her teams tested every step of the funnel, from the moment a shopper searched for a couch on Google to the moment they checked out. Today she is VP of Enterprise Analytics & Data Science at Early Warning, the 30-year-old consortium owned by the seven largest US banks that fights identity and payment fraud, and the operator of Zelle, which processed a trillion dollars in payments last year.

On The Experimentation Edge, she told host Ashley Stirrup something most experimentation teams don't want to hear: the majority of testing dysfunction happens before the test ever launches.

Listen to the episode here.

The framework: can you test this validly?

Singhee's decision framework starts with a reframe. The question is never whether a change deserves a test. "It's really not about should we test, but it's more about can you test this validly?"

Four questions decide it.

First, can you truly randomize? "It truly only works when the randomization is clean," she says — clean separation between who gets treatment and who was going to succeed anyway. If cross-pollination, unrandomized seeds, or a mismatched randomization unit contaminate the split, there is no case for a standard A/B test. The unit question is subtler than it looks: teams routinely randomize on one unit — session, user, device — and then read out results on another. That mismatch alone invalidates a readout.

Second, is your effect size plausible given your traffic? "If you only get 5,000 checkout sessions a month, you're hoping for a half percent lift in conversion — you'll need months of data." Most people's guesses about detectable effects are wrong, and the power math has to happen before launch, not as an afterthought.

Third, is the change reversible and cheap to test? A return-policy change, for example, contaminates so much of the experience that a clean split becomes nearly impossible. Some changes simply aren't testing candidates.

Fourth, and the one Singhee weighs most heavily: do you actually have a hypothesis, or are you just poking around? "A test without a specific falsifiable hypothesis is just like a fishing expedition in my mind." Writing it down — if we cut checkout from four steps to two, conversion improves by 1.5 points — forces the team to articulate a mechanism and creates the standard the result gets checked against later. It is also, she notes, the single best vaccine against p-hacking.

The statistic nobody builds their culture around

Here is the number that should reshape how every experimentation program measures itself: 85 to 90% of tests fail. It's a well-known statistic. Yet in most companies, the pressure runs entirely the other way — hunt for wins, stack up wins, report wins.

Singhee flips the incentive. "You have to understand the A/B test is a learning agenda. If you've learned something, it's good enough." And the corollary cuts deeper: "If you're winning too many, I would be very skeptical, because really 85 to 90% of them are supposed to fail." A suspiciously high win rate isn't evidence of a brilliant team. It's usually evidence of a broken measurement pipeline — novelty effects read too early, hidden segment effects miscalled as wins, or multiple comparisons quietly inflating false positives.

Her prescription is to decide the endings before the story starts. Every pre-registered analysis plan should answer three questions: What will we do if it succeeds? What will we do if it fails? What will we do if the primary metric wins but guardrail metrics decline? Getting VP-level approval on those answers up front means nobody is left "desperately trying to prove it to be a win" after the fact.

The best organizations she has seen go one step further: they log every test and every learning in a place the whole company can see. "That is a world-class organization where there's this reinforcement loop going on... And that's how you learn as an organization."

Pre-registration is the antidote to p-hacking

P-hacking rarely announces itself. In Singhee's experience, it arrives politely, after a test has gone sideways, in the form of requests to the analytics team: Can you look at just new users? Just the mobile segment? Just this one slice?

"You will start with your power and duration on a particular sample and a detectable size," she explains, "but nevertheless, you'll then start wondering, 'Oh, how would it do on new users? How would it do on just the mobile traffic?' You never pre-registered for that, so you can't be reading out your experiment results on those."

The fix has to be installed before launch. Her analyst checklist: the hypothesis, the mechanism, the primary metric, the exact statistical test that will be run, and every subgroup analysis the team plans to perform. Anything outside that list is exploration, not evidence.

Then comes the discipline most teams skip entirely: a pre-registered kill criteria. "What would make you say out loud that this idea was wrong, we're not shipping it?" Without a written answer, the default human behavior is to hunt for a subgroup where the losing idea secretly worked.

None of this is about winning more. "The biggest thing for me in a learning agenda is what did you learn from this test? I don't care if it won or lost."

Losses are where the money is

Singhee closed the conversation with the argument she wishes every skeptical executive would internalize. If 85 to 90% of ideas fail, then a company that ships everything without testing is silently absorbing all of those losses. "Imagine if you didn't A/B test... you'd actually be losing revenue." The wins get the headlines, but loss avoidance pays the bills: ship ten features untested and the two losers can erase everything the two winners gained.

Her sincere request: "People do more A/B testing, not less."

The teams that get this right won't be the ones with the highest win rates. They'll be the ones who can answer, for every test they've ever run, one simple question: what did we learn?

Ready to put a real decision framework behind your experimentation program? Start for free or get a demo at growthbook.io.

Feature flags are one of the few tools you can use to change production behavior in seconds, without deploying code. As powerful as that sounds, it also comes with its own risks.

Your team could have dozens of engineers on board who create flags across multiple environments. The same flexibility that sped up your deployment process now becomes the liability that works against you. The reason for that is that if anyone ships a change without approval or leaves an ungoverned flag in the codebase, that’s a potential point of failure that could cost you thousands, and potentially millions, of dollars.

The 2012 Knight Capital incident is one example of this. The company lost $460 million in 45 minutes after a deployment activated dead code on its production servers. When the SEC finished its investigation, it said that the lack of governance controls was one of the contributors to the incident.

That’s why you need a governance framework for feature flagging. In this guide, we’ll explain what it is, what you need to create one, and how to create one for your organization.

What is feature flag governance?

Feature flag governance is the organizational layer of policy and automation that controls how your team manages feature flags across their lifecycle. It covers who can modify a flag in each environment and what review a production change requires before it ships.

It goes beyond being just a best practice for software development. Rather, it's about enforcing a strict process with the right tools and frameworks to determine how your team uses flags. That’s because if you don’t account for this while using feature flags at scale, you could end up with:

• Flag-related technical debt that multiplies quickly.
• Changes in production that have no audit trail.
• Production incidents that could result in unnecessary audits.
• Massive penalties from regulatory institutions for lack of compliance.

The 6 pillars of a feature flag governance framework

Before you even think about implementing a governance framework for feature flags, you need to know what controls it stands on first. Here’s a list of the most important pillars to start with:

1. Flag categories and lifecycle

Every feature flag you create needs a purpose, a designated owner, and a lifespan. This helps you categorize its type and document it properly. If you don’t have this recorded, you won’t be able to tell whether a release flag needs to be deleted or an operation flag (like a kill switch) lives forever.

You can classify feature flags along four axes:

• Purpose: What the flag does (release, experiment, operational, permission)
• Lifespan: How long it should exist in your codebase (short-lived vs. long-lived)
• Scope: Whether it applies uniformly to all users (system-level) or varies per user based on attributes like plan, geography, or device (user-level)
• Value type: The data the flag carries (boolean, string, number, or JSON)

Purpose determines who owns the flag and which lifecycle policy applies to it, whereas the lifespan determines when to clean it up.

If you’re using a feature flagging platform for this, they may not have a built-in category field, as it varies based on how you define it. So you can build a taxonomy by creating the required fields that enforce classification and, as a result, its categorization. For example, an enum for a subscription status.

2. Naming conventions and metadata

A flag’s name should tell you what team owns it and what it does. If you can’t answer both questions from the key, that’s a discoverability issue that scales with every flag you create.

Consider defining your own naming convention to avoid this problem. You can use a structure like team.scope.feature.type and enforce it with a regex validator. You can pair it with the required metadata so every flag has a dedicated owner and review or expiry date.

For a new trading UI, it could look like this:

• Name: tradingUI.redesign.release
• Owner: Matt Hodges, Engineering
• Review date: 10/10/2026

To control flag sprawl, make it mandatory to assign every flag to a project. So nobody can create one without assigning it to a team, which makes ownership a default requirement.

Tip: Some feature flagging platforms enforce naming rules only in the UI, not in the API. If your team or AI agents create flags programmatically, ensure your naming validation runs on every write path.

3. Role-based access control (RBAC)

Depending on the environment the flag is in, the unit of risk changes completely. For example, a flag in staging just means your engineers can see the effects of the changes, but in production, a negative impact could trigger a full-blown incident.

In fact, the OWASP Top 10 found that broken access controls are among the top reasons for application security risks. That’s why you should scope permissions by environment, so there are more restrictions where needed.

The same flag change carries different risks depending on the environment, and your permissions should reflect that. Also, the ability to draft or edit a flag should be separate from the ability to publish that change to your SDKs.  It’s recommended because developers can iterate on changes in staging, but that doesn’t mean they have the authority to push them to production. If your platform doesn't separate these natively, enforce the split at the CI/CD layer by requiring a different deployment token for production-related changes.

4. Approval workflows and change control

Any production change you make using a feature flag reaches your entire user base in seconds. If it results in an incident, you won’t have any record of who did it, who approved it, or why.

That’s why it’s better to gate flag changes behind specific approval layers so that everything requires approval before your SDKs see it.

A good rule of thumb to use is the four-eyes principle. Here, the person who requests a change shouldn’t be the one to approve it. To avoid applying the same unit of risk to all changes, ensure that lower environments require approval only for changes with drastic consequences. Otherwise, limit layered approval to production only.

In fact, in the Knight Capital incident, the SEC noted that the company lacked written procedures requiring review for critical deployments. The missing control was a second pair of eyes to validate the change.

Tip: If you’re conducting high-risk rollouts, embed approval gates directly into the rollout itself. For instance, a staged Ramp Schedule can require sign-off before moving on to the next rollout percentage if all the guardrail and signal metrics look like they’re regressing.

5. Automated validation and guardrails

In July 2024, CrowdStrike experienced a global outage that took down 8.5 million Windows devices due to a configuration update that caused the validator to check against the wrong input count. Validation only works when the schema the validator enforces matches the schema the runtime consumes.

This is why you need automated validation that works with guardrails like human review. The former asks questions like “Is the configuration valid?” while the latter asks questions like “Should you even make this change?”

If you’re building a homegrown tool or using a feature flagging platform, check if it has capabilities like:

• JSON schema validation for non-boolean flag values, so nobody can save a configuration that violates the expected shape.

• Unreachable rule detection to warn you during editing when a targeting rule will never fire because a higher-priority rule already covers the same population.
• Attribute hygiene that rejects unregistered targeting attributes. It prevents typo’d attribute names from creating rules that appear correct but match no users.
• Publish gates that collect every active validation check at the time of publishing and return them in a single response. As a result, the audit trail records which policies a change bypassed.

6. Lifecycle cleanup and audit trails

Any flag that outlives its purpose is considered technical debt. Your governance system should surface them proactively rather than relying on someone to clean them up.

You can either set up reminders in Jira or similar tools to notify you when it’s time to review the flag. Or use a platform with automated stale flag detection to flag stale flags. You can pair this with Code References, a CLI that scans the codebase and where the stale flag lives. So you can pinpoint the exact line and delete or archive the flag.

All of these changes should be recorded in a clear audit log, including a description of who made the change and what it was. Platforms like GrowthBook even differentiate between human-made changes and automated ones. Typically, automated ones show an API badge, but human-made ones show the user based on their Personal Access Token (PAT).

How governance works in the agentic era of development

We’re living in a time where AI coding agents can create and modify feature flags on your behalf. As wonderful as it sounds, it comes with its own risk.

A recent DORA report found that even though AI adoption improves software delivery throughput, it also increases the instability of these releases. As engineering teams ship faster, it has become harder to enforce governance at scale—especially as 30% of respondents report little to no trust in AI-generated code.

Here’s how each AI-driven change maps to a governance control:

That’s one of the reasons we’ve built agent governance into GrowthBook, too. Since users can connect agents like Claude Code and Cursor to GrowthBook using Agent Skills and the MCP server, they hit the same REST API that powers the dashboard. So every agent-initiated change runs through the same governance controls as a human-made one would.

Note: You can see how GrowthBook has evolved its governance controls to keep up with the agentic era.

How to implement feature flagging governance in your organization

You don’t need to implement every single pillar at once. Start with the controls that take the least effort but offer the most value. You can use the following progression as well:

Step 1: Lock down naming and ownership

Make it mandatory for every new flag to belong to a project and enforce a naming convention through your flag creation flow.

In GrowthBook, you can enable the require-project setting in your organization settings and configure a regex validator with an example key format under Settings → General → Features.

It takes two minutes and immediately prevents unowned flags from entering the system.

Step 2: Scope permissions by environment

Assign roles so that developers can toggle flags in development and staging, while only release managers or project leads can publish changes to production.

In a platform like GrowthBook, the permissions for drafting a change and publishing it to SDKs are separate. So you can scope each permission to specific environments and projects.

Step 3: Turn on approval requirements for production

Enable review requirements for production flag changes so every change goes through a review cycle before it reaches your users.

In GrowthBook, you configure approval rules scoped by environment and project. Plus, you can block self-approval and reset reviews when a draft changes after approval. RBAC for flags looks like this in GrowthBook:

Step 4: Add validation guardrails

Attach JSON schemas to your non-boolean flag values and enable attribute hygiene to reject unregistered targeting attributes.

GrowthBook supports JSON Schema validation and Custom Hooks, which run server-side on every write path to enforce organization-specific policies. For example, if you want to make sure your team adds description or other types of metadata while creating flags, you can do so using Custom Hooks.

Example of an approval policy that requires a service-account approval from at least one human being:

if (revision) {
  const approvals = (revision.reviews || []).filter(
    (r) => r.decision === "approve" && !r.stale
  );
  if (!approvals.some((r) => r.userId === "key_abc123")) {
    throw new Error("Publishing requires approval from the release-bot service account.");
  }
  if (!approvals.some((r) => r.userId !== "key_abc123")) {
    throw new Error("Publishing requires at least one human approval.");
  }
}

Step 5: Schedule cleanup from day one

As you create flags, add an expiry date and owner. If it’s manual, you can add a ticket with specifics in your platform of choice. If you’re using a platform like GrowthBook, you get stakeholder detection and code references in one place.

Here, it scans the codebase and surfaces dead flags whenever the enabled environment yields a one-sided result, and it has been more than 2 weeks since it has been untouched. And Code References connect each flag to its call sites in your codebase via a GitHub Action or CLI.

Step 6: Make governance enforceable

There are process elements to the entire feature flag lifecycle too. Uptime Institute’s 2025 Annual Outage analysis found that 85% of human-error-related outages occur because staff don’t follow procedures or because procedures themselves are flawed.

Even if you have a governance policy, it doesn’t mean everyone follows it. While the platform can enforce it by virtue of its capabilities, you still need to build enforcement into your workflows. For instance:

• Add flag ownership checks to your code review process. If a pull request is created without an owner, it shouldn’t pass review.
• Schedule a recurring flag audit (monthly or quarterly) to review stale flags and verify ownership is up to date.
• Wire governance alerts into your team’s existing channels, such as Slack or Teams, to make them visible.
• Assign a governance owner who holds the team accountable when certain rules or frameworks are bypassed.

Build governance you trust to truly reduce organizational risk

Feature flags give your team the power to change production behavior without redeploying code, but if you don’t use them correctly, they become a liability.

The Knight Capital incident began with a flag no one thought to remove from the codebase. But it only got pushed into production because nobody reviewed that change or bothered to cross-check if there was a dependency that could trigger a glitch.

It’s the reason governance has become the focus for engineering teams today and why feature flagging platforms choose to build these features in. GrowthBook offers the tools you need to implement every pillar of this framework:

And since GrowthBook is open-source and can be self-hosted with a warehouse-native architecture, your audit data goes to your own data warehouse, and nothing leaves your environment. If you’re in a regulated industry that mandates SOC 2 or HIPAA compliance, the governance system itself is under your control.

Try GrowthBook for free or book a demo to see how the governance system could work for your organization.

Getting your first experimentation win is one thing. Scaling A/B testing to hundreds of experiments a year across many teams is a distinct discipline with its own challenges. Programs that add velocity without changing how they operate often run into similar problems: teams rerun the same tests without realizing it, small mistakes in assignment and tracking distort results, different teams define the same metric differently and reach conflicting conclusions, false positives accumulate as volume increases, and new launches get backed up in the same engineering queue.

The pressure to scale is also rising. Kevin Yang, executive director and head of experimentation at JPMorgan Chase, described that pressure on The Experimentation Edge:

"With AI coming through, everybody is going to ship faster. But do you have the right infrastructure in place to measure the things you're shipping out? If you don't measure them right, your mistakes are going to compound."

Shipping faster only pays off if you can trust what you're measuring. And when teams optimize for test count, easy tests crowd out important ones, so the program learns less. The point of scaling is faster, more reliable learning that compounds across teams and keeps the product improving.

What "scaling experimentation" actually means

Scaling experimentation means moving from isolated, ad-hoc tests to a standardized, company-wide system. More teams run more experiments, and the data stays as trustworthy as it was when one analyst reviewed every result. The test count grows as a consequence of that system. You're building an operating model that lets measurement keep pace with how fast your teams ship. Shared metric definitions, automated health checks, and approval gates replace the quality control that used to live in a few experts' heads.The culture also has to scale alongside the system so that testing becomes the default for every new feature, and what team learn informs each other's next experiments.

How to scale your experimentation program: 5 pillars for success

Reaching hundreds (or even thousands) of experiments a year without losing trust in your results rests on 5 foundational pillars: infrastructure, prioritization, institutional memory, statistics, and governance. Whichever of the 5 is weakest sets the limit on how far your program can scale.

Pillar 1: A scalable technical foundation

Every experiment adds analysis work, so experiment volume is limited by how much of that work you can reuse. With a warehouse-native architecture, experiments are analyzed directly against the data warehouse you already maintain. There is no copied dataset in a vendor's system to reconcile. Fact tables and metrics are defined once in SQL against your warehouse tables, and that single metric definition serves feature flags rollouts, experiment analysis, and analytics. When the growth team and the checkout team both measure "conversion rate," they are running the same SQL, and you can compare their results.

Agreeing on which metrics to use is as important as agreeing on how they're defined. There are different schools of thought on the right structure. Some programs align the whole company on a single overall evaluation criterion (OEC) or North Star metric, while others set different primary metrics for different parts of the business. Many structures can work as long as the choice is deliberate. If every team picks its own success metrics, you lose the ability to compare results across the program. Whichever structure you choose, require experiments to declare their primary metrics up front, so results are judged against the metric the test was designed to move rather than whatever happened to improve.

It's also important to confirm that nothing in your experimentation pipeline is broken before you scale up. A/A tests (experiments where both groups receive the identical experience) are the gold standard for validating your pipeline. Any statistically significant difference between the groups is either chance or a bug. With GrowthBook's default thresholds, about 10% of A/A tests will flag a single metric as a winner or loser, so 1 flagged metric in 1 test is plausibly chance. Several flagged metrics in the same test, especially with large measured effects, point to broken assignment or tracking, a problem that’s much better to discover at 10 experiments a year rather than at 300.

Pillar 2: Ruthless prioritization

As a program grows, test ideas usually accumulate faster than the capacity to run them. A scoring framework such as ICE (impact, confidence, ease) or PIE (potential, importance, ease) forces every proposed test through the same filter. Most frameworks weigh how large of an effect is plausible, how strong the supporting evidence is, and what the test costs to build, run, and analyze. The scored ideas form a ranked experiment backlog. AI can help at both ends of that process, brainstorming test ideas and scoring them against your framework, with a human in the loop making the final determination. The exact score matters less than the habit of weighing expected value against cost. Low-value tests consume capacity and analyst time, so a mediocre idea that is cheap to run can still cost more than the information it returns.

Sometimes you can even measure an idea's impact without building it. Crystal Ammari, digital product optimization strategist at Disney, described on The Experimentation Edge how a customer service team she worked with tested demand for a video-chat support feature before committing to build it:

"We did what I call dry testing, which is essentially testing a feature without that feature actually being built."

The team added a button for the unbuilt feature to the help page and measured clicks.

"I believe there was somewhere around 4 million people that had entered the test, and only 106 people clicked on the button. I will never forget that 106 because it was such stark, obvious evidence that this is not something people wanted." She estimates the result saved the business millions of dollars in build, hiring, and training costs for a feature almost nobody would have used.

Pillar 3: A living experiment repository

As the experiment count grows, "didn't we already test this?" becomes harder to answer from memory. Some teams track past tests in a spreadsheet or shared doc, but this is only as good as your ability to keep it updated. A useful repository has to be part of the experimentation platform itself. When every experiment runs through the platform, the hypothesis, audience, dates, variations, and results are all captured automatically in a queryable record. That record is the single source of truth for what has been tested.

The repository has 2 uses: teams catch redundant tests before launch, and the accumulated record becomes a portfolio you can analyze for experiment velocity, win rates, average lift, and the scaled impact of shipped winners (the projected effect once a winning change rolls out to everyone). In GrowthBook 5.0, meta-analysis blocks on Product Analytics dashboards track what's running, win percentages, lift, and cumulative impact across the program. The new Learnings feature goes a step further and captures patterns across experiments, with each learning tied to its supporting and contradicting evidence, so people and AI agents can build on what the program has already discovered.

Insights still need to circulate outside the platform. Guests on the Experimentation Edge frequently describe the same two habits: a shared Slack channel where teams post results, and recurring meetings where they review experiment results as a group. Ilya Izrailevsky, the senior engineering manager leading DoorDash's experimentation platform, described how far that can go:

"After we run an experiment, no matter the results, whether ship or no ship, we send the results out across the company and have a discussion. Our leadership, including our CEO, Tony, would read and reply to those experiment emails and would congratulate folks, but also encourage them to try some alternative ways."

Pillar 4: Statistical rigor and clean data at volume

False positives multiply as experiment volume grows. Some of your winners are false, and the share depends on your program's true success rate. When the true success rate is around 10%, about 22% of statistically significant wins are false, roughly 1 in 5, even in a properly powered program. For a program running 300 experiments a year, that works out to about 30 winners, with roughly 7 of them being false positives.

Several techniques protect statistical rigor as experiment volume grows:

• Pre-committed experimental designs: Set the sample size and duration before launch. Stopping a test as soon as it crosses the significance threshold inflates the false positive rate, because at each look, random variation alone can produce a statistically significant result. When teams need the flexibility to monitor and act on results mid-experiment, sequential testing widens the confidence intervals so that repeated looks keep that rate under control.
• Health checks on every experiment: A sample ratio mismatch (SRM) (where the observed assignment ratio differs from what was configured) signals a data problem serious enough that the results can't be trusted. Other checks flag problems like multiple exposures and suspicious uplifts. At scale these checks must run automatically to catch problems before teams act on the results.
• Multiple testing corrections: Evaluating many metrics and variations per experiment multiplies the chances of a false positive. There are statistical corrections that you can implement to compensate for the extra comparisons. At scale, they should be a platform default rather than remembered analysis by analysis.
• Variance reduction: CUPED (Controlled-experiment Using Pre-Experiment Data) uses each unit's pre-experiment behavior to reduce noise in the outcome metric, so an experiment can detect the same effect with a smaller sample. When each test needs fewer users, the same user base can support more experiments at once.
• Isolation for concurrent tests: Concurrent experiments can contaminate each other's results when their changes interact, because some users experience both at once. Namespaces make conflicting experiments mutually exclusive, so a unit is only ever assigned to one of them, while unrelated experiments can continue to overlap freely. 

When a result is significant but can't be explained, rerunning the test is the best way to validate the results. Medha Umarji, VP of growth and experimentation at Fanatics, described catching a false winner: a change the team had tested 6 to 8 times before, always with flat results, suddenly showed a statistically significant revenue lift.

"Another thing that we do here to reduce our false positive risk is we replicate our outcomes. We try to replicate it if we are not able to explain it. And so we ended up turning off this test and rerunning it, and we weren't able to replicate it. It was really flat."

Pillar 5: Open access without losing governance

Engineering capacity limits experiment velocity in a way prioritization can't address. If every experiment requires engineers to implement variations, review the setup, and launch, velocity is capped by their sprint capacity. PMs, marketers, and analysts with testable ideas wait on engineering time. Self-serve tooling removes that dependence with features like a no-code Visual Editor for frontend changes and a simplified experiment creation flow.

Self-serve only works with a quality gate, because opening creation to more people also opens it to more setup mistakes. A draft-first approval flow provides that gate. Anyone can build an experiment, and a reviewer approves it before it reaches production. The same control extends to software agents. In GrowthBook 5.0, AI agents can brainstorm, create, and launch experiments, and their production changes wait for the same human review as everyone else's. When the review step is enforced by the platform, nothing slips through accidentally, even as the number of experiments grows.

What scaling experiments looks like in practice

These 5 pillars are visible in every large experimentation program. Izrailevsky put numbers on DoorDash's current scale:

"We run about 12,000 experiments per year. Over 42 million monthly active users on DoorDash's platform."

At peak, the platform evaluates experiment feature flags around 300 million times per second. No team can review 12,000 experiments a year manually. DoorDash balances a success metric against multiple guardrail metrics for each experiment and shares every result across the company.

The Home Depot's online business recently passed $25 billion in revenue. Kim Ting Li, senior manager of online experimentation there, described on The Experimentation Edge how much attention leadership pays to the tests:

"All the executives and leaders love to learn about the tests. They really truly understand A/B testing is the golden rule to understand incrementality, the real impact."

The same foundations show up in much smaller programs too:

• Chess.com ran about 400 experiments one year and set a goal of 1,000 for the next year, according to Nafis Shaikh, director of product management there. Each team aligns on and owns its own metric area, with the gameplay team optimizing the core experience and the monetization team optimizing revenue.
• Fyxer, an AI email assistant, ran 541 experiments in one year with a small team while growing from $1M to $35M in ARR. AI agents make the code changes and engineers sign off on video previews, so each developer can run several experiments in parallel.
• The Philadelphia Inquirer built a centralized program where marketing, the newsroom, and product all test under one set of standards.
• Lingokids, a kids' learning app with over 3 million weekly active users, roughly doubled its parallel experiment volume to about 15 per month after adopting namespaces to keep concurrent tests from colliding.

How to scale experimentation without sacrificing quality

An experimentation program can hit its velocity goals and still learn very little. Test count is only one input into learning, and once the count becomes the target, teams find ways to raise it that don't necessarily produce useful insights. Judge your program on its outputs instead: the winning changes it shipped, the harmful ones it caught before rollout, and what the results taught you about your users.

GrowthBook is built to implement these 5 pillars, with a warehouse-native foundation, SQL-defined metrics, health checks and false positive controls in the stats engine, a living portfolio view of every experiment, and governed self-serve access for people and agents alike. Ready to scale your experimentation program? Explore GrowthBook's experimentation platform or try it for free.

Three hundred million people play a Supercell game every month. Clash of Clans, Clash Royale, Brawl Stars, Hay Day, Boom Beach: some of the most successful mobile games ever made, built by a company of about a thousand people in Helsinki. A player base that size could justify thousands of A/B tests a year, and plenty of companies a fraction of Supercell's scale run exactly that many.

Supercell runs fewer than a hundred a quarter.

That number is not a program falling behind. It is the most deliberate thing about how the company experiments, and understanding why requires understanding two things Supercell refuses to compromise: its creative culture and its players' trust. On The Experimentation Edge, Shan Huang, a data scientist on Supercell's central experimentation team, walked through how both survive contact with rigorous testing, and what happens now that AI lets anyone in the company analyze an experiment on their own.

Listen to the episode here.

A company where decisions are made bottom up

Supercell's structure explains almost everything about its testing philosophy. "Supercell is a very decentralized company, where decisions are made bottom up," Shan explained. "Each game teams are completely independent in what games they develop, what decisions they make in terms of their games."

Shan's central experimentation team has two jobs. The first is the platform: giving game teams a shared way to run and analyze A/B tests without repeating setup work, with rigor built in. The second is harder to put on a roadmap. "The second part of our team's role is to drive the experimentation culture at Supercell," he said. Because for most of the company's history, A/B testing simply wasn't how games got made. "Historically A/B test was not part of the culture of developing a product because we consider ourself a very creative company, and we still are."

That framing produces the question that anchors the whole episode: "How can we be more hypothesis-driven while keep being creative."

Note the word choice. Not data-driven. When Ashley described the company as baking a data-driven approach on top of its creative core, Shan offered a gentle correction: "I think we prefer to use the word hypothesis-driven." The difference is not semantic. Data-driven can mean chasing whatever the dashboard rewards. Hypothesis-driven means a designer's creative conviction comes first, gets written down as a falsifiable claim, and then meets the players. In Shan's telling, the contradiction is the prize: "If you see the data contradicts with your hypothesis, that's actually the best part of experiment where you actually learn something completely new."

So the volume stays low on purpose. Four or five big games plus a few in testing, ten to twenty experiments per game per quarter, each one chosen because the answer matters. Retention is the North Star, and revenue is deliberately not the goal: "If we make the experience of the game better, business outcomes comes naturally. It's not our first goal."

The scale is what makes the discipline necessary rather than optional. "You can design a very creative and very good mechanism for players," Shan said, "but you don't really know what setup works best for the masses." Intuition produces the idea. Only an experiment can tell you how it lands across the average of hundreds of millions of people.

Telling players they're in the test

Here is where Supercell diverges from nearly every experimentation program you've heard of: they tell players about A/B tests before running them.

"We always try our best to be as transparent as we want to the player community," Shan said. When a test is coming, community managers announce it upfront: the team will be experimenting with features in this part of the game over the next month or two, because they are not yet sure which design makes the experience best. The message to players is an invitation, not a disclosure. We want to try a few things and learn from you.

Then comes the part that turns transparency into policy. "We also want to make players rest assured that if they don't get the better experience during the test period, they will always have a make-up event later on. We take fairness very seriously."

Why go this far? Shan's answer draws on his previous life in e-commerce. "Users come to the website to buy something. But game players, they go to the game to have fun. They are trying to entertain themselves using this time. So if we are testing them by giving some people a disadvantage, it's not a good experience because their main motive is just to have fun."

A shopper who lands in a losing checkout variant loses a few seconds. A player who lands in a losing game variant loses some part of the thing they came for. Supercell's community is famously vocal, and "I don't want to be tested on" is a sentiment any experimentation leader will recognize. Ashley put the underlying math plainly on the show: most new features lose, which is exactly why measuring matters, but to an individual player who doesn't see that math, testing can sound like a scary thing.

Supercell's answer is to treat the fear as legitimate and design around it. Announce the test. Explain the uncertainty. Guarantee the make-up. The result is an experimentation program that a skeptical, passionate community tolerates, and even participates in, because the fairness contract is explicit.

Everyone can analyze an experiment now

The third act of the conversation is the newest, and it will sound familiar to any team watching AI reshape their workflow.

"It's a really new era," Shan said. "We have some experiment skills they can just import to Claude, and then they can ask Claude, okay, use this skill and now I have been set up this experiment and data. You can find the data yourself and analyze it. Tell me the result. What shall I do?"

The consequence is a wholesale change in who does analysis. "It's not restrict to only data analyst or product manager with experiment knowledge. Basically everyone can run and analyze experiment on their own now."

For a central team whose mission is spreading experimentation culture through a decentralized company, this is the dream scenario. Analysis used to queue behind a small number of qualified people. Now it doesn't. But Shan is candid about what got traded away. "It also creates a new challenge that we don't know the quality of running those AI-driven analysis. I'm sure that people with experience, they can guide the AI to run the analysis correctly, but we don't know everybody who's running analysis using AI."

When quality lived inside the gatekeepers, removing the gate meant removing the guarantee. And AI's failure mode is uniquely dangerous in this domain, as Ashley noted: it can give you a very confident answer built on half the data. A false winner doesn't just waste one launch. It quietly redirects investment and, eventually, undermines trust in the whole program. The emerging answer, the one GrowthBook is investing in, is to move rigor from people into the system itself: experiment templates, guardrails, and defaults that make the AI behave the way your best data scientist would.

Trust is the infrastructure

Pull the three threads together and Supercell's approach resolves into a single principle: experimentation at scale runs on trust, and trust has to be engineered as deliberately as the statistics.

The game teams trust the central team because it partners and shares learnings instead of dictating. The creative culture trusts experimentation because hypotheses serve the craft rather than replacing it. Players trust the tests because the company announces them and guarantees fairness. And the next frontier, AI-driven self-serve analysis, will earn trust the same way, through guardrails that make every analysis as rigorous as the expert-run ones.

Fewer than a hundred tests a quarter for 300 million players sounds like restraint. It is actually the cost of doing every one of them in a way nobody, inside the company or out, has reason to doubt.

Ready to bring that kind of rigor to your own experimentation program? Learn more at growthbook.io.

You ran a solid experiment. Good sample size, clean randomization. The overall effect is clearly positive, and you're ready to write it up.

Then you break it down by the new customer segmentation, and something is off. Every segment sits well below the overall number. Not one or two. All of them. How can the whole be bigger than every single one of its parts? Is this the famous Simpson's paradox?

Or maybe you toggled variance reduction on and off and the estimate moved a lot. Either way, you may have walked into one of the worst traps there is: splitting or adjusting your sample on a variable that is itself an outcome. The tell is that it is defined after users entered the experiment, which means the treatment could have moved it. This post is about how that happens, how it poisons your estimate, and the one simple rule that keeps you out of it.

Where bad controls sneak into experiments

Three analysis errors do this, and we will work through each below.

The first is splitting by a segment, the everyday analysis of who the feature helped (there is a whole piece on segment analysis without fooling yourself that guides you past other traps). You break the results down by a dimension computed over the experiment window, like a revenue tier or an engagement segment. It looks like a fair comparison, but the feature itself pushed users between tiers.

The second is conditioning on a funnel step. You restrict to the people who reached it, or you divide by them, and either way the group you compare across is one the treatment helped select.

The third is variance reduction. You tighten your estimate by adjusting for a covariate, except the covariate was measured during the experiment instead of before it, so adjusting for it strips out part of the very effect you are trying to measure.

All three share a red flag. The variable got its value after users were assigned, so the treatment had a chance to change it. That is what makes it a bad control.

What is a bad control? A bad control is a variable you split on, adjust for, or filter on that the treatment affected. Anything that is measured after the experiment starts risks biasing your results.

Why does every segment look worse?

Let's imagine a reading app that ships a new recommendation feature. It increases reading by two extra articles for every user, on top of a base of about ten articles a month. The effect is identical in every segment, so there is no real heterogeneity for a breakdown to find.

You run the experiment for 30 days and collect the data. You want to know who the feature helped most, so you split the results by engagement. There is no ready-made segment for it. So you reuse the definition from the dashboards. The Engagement segment is based on days active in the last 30 days, cut into four tiers from At-risk up to Power. It is a different quantity from the metric you are measuring, which is what makes it feel safe.

You run the split, and the tiers come back with effects at 1.8, 1.4, 1.4 and 1.1 articles. Every one of them sits below the overall estimate of 1.95, and the gap widens as engagement rises. It reads like a real finding: the feature helps your at-risk readers most and does little for power users. Might be worth targeting. How did that happen?

The error is hiding in the segment definition. "The last 30 days" counts backward from the day you run the analysis. That is exactly the experiment window. Better recommendations bring people back a little more often, so the feature moved some users up a tier. You sorted people on something the feature had already changed, then measured the effect inside the resulting groups.

The data here is simulated, so we know the true answer. That also lets us run the split correctly. Set the tiers using the 30 days before the experiment started, so the groupings cannot depend on the feature. Now every tier returns an estimate close to two articles. Not exactly two, because assignment is random and each tier holds a limited number of users, but every confidence interval covers the true effect. The targeting story disappears with it. Same data, same experiment. The only thing that changed was the timing of the segmenting variable.

Figure 1: The same simulated experiment, split two ways. Tiers fixed before the experiment (purple) recover the true effect. Tiers recomputed from days active during the experiment (blue) fall short in every tier, and fall furthest for the most engaged.

The arithmetic looks like Simpson's paradox, and the diagnosis is the opposite. In Simpson's the subgroups are real and conditioning on them is what repairs the picture. Here the subgroups are the damage, because the treatment helped define them. The overall number is the trustworthy one, and the breakdown is what introduces the bias.

When you group by engagement measured during the experiment, you are grouping by something the treatment itself moved. It does not have to be your intended goal metric. It can be any variable the feature affected, perhaps unintentionally. The treated users in a tier got there partly because the feature pushed them up, so you compare them against control users with a higher baseline intent. You silently get the treatment effect plus a bias term.

What the within-tier split actually estimates when defined post-treatment‍
Inside a tier you do not get the effect for that tier. You get the effect for the users who would have been there anyway, plus a selection gap: the difference in articles read under treatment between the users you actually observe in the tier and the users who would have been there without the feature. The feature reshuffles who ends up where, so those two groups are not the same people. The gap is zero only when the treatment doesn't move the tier, which is exactly what a pre-experiment definition buys you.

The reshuffle runs both ways, which is easy to miss. Every tier except the lowest gains users the feature pushed up. Every tier except the highest loses users the feature pushed out. At-risk gains nobody at all, so its bias comes purely from departures: the users most able to climb out do, and the treated group left behind is less active on average than the control group it gets compared with. The notation in the next section covers both cases without comment, because X(1) and X(0) differ whichever direction someone moved.

Why you end up comparing different people

Before talking about the two other failure modes, it is worth seeing why this happens in general, and where that bias term comes from.

Randomization buys you one thing: on average, the treatment and control groups are comparable on everything apart from the treatment feature. Write each user's potential outcomes as Y(0) and Y(1), the result you would see without and with the feature. If that notation is new, the guide to treatment effects walks through it. A clean experiment compares the average Y(1) among the treated against the average Y(0) among the control, and because the groups are otherwise alike, the difference is the real effect.

Segment analysis is to estimate conditional average treatment effects (CATEs). It is the effect inside each subgroup rather than across everyone. Take a characteristic X, like the engagement tier. The CATE at X = x is the average of Y(1) − Y(0) among the users in a given tier, x.

This is well defined when the treatment cannot move X. Usually that means fixing it before launch, but a variable recorded later is fine too, as long as the feature could not have touched it. The users with X = x are the same set whether or not they were treated. Within them you compare Y(1) against Y(0), and the difference is the CATE you were after.

It breaks when the treatment can move X, which is what typically happens when the tier is recomputed from activity during the experiment. Now each user has two versions of it, X(0) without the feature and X(1) with it. The treated users you see at X = x are the ones with X(1) = x. The control users you see there are the ones with X(0) = x. So the comparison you run is

A real effect would hold the group fixed across both terms. This one conditions on X(1) = x on the left and X(0) = x on the right. Those are different groups of people, so the difference in Y is meaningless.¹

How big is the error? The bias is

the reading gap between two sets of treated users. One set is the users the feature lifted into the tier, with X(1) = x. The other is the users who would have been under control, with X(0) = x. The first set arrived from a lower baseline, so even treated they read less than the already-heavy readers. That gap is selection, not a treatment effect. Define the tiers with pre-experiment data so X(1) = X(0), and the bias is zero. More notation details in the appendix.

This is a known failure mode called post-treatment bias, and it is not bounded by the size of the real effect. It can exceed it and flip the sign.² It is also common, well documented in research. A review of published experiments in political science found that nearly half had conditioned on a variable measured after treatment.³ If it slips past peer review that often, it slips into a rushed product readout even more easily. I've seen it many times.

What about funnel steps and conversion rates?

Back to the failure modes now. There is a second way in, and it may be the most common of all. Say the reading app tracks a funnel: saw the recommendations carousel, opened something from it, finished the article.

The direct version is the one we just derived. Someone proposes cutting the analysis down to users who opened something from the carousel, to buy back some power. Most people scroll straight past it whatever is in it, so dropping them looks like dropping noise, and the estimate should tighten.

But that is a bold bet on how the feature works. Better recommendations can make the people already browsing the carousel open something more relevant, which leaves the set of openers untouched. They can also make someone who would have scrolled past stop and open something instead, which does not.⁴ A recommendation feature is usually built to do both. You would be assuming half the effect away in order to measure the other half. The openers left in the treatment arm are partly the ones the feature pulled in, and they have no counterpart on the control side. It is the segment split again, with a filter instead of a tier.

You might do the same thing in a subtler way, simply because that's how some important metrics are defined. A funnel conversion rate divides each step by the step before it. Of the people who opened something, what share finished it? Nobody was dropped and both arms are whole, so it feels safe. But when finishing implies opening, finishes divided by opens is exactly the finish rate among openers. It is the same quantity the filter gave you, reached by a different route. The denominator does the conditioning for you.

E-commerce teams meet this constantly, because almost every rate they care about has a moving denominator. Click-through rate divides by impressions. Conversion rate divides by sessions. Any feature that changes how often people show up, or how many of them reach the step before, moves the denominator. This changes the ratio but not necessarily the final step, which is the one you actually care about.

The fix is not a pre-experiment version of the step, because there isn't one. It is to put the randomized group in the denominator. Finishes per assigned user, purchases per assigned user. That is what funnel metrics in GrowthBook do, measuring every step against everyone who was exposed, and why they deliberately do not hand you the step-to-step rate as a decision metric with a confidence interval.

Can variance reduction backfire?

Splitting and dividing are the visible ways in. Adjustment is the quiet one. Most variance reduction methods, whether CUPED, post-stratification, or plain regression with covariates, work by feeding in a variable that predicts the outcome. The PE in CUPED is for Pre-Experiment, so standard CUPED is by definition safe. But the hunger for power has made it common for tools to offer more extensive variance reduction, by including more covariates that predict the outcome. Post-stratification is one approach.

Take retention as the outcome, whether a subscriber renews next month. The experiment runs on existing subscribers only, everyone who was already paying when it started. Standard CUPED would adjust using last month's retention, the same metric one period back. But that is one for every user in the sample. If it were zero they would have churned and never entered. So the covariate is constant, and there is nothing to adjust with. You grasp for another variable that predicts next month's retention. Reading volume seems like a natural one. Heavy readers renew more often, so reading predicts retention. But the feature raises reading too. Adjust on the during-experiment count and you subtract a difference the feature created, not one that was there to begin with.

But reading volume comes in two snapshots, and the choice matters. Reading from before the experiment is clean. Reading during the experiment is not, because the feature itself raised it. Adjust on that during-experiment number and the estimate collapses, from around four percentage points on retention to roughly zero.

Figure 2: Reducing variance on retention. A pre-experiment covariate (purple) tightens the estimate around the truth. The same signal measured during the experiment (blue) drags it to zero and slightly past.

An imbalanced sample would produce the same gap, but then the adjustment would be repairing the estimate rather than wrecking it, and the pre-exposure balance check would have said so. When balance passes and the estimate still moves this much, suspect the timing of the covariate.

How do you avoid bad controls?

The rule is simple. Check the definition of every variable you split or adjust on, and ask when it was measured. If any part of its window falls after the experiment started, it's a hazard. Condition on pre-experiment data only.

One check is the SRM inside each segment, which GrowthBook runs on the experiment health page. It helps, but it's not bullet proof. In our simulation it fires for the top and bottom tiers but not for the two in the middle. An SRM test spots the net count across a boundary, while the bias comes from who crossed rather than how many.

The most relevant check is pre-exposure balance inside each segment. Randomization guarantees the groups are comparable on anything measured before assignment, so a gap inside a segment could be because you let the segmentation use post-assignment data.

The reading example is fairly simple, as the example segmentation is based only on one metric only that you would think twice about after reading this. But often segmentation is more complex, where a model determines the segments based on many different input variables. In such cases it is easy to think of segments as stable over time, when in fact they depend a lot on recent user activity. Make sure to fetch the segment assignment from before the user entered the experiment.

A striking result in a single segment should raise suspicion. Before you build a story on it, check the timing of the variable underneath. You ran a clean experiment. Keep it clean by splitting and adjusting only on what was already known before it started.

Appendix

The simulated experiment in numbers

Here is that simulated experiment tier by tier, with the users in each arm and their average articles read over the window. The overall estimate across all 20,000 users is +1.95, very close to the simulated +2. The setup details of the simulation are at the bottom of the appendix.

Tiers defined before the experiment

Tiers recomputed from days active during the experiment

First, look at the counts. With pre-experiment tiers the two arms are balanced inside every tier, which is what randomization should give you. With experiment-period tiers they are not. At-risk holds 1,775 treated against 2,111 control, and Power holds 3,200 against 2,859. The feature brought people back a little more often, which pushed treated users upward through the tiers.

Tier switching in itself is not the problem. Nearly 40% of users switch tiers across the time periods, resembling normal segment behavior when based on recent activity. What breaks the analysis is that the switching has a direction correlated with the treatment. Net upward moves run +1,131 in the treated arm against +111 in control. That is the kind of imbalance a per-segment sample ratio check is meant to catch, though as the main text notes it doesn't necessarily in every segment.

Take the Power tier, for example, where the estimate came back at +1.13 instead of +2. The treated users counted there are of two types.

The first type entered the segment only because of the treatment. These users had on average been less active in the pre-experiment period. They got pushed over the tier boundary, and they are also less active during the experiment than the Power-anyway users who would have been there even without the treatment. Pooling both groups drags the treated Power average down compared to the Power-anyway type.

The other tiers behave similarly but also have the highest-propensity users in each tier pushed up to the next tier.

Where the bias comes from

First, the notation. Let D be the assignment, with D = 1 for treated and D = 0 for control. Each user has potential outcomes Y(0) and Y(1), and potential tiers X(0) and X(1). We observe Y = Y(D) and X = X(D), and random assignment makes D independent of all four.

We compute the within-tier estimate at tier x, the treated mean minus the control mean among the users observed there. By randomization, conditioning on the observed tier is the same as conditioning on the potential tier in each group, so

A real conditional effect compares Y(1) and Y(0) within one fixed group. This compares them across two. The treated term conditions on X(1) = x, the control term on X(0) = x. The only way those are the same type of people is if the treatment leaves the tier memberships alone, X(1) = X(0). The real requirement is to condition on a tier the treatment did not set, and a pre-experiment tier meets it by construction. It is the same idea as principal stratification, where the legitimate conditioning is on a treatment-independent quantity.

To size the gap when X(1) ≠ X(0), add and subtract the treated mean on the control stratum, E[Y(1) | X(0) = x].

The first bracket is the ATE for that group, the actual effect among the users who would have been in tier x without the feature. The second is the selection gap from the main text. When X(1) = X(0) it is zero, and the estimate is exactly that ATE. Otherwise it is the difference in treated outcome between the users the feature moves into tier x and the ones who would have been there anyway.

How the simulation works

Here is the setup behind the simulation. The exact constants are in the script.

I generate 20,000 users. Each has a baseline reading count, the articles they read in the 30 days before launch, drawn from a right-skewed negative binomial. The mean is 9.8 articles with a standard deviation of 7.6, the median is 8, and the 90th percentile is 20. That spread is the point: a realistic engagement distribution has a mass of near-dormant users and a long tail of heavy ones, which is what gives the tiers something to separate. Each also has a habitual visit propensity that rises with that reading count, with normal noise of 0.05 on top. Days active is a binomial draw over 30 days at that propensity, averaging about eleven. I sort users into four engagement tiers by the quartiles of days active in that pre-experiment window, from At-risk up to Power. The cut points are set once and never change, the way a real segmentation would be. What moves later is which side of them a user falls on.

I simulate the potential outcomes directly. Y(0) is what a user reads over the experiment window with no feature, a Poisson draw whose mean is that user's own pre-experiment article count. Y(1) is Y(0) + 2, for every user, with no variation. I assign treatment at random to 50%, and observe whichever of the two applies.

The feature also makes people come back slightly more often, enough to add about 0.8 days on a base of eleven. That is deliberately modest next to the 20% effect on reading, because most of the extra reading is more articles per visit rather than more visits. Days active during the experiment is a second binomial draw at the raised propensity, plus month-to-month drift with a standard deviation of 0.04, same for both treatment groups. The biased results in Figure 1 come from re-sorting users into the same four quartile cut points using this experiment-period day count.

About 40% of users end up in a different tier than they started in. Most of that is drift and affects both arms equally. The part that matters is the direction: net upward moves run +1,131 among the treated against +111 among control. Driven by the treatment effect on days active.

Figure 1 reports the within-tier effect on articles read, once for tiers fixed before the experiment and once for tiers recomputed during it. The true effect is two articles in every tier, as simulated.

Figure 2 uses a second outcome, retention. Every user in the experiment is already a subscriber, so pre-period retention is one for everyone. That makes it a covariate with no variance and no use for variance reduction. The natural fallback is reading volume. I model retention as a logistic function of pre-period reading and treatment, tuned to about a four percentage point effect. Then I reduce variance two ways, with CUPED and with post-stratification. Each time I use either the pre-experiment reading count, which is safe, or the experiment-period count, which is the bad-control case. The safe version tightens the estimate around the truth. The bad control drags both downwards. The strata for the post-stratification run are quartiles of that same reading count, so the two methods condition on the same variable, one continuously and one in four bins.

¹ Conditioning on the joint pair {X(0), X(1)} rather than the observed X does define a valid effect, the idea behind principal stratification (Frangakis and Rubin, 2002). The subgroups are latent, though, so they are hard to use in practice.

² Two formal treatments of bad controls. Wooldridge, "A Formal Investigation of 'Bad' Controls" (SSRN working paper 4688295), works in potential outcomes. Cinelli, Forney, and Pearl, "A Crash Course in Good and Bad Controls," Sociological Methods & Research 53, no. 3 (2024): 1071–1104, give the causal-diagram (collider) version.

³ Montgomery, Nyhan, and Torres, "How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It," American Journal of Political Science 62, no. 3 (2018): 760–775. They found 46.7% of the studies they reviewed, 35 of 75, conditioned on a post-treatment variable.

⁴ Economists call these the extensive and intensive margins: whether someone acts at all, versus how much or how well they act if they act at all. The filter here is safe against movement on the intensive margin and broken by movement on the extensive one.

Dimension splits can surface what your Average Treatment Effect hides, or fool you with noise. How to avoid cherry-picking your results, correct for multiple tests, and steer clear of bad controls.

Why your experiment results look different across segments

Your experiment's Average Treatment Effect (ATE) is just that, an average. As we wrote in the previous piece, behind a flat or modest headline result, users can be having very different experiences. Real gains for some, real losses for others, with the effects canceling out in aggregate. How do you actually find those differences, and how do you know when you've found something real?

The answer, in most experiments, is dimension splits: cutting your results by user properties like market, device, plan tier, or account age. It's a natural instinct and a reasonable one. The problem is that the same dataset that contains real signals also contains noise, and noise has a habit of looking like signal when you cut the data enough ways.

That doesn't mean you shouldn't do it. There are three good reasons to look at segment-level effects:

1. Understanding the mechanism. Why did this feature work, and for whom?
2. Informing a rollout decision. Should this go to everyone, or only to certain users?
3. Assessing commercial value when short-term metrics fall short. Some features matter most to your most valuable users, and a single headline number won't show you that.

To make these questions concrete, we'll use a running example throughout: an upsell banner prompting users to join a paid loyalty program.

Whatever your reason for splitting, what you're estimating in each segment is its own Conditional Average Treatment Effect (CATE): the treatment effect for users in that subgroup rather than for the full population. And one caveat applies no matter how you slice: segment samples are smaller than the full experiment population, so estimates are noisier and you have less power to detect an effect of a given size.

This piece covers three ways those splits can mislead you. Each calls for a different response.

Before the experiment: pre-specify your segments

The most defensible approach to segment analysis starts before the experiment runs. You identify the dimensions you want to examine and write down the hypothesis behind each. For the loyalty program upsell banner example, at least two dimensions make sense: recent buyers (those who made at least one purchase in the 30 days before launch), who may respond more strongly because they are already more loyal; and market, because the loyalty program is more developed in some countries. We pre-specify these dimensions because we have clear hypotheses.

Why write the hypothesis down?

Pre-specifying protects against a specific failure mode: cherry-picking. When you choose which dimensions to examine after seeing the results, it's easy, even unintentionally, to frame a fishing expedition as a deliberate analysis. You report back the dimensions with interesting results, but quietly discard the ones without. Writing down your dimensions and hypotheses before the experiment starts makes that impossible: everyone can see you made the choice before the results could influence you. 

Pre-specifying doesn't solve everything, though. Pre-specify 20 markets and you'd expect roughly one to come up significantly by chance, even if the feature does nothing anywhere. That's the multiple testing problem, and pre-specifying only means you're facing it honestly.

Which correction fits the decision?

When the goal is a per-segment rollout decision rather than understanding the mechanism, the stakes change. You're making an independent call for each segment: does this market get the feature, or not? A wrong call ships a feature to users it doesn't help.

The obvious fix is to raise the bar. Instead of calling a result significant at 5%, demand 1%. That instinct is right, and it's essentially what a multiple testing correction does. The difference is that a correction sets the bar for you, based on how many segments you tested and which kind of mistake you're trying to avoid. Set it by hand and you're guessing. Too loose and you roll out to segments that were noise. Too strict and you miss the segments where it worked, which is a real risk when each segment is a fraction of your sample.

So the question isn't whether to correct, it's which correction matches the decision you're making. The table below summarizes the two standard approaches. Both are frequentist, framed around p-values. GrowthBook's Bayesian engine has a direct counterpart for each, which we note as we go.

For per-segment rollout decisions, FDR is probably more natural. When you're making independent calls for potentially many segments, a small proportion of incorrect rollouts is an acceptable cost as long as most are right. The principle generalizes: match the correction to the decision rather than applying one uniformly.

The Bayesian equivalent

The same decision carries over to GrowthBook's Bayesian engine without any p-values. Each segment reports a Chance to Win, the posterior probability that the variation beats control. The posterior probability that a given rollout is a mistake is therefore 1 − Chance to Win. Rank the segments by Chance to Win and roll out down the list until the running average of (1 − Chance to Win) reaches the false discovery rate you're willing to accept. That average is the expected share of your rollouts that aren't real. It is the same notion of error Benjamini-Hochberg targets, read off the posterior instead of from p-values.

One assumption underpins that number. If your prior is flat, you are treating every effect size as equally plausible before you look. Scan twenty segments on that assumption and the top of your list will look better than it deserves to. A prior centered at no effect corrects some of this, because it pulls the noisiest segments toward zero hardest. Exactly how hard to pull is an open debate.

In practice, very few experimentation platforms apply any correction to dimension splits at all; most treat them as exploratory by default. GrowthBook applies correction within each dimension breakdown when the metric is a primary metric. However, it corrects within each dimension separately, not across all dimensions at once. For FWER users that's something to bear in mind: each additional dimension you run adds more chances for a false positive to slip through. For FDR the concern is much more limited, since you're controlling a proportion rather than a count.

After the experiment: how should you explore post-hoc?

Once an experiment finishes and you see the main results, you instinctively want to start slicing. Maybe the overall effect was smaller than expected, or it moved in an unexpected direction, or you're just trying to understand who moved the number. That's a useful impulse. This section isn't an argument against exploratory analysis, it's about how to do it without fooling yourself.

Exploratory analysis is harder to keep honest, because the protection you had before is gone. Pre-specifying made cherry-picking impossible to hide. Post-hoc, you're slicing after the fact, which is, in all but name, a fishing expedition. Even though it might not feel like one. The multiple testing problem hasn't gone anywhere either, and now nobody's counting. With pre-specified markets you knew how many tests you were running. Post-hoc slicing is open-ended: you cut by device, then tenure, then market, and by the time you cut by purchase frequency you have forgotten about device.

How do you explore honestly?

What changes is the goal. You're generating hypotheses now, not confirming them, so the question is no longer whether a segment cleared a significance threshold, and formal corrections aren't the tool for that job. Read the unadjusted intervals directly: the point estimate is the candidate signal, its width is the noise around it, and you judge each segment by how far it stands out. In GrowthBook's Bayesian engine, rank by Chance to Win and read the list the same way. Then say it out loud, to yourself and your audience, that you're ranking candidates rather than confirming findings. Keep one caveat in mind: the more segments you scan, the more spurious front-runners appear at the top of the list. Thus, a strong result is a reason to run a confirmatory test, not a finding in itself.

Report every cut you ran

Be transparent about what you actually ran. Report how many cuts you made. "We sliced by five dimensions post-hoc and the recent buyer segment showed the strongest signal" gives anyone reading the analysis the context to calibrate. Omitting that information, even unintentionally, is how people mistake exploratory findings for confirmatory ones. A shared results interface helps, since what you ran is logged and visible to the whole team rather than only to whoever ran the analysis.

One thing should raise your confidence: convergence. When independent dimensions point to the same story, that should move you more than any correction can. Just watch for the temptation to grasp for whatever story fits the results in front of you.

Treat every post-hoc result as a hypothesis, not a finding.

Replicate. That's the bottom line for exploratory segment analysis. Design a follow-up experiment that tests the result properly: pre-specified, sized for that segment, with the appropriate correction if a rollout decision is on the table. If you're not willing to run the follow-up, you're not ready to act on the finding. A result that survives replication is something you can stand behind. What doesn't replicate remains a hypothesis.

Any segment: avoid bad controls

There's one mistake that ruins a segment analysis no matter how careful you were everywhere else: splitting on a variable that the treatment itself changed. It has nothing to do with how many tests you ran or whether you pre-specified them. If the treatment moves the dimension you're splitting on, the comparison is worthless before you start.

The loyalty program example makes this concrete. After the experiment ends, you want to know whether Top Customers responded differently to the upsell banner. Makes sense: these are the users whose behavior matters most commercially. So you define "Top Customers" as anyone who spent more than $200 over the 60 days before the analysis date. Well intended, but that window overlaps the experiment period. Some treatment-group users saw the banner, joined the loyalty program, and then bought more than they otherwise would have. The program's perks gave them a new reason to spend. Some of those users crossed the $200 threshold because of the program, not because of any prior purchasing pattern.

In the control group, Top Customers are established high spenders. In the treatment group, Top Customers now also include people the program pushed over the line during the experiment. So within the same segment label, you're comparing established high spenders against a mixed group that includes newly converted ones. Any CATE estimate for Top Customers from this comparison is biased: the two groups share a label but have different baseline intentions.

The fix is simple: segment only by pre-treatment variables. Any dimension that could plausibly have shifted in response to the treatment is off-limits, regardless of how natural it feels to look at. Top Customer status defined using spend from before the experiment started is fine. Defined using a window that overlaps the experiment period is not. With dimensions that change over time, this happens by accident. A join that picks up a user's current attribute value instead of the value at assignment time quietly reintroduces the problem.

Why post-treatment splits backfire covers the full mechanics, with a simulated experiment where every segment comes back understated.

What good segment analysis looks like

Pre-specify when you can, and apply the right correction for what you're deciding. When you explore post-hoc, do it honestly: report what you ran, treat results as hypotheses, and replicate before acting. That covers the first two reasons for running dimension splits. The third is simpler.

Think back to the Top Customers segment, defined cleanly on spend from before launch. Those users' long-run value far exceeds what a normal experiment duration can show. A feature that concentrates its effect on them can be worth shipping even when the headline ATE looks modest. That's the third reason to split: not to act on each segment, but to see whether the case for shipping to everyone is stronger than the headline number suggests.

Post-hoc slicing is exactly the kind of analysis statisticians warn you against, and the concerns above are precisely why. But the cost of never looking is just as real. Some of the most important segment effects are ones nobody had a hypothesis for, and exploration is the only way to find them. There is a more systematic way to do this. Methods like causal forests search across many dimensions at once, looking for heterogeneous treatment effects without asking you to pre-specify where to look, which takes much of the cherry-picking problem off the table. That's a topic for its own piece.

One thing worth doing this week. Pick three experiments you've already called, ideally ones where the headline came out flat or ambiguous, and open them up in exploratory mode. Not to relitigate the decision, but to build a list of segment hypotheses worth following up on. Past experiments are the cheapest source of hypotheses you have.

As Hattori Hanzō would say: any fool can pick up a blade. Knowing where to cut is the craft.

Running an experiment gives you an answer to a question. Running thousands of experiments gives you a lot of answers, but also something much more valuable: a body of evidence about how your product, users, and business actually behave.

The problem is that this knowledge is surprisingly difficult to use in practice.

A PM working on onboarding might not know that another team tested a similar idea six months ago. An engineer building a new checkout flow might not know which patterns have consistently helped conversion on other parts of the product. Even when people remember that relevant experiments exist, reading through dozens or hundreds of them to find the important patterns is rarely practical.

Today, we are launching Learnings in GrowthBook to help solve this problem.

Learnings let teams capture what they have learned across experiments, user research, and other sources of evidence, then make that knowledge available to both people and AI agents when they are building something new.

Experiments produce more than winners

The most obvious output of an experiment is a decision. Those decisions create incremental gains, and over time those gains compound.

But there is another output from experimentation that is easier to overlook: knowledge. Every experiment produces it, including the ones that lose or move nothing at all.

You might learn that:

• Showing pricing earlier in the funnel consistently improves qualified conversions.
• Simplifying onboarding helps new users but hurts activation for experienced users.
• Social proof matters on acquisition pages but has little effect inside the product.
• Asking users to configure everything upfront creates friction, while progressive configuration performs better.

None of these conclusions necessarily come from a single experiment. They emerge after five, twenty, or a hundred experiments, once somebody notices the pattern.

This is where an experimentation program becomes more powerful than a sequence of isolated A/B tests: individual tests give you answers, but the program gives you a model of how your users behave. You are gradually identifying patterns and putting your learnings to work for future growth.

Learnings compound your experimentation program

If an experiment improves conversion by 2%, that improvement can continue generating value for as long as the change remains in the product (novelty effects aside). That’s real compounding, and it’s the return most programs measure.

But that win only compounds one thing: a metric. A learning acts on something different: the quality of the next decision.

Imagine your team learns through repeated experiments that users perform better when complex actions are introduced progressively rather than all at once. That insight might influence your next onboarding flow. Then your settings experience. Then a new AI feature. Then the way an agent designs a workflow six months later.

It doesn’t stay with one team either. An insight can travel to whoever picks your onboarding feature or settings work next. Your experimentation program can widen, since everyone starts from organizational knowledge.  

The value is not limited to the experiment that produced the learning. It changes the starting point of future work. Instead of beginning every project from first principles, your team starts with a set of evidence-backed assumptions about what tends to work (and what doesn’t).

Learnings in GrowthBook

GrowthBook Learnings are designed to capture this organizational knowledge explicitly.

A learning can reference evidence from multiple experiments, rather than being tied to the result of a single test. That matters because many useful conclusions only become visible across a collection of experiments.

You can use Learnings to document things like:

• Patterns that repeatedly improve a metric
• Approaches that consistently fail
• Differences between user segments
• Design principles supported by experimentation
• Unexpected behaviors observed across multiple tests
• Areas where the evidence is contradictory or uncertain

The same change isn’t universally applicable, though. Streamlining a flow can lift conversion in one part of your product and lower it in another, and the right amount of friction depends on what the user is trying to do. Learnings can be scoped to specific projects or tags, so that patterns that have been tested for your self-serve signup aren’t applied to your enterprise onboarding flow. 

And learnings do not have to come exclusively from experiments either. You can capture qualitative findings from user research, customer interviews, support conversations, or other sources and combine them with quantitative evidence.

The goal is not to turn every observation into an immutable rule.

It is to give your organization a shared, evidence-backed memory. Each learning can be updated if new evidence comes in, and also have a specific status for when the learning is not verified yet, or if it’s no longer relevant. Statuses for learnings are entirely customizable. Once captured this way, that memory becomes something both people and AI agents can draw on, which raises the question of how much of it to hand an agent at once, and in what form.

The context window problem for organizations

There is a useful analogy to working with AI coding tools.

If you give an agent every line of code your company has ever written, you have technically given it more information. But you have not necessarily given it better context.

The useful question is: what does the agent actually need to know to make this decision well?

Organizations have the same problem. After thousands of experiments, nobody should need to read thousands of experiment reports before starting a project. They need the relevant conclusions.

Learnings act as a compressed context layer over your experimentation history. The underlying experiments are still there as evidence, but people and agents can work from the higher-level patterns that those experiments have established.

For example:

Learning: Users are more likely to complete complex setup flows when advanced configuration is deferred until after initial success.

Evidence: Seven onboarding experiments across three product areas.

A PM planning a new onboarding experience can start with that knowledge instead of rediscovering it. An engineer can incorporate it while designing the implementation. The same applies to agents, even more strongly. An agent without access to your evidence will still produce a confident answer, but it will be generic and maybe an approach you’ve already disproved. Point your agents at your Learnings, and they start from what you already know.

Learnings are available through GrowthBook's APIs and MCP support, just like your experiments and other experimentation data. Through GrowthBook Skills, you can instruct agents to retrieve relevant Learnings before they design or implement something.

Instead of giving agents generic product-development best practices, you can give them exactly what they need to know to make this decision well: your company’s own compressed, decision-relevant knowledge, grounded in real outcomes and updated as new evidence comes in.

AI can help find the patterns humans miss

As experimentation programs grow, manually identifying patterns becomes harder if not impossible. A team running ten experiments a year can probably remember most of them. A company running thousands cannot. GrowthBook can use AI to analyze your experiment history and surface commonalities across results.

Perhaps a certain type of messaging consistently works for new users but not existing customers. Maybe several unrelated experiments show that reducing perceived commitment improves activation. Maybe a UI pattern that teams keep proposing has actually failed in four different areas of the product.

These patterns are easy to miss when every experiment is analyzed independently.

AI makes it possible to search across a much larger body of evidence, while Learnings give teams a place to review, refine, and preserve the conclusions that matter.

Good experiment hygiene becomes even more valuable

There is an important prerequisite.

AI cannot infer very much from an experiment called "Homepage test 7" with no hypothesis, description, or conclusion.

The better your experiment documentation, the more useful your accumulated knowledge becomes.

This is one reason GrowthBook has increasingly invested in experiment quality and hygiene, including checklists and workflows that encourage teams to document the hypothesis, context, results, and conclusions of an experiment. Good documentation has always made individual experiments easier to understand.

With Learnings, it also makes the entire experimentation history more valuable. Every well-documented experiment becomes another piece of evidence that can contribute to future decisions.

From experimentation history to organizational memory

The long-term value of experimentation is not just making better decisions today. It is making every future decision from a stronger starting point.

Experiments that win improve the product. All your experiments improve the ideas, assumptions, and decisions that come next, increasingly including the ones your agents make. That’s the part that really compounds. Firmer footing leads to better experiments, which produce better evidence, which produces better judgment. Over time, your experimentation program is not only improving your product, it’s improving your organization’s ability to build one.

Experiments should not disappear into a results archive once a decision has been made. The best ones should keep teaching you.

‍

Running an experiment is the easy part. The hard part is knowing which experiments are worth running at all, and what to do when the textbook playbook, the classic 50/50 production split, is simply off the table.

That's the situation Ben Schein lives in every day. On this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Ben, Director of Product Management at Clover, one of the largest point-of-sale and payment processing providers in the world. Ben oversees product strategy for the on-premise and digital tools that restaurants run their businesses on: the point of sale a server uses on the floor, the handhelds they carry between tables, and the systems behind menus, promotions, reservations, online ordering, and catering.

The scale is staggering. Clover serves more than 300,000 merchants in the United States alone and processes billions of dollars every day. Ben can watch spikes ripple through the system in real time when a World Cup match or a major concert hits a city. And that scale shapes every decision his teams make about testing.

Listen to the episode here.

You can't A/B test a work tool

Here is the constraint that makes Clover's experimentation program different from most: the product is someone's workday.

"Testing is not really 'let's put something in production and see what happens,'" Ben explained. "The worst possible thing is to show up to work the next day and suddenly the tool that you use for work is totally different and no one told you why. You can't do an A/B test in that environment."

A consumer app can quietly ship a variant to 5% of users and watch the metrics. A server picking up a handheld during the dinner rush cannot be a surprise test subject. So Clover inverts the standard model. Instead of testing in production, everything is proven before rollout: structured pilots, ground-level validation, and a detailed go-to-market plan for every feature that ships.

The consequence is that testing carries real financial weight at Clover. "If we don't know that this thing is gonna be successful, we're not gonna invest all those dollars in getting this thing to market," Ben said. Testing is not a stage gate that slows the roadmap down. It is the evidence that justifies the rollout budget in the first place, part of the company's vernacular and culture, as Ben puts it.

Uncertainty and downside set the testing depth

If you can't test everything in production, you have to be ruthless about what earns deep experimentation. Ben's triage comes down to two variables: how much uncertainty is built into the change, and how much real downside exists if it goes wrong.

At the high end sit things like payment authorization flows or the entire funnel a server uses to place an order on a restaurant floor. Those changes are high touch and central to the core experience. They get intentional, ground-level testing before any system-wide change, because when something less proven trips up, the impact is significantly bigger.

At the other end sit table stakes features. Adding Apple Pay matters to the business, but it is an industry standard; the uncertainty was resolved years ago by the rest of the market. You monitor the impacts after rollout, and you move on.

The lesson for experimentation teams: testing depth should follow the risk reward math, not the visibility of the feature. A flashy redesign might need less rigor than an invisible change to an authorization flow. Most roadmaps get this backwards.

Ben applies the same discipline to what happens after a test reads out. His advice to PMs who are new to experimentation is to know the strategic value of a test before running it. "We want the learnings of those tests. We don't always know why we want those learnings," he observed, and a junior PM staring at a significant result with no idea what to do next is the predictable outcome. His fix: structure tests for durable business value rather than pass-fail verdicts, pair every headline metric with counter metrics, and anchor on ground-level measurements like items per check that stay resilient whether $100 or $100 million is being spent on marketing that quarter.

What a burger chain teaches about checkout funnels

Before Clover, Ben led product at Shake Shack, joining at the tail end of the pandemic when the brand's digital channels had gone to market fast and the open question was whether they were any good.

The team's answer became a masterclass in treating a conversion funnel as a brand channel. Shake Shack is built on hospitality, so the question Ben's team asked was strange and productive: what does hospitality look like in a checkout flow?

Some of it was classic optimization: removing redundant steps, and fixing customers who accidentally ordered from the wrong location as store density grew. But the more interesting work was additive. The team tested how presenting prep-time expectations affected conversion, because a Shake Shack burger takes longer than typical fast service and an unexplained wait erodes trust. Set the expectation honestly at checkout, and NPS downstream reflects that the promise was kept.

Then there were the loading screens. Those interstitial moments after a purchase became deliberate brand real estate: the quality of the beef, the fact that the patty is never frozen, conveyed in a passive moment the customer was already spending rather than buried third in a product description. The same thinking carried to the kiosks in the restaurants themselves.

The thread connecting it all is context. "I don't even think of it as personalized, it's contextualized," Ben said. Nobody opens a restaurant app just to browse; they are hungry, they are in the car, they are standing somewhere they can't see the menu. Funnels that acknowledge that intent outperform funnels that don't.

Why this matters for experimentation teams

Ben's career runs from hand-writing test parameters in Firebase config files at NPR, with weeks of work per readout, to a world where the same test runs in an afternoon. His prediction is that the democratization keeps going: interns running hypothetical experiments in sandboxed environments without asking permission, with AI tools making analysis approachable to anyone regardless of background.

But the through line of the conversation is that tooling was never the hard part. The hard part is judgment: knowing which changes carry real downside, structuring tests so their learnings have durable value, and understanding the context your users bring before you measure what they do. Machines may beat humans at allocating ad spend, as Ben readily concedes. Knowing your audience well enough to see where they are going next is still a human edge.

Hear the full conversation with Ben Schein on The Experimentation Edge. And if you're ready to bring that kind of rigor to your own rollouts, visit growthbook.io to see why the open source experimentation platform leader powers testing programs at any scale.

‍

Every experimentation program has a story it keeps coming back to. For Edd Saunders, product experimentation manager at JobLeads, it involves pizza.

On this episode of The Experimentation Edge, Edd walked through the experiment that looked like a guaranteed winner, failed completely, and permanently changed how he designs tests. He also shared the problem mapping framework he uses to train new experimenters and explained how JobLeads multiplied its experiment velocity almost tenfold in about a year.

Listen to the episode here.

The sure thing that went nowhere

A few years back, during his consulting days, Edd worked with a large, established pizza company that wanted to explore personalization as a way to improve its web experience. The research was thorough. Customer journey maps showed that the average customer needed around 22 distinct inputs from starting a session to completing an order. About half of all traffic came from returning users, and those returning users ordered the same pizza week after week. Previous experiments had shown that customers kept two or three competing delivery sites open at once, racing whichever basket filled first, hungry and increasingly irritated as their blood sugar dropped.

The logical conclusion practically wrote itself: reduce the number of clicks. The team built a database that captured every item a customer ordered, saved it against their user ID, and served their usual order back in a nicely designed widget on their next visit. One click added the entire order to the basket and sent the customer straight to checkout. Twenty-two steps became five or six.

"It had absolutely zero impact, zero impact on user behavior," Edd said. "It didn't increase purchases. It didn't decrease purchases. People saw it and just thought, nah, I'm not using that."

The team iterated a few times, then drew the honest conclusion: forcing personalization on these customers wasn't worth the money. But the deeper learning was about why the feature failed. "The exploration is still a massive part of customers' delight," Edd explained. Customers wanted to feel in control of their experience. "By making it easier to find their order again, we actually took away some of their control." People would tell themselves they wanted something different this time, browse the menu, and land on the same pizza they always ordered. The browsing was the point.

For Ashley, the story hit close to home. He admitted he personally would have wanted the feature, and that its failure would have kept him up at night. That is precisely the value of the experiment. It saved the company from continuing to invest in a direction that data, logic, and intuition all endorsed, and that customers quietly rejected.

Getting out of the solution space

The pizza story sets up the question every experimentation leader eventually faces: how do you guide someone who has never run a test before? Edd spent about six years consulting and training people who were new to experimentation but full of ideas, and he has a clear diagnosis of where they go wrong.

"The trickiest part is getting people out of the solution space thinking and moving them into the problem space thinking," he said. Solution space thinking means generating ideas and throwing them at the wall. And because everyone carries a natural bias that their own ideas are better, everyone new to testing eventually has the same humbling experience. "Everyone new to this has to go through this kind of ego death, where their brilliant idea actually has zero impact whatsoever."

His alternative starts with the customer. Edd runs an exercise called customer journey mapping that doubles as an introduction to analytics tools. First, understand where people come from, where they land, and every micro step between acquisition and activation, then calculate funnel drop-off rates in granular detail to find the biggest leak. Second, layer qualitative data on top: heatmaps, session recordings, click maps, and scroll maps that reveal the how behind the what. Third, add user research, from interviews to third party reviews. At JobLeads, social media reviews alone provide a huge amount of intelligence about where the product should go.

From those three sources, problems get written as statements: as a new user, I don't register for JobLeads because I don't trust the brand. Then comes the exercise Edd loves most, problem mapping. Each problem lands on a 2x2 matrix. The horizontal axis measures evidence, from pure assumption on the left to fully validated on the right. The vertical axis measures impact. The top right quadrant, high impact problems you know are real, becomes the first batch of hypotheses to test. High impact unknowns get logged for additional research, or simply tested if they're cheap to run.

The payoff goes beyond win rate. Validated problems are useful to marketing, customer support, and leadership, not just product teams. And the approach solves the fear that haunts every new experimenter: running out of ideas. "If you live in the solution space, your ideas run dry pretty quickly," Edd said. "If you understand the problems you're trying to solve, naturally you'll come up with multiple hypotheses for each problem. One problem can spring a nice tree of ideas."

Democratizing experimentation

JobLeads only started testing seriously in early 2025, and the program's growth has been steep. Velocity has climbed from roughly 0.3 experiments launched per month to about 2.8. Edd, who was the second dedicated experimentation hire, has spent much of his time on experiment operations: standardized workflows, making each departments roadmap accessible to everyone, and a company-wide knowledge base that synthesizes the learnings from every experiment for anyone in the organization, whether they sit in marketing, customer support, or product.

That knowledge base serves a second purpose: keeping people excited. Edd has watched interest in experimentation spike at launch and then dwindle as people discover that most of their ideas won't win. His counter is to reframe what a result means. "It isn't look at what's won, look at what's lost. It's look at what we've learnt and look at what better decisions we're empowered to make because we've run an experiment."

Ashley summarized the compounding effect with a baseball metaphor: experimentation isn't about home runs, it's about stacking singles. Edd agreed. "Your one test might not have a huge amount of impact, but when you look back over a year, the 100 or so tests you run might. The growth will have compounded over time."

Looking ahead, Edd's ambition is to push experimentation beyond the specialists. He wants content managers and everyday marketers to take an idea from problem to live experiment themselves, then watch it get validated and absorbed into production. "I guess you could call it democratizing experimentation, giving the power to the people."

The obstacle isn't primarily education. It's habit change. People are already under pressure, and mandating experimentation on top of their day jobs breeds resistance. Edd's answer is to make the value of each contribution visible: "Hey, you had that great idea a few weeks ago. We ran it as an experiment, and the company now knows X, Y, Z because of your insight."

AI plays a quiet supporting role throughout. Edd uses it to validate MVPs cheaply before features get built properly, and for the unglamorous automation that keeps the machine running: drafting tickets, standardizing documentation for the knowledge base, and small workflow automations like extracting Figma links between ticket statuses. Little stepping stones, as he put it, that ease the burden across the organization.

The takeaway

The thread running through the conversation is humility in the face of evidence. The most logical feature can have zero impact. The most brilliant idea can flop. What compounds isn't any single win but the accumulation of validated learnings, shared widely enough that the whole company gets smarter with every test.

Hear the full conversation with Edd Saunders on The Experimentation Edge. And if you're ready to raise your own experiment velocity, visit growthbook.io to see why the open-source experimentation platform leader powers testing programs at any scale.

In 2025, Google Cloud went down globally for roughly three hours. This happened because of a simple configuration change that left blank fields, and within seconds, it was replicated across the entire service.

Google’s own incident report was blunt about the root cause. They admitted they did not have appropriate error-handling measures, such as feature flags, in place.

A config value broke production. But a feature flag would’ve stopped it.

A remote config and feature flag both change production behavior without a redeploy. That’s what makes the boundary hard to see.

In this article, we’ll explore the differences between feature flags and remote configs and when it makes sense to use either or both.

What’s the difference between a feature flag and remote config?

A feature flag decides whether something happens with your code. Remote configuration decides what a value is. Both of these functionalities live outside your codebase, and both change without redeploying code. That’s why many developers conflate them.

Let’s say you’re rebuilding a checkout feature. The initial rollout is 10% of users while you monitor error rates, and the payment service it calls has a rate limit you can tune as traffic increases. The checkout page has a JSON object that controls field order and input density. So, in this case, the flag asks who should experience the new checkout flow, while the config values such as the rate limit and layout object define how the system behaves once you’re in it.

Now, the difference is clear in how you target different aspects of your infrastructure and what the payload is meant to do.

For instance, flags evaluate only against specific user attributes, such as geography or device type. If two users hit the same endpoint, they see different answers depending on their attributes. Remote configuration can do the same thing. You can serve the same value across an entire environment or segment them based on certain attributes like geography. But the focus is on what value a setting holds (the payload) and not on who should see a feature. In either case, the value is permanent and continues to stay in your codebase, unlike a feature flag.

Here are a few other differences between feature flags and remote configuration:

The method you reach for works the same way in the codebase, too:

// Feature flag: should this user see the rebuilt checkout?
if (gb.isOn("new-checkout-flow")) {
  renderNewCheckout();
}

// Remote configuration: what rate limit applies here?
const rateLimit = gb.getFeatureValue("payments-rate-limit", 100);

isOn() returns a boolean and gates a block of code. getFeatureValue() returns a value your application then uses—along with a fallback for the moment before the SDK has loaded.

Typically, developers see this type of configuration in Firebase Remote Config for the first time, but it’s possible on other platforms like GrowthBook as well.

What happens when feature flags and remote config overlap in your infrastructure?

Many feature flagging platforms (including GrowthBook) now support string, number, and JSON values. So if you put a specific feature, for example, your pricing tiers in a JSON flag, you’ve built a remote configuration service inside a flagging platform.

Firebase had covered similar ground—but in reverse. So, it added progressive rollouts and per-user targeting to an existing config service.

You can no longer separate a platform based on whether it’s best for feature flagging or remote configuration. In fact, several studies treat feature toggles and remote configuration as belonging to the same family of techniques. If the academic community can’t always clearly differentiate between the two, engineering teams shouldn’t expect to either.

That said, here’s what happens when they overlap within the same platform:

1. Intent and lifecycle

The intent of the capability changes how long it stays in your codebase. Remote configuration is usually permanent by design, while toggles are (mostly) temporary—unless it’s a kill switch.

For example, a rate limit is tuned over years and stays useful the whole time. But a rollout feature flag’s utility ends the minute you reach 100% rollout. If you serve the same value from a flag for more than 12 months and it stops working as a release control, then that’s just an undocumented config.

In either case, if flags or config remain in your codebase with no use whatsoever, you’re dealing with a stale flag. And once they pile up, you’re leaving more room for unnecessary incidents.

Tip: If you’re using a feature flagging platform like GrowthBook for both purposes, take advantage of the Stale Detection feature to clean up unnecessary config or flags. It’ll alert you if the flag has no active environments or if it sends all its traffic to a single variation. That’s your cue to clean it up.

2. Targeting

If you’re unable to figure out whether you have a feature flag or remote config in place, just open the flag and review its rules.

If the value is making your code branch an if/else where only one path runs, you have a feature flag, whether it's temporary or permanent. For example, it could be a checkout button rollout which is temporary or even a kill switch that's more permanent.

But if the code consumes the value directly, you have a remote configuration. For example, a rate limit for your service or a layout JSON your UI renders. Here, the code runs on the same path but only the parameter or payload changes. There's no dead branch to clean up because you didn’t create any. It’s a permanent value.

3. Auditability and ownership

Whether you’re working with feature flags or configs, you need versioning.

When someone on your team drops your checkout service’s rate limit from 100 to 50 requests per second and latency spikes, the first question is who made that change and what the previous value was. An audit log solves that problem because an unversioned config can create production failures.

But feature flags need lifecycle management in addition to versioning. When you’re creating them, you need to assign an owner and expiry date. If you don’t do this, you’ll increase your software’s technical debt, which only increases the surface area of failure over that period.

This is all to say that both of these techniques need governance, but the measures look different.

When does using feature flags as remote config become a problem?

You can run flags and config through the same platform. The problem starts when you run them through the same mental model. Here’s why:

1. Stale flags that become undocumented configuration

As you keep creating more flags or configurations, your codebase becomes more cluttered over time. In fact, a 2026 study found that feature toggles in Kubernetes stay in the codebase for a median of 734 days, while they stay for 185 days in GitLab. Stale flags are the norm, so you need to prepare for that.

Within GrowthBook, you can identify dead code snippets using the Stale Detection feature. If you’re using the MCP, the platform also offers agent skills to find flags and clean them up without leaving your AI agent.

2. Too many active flags create points of failure

Every flag you create results in 2ⁿ possible code paths. For example, 10 active flags create 1,024 possible code paths while 20 create over a million. Only a few of those combinations are valid states for your product, and you can’t test the rest. Even though only 7% of toggles interact directly with each other, toggle interactions grow by an average of 22% over time, so complexity increases with every flag you leave in place.

3. Storing secrets and API keys in flag payloads

Since feature flag payloads are cached client-side, anything inside them can appear in browser dev tools. Typically, API keys and credentials should be stored in a secrets manager or in environment variables to avoid this issue.

If you’re using a feature flagging or remote configuration platform, make sure it offers the necessary protection. For instance, within GrowthBook, the server-side SDKs keep the full payload in a secure environment where it never reaches the browser.

Remote evaluation takes it a step further by evaluating them server-side and returning only the resolved values to the client—so rules and unused variation stay hidden.

4. Config changes that ship without release controls

In 2024, CrowdStrike shipped a content configuration update to every Falcon sensor, and the change didn’t contain any new code. It took down millions of Windows systems globally, resulting in a $5.4 billion loss for the company.

Even Cloudflare’s November 2025 outage happened because an internally generated config file doubled in size past a hardcoded ceiling. The company responded by hardening config generation and enabling more kill switches.

In both cases, the company shipped a config change globally and eventually concluded that the config needs the same controls that feature flags already give you. That’s because config carries the same risks as standard deployments, so you should treat it the same way.

How to choose between feature flags and remote config for development

Here are a few questions you can run through before choosing the right technique:

If you get the most yeses in the feature flag column, create a flag and use it for your rollout. If most of the yeses sit in the config column, use your flagging platform or a dedicated remote config system to create the config.

For the latter, you can use a string or JSON flag with a clear permanent-value intent and give it a rule that applies to every user.

How GrowthBook supports feature flagging and remote configuration

The Google Cloud outage started with a config value and eventually led to the realization that feature flagging would’ve solved this problem. That’s one of the reasons GrowthBook was built as a single feature primitive with four value types rather than two products bolted together.

A few ways GrowthBook does both jobs well:

• Feature flags and remote config: Every feature has a key, a type, a default value, and rules per environment. Boolean features run through isOn() for releases, while string, number, and JSON features run through getFeatureValue() for configuration.
• JSON schema validation: It enforces structure on non-boolean flag values—you define the allowed shape, and the platform rejects invalid entries before they reach production. The raw JSON editor becomes a generated form with typed fields and inline validation errors.
• Rules: You can use targeting rules across all four value types and serve different configurations per segment when you actually need them.
• Audit logs: You can review draft revisions and audit logs to answer who changed a value and when, with approval flows for changes that need a reviewer.
• Stale detection: This feature surfaces flags that stopped making decisions, and Code References show you where the dead code is present. You can decide whether to keep or remove it.
• Configs: The platform also ships dedicated Configs, which are typed and validated JSON objects with schema enforcement and per-environment overrides.
• Constants: It’s present alongside Configs and is a reusable value you define once and reference across multiple flags. These give long-lived configuration its own first-class object inside the platform rather than treating it as a side effect of a non-boolean flag.

If you want to see how GrowthBook handles feature flags and remote config in one place, start for free or book a demo with us today.

‍

Every experimentation team has had this experience: An A/B test showed a statistically significant lift, but after the change was rolled out, the primary metric didn't budge. It turned out the intervention didn't have an effect and the lift was caused by a flaw in the experimental design that introduced bias. By the time you're analyzing an experiment's results, many of the decisions that determine whether those results are reliable have already been made. 

We've already looked at several components of trustworthy experiments in depth: reducing false positive rates for more reliable wins, reducing variance to run conclusive experiments more often, and diagnosing sample ratio mismatch when you detect sample bias.

This article is a pre-launch framework for the decisions you need to make and the checks you should run before your experiment starts collecting data to ensure your results are trustworthy.

The 4 validity types that matter for experiment design

Validity describes the trustworthiness of an experiment's conclusion, whether the effect it reports is real or just a flaw in how the test was built. The standard experimental-design framework, from Shadish, Cook, and Campbell, breaks it into 4 types.

1. Internal validity

Randomization is supposed to guarantee internal validity, meaning the change you tested, rather than some other factor, caused the observed difference between the test and control groups. But two randomized groups stop being comparable when the intervention itself causes a change in their composition.

For example, if a new checkout flow loads half a second slower, impatient users may abandon it before they're counted in the analysis, leaving the test group without the users who would drag conversion down. Novelty, network effects, selective enrollment, and attrition can all threaten internal validity when an experiment isn't designed properly.

2. External validity

A 2-week test run over the winter holidays can show a statistically significant lift that disappears in January when normal user behavior returns. When that happens, you have an issue with external validity.

External validity requires that an experiment's measured effect apply to more than just the users and the time period you tested. An effect won't generalize when the conditions during the test differ from the conditions after rollout. Plan to run your experiment during a representative period on a representative sample to get reliable results.

3. Construct validity

An experiment can be internally and externally valid while still measuring the wrong thing. Construct validity means the metric you measure actually reflects the outcome you care about. Metrics are often just proxies for something harder to measure, and they can improve even when the outcome they represent does not.

For example, a recommendation carousel test might measure click-through as a stand-in for purchases, but a variation can lift clicks while purchases stay flat. Choosing the right metric up front prevents issues with construct validity.

4. Statistical conclusion validity

Statistical conclusion validity depends on your analysis supporting your conclusions. Checking the dashboard each morning and stopping when the result looks significant raises the chance that the winner you declare is a false positive, even when the design itself is sound. You can protect statistical conclusion validity by setting the sample size in advance and pre-specifying the analysis details before launch.

GrowthBook's statistical validity guide covers 2 more types as well as common threats to validity like p-hacking, peeking, and regression to the mean.

How to design for experiment validity before you collect any data

The decisions that determine an experiment's validity happen at the planning stage. Pre-registering an experiment means fixing all the details in advance, such as the hypothesis, the primary metric, the sample size, and stopping criteria. Committing to them before launch means you can't go looking for a significant result by peeking at the data or choosing what to measure after you've seen the numbers.

Start with a hypothesis that has a mechanism

Without a stated mechanism in your hypothesis, you have no way to pick the right metric or to judge whether a result makes sense. Write your hypothesis so the reason is explicit. A clear research hypothesis names the change, the metric, the expected direction, and the mechanism: we believe [change] will move [metric] [direction] because [mechanism]. The because is easy to leave out, and it's the part that makes a null result informative. When the metric doesn't move, you can use the mechanism to decide which link in the causal chain to check first. Stating it up front also constrains the metric you pick next, because it names the specific behavior the feature is supposed to change.

Match your randomization unit to your analysis unit

Randomization creates balanced groups, but this balance only applies to the unit you make assignments on. For example, if you randomized by user, the user groups would be balanced. If you then analyzed that experiment using sessions instead, you would no longer be comparing the same groups you balanced. Since one user can generate multiple sessions, a variation that alters user behavior (like visit frequency) can skew the session count in one arm for reasons independent of the metric you are testing.

A mismatch between identifiers can cause the same problem. Many metrics rely on anonymous IDs, because the tracking events fire before users sign in. If your SDK assigns on a user ID, one user can map to several anonymous IDs in your data, and those IDs may not split evenly across your groups. To prevent this, assign and analyze on the same identifier. You’ll need to choose the one that's available when users first see the change: the user ID on signed-in surfaces or the anonymous ID before sign-in.

Randomize whole groups when users affect each other

Some treatments change how users affect each other. Standard randomization assumes that one user's assignment doesn't change another user's outcome. Statisticians call this the stable unit treatment value assumption, or SUTVA.

For example, imagine a referral test where the treatment arm sees an offer for $10 if they refer a friend, and the control arm doesn't. Treatment users send invitations. When the invited friends arrive, some are assigned to the control group, and of course they sign up. The control group's conversion rate rises because of the treatment, so the gap between the arms shrinks and the measured effect understates what the feature does. The split is still 50/50, and every health check passes, but the estimate is wrong. 

Marketplaces where buyers and sellers draw on shared inventory have the same problem, as does any social feature where treated and control users interact. The measured effect is still biased, and you can't easily predict the direction.

There's no way to correct for interference between groups in the analysis afterward, so you have to design for it in advance. Cluster randomization assigns whole groups to the same variation, like every user in a city or a connected social graph, so the effect stays contained inside a cluster instead of leaking between groups. Geographic holdouts and time-based designs, where the treatment switches on and off across time periods, do the same job when the groups don't divide neatly. The best option depends on how your users connect to each other. Whichever design you choose, the decision has to happen before launch.

Re-randomize when you reuse a surface

Reusing the same randomization across back-to-back experiments lets users keep their old assignment, so anyone in the treatment arm last time stays in it for the new test. If the earlier treatment caused a lasting effect, that effect is now correlated with the new assignment and biases the result. For example, if a previous feature redesign confused users, their temporary frustration will artificially drag down engagement with the new tool. Re-randomize with a fresh seed for every experiment, so each user's new assignment is independent of any prior test. GrowthBook does this automatically when you start a new phase with re-randomization or create a fresh experiment.

Keep the arms identical except for the change you're testing

A new variant introduces new code, and new code can differ from the control in ways you didn't intend. It might degrade performance or have a bug the control doesn't, and suddenly you’re not just measuring the intervention, but also everything else that came along with it.

For example, a redesign might appear to fail simply because the new page loads more slowly, confounding the design change with added latency. Compare latency and error rate across the arms so that a performance issue doesn’t bias the treatment effect.

You also need to measure all experiment arms the same way. They have to fire the exposure event (which records that a user entered the experiment) at the same point and use identical metric definitions. If the exposure event fires after variation-specific code runs, the slower arm will lose users before they're counted, creating a sample ratio mismatch (SRM) and a biased result. 

An A/A test, where both arms serve the same experience, can help you check your measurement setup. With no real difference between the arms, any difference the analysis reports came from your assignment or tracking. But it can't catch problems that only appear when the real variant runs, so monitor latency, error rates, and SRM warnings during the live test.

Choose a single primary metric that measures the right thing

The more metrics you analyze, the more likely you are to find a false positive. You can control that risk with a multiple comparisons correction, but you lose some sensitivity when you do. Declare a single primary metric that directly tests your hypothesis. Ideally, it should connect to your Overall Evaluation Criterion (OEC), the organization-wide definition of success that experiments are judged against.

Three questions help with primary metric selection: 

1. Is it proximate to the feature, meaning it measures something the feature directly changes? 
2. Does it predict the long-term outcome you want, rather than a short-term proxy that can rise while the real goal stays flat? 
3. Is it a rate or a binary metric, rather than a total that a handful of high-spending users can disproportionately influence? 

With a proximate, predictive, outlier-resistant metric, you can detect a real effect with far less data than with a noisy one.

In addition to your primary metric, you can also declare guardrail metrics. These are the metrics that must not degrade. Revenue, support tickets, latency, and error rate are all common here. You don't use a guardrail metric to call a winner, even when it improves. Instead, its purpose is to detect when the variant makes something important worse that the primary metric wouldn't catch. When you declare guardrails before launch, you see the decline during the test instead of after rollout.

Set your decision criteria before launch, too. Your metric choices describe what to measure, not when a result is good enough to ship, so document which combination of outcomes means you ship, roll back, or iterate. For example, you might only ship if the primary metric improves with 80%+ confidence and your guardrail metrics don’t degrade. Committing to your criteria up front is part of pre-registration, and it helps make sure all your stakeholders are aligned on what success looks like. 

Calculate your sample size and set a minimum duration 

Peeking at results early and acting on what you see is one of the easiest ways to accidentally increase your false positive rate. Without a sample-size target set in advance, every early look triggers a decision about whether or not to stop. As you're designing your experiment, run a power analysis to determine your sample-size target.

Power analysis is built on 3 inputs: 

• Minimum detectable effect: the smallest increase in a metric's performance that would be meaningful for your business.
• Baseline value: your primary metric's current value for the same audience, from 2-4 weeks of pre-experiment data.
• Power target: the chance of detecting a real effect if there is one. 80% is standard for most decisions, but you may want to go higher if missing a real effect would be costly.

Split the two arms evenly unless you have a reason not to, because a 50/50 allocation gets the most power out of a given sample. Using a ratio like 90/10 to limit exposure is a reasonable safety choice on riskier changes, but it will cost you sensitivity and lengthen the test.

If your test enrolls users continuously, like most product and website tests, it also needs a minimum duration, regardless of when you reach your sample-size target. A 2-week minimum is common. User behavior varies across the week, so running experiments in whole-week increments gives every day equal representation. Duration also needs to outlast the burst of first-time curiosity a new feature draws, and features that change habitual behavior, like navigation or a core workflow, need longer still.

Some metrics also take time to mature. If your primary metric is completing a 7-day trial, users who enroll in the test's final week haven't had time to complete it. Set a conversion window on the metric, which counts each user's events only within a fixed period after their own exposure, in this case 7 days, so a user who enrolled on day one and a user who enrolled in the final week are measured on the same terms. You’ll need to wait for the last enrollees' windows to close before analyzing. The window keeps the measurement period equal rather than shortening the wait.

Together, the sample size, the minimum duration, and metric maturity determine your analysis date. Plan to analyze the results once, when all 3 are met. If you know you'll want to act on results as they come in, enable sequential testing at design time, which adjusts the statistics so that repeated looks don't inflate the Type I error rate. 

Plan how concurrent experiments share users

When several experiments run on the same audience at the same time, they will inevitably share users. That's usually fine because experiments assign users independently, and shared users spread evenly across each experiment's arms. When Microsoft analyzed every pair of concurrent tests running across 4 of its products, 3 products showed no statistically significant interactions at all, and the fourth showed them in 1 in 50,000 test-pair metrics.

But when you expect two treatments to interact, for example if both are making changes to the same onboarding flow, you can design the experiments to be mutually exclusive so users only get enrolled in one. GrowthBook supports this with namespaces. Since mutual exclusion spreads your sample further, reserve it for experiments that actually interact.

Run pre-experiment checks before launch

A final round of checks catches potential validity issues while they are still relatively simple to correct. Before the experiment goes live, confirm the following:

• A/A test passed: Run an A/A test on any new experiment surface or after making tracking changes. Since both arms are identical, any statistically significant result points to a problem in your instrumentation or assignment.
• Pre-experiment balance check: If you’re assigning a fixed cohort before launch, confirm the groups don’t already differ on your primary metric (like revenue per user) before the treatment starts. An imbalance that predates the experiment means your results will be biased from the start. This is only possible for batch-assigned experiments on existing users who have historical data. 
• Enrollment overlap reviewed: Check which live experiments this one will overlap with. Having users in multiple experiments is fine as long as they change independent parts of the product. If two do interact, make them mutually exclusive so users only get enrolled in one experiment. You can do this in GrowthBook using a shared namespace.
• Runtime confirmed: Given your user volume and minimum detectable effect, confirm the experiment can reach its required sample size in a reasonable amount of time. A test projected to take 9 months at your current volume needs a different design.

Common threats to internal and external validity in experimentation

The design decisions above reduce the risk of a validity failure, but they can't eliminate it entirely. Once the experiment is live, watch for these common threats to validity, especially when the results don't seem to make sense.

Novelty and primacy effects

Users behave differently when they encounter something new, so the first week of data often shows a bigger effect than later weeks will. Novelty pushes the early effect up, and its opposite, the primacy effect, pushes it down. This is because experienced users need time to adapt before a change starts helping them. When a test is too short, it measures the reaction to a change rather than the change itself. You can spot novelty after rollout, when a winning result decays over the following weeks.

Network effects and SUTVA violations

Network effects are hard to spot because they pass standard health checks. You'll see a balanced split between groups, but if users are influencing one another, the results are skewed anyway. Beware of SUTVA violations anytime your experiment involves referrals, social interactions, or shared inventory, especially when the results are surprising. Since you can't fix this bias after the fact, it's best to catch it early and rerun the test using a group-based design, such as cluster randomization.

Interference from concurrent experiments

Concurrent experiments usually only contaminate each other when their treatments interact, like two changes to the same algorithm. As covered earlier, independent assignment keeps ordinary overlap from biasing either result. That said, you should suspect an interaction when two teams have shipped changes to the same surface at the same time, and either result looks implausible. To verify your results, make the experiments mutually exclusive with a namespace and rerun them, so that each test measures its own change.

The pre-launch checklist for trustworthy experiments

This checklist is your pre-registration record. Answer each item with a yes or no before the experiment goes live. Each item corresponds to an A/B testing best practice covered in the sections above.

Hypothesis written with its mechanism. We believe [change] will move [metric] [direction] because [mechanism].

• Decision-making criteria written down before launch, including which combination of goal and guardrail metrics means you ship, roll back, or iterate. 

• Guardrail metrics declared for anything that must not degrade, like revenue, support tickets, latency, or error rate.

• Randomization unit matched to the analysis unit, on both grain and identifier.

• Randomization re-seeded if reusing a surface or the same users from a prior experiment.

• Exposure event confirmed to fire at the same point in both arms, with latency and errors compared across arms.

• Network effects assessed: does this feature change how users affect each other? 

• User-sharing plan set: overlap allowed for independent experiments that change different parts of the product, or mutually exclusive namespaces for tests that interact.

• Sample size calculated from the minimum detectable effect, baseline, and power target, with an even split unless there's a reason otherwise.

• Minimum duration set for continuously enrolling tests, on a window without holidays or major promotions.

• A/A test passed on new surfaces and after tracking changes.

• Pre-experiment balance check run, with no significant imbalance between groups.

• Concurrent-experiment enrollment overlap reviewed.

• Stopping criteria written down. Analyze once at the committed sample size and date, or use sequential testing if you need to look early.

How GrowthBook supports pre-launch validity

GrowthBook's pre-launch checklist lets you encode these checks as required steps before an experiment moves to production. Use the power calculator to project power over time before launch and commit to a realistic runtime.

GrowthBook automatically flags SRM and pre-experiment imbalance on the Results tab once the test is live, the opt-in Health tab adds deeper checks and visualizations, and sequential testing keeps results valid if you need to monitor early.

GrowthBook’s Decision Framework lets teams define decision criteria directly in the experiment, so the readout includes a shipping recommendation. The stats engine is open source, so you can see exactly how each check is computed.

Trustworthy results begin during the design phase, long before data collection starts. GrowthBook's experimentation platform puts these safeguards in place by default. You can try it for free or book a demo with the team.

‍

Running 100 tests a year is the easy part. The hard part is knowing what a win actually means when the "customer" on the other side of your website is 1,100 different businesses.

That is the problem Arie Polycarpou works on every day as Senior Manager of Test and Learn at Aspen Dental. On this episode of The Experimentation Edge, he walked host Ashley Stirrup through twelve years of building experimentation programs at Kohl's, Marriott, Total Wine, and now the largest company in The Aspen Group's healthcare portfolio. Along the way he shared the two numbers he actually manages to: a win rate he deliberately expects to fall, and stacked win math with a haircut built in.

Listen to the episode here.

Twelve years from volunteer to program builder

Arie's route into experimentation started with a hand raised at Kohl's Department Stores. He had landed in digital analytics out of a marketing analytics background, drawn to digital because, unlike most data work, you get to see a whole customer experience start to finish. A/B testing was a growing field at Kohl's with exactly one person working on it. That person needed help. Arie volunteered.

Twelve years later, the pattern of his career is remarkably consistent: join as an individual contributor, connect the data sources, install the rigor and the processes, then build the team. At Marriott he helped bring measurement discipline to A/B tests. At Total Wine, a company that grew tremendously during COVID, he took a program doing a couple of big product tests a quarter and turned it into a full-fledged, hundred-test-a-year operation. After graduate studies in digital transformation, he joined Aspen Dental in October 2025 to bring its agency-run testing in-house and accelerate both the volume and the depth of what the program can measure.

Now he leads a team of three: himself, a manager of analytics, and a coordinator of testing and analytics. The descriptive work matters, but the bread and butter is turning known business opportunities into tests, then working with product managers, the merchandising and production group, and a team of front-end developers to build them. The target: about 100 tests a year, with room to grow.

Healthcare retail: one website, 1,100 offices

Aspen Dental is not a typical B2C business. It is part of TAG, The Aspen Group, which owns several healthcare organizations, and Arie describes the model as healthcare retail. Aspen Dental has about 1,100 offices across the country, and none of them are corporate-owned. Private dentists own the practices, in a model Arie compares to Marriott: the brand provides resources, training, and education, but the practice owners run the businesses.

The website's job is focused: attract new patients, help them learn about the offering, and get them on the books. Appointment bookings per location is the North Star. As Arie puts it, the website's role is to "give the offices a chance to serve a user."

But that franchise structure changes what a win means. Every office has its own star rating, level of service, show rate, and patient return value. A test that lifts bookings overall can still be a bad trade if it routes more patients toward weaker offices or lower-value visit types. So Arie's team monitors what happens after the booking: the value of a patient, whether the mix of users is changing, and whether the mix of offices receiving those patients is changing. Promoting stronger offices, with higher show rates and higher return value, is simply better for the business than spreading demand indiscriminately.

For experimentation teams, the lesson generalizes: when your business is really a network of businesses, the unit of analysis has to follow the value, not just the conversion.

The win rate that should fall as you mature

Ashley asked Arie a question every experimentation leader gets from their executives: what's your win rate?

His answer runs against instinct. Arie strives for a true win rate of about 25%, and he treats the roughly 40% win rate that new programs often post as a symptom of youth, not excellence. Early programs feast on low-hanging fruit: the obvious fixes, the long-postponed ideas, the big value opportunities. As those get knocked out, the tests get more ambitious and the win rate should decline.

The logic cuts both ways. Too low a win rate means you're wasting effort testing the wrong things. Too high means you're not testing anything hard. And the baseline is stiffer than a coin flip suggests: your current experience already embodies years of descriptive data, UX best practices, and accumulated knowledge. Beating it significantly takes real work.

The industry's most mature programs bear this out. Ashley noted that Bing has been quoted around a 20% win rate, and Airbnb as low as 10%. The more mature the product, the harder it is to move the needle. A falling win rate, in other words, can be evidence that your program is finally testing things worth testing.

The haircut rule: two 5% wins don't make 10%

The second number Arie manages carefully is the one that gets reported upward: the stacked, annualized value of the program's wins.

His rule is to be optimistic about better and conservative about how much. A 5% lift measured during a test window rarely survives a year in market. Customers acclimate and the novelty gets baked in. New changes get layered on top of old ones. And lifts don't compound the way arithmetic implies: a 5% homepage win followed by another 5% win does not produce 10%. "I know the math should make it seem like it does work like that," Arie says, "but it doesn't."

So depreciation gets baked into estimates before they reach leadership, in the form of haircuts. The approach echoes what Ronny Kohavi described on a GrowthBook webinar: apply a 20% haircut when you stack wins. Arie goes further when the evidence is thinner, taking a bigger haircut on tests that ran shorter or won with weaker confidence.

The alternative, long-term holdouts, is statistically cleaner but operationally messy. Holdouts force you to maintain two experiences, effectively two sets of code, and the longer they run, the harder they get. For most programs, disciplined conservative estimation is the more sustainable path, and it pays a cultural dividend: conservative numbers that hold up build trust with stakeholders in a way inflated dashboards never do.

Where the program goes next

Aspen Dental's program is still young; Arie joined less than a year ago and the muscle is still being built. The road map runs toward specificity: localization, where each of 1,100 offices can be served differently, and service lines, where the experience that books a denture repair is very different from the one that serves a dental emergency.

On AI, Arie is interested but deliberate. Testing tools are adding AI to analyze tests faster, look up results faster, and even help build tests. He sees the opportunity, and Aspen is looking into it, but with guardrails: keep a clean code base, keep top engineers in control of the site, and use AI as additional analytics hands that free the team to think more strategically, not as a replacement for rigor.

Why this matters for experimentation teams

Three disciplines run through this conversation. Follow the value past the conversion event, even when that means office-by-office analysis. Read your win rate as a diagnostic of maturity, not a scoreboard. And report stacked wins with a haircut, so the number you promise is one that holds.

Ready to take control of your experimentation program? Start for free or get a demo at growthbook.io.

Listen to the full episode of The Experimentation Edge

Engineers never deliberately decided to couple deployment and release. It happened by default, because there was no way to draw a hard line between merging code and exposing it to users.

And because of this process, your release schedule depends on whether you’re ready to deploy and not on whether your product roadmap requires it.

Well, that’s not the case anymore.

Feature flags allow you to decouple deployment from release by controlling what gets exposed to users, even after deployment. In fact, a 2026 study found that coupled deployments produce unsafe activations 60% of the time. But decoupled ones produced none for rollouts with gated exposure.

In this guide, we’ll explain how feature flags decouple deployment from release and why you should consider adopting them.

Deployment vs. release: What’s the difference?

Deployment is a technical event in which you push code to production, where it runs on your servers, but real users don’t see it yet. On the flip side, release is a business decision. Here, you make the feature visible to users, and it depends on your organization’s roadmap rather than when the code was shipped.

Without feature flags, deployment and release happen together. Even a half-finished feature gets released to users if you deploy it. With feature flags, you separate these events.

Why you should consider decoupling deployment from release
‍

When you separate deployment from release, it changes the risk profile of every change your team ships. Here’s how:

1. It makes releases low-stakes

A bad release doesn’t have to be a crisis anymore. In the traditional software development process, if something goes wrong, the fix also runs through the same pipeline that caused the problem. And your users continue to experience issues until you rectify the problem.

But when you decouple deployment from release using feature flags, the risk becomes containable. The code stays deployed and only the exposure changes. If someone turns the flag off, users can see the older version of the code in seconds while your team looks into the problem.

For instance, in a platform like GrowthBook, because flag evaluation happens locally from a cached payload, a simple toggle switch sends the updates to every SDK in real time. As a result, the changes show up on the next check, which is almost immediately.

In short: the act of reverting a change becomes boring, and that’s the goal.

It lets you ship faster

The fear of breaking production also pushes teams toward batching releases. When every release is an all-or-nothing bet, you hedge by bundling changes into bigger, less frequent releases. But that also increases your risk and slows down development.

Decoupling releases solves that issue. The 2025 DORA survey shows what teams do once it’s gone:

• 22.7% of organizations deploy to production at least once per day
• 44.6% deploy at least once per week. This is the type of cadence a decoupled release offers. You can deploy continuously, but release only when you’re ready.

In fact, trunk-based development becomes the norm, with every merge going straight into main behind a feature flag. So you don’t have to deal with long feature branches or the bugs they introduce.

It gives product teams control over the release moment

Once you separate deployment and release, you can also change the ownership of the release process to the team that should hold it. While engineers deploy code, product managers can decide when users actually see the new change or feature, without relying on the development team.

Here’s how it differs:

• Engineering answers, “Is this code safe to run in production?”
• Product answers, “Should users see this now?”

Let’s say a product manager for a telehealth platform wants to release a new appointment-booking flow in one region before rolling it out nationwide. If engineers already deployed the feature two weeks back, all the PM has to do is set the targeting rules using a feature flag (such as geography or plan type) and launch it.

Tip: Choose a platform that gives non-technical teams full control too. For instance, GrowthBook’s dashboard is built so that non-engineers can manage flag states. If they need attribute-based targeting or want to build Saved Groups, they can do that without writing code.

How feature flags decouple deployment from release

Feature flags create the missing layer between deploying code and releasing features. Here’s how you can set them up as is or using GrowthBook:

Step 1: Wrap new code in a flag

Attach an if/else wrapper to your new code so that when you deploy the feature, it defaults to an “off” state. In this case, the code runs in production while it still allocates memory and initializes its dependencies. However, users can’t see or experience the feature yet.

if (feature.isOn("new-booking-flow")) {
  renderNewBooking();
} else {
  renderCurrentBooking();
}

This process is called a “dark launch.” A 2026 study found that these kinds of shadow deployments identified 40% of potential regressions that the sandboxing stage didn’t catch. You can see what goes wrong before you release the feature.

Note: Within GrowthBook’s feature flagging platform, every feature wrap dark launches by default. Dark launches still rely on the flag’s state so the performance matters here irrespective of whether your users see it. GrowthBook's SDK evaluates flags locally from a cached payload rather than making a network call per check, so there’s no latency between each call. Also, if its servers become unreachable, the SDK continues working with the last cached state. As a result, your dark launch doesn’t break because the flag service went down. 

Step 2: Choose your release strategy

Once your code sits dark in production, the flag becomes the release mechanism. Now, whoever owns the flag owns the release moment.

You can use any of the following release strategies to launch the new change, depending on the risk level and level of validation you’ve done:

If you’re launching a minor change, you might want to do an instant release. In GrowthBook, it uses a simple Forced Value rule where you define the targeting conditions and release the change. If you need more control over who sees the feature, you can add targeting conditions based on any user attribute using the same Forced Value rule. You can also add prerequisite features to ensure the flag only activates when another feature or rule is already in effect. If you target the same segment frequently, use Saved Groups to create reusable segments.

Here’s a list of attributes you can use in GrowthBook:

Even though targeting lets you choose a specific audience, it can’t tell you whether it actually works for them. The safest way to test that is to use a Percentage Rollout along with an experiment. Here, you define the percentage of users to run the test with and then monitor the metrics you need to see whether the feature improves or degrades them. You can do more granular targeting using Saved Groups or Prerequisite Features.

But if you roll out the feature while testing at every step, Ramp Schedules and Safe Rollouts are the best ways to do that. It uses a combination of percentage rollouts and observability to make that happen. A Ramp Schedule is a release plan you attach to a targeting rule, and at each step, all you have to do is define the percentages, timeframe, and approval workflows. GrowthBook automates the step-up on a timeline you define, so you don’t have to manually go back and ramp it up. If you attach guardrail metrics like error rates or latency spikes to it, it becomes a Safe Rollout (Monitored Ramp-up) where the rollout can be paused or reverted automatically if the metrics degrade.

For instance, if you’re monitoring the new booking flow feature, review metrics such as conversion rate and error rate before the rollout starts.

Step 3: Monitor the rollout and remove the flag

Once the feature flag and accompanying code go live, it’s time to monitor the rollout. This means the guardrail metrics you defined earlier have to be monitored. Here, you’ll do a percentage rollout and monitor performance at each stage.

Let’s say you’re monitoring error rates for a new booking flow. If you notice a spike at 10% rollout, you can revert to the old version immediately. If nothing goes wrong, move on to the next step of the rollout. Within GrowthBook, the platform monitors these metrics at every stage and pauses or reverses the rollout if they cross the defined thresholds.

After you hit 100% with stable performance, archive the flag or remove it from your codebase.

How does governance work when deployments and releases are decoupled?

When you start decoupling your delivery process, you’ll contend with two questions:

1. If any engineer can toggle a flag in production, what stops an accidental release?
2. And if PMs own the release, what stops someone from exposing a feature that isn’t finished?

Both of these questions have one answer: approval workflows.

On platforms like GrowthBook, you get this built in, so every change is treated as a draft that eventually goes out for review. It only gets merged into production when the right person approves it. And the published versions “lock” because every time you make a change, the platform takes an immutable snapshot of the change. If any changes cause an issue, you can always roll back to the previous version, and that gets recorded in the audit logs too.

As a result, the staging process runs without interference, and you involve stakeholders only when the flag needs to go live.

That said, the “who” changes as more organizations use AI to deploy and release features. You can now use AI coding agents to draft flag configurations and release plans programmatically through GrowthBook’s MCP server. Even though the agent handles the setup, a human stakeholder still approves them before anything goes live.

Note: GrowthBook blocks authors from approving their own requests, so you use the four-eyes principle by default. If someone edits a draft after approval, the review process is reset (unless you’re an admin).

What does the feature flag lifecycle look like?

The decoupling process doesn’t end when you hit 100% rollout. The full lifecycle runs through four stages:

• Deploy
• Release
• Expand
• Clean up

We’ve seen that most engineering teams forget the fourth stage, which is the clean up. In fact, Uber built their own automated refactoring tool, Piranha, to clean up over 2,000 stale feature flags. It’s expected when you haven’t baked in governance during the deployment process. Your team keeps adding flags without removing them until something goes wrong.

In a nutshell, the control layer you added for safer releases becomes technical debt that increases risk.

Your “definition of done” should include clean-up, too. Once the feature serves 100% of users with stable metrics, it shouldn’t exist in your codebase. Either archive it or remove it altogether.

Platforms like GrowthBook automatically surface stale flags. If a flag is untouched for more than 2 weeks and is disabled everywhere or routes all traffic to a single variant, it’ll show up in the dashboard. You can use Code References to pinpoint exactly where it is in your codebase. 

And then use the GrowthBook MCP server and ‘flag-cleanup’ GitHub skill to clean it up without ever leaving your AI agent’s interface. Once you’re done cleaning up the stale flags, document it and move on to the next release.

5 mistakes engineering teams make when decoupling deployment from release

Even though decoupling solves the issue of slow development, you still need strong governance in place to implement it effectively. Here are a few mistakes every engineering team should avoid while using feature flags for decoupling:

1. Leaving flags in the code indefinitely

It’s easy to create a flag, but it’s also just as easy to forget about it after you’re done releasing a new update or feature. The problem is that it’ll continue to quietly serve a single variant of the change to every user via a hidden conditional branch. That’s why you should continue to clean up stale flags once you’re done using them or at least every 90 days. Assign an owner and a removal date while you’re creating the flag to avoid this issue.

2. Skipping approval workflows in production

Without a proper approval workflow in place, there’s a good chance unauthorized users can create a flag that accidentally releases something. Gate production behind sign-off and let staging happen freely so that the risk lives in an environment where real users don’t experience the bug or a premature feature.

3. Using flags as permanent configuration

Feature flagging systems are usually built for even minor configuration changes, but the problem is that engineering teams assume that when they create the flag, they’ll eventually settle on one path and remove the other. In reality, that’s rarely the case. When you forget to clean up your flags or even switch off (if needed), that’s just not the case. That’s why we’ve seen incidents like the Knight Capital accidental trading glitch happen. If you are using feature flags as a permanent toggle, like with kill switches or toggles for location-based capabilities and plan-based tiers, mark it as permanent and document it. If it’s not permanent, it should have an expiry date.

4. Releasing without a rollback criterion

Before you flip the flag, define what “something went wrong” looks like for this specific feature. That means identifying the issues your users could experience and tying those issues to guardrail metrics that would degrade if the feature has problems. If you don’t, your users might experience severe app performance issues, while you won’t even know that something broke in production. Consider defining your guardrail metrics and their thresholds while creating flags, and use a Ramp Schedule (Safe Rollout) to monitor performance as you slowly ramp up the rollout.

5. Treating decoupling as an engineering-only win

Decoupling deployment from release is a cross-functional change, and that’s also why it’s a sociotechnical problem. In addition to engineers, even quality assurance (QA) teams and product managers are involved in the process, so each team needs to have the right level of access and visibility into the release. The goal is to give each team self-serve access to the data and controls they need. For example, product managers should be able to target segments and schedule releases, while QA teams should be able to monitor the metrics and roll back if needed. Ideally, you’re working with a platform that gives even non-technical users the right access to make changes without writing code.

It’s time to deploy continuously, but release features intentionally

Nobody decided to couple deployment and release. It was the default for the longest time, but that doesn’t mean it should continue to be.

You have tools like feature flags that help you decouple deployment from release successfully without breaking things. In fact, you can ship fast without breaking things. Deployment becomes a non-event, and release becomes a deliberate choice made by whoever’s closest to the customer.

That’s why the engineering teams that adopt feature flags improve deployment frequency while reducing change failure rates.

Curious how feature flags could improve your development process? Try GrowthBook for free today.

Running an experiment is the easy part. The hard part is running one at a 150-year-old financial services company, where customer journeys stretch across months, risk tolerance sits near zero, and half the organization believes "we tried that years ago" settles the question.

That is the environment Erika Dunn works in every day. As assistant director of data science at Principal Financial Group, she built the company's experimentation center of excellence and, more recently, something most enterprise teams are still only reading about: synthetic digital audiences, built entirely in house, that just picked a winner in a live A/B test. On this episode of The Experimentation Edge, she walked host Ashley Stirrup through how it all works, and what experimentation leaders at any company can borrow from it.

Listen to the episode here.

From observational studies to clean 50/50 splits

Erika's path into experimentation started in quantitative psychology, which gave her the statistical grounding long before she had the corporate title. Her first job studied childhood obesity: observational work with no control groups, just trying to understand what happens when an intervention lands in the real world. At H&R Block, Adobe Analytics brought regular experimentation into the marketing space, and she realized the work she loved in her graduate program had a corporate career attached to it. Then came Amazon, where, as she put it, they can turn anything into an experiment. Eventually she landed at Principal, where she got to dig in and build her own center of excellence around a simple pair of questions: what is experimentation, and how do we share what it teaches us?

That range matters. Someone who has run studies with no control groups and clean web tests with 50,000 users in each arm knows exactly what rigor buys you, and what you lose without it.

When testing gets watered down

The biggest challenge Erika names is not tooling or statistics. It is vocabulary. In more than one company, she has watched testing get watered down as a concept, where "let's try out something" passes for an experiment even though nothing is controlled and nothing is compared. She is careful not to dismiss it: trying things means a team recognizes their ideas need contact with reality. But the difference between trying and testing is a control group, and getting people to appreciate the advantage of rigor takes time, intentionality, and sometimes money. Her core argument is that with just a little pre-planning, you get so much more out of every test.

The second enemy is tribal knowledge. Every organization has it: the insight that lives in one silo and never travels, or the confident claim that "we did that years ago." Ashley recalled a guest from Twitch who spent years being told pricing had already been tested and was off limits, until a rigorous retest showed a huge impact on the business. Erika hears the same pattern inside companies that are 60, 100, or in Principal's case 150 years old. Things change. The segment that ignored an offer three years ago is not today's segment, and a test that lost on the whole population may be a clear winner with high-revenue customers.

Her structural answer is a center of excellence where teams share experimentation wins and losses in one place. Losses are the important part. A visible record of what did not work, and when, is what turns "we tried that" from a conversation ender into a data point with a date on it.

Turning frustration into a metric

Erika's team supports Principal's marketing organization, where the goal is getting the right message to the right people across email and web. The data science advantage, as she describes it, is simple: come in with additional data, or help teams see the data they already have in a new way.

Two examples from the episode stand out. The first is content analysis. Her team broke marketing copy down with readability statistics, including reading level, sentence counts, nouns, verbs, and possessives, and correlated those features with how people responded. That gives content writers something concrete to think about beyond instinct.

The second is the looping metric, developed with her longtime collaborator Josh Ellington, who she is quick to credit. Built in SQL on top of existing web behavior data, it measures how many times a user winds up back at the same main page within tight time frames. When that ratio climbs, it flags desperation behavior: searching, backtracking, revisiting the same section over and over. The team confirmed the signal against session replay tools, and it holds. In tax services, looping predicts bounce. In finance, it predicts something quieter and more expensive: customers who slowly disengage from tools they find frustrating, dragging down engagement and retention over time.

The same discipline shows up in how the team handles assumptions. When they set a nine-second window between an email click and a website visit, it is because Josh dug into the data and found that is how long the trip actually takes. Not five minutes, because a lot can happen in five minutes, and the behavior you attribute across that gap may have nothing to do with your email. Keeping behaviors linked closely in time is unglamorous work, and it is the difference between data and noise.

Customers that look like your customers

The centerpiece of the conversation is Principal's synthetic digital audience program, which Erika describes, for better or worse, as completely Erika built. Drawing on her probabilistic Bayesian background, the system uses real customer data to drive a synthetic data normalization process, producing bell-curve populations of profiles that represent Principal's actual segments without using any real customer data. Each audience is tuned to a specific experimental space: the profiles look like the employer audience in that particular marketing domain, not like a generic average person.

The workflow is deliberately conservative. Before profiles are set loose on new material, they are trained on existing similar content so they replicate known response patterns. Then, through a gen AI process, the team asks each profile its likelihood of opening, clicking, or engaging with a piece of content. Partners can hand over 20 subject lines or banners and get back a ranked list. Teams that want to go wider can generate 50 AI-created variations and let AI duke it out against AI, without burning weeks of human review time.

Erika is precise about what the system does not do. It will not tell you what results to expect. It creates a priority process, narrowing a long list of ideas to the short list worth spending real traffic on. That distinction is what makes the program credible inside a risk-averse company, and it addresses two problems at once. It compresses time in a business where financial journeys are long and feedback is slow, and it creates a safe playground for teams whose tone has always been neutral to finally test new voices, new phrasings, and new content without betting a live campaign on them. As Erika puts it, an experimentation mindset requires that people cannot get punished for making mistakes. They need guardrails, and they need room to learn.

The first win, and three more in August

The credibility test came recently: the first traditional A/B test using a synthetic audience selection ran against a control, and it won. An earlier partner had already seen lift, but without a traditional test design, and Erika's team held their applause until the clean result landed. That partner, working with a small population and a small engagement window, had previously paid outside consultants without moving the needle. The in-house synthetic process is now outperforming that spend.

Three more tests launch in August, after a year spent helping teams across Principal build their audiences. The roadmap points toward agents: Erika sees synthetic audiences moving into agentic workflows, and more broadly, she sees AI helping teams answer the question she hears everywhere now that everyone finally has the data they asked for. We are drowning in information, so where should we experiment next? Her bet is that the next edge in experimentation is efficiency: agents that read your roadmap, your data, and your ideas, and point you at the gains you cannot see from deep in the weeds.

What this means for your experimentation program

Erika's playbook at Principal compresses into four moves any team can apply:

1. Draw the line between trying and testing. A control group and a little pre-planning turn "let's try something" into an experiment worth learning from.
2. Turn behaviors you already log into metrics. A looping metric built from existing web data found pain points no heat map had surfaced, and checked assumptions as basic as how long an email click takes.
3. Use synthetic audiences to prioritize, not replace, live tests. Rank 20 ideas cheaply, then spend real traffic on the strongest, and confirm with a clean A/B lift before celebrating.
4. Share losses as loudly as wins. A center of excellence with a visible record of failed tests is the only durable cure for "we did that years ago." None of this required buying a platform her company was not ready for. It required statistical rigor, creative metric design, and the patience to build trust one clean win at a time.

Ready to take control of your experimentation program? Listen to the full episode of The Experimentation Edge

Running an experiment is the easy part. The hard part is running hundreds of them when every request routes through one central team, every customer interaction carries regulatory weight, and even a one percent gap in coverage is unacceptable.

That is the reality Vijay Lal manages every day. As Lead Product Manager for Experimentation at US Bank, he sits at the intersection of marketing partners who want to test everything, engineers who have to keep a banking platform secure, and customers who expect their login page to work every single time. On this episode of The Experimentation Edge, he walked through how a regulated bank scales experimentation without sacrificing rigor, and what product managers everywhere can take from it.

Listen to the episode here.

The bottleneck every experimentation program hits

Vijay's experimentation career started in 2016 at Comcast, after a redesign project for MassMutual convinced him that customer experience was where he wanted to build. At Comcast he supported sales and marketing teams running tests for prospective customers at enormous scale, on traffic volumes so large the platform itself had to be upgraded, with Adobe Analytics and Adobe Test & Target underneath.

Six and a half years later he moved to US Bank, into a similar role in a very different industry. Financial services is more regulated than telecom, and the stakes of a broken experience are higher. When money is involved, nobody gets to shrug off an error.

The problem he found is one nearly every experimentation program recognizes. Demand for experiments outgrew the central team's capacity to run them. Marketing partners wanted to test at a volume that Vijay's team could never cater to alone. Most organizations respond by hiring, queueing, or saying no.

Self serve, with guardrails

Vijay chose a fourth option: make the platform self serve. In his words, why not make it so that anyone who does not know anything about technology can start using the platform and run experiments for customers?

That decision sounds simple. Executing it is not, and Vijay was clear about where the real work lives:

Step 1: Simplify the complex. An experimentation platform is not a simple tool. A friendly UI does not make experiment execution easy for people who are not proficient with the technology. The platform team's job is to compress that complexity into something a marketer can safely operate.

Step 2: Train continuously. Technology evolves every month. New capabilities, platform upgrades, tool changes. Self serve only works if the people using the platform stay current, so training is not a launch activity, it is a standing commitment.

Step 3: Attach responsibility to capability. Handing someone the power to put changes into a customer facing production environment is not a small thing. Every self serve user needs to understand the implications of what they ship, and guardrail metrics need to catch what they miss.

That last point is the one experimentation leaders should sit with. Democratization is not just access. It is access plus guardrails plus the knowledge of what the implications will be. Get all three right and a small central team can support an enormous testing program.

Every customer accounted for

The best story of the episode is about a login widget, and it shows what experimentation discipline looks like when the stakes are real.

US Bank's login widget historically loaded after the page finished loading. Vijay's team wanted it embedded, loading with the page, for a faster and cleaner experience. The complication: the login widget is one of the most secured components a bank has, because customers enter their user ID and password into it. Rebuilding how it loads meant working through security constraints, multiple technology partners, and an architecture where the component is delivered separately as an experience fragment.

Then testing surfaced a harder problem. For customers on slower internet connections, the embedded widget might not load properly. Most teams would look at the percentage affected and call it acceptable. Vijay's team did the opposite. In his words, every customer matters. Even if five percent of customers cannot see an experience, that is a big deal, and slippage of one or two percent is not acceptable.

The solution was elegantly simple: a fallback. Any customer whose page did not load within two seconds got a front end that looked the same, with the login widget displayed slightly differently, so it always worked. Every customer was accounted for in either control or challenger. No one fell through the cracks, the experiment stayed clean, and the security bar never moved.

There is a lesson here that goes beyond banking. Your users are on fast connections and slow ones, new devices and old ones. An experiment that only works for the top deciles of performance is not a complete experiment. Designing the fallback is part of designing the test.

Hypothesis first, metrics second

The third theme of the conversation is about how experiments get measured, and Vijay drew a sharp line between two approaches.

The traditional approach: a leader wants to run an experiment or ship a product, and a KPI gets attached to justify it. The modern approach: every experiment starts with a hypothesis, and the hypothesis drives the metrics. Not one metric in a silo, but a primary KPI for the immediate behavior, such as engagement on first visit, paired with secondary KPIs for what happens later, such as whether the customer is still engaged when they return.

Why does the order matter? Because metrics without a hypothesis can tell you what happened but never why. Vijay's example: data analysts can see customers rage clicking on a page and revenue falling. What the dashboard cannot say is that the button is disabled and the customer expects it to work. The hypothesis is what connects the number to the behavior behind it.

This also connects to a point Ashley raised about metric distance. The further a metric sits from the feature, the more noise other factors introduce. A North Star metric like revenue matters, but the experiment needs a goal metric close to the feature's immediate behavior, plus guardrail metrics to catch degraded experiences across segments, devices, and platforms that a topline average would hide.

What this means for your experimentation program

Vijay's approach at US Bank compresses into four moves any team can apply:

1. Make the platform self serve so your central team stops being the bottleneck for testing velocity.
2. Pair capability with guardrail metrics so non technical users can ship safely to production.
3. Design for every customer, including the slow connections and edge cases, with fallbacks built into the experiment itself.
4. Start from the hypothesis, then derive a primary KPI and secondary KPIs, rather than letting a leader's preferred number define the test. None of this requires a bank's budget. It requires treating experimentation as a product with users of its own, and treating rigor as non negotiable even when only one percent of customers is on the line. Especially then.

Ready to take control of your experimentation program? Listen to the full episode of The Experimentation Edge

GrowthBook is the industry's only open-source, warehouse-native experimentation platform, giving you predictable pricing, full data ownership, and results you can verify. Here's why Statsig customers are switching to GrowthBook.

Statsig has changed hands twice in eight months. OpenAI acquired the company in September 2025, and founder Vijaye Raji became OpenAI's CTO of Applications. In May 2026, Amplitude took over the Statsig brand and customer base, while the engineering team that built Statsig stayed at OpenAI.

Statsig isn't the same company today that it was a year ago. We're hearing questions about who owns the roadmap, whether experimentation will stay a high priority within a broader analytics and self-improving product company, and who'll handle support with so many product experts gone. Amplitude has published a phased plan to converge the two products, which answers some questions and opens more. Whether to sign on or renew with Statsig is an open question for teams now.

GrowthBook is where many of them land. This post explains what GrowthBook offers, what the migration looks like in practice, and how to decide whether GrowthBook compared to Statsig makes sense for your team.

What is the GrowthBook open-source platform?

GrowthBook is an open-source feature flag, experimentation, and product analytics platform. It is the original warehouse-native platform, trusted by more than 3,000 companies, including Dropbox, Khan Academy, Upstart, Sony, and Wikipedia. It handles over 1 trillion feature flag lookups per day.

The warehouse-native architecture is the defining design choice. Rather than copying your data into its own system, GrowthBook queries your data where it already lives — Snowflake, BigQuery, Databricks, Redshift, ClickHouse, Postgres, and more. Analysis runs in your warehouse with read-only access. Every SQL query is visible. Every result is reproducible.

That's the short version. The longer version explains why it matters when you're evaluating a replacement.

4 reasons product and engineering teams switch from Statsig to GrowthBook

1. Statsig's ownership, roadmap, and support have changed

Statsig is now a brand and customer base inside Amplitude, running on a perpetual non-exclusive license to tech whose original engineering team stayed at OpenAI. Amplitude has committed to maintaining it and published a plan for the product. That may work out great. But, it's a different plan than most Statsig customers made a bet on, so teams are asking about the roadmap, whether experimentation stays a priority in the broader company, and whether support relationships survive with the same quality.

GrowthBook is independent and remains focused on feature flagging and experimentation. We can't promise who will own us in ten years, but our core is MIT licensed, so every version you deploy stays yours to run and modify.

And because GrowthBook queries your warehouse with read-only credentials instead of ingesting your events, your event history and experiment data already sit in infrastructure you control. There is nothing to extract if the relationship ends. Raw PII never leaves your environment either, which for teams under GDPR, HIPAA, CCPA, COPPA or data residency requirements is operational, not philosophical. Self-hosting goes further, with zero external data egress and air-gapped deployments available for the most sensitive environments.

John Resig, Chief Software Architect at Khan Academy, described exactly this concern: the ability to retain data ownership was, in his words, "very, very important," because most platforms require passing user data to a third-party service.

2. Statsig is expensive to scale

Statsig prices on events and traffic. That structure makes sense when you're running a handful of experiments on modest traffic. At scale, it penalizes the behavior you want to encourage: more experiments, more feature flags, more coverage.

Teams using Statsig often end up managing their experimentation volume to manage their bill — sampling down traffic, avoiding flagging minor changes, skipping experiments on low-stakes features. That's the opposite of a healthy experimentation culture.

GrowthBook uses per-seat pricing. A team that runs 10 experiments a month pays the same as one running 100. Feature flag evaluations don't generate a cost event. The experimentation ROI calculator can model your specific usage to show expected savings.

3. Statsig has limited visibility into underlying results

Statsig's statistics engine is proprietary. You can see the outputs, but not the logic that produced them. When a result is surprising, your options for investigation are limited to the interfaces Statsig exposes.

GrowthBook's engine is fully open source on GitHub (8,000+ stars). Every calculation is inspectable. Every query is visible in your warehouse. If a result looks off, you can drill into the underlying SQL, check the raw data, and confirm or refute the calculation on your own terms.

Diego Accame, Director of Engineering at Upstart, put it this way: "Our strength is as an AI-powered lending marketplace, not an experimentation framework company. GrowthBook lets us focus our resources where they matter most — on growing our core business."

That confidence comes partly from owning the infrastructure and partly from being able to verify what the infrastructure is doing.

4. Statsig has limited statistical depth

GrowthBook supports Bayesian, frequentist, and sequential testing, with CUPED variance reduction and post-stratification. Statsig supports a similar range, but without post-stratification and the ability to inspect or reproduce the calculations.

For data science teams that care about methodology — particularly at companies where an experiment result drives a significant product or business decision — the ability to validate the math is a meaningful advantage.

How to migrate from Statsig to GrowthBook

GrowthBook ships a migration kit specifically for Statsig customers, including an AI-powered assistant that can transform your existing codebase.

Here's what migrates:

Projects, teams, and tags carry over cleanly, preserving your workspace organization so teams can keep working without rebuilding their context.

Feature gates from Statsig map to GrowthBook feature flags, which support multiple environments, targeting rules, gradual rollouts, and instant kill switches.

SDKs migrate automatically. The AI migration assistant points at your codebase and handles the transformation — feature gates, dynamic configs, and user attributes converted to GrowthBook equivalents. JavaScript, TypeScript, and React are supported today, with more coming. 

This is the step that usually takes weeks; the assistant reduces it to minutes.

Experiments transfer, including past experiments run on Statsig. You can generate custom reports from past Statsig experiments in GrowthBook, which preserves institutional knowledge.

Targeting rules transfer with full visibility into conditions and rollouts. GrowthBook includes debugging tools that simulate flag values for specific audiences, making it straightforward to verify that migration behavior matches pre-migration behavior.

Safe rollouts remain a first-class concept. GrowthBook supports gradual exposure with automatic monitoring of guardrail metrics, so regressions trigger alerts before they reach your full user base.

The SDKs themselves don't require replacement during migration. If you're moving from Statsig cloud to GrowthBook cloud, or from Statsig to self-hosted GrowthBook, your feature flag configuration and experiment setup carry over without requiring SDK changes or redeployment of your application code.

The full GrowthBook platform you're migrating to

Migration is the starting line, not the finish line. Here's what GrowthBook offers beyond Statsig feature parity.

Feature flagging that doesn't cost per evaluation

GrowthBook's feature flags run through zero-network-call SDKs. The SDK downloads a payload at startup and evaluates flags locally, so each flag evaluation adds sub-millisecond latency without generating a billable event. You can flag every feature in your product — including low-traffic, experimental, and internal-use features — without worrying about cost.

GrowthBook supports 24+ SDKs: JavaScript, React, React Native, Node.js, Python, Ruby, Go, PHP, Java, Kotlin, Swift, and more. The Chrome debugger lets you inspect flag state and experiment assignment in real time without touching application code.

Experimentation with SQL you write and own

GrowthBook's metric system is SQL-first. You write metrics using your warehouse's SQL dialect, join against any tables in your schema, and apply whatever business logic your team uses. A metric for revenue per activated user might join your experiment assignment table to your payments table to your activation events — all using the same logic your data team uses everywhere else.

Forgot to add a metric before an experiment started? Add it retroactively. The data is already in your warehouse. Just define the metric and run the analysis against the historical assignment data.

Metrics can be standardized in a library, enabling every team to measure success consistently. They can be scoped to specific experiments or applied globally as guardrails.

Deployment on your terms

GrowthBook Cloud runs on AWS with automatic updates, encrypted data at rest and in transit, 99.99% uptime SLA at Enterprise tier, and SOC 2 Type II, ISO 27001, GDPR, COPPA, and CCPA compliance.

GrowthBook Self-Hosted runs on your infrastructure, choose any major cloud provider or on-premises, deployed with Kubernetes or any container platform. Same codebase. Same features. Same development roadmap. The only difference is who manages the infrastructure.

Many teams start on GrowthBook Cloud for the fastest path to running experiments, then migrate to self-hosted when compliance requirements or internal policy require it. GrowthBook's SDK and configuration structure don't change in that migration, so the transition preserves everything you've built.

If you don't have a data warehouse yet, GrowthBook's Managed Warehouse gives you a fully functional environment immediately, with the option to migrate to your own warehouse at any time.

AI-ready experimentation

Three of the five leading AI infrastructure companies use GrowthBook to test and optimize their products. The platform handles the non-deterministic, high-variance nature of AI feature testing well. 

• Sequential testing reduces false positives
• CUPED variance reduction accelerates decision-making
• Fully custom SQL metrics capture what matters for AI outputs (task completion, output acceptance, engagement depth) rather than just clicks

The other half of AI readiness is whether your agents can operate the platform, not just whether the platform can test AI features.

With GrowthBook 5.0, we shipped 25 open-source agent skills covering flags, experiments, and product analytics. Each is a readable Markdown playbook for a single task (flag-create, flag-ramp, experiment-design, experiment-analyze), so an agent follows your order of operations instead of inventing its own. They install through your agent's plugin system and run without an MCP server. A rebuilt CLI covers 100% of the REST API, and the MCP server is there for Cursor, Claude Code, and other MCP clients that want it.

The part that matters for adoption is what happens when an agent gets it wrong. Agent changes land as drafts for a teammate to review rather than as live tests, inside the same approval policies and audit trail your team already uses. Deletes take two steps. Analysis skills are read-only. And you can read any skill before you run it, then fork it if your review process differs.

Getting started for free

You can evaluate GrowthBook without a contract or commitment. The migration kit, including the AI-powered SDK migration assistant, is available immediately.

The practical starting path for most teams:

1. Connect GrowthBook to your data warehouse. Pre-built SQL templates get you to first results without custom data engineering. Customize from there.
2. Run the AI migration assistant against your codebase. It transforms Statsig feature gates to GrowthBook equivalents and generates a diff for your team to review.
3. Import your Statsig experiments. Historical results carry over so you don't lose the record of what you've learned.
4. Start your first GrowthBook experiment. The Chrome debugger and visual editor make the first experiment accessible to non-engineers.

The decision to switch to GrowthBook

If your team depends on Statsig and the OpenAI and Amplitude acquisitions raise questions you can't yet get answered, about roadmap continuity, company priorities, and who supports you through a transition, then GrowthBook is the switch that costs the least to evaluate and offers the most structural independence.

Open source lets you inspect and audit what you're running. Warehouse-native architecture gives you data ownership that doesn't depend on a vendor relationship. Per-seat pricing gives you the freedom to run more experiments without watching a meter.

The migration kit makes the practical barriers manageable. The question is whether the reasons to switch outweigh the friction of switching. For most Statsig customers evaluating the post-acquisition landscape, that math is becoming clearer.

Ready to get started?

Schedule a consultation →

Estimate your savings →

Read the GrowthBook vs. Statsig comparison →

‍

Buying an experimentation tool is the easy part. The hard part is admitting when that tool costs more than it teaches you.

Luis Trindade has watched that math play out from the inside. He joined Farfetch twelve years ago as employee 700, just as the luxury marketplace opened its second tech hub in Lisbon. The company grew to 7,000 people worldwide, was acquired by Coupang, and refocused on the marketplace that connects luxury boutiques and brands to customers in markets Amazon has tried and failed to crack. Through all of it, Luis built the experimentation program that now runs a couple hundred experiments a month in low season, on a platform Farfetch built entirely in house.

On The Experimentation Edge, Ashley Stirrup asked Luis how that happened and what it cost. The answer is a case study in build versus buy, and in the difference between owning a testing tool and running an experimentation program.

Listen to the full episode

A hybrid setup that stopped making sense

Early Farfetch looked like most companies at hyper growth. Engineering ran tests on a tool it had implemented itself. Marketing used an external vendor because that was the tool it could get access to. Some teams did not know experimentation capabilities existed at all.

"How could we make sure that we talked the same language all as a company?" Luis asked. His answer was organizational before it was technical. Farfetch stood up an experimentation center of excellence, but rejected the standard version of one from day one. "Let's wait for somebody to ask us to execute for them and deliver just the results. I never believed on that," he said. The center would enable teams to run their own experiments, not run experiments for them.

On the engineering side, that meant Fabs, the Farfetch A/B testing system. Split engine, setup engine, stats engine, all built internally, layered on top of an omni-tracking system designed to follow a customer across web, app, and even into physical boutiques. For a while, Fabs ran alongside the external vendor that marketing used, a hybrid arrangement many companies would recognize.

Then the bill for the hybrid came due. The injected JavaScript hurt page performance. The results were inconsistent. And the team was already doing rework to pull data back internally for the deep dives the vendor could not provide. "Especially with tools that are based on JavaScript and code injection, they break a lot," Luis said. "They create inconsistent results." For a tech heavy company at Farfetch's maturity level, the external tool had become pure overhead. They phased it out and consolidated everything on Fabs 2.0.

One door for every experiment

The most distinctive choice in Fabs 2.0 is architectural. Farfetch's feature toggling system is the only entry point for running an experiment at the company. Every test, from any team, goes through it.

This is not a typical feature flag setup with on and off key values. The toggling system is deeply interconnected with Farfetch's segmentation service, benefits service, and user systems, which means teams can define exactly who sees what and when, then connect that definition to a randomized split. It also plugs into the CMS, the recommendation system, and the messaging systems, so marketing and content teams run their own experiments with full control and zero code injection.

The single door did more than clean up the architecture. It gave every team the same language. A single hypothesis template is used across the company. Weekly experimentation clinics, which Luis describes as group therapy sessions, put hypotheses in front of peers to be challenged before they ship. A monthly test and learn session opens the results to everyone from junior developers to C level executives. New joiners upskill just by sitting in the room.

That is where the center of excellence spends its effort now. The central team has shrunk over time while experiment volume keeps growing, and Luis considers that the point. "Practices, for you to be good, you need to practice, practice and repeat," he said. Roughly 80 percent of his time goes to enablement: ceremonies, coaching, and the shared knowledge base of learnings.

The metric Farfetch refuses to manage by

Ask most experimentation leaders about their win rate and you will get a number. Ask Luis and you get a shrug. "Oh, what is your winning rate? We don't even track it," he said. It exists on a dashboard somewhere. Nobody manages by it.

Instead, Farfetch rebranded the outcome of every experiment, in the tooling itself. The question is not whether a test won or lost. It is whether the team was able to learn from it, yes or no. "A failure is actually a test that was badly set up, wrong metrics, created many biases like sampling biases," Luis said. "That was a failure test. All the other tests are opportunities to learn."

The target that follows is unusual: a learning rate of 100 percent, meaning zero failed tests. A disproven hypothesis is not a failure. It is money the company did not spend on an idea that was not worth pursuing, and teams are encouraged to announce that proudly.

The reframe matters because it changes what people are willing to test. When losing counts against you, teams protect safe hypotheses. When learning is the metric, they bring their riskiest, most interesting ideas to the clinic.

Two years to beat the market leader

The hardest test of that philosophy came from Luis's own area. Farfetch was paying the world's leading recommendation engine vendor while sitting on a lake of behavioral data the vendor could never fully use. Strategically, the company wanted the dependency gone. So the team built its own engine, called Inspire.

"Surprise, at the beginning, it was completely losing against the world leader of recommendations," Luis said.

A strict fail fast reading says kill it. Farfetch did not, because the vision was strategic rather than tactical. What the team refused to do was run the bet blind. Every additional dollar spent on Inspire had to be justified while the vendor contract was still being paid, so the team ran hundreds of small, quick, directional iterations: A/B tests where volume allowed, quasi experiments where it did not, qualitative insight wherever it sharpened the picture.

"We all have a tool belt of experimental tools that you can use. All of them are valid," Luis said. "We just need to understand the different capabilities of them and their limitations."

Around the halfway mark of what became a two year effort, the tide shifted. Inspire started winning. Farfetch increased its investment, phased out the vendor, and today the engine powers recommendations across the business. Even accounting for full maintenance costs, the economics compensated.

"Strategy is key when we are doing experimentation," Luis said. "But at the same time, we need to do it in multiple and small, quick learning iterations." Keep the vision fixed. Let the iterations decide the path.

When building is the right call

Luis is not dogmatic about building. His own caveat: "I would not recommend to do this for a company that is not core for them to have a tech team." Farfetch is a product and technology company by DNA, with the maturity to maintain a statistical engine, a tracking layer, and a toggling system as first class products. For a company without that core, the same decision could be a costly mistake.

The honest version of build versus buy is not about features. It is about whether the platform is strategic to your business, whether you have a data advantage to exploit, and whether the hidden costs of an external tool, in performance, control, and rework, exceed the visible cost of engineering time.

Steps to take from Farfetch's playbook

1. Audit what your external testing tools actually cost. Count page performance, inconsistent results, and every hour spent pulling your own data back for analysis.
2. Route every experiment through one entry point. A single door creates a single language, and a single language is what makes learnings transferable between teams.
3. Measure learning rate, not win rate. Define failure narrowly as bad experimental design, and celebrate disproven hypotheses as avoided spend.
4. Give strategic bets a longer clock. Use quick directional iterations to decide whether to keep investing, and reserve fail fast for tactics, not vision.
5. Spend your central team on ceremonies, not execution. Clinics, shared templates, and open review sessions scale further than a service desk ever will.

The experiment you spent weeks making the case for is finally live. It runs its planned two weeks, the treatment comes out ahead, and you open the results to write up the win. At the top of the page sits a warning: sample ratio mismatch detected.

That sample ratio mismatch (SRM) warning means the observed traffic split doesn't match the one you configured, and until you know why, you can't trust the experiment results.

Diagnosing an SRM is much harder than detecting it. The root cause can sit anywhere from assignment to how the data gets processed and analyzed. This article covers the 5 types of SRM and how to prevent them, then walks through the investigation step by step: how to isolate the cause and how to decide if the experiment can be saved.

What is sample ratio mismatch, and why does it matter?

A sample ratio mismatch is a statistically significant gap between the traffic split you configured and the split your experiment produced. If you implemented a 50/50 test and the realized allocation is far enough from 50/50 that chance can't explain it, you have an SRM. For example, if a 50/50 test with a few thousand users in each arm comes back 45/55, that gap is far too large to be chance.

Even the most mature experimentation programs experience SRMs. For example, roughly 6% of experiments at Microsoft had SRM issues and LinkedIn has reported rates around 10% for some kinds of experiments.

Random assignment is what makes an A/B test valid. It guarantees the groups are comparable, so a metric difference can be attributed to the treatment rather than to who ended up in each group. Whatever caused the mismatch added or removed users non-randomly, which introduces bias.

For example, a slower treatment loses users who bounce before the exposure event fires. Those impatient users drop out of the treatment group, but their counterparts stay in the control group. The treatment group is left with more patient, engaged users than the control, so its metrics look better even if the treatment didn’t have an effect. The result is a false positive.

A chi-squared goodness-of-fit test detects an SRM by comparing the observed unit counts to the expected ones and returning a p-value, the probability of seeing an imbalance this large or larger if the samples were randomized correctly. GrowthBook runs this check automatically on every experiment. The test tells you an SRM exists. The rest of this guide is about finding out why.

The 5 types of sample ratio mismatch

Every SRM traces back to one of 5 types. This grouping is based on the taxonomy from a 2019 study of experiments run by four companies, which classified SRM root causes by the experiment stage where they enter. The diagnostic steps below tell you which type you have.

1. Assignment SRMs

With assignment SRMs, the randomization assigns users to variations in the wrong proportions, before any data is collected. Causes include a biased randomization function, eligibility criteria applied unevenly across variations, a hash attribute (the identifier used for bucketing) that's empty for some users, and 2 concurrent experiments whose bucketing isn't independent (users from one arm of the first test cluster into one arm of the second). Running an A/A test on a new experiment surface catches most of these before a real experiment depends on them.

2. Exposure logging SRMs

When users are assigned correctly, but not all of them get recorded, you have an exposure logging SRM. This is one of the most common types of SRM, because assignment often happens server-side while the tracking event (in GrowthBook, the SDK's `trackingCallback`) fires client-side. Because bots, ad blockers, and users who bounce before a page loads often prevent tracking code from executing, some users are assigned to an experiment but never recorded. This creates a data imbalance (SRM) if the experiment’s variations cause these issues at different rates. For example, a slower variation loses more users to bounces before the tracking event can fire. Firing the exposure event from the backend, before any variation-specific code runs, is the most reliable way to prevent this.

3. Analysis and filtering SRMs

The data is collected correctly, but the analysis drops users unevenly, usually through a filter such as an activation metric, which limits the analysis to users who reach a certain event. If the treatment changes who reaches that event, the filter removes more users from one arm than the other. Choosing activation metrics the treatment can't influence prevents most of these.

4. Data processing SRMs

The exposure data is right and the analysis is configured correctly, but rows for one variation get dropped or duplicated in the pipeline, by a faulty join, a deduplication step, or bot filtering. Monitoring how many units each join and filter drops per variation catches these early.

5. Interference SRMs

Someone or something acts on the running experiment itself. A variation's traffic percentage gets changed mid-run, an internal team assigns itself into a variant, or an attacker submits injection strings through a form field that gets recorded in your telemetry, corrupting events in whichever variation they were assigned to. Alerts on changes to running experiments prevent most interference from your own team.

How to identify the root cause of an SRM

Identifying the root cause of an SRM is a process of systematically ruling things out. Each step below eliminates one or more causes or points directly at the culprit. Go through each step in order, because each result tells you what to look for in the next step.

Step 1: Confirm the SRM is unexpected

Not every SRM warning points to a data-quality problem. There are 2 configuration issues you can try and rule out right away.

Check that your metric matches the assignment grain

An SRM warning can be caused by metric choice rather than the experiment. A metric needs to be built at the grain you randomized, meaning its denominator counts the same unit, and it uses the same identifier.

Sessions per user, for example, is fine in a user-randomized experiment, because the denominator is users, and users are what the randomization balanced. Conversion rate per session is not, because its denominator is sessions, a unit the randomization didn't balance. A variation that changes how often visitors return generates more sessions in one arm. The session counts diverge for a legitimate reason, but the divergence still triggers an SRM warning.

The same thing applies to identifiers. Randomization only balances the specific identifier you use for hashing. If your metric relies on a different identifier, it may count units that weren’t part of the original randomization. For example, if you randomize by user ID but your metric tracks anonymous ID (common for pre-login events), a single user could map to multiple anonymous IDs. Since these IDs aren’t guaranteed to split evenly across variations, your groups will be imbalanced. 

In either case, you need to rebuild the metric on the unit you actually randomized and rerun the analysis.

Check whether the experiment was altered mid-run

Editing the assignment ratio or targeting, or starting a new phase without re-randomizing, can also cause the imbalance on its own. If that's what happened, there are 2 solutions, and the choice depends on the data collected before the change:

• End the analysis at the point of the change: Choose this option when the truncated experiment still satisfies your experimental design. Cutting an experiment short can violate the experimental design in ways that bias or weaken the result. For example, ending mid-week introduces day-of-week bias, an incomplete business cycle changes which users you capture, and a smaller sample can leave the test underpowered.
• Re-randomize and restart the experiment: This is the only solution when the data collected before the change doesn't satisfy the experimental design.

If the metric matches the assignment grain and the experiment wasn't altered mid-run, the imbalance is unexplained. Move on to the next step.

Step 2: Determine whether the SRM is isolated or systemic

An SRM can be systemic or specific to one experiment. It's systemic when other live experiments show SRM warnings at the same time, or when an A/A test (an experiment that serves the same experience to both arms) shows one. Systemic problems live in infrastructure shared across experiments, such as the assignment system, the tracking code, or the data pipeline. Comparing experiments identifies systemic problems much faster than debugging a single experiment. GrowthBook runs the same SRM check on every experiment, so checking your other recent experiments for warnings only takes a few minutes.

• Several experiments show an SRM at once: The cause is likely in shared infrastructure, not in any single experiment, so start the investigation there. The 2019 study describes a bug in Microsoft's assignment service, which randomized users into 1,000 buckets that each held 0.1% of traffic. The bug gave the control one bucket fewer than the configured split called for, so every 50/50 test on the platform ran at 49.9/50. An A/A test surfaced it.
• Only this experiment shows an SRM: The cause is local to this experiment's setup, data, or treatment. Continue to the next step.

Step 3: Read the severity and direction of the SRM

Severity describes how far the observed split sits from the one you configured, and direction describes which arm is missing units. In GrowthBook, the Health Tab's balance check shows observed and expected units side-by-side for each variation so you quickly identify both.

• The imbalance is small but statistically significant: The cause is likely narrow, like a bug that affects a specific browser or a brief tracking outage.
• The imbalance is large: Causes this big tend to be structural, like a redirect that fails for a whole variation or a tracking event missing from one code path.
• The treatment arm is short: Treatment users are being lost before they're counted. This usually means an exposure or performance problem, like a slower variation where users bounce before the tracking event fires.
• One arm is at or near zero: The exposure event or trigger isn't firing for that arm at all, and there is probably a telemetry problem that needs to be fixed.

Step 4: Pinpoint when the SRM started

The timing of the imbalance separates causes that were present at launch from causes that arrived later. GrowthBook's Health Tab plots units by the date they were first exposed.

• Present from day one: Assignment or exposure logging broke before anyone entered the experiment.
• Appears mid-run: Something changed partway through the experiment. Check for things like a targeting change, a delayed variation start, or a change to the data source behind the experiment.
• Strongest on day one, then fades: A rollout effect is usually the cause here. For example, one variation starting a few hours late or cached pages serving the old experience until the cache expires. Users who joined the experiment during that window end up in one arm disproportionately, but the gap closes as more units accumulate.

Step 5: Segment the data to localize the SRM

Many SRMs can be localized by comparing data: the filtered analysis against the unfiltered exposure counts, and the population broken down by segment. Use what you learned in the previous investigation steps to decide where to look first.

• The SRM appears only in the triggered or filtered view, not in the unfiltered exposure counts: The filter is likely the cause and the problem is in the analysis. Activation metrics that sit downstream of the treatment are a common culprit, because a slower or heavier variation can reduce how many users reach the activation event.
• The imbalance originates in one dimension: The cause is something that varies with that specific population rather than the experiment as a whole. For example, if one browser has an SRM while the rest are fine, you probably have an implementation bug in that browser. GrowthBook's Health Tab makes it easy to break units down by the dimensions you’ve defined. 

Most dimensions have typical causes to check for:

• Browser: Ad blockers or tracking-prevention features block the exposure event in that browser.
• Device: An SDK bug affects one app version, or slower devices lose users before the event fires.
• Region: Consent requirements suppress tracking for some users, or bot traffic concentrates in one geography.
• Channel: A campaign link sends users directly into one variation, bypassing randomization.
• New vs. returning users: An imbalance concentrated among returning users points to a feedback loop, like a treatment that changes how often users return and re-enter the experiment.
• Login status: Signing in mid-experiment switches a user from an anonymous ID to a user ID, which can hash into a different variation, so the user is re-assigned, double-counted, or removed by the multiple-exposure filter, which excludes users who were seen in more than one variation.

The login-status case only skews the split when the treatment changes how often users sign in, like a checkout that requires an account. With an uneven split like 90/10, the multiple-exposure filter compounds the loss. A re-evaluated user from the small arm usually lands in the large arm and gets removed, while large-arm users mostly re-hash into the same arm and are never flagged.

Localizing the SRM also tells you which subject matter expert in your company to reach out to, because they can often explain what happened faster than further analysis will. For example, if there are more organic search visitors in one variation of a URL redirect test, your SEO manager may know that Google indexed that variation's URL, so searchers land on it directly, bypassing randomization. Or if an imbalance is limited to one app version, the lead engineer can quickly check if a recent release affected exposure logging.

Step 6: Compare performance and engagement metrics

Comparing performance and engagement across the arms separates a treatment-driven difference from an instrumentation one. In GrowthBook, if you track load time or errors as guardrail metrics, they will be visible alongside your goal metrics in the results table.

• The treatment arm is short with worse load times, or long with better ones: The treatment's performance change is altering who gets recorded, causing selection bias. A variation that adds 800ms of load time loses its most impatient users before the exposure event fires, while one that gets faster gains users instead. Either way, the two groups are no longer comparable.
• Engagement per user differs between arms: The direction tells you who is missing. A higher average in the short arm means less-engaged users are being lost, which points to bounces or tracking loss. A lower average means the most-engaged users are being removed, which points to filtering, like a bot detector catching heavy users.

Step 7: Inspect the data pipeline

If nothing upstream explains the imbalance, the cause is likely in the data pipeline itself. Between the raw tracking events and the results table, the data passes through steps that join, filter, and deduplicate it, and any of those steps can drop or duplicate rows for one variation. To locate the faulty step, count the units per variation after each one. The first step where the split becomes imbalanced is the one introducing the SRM. In GrowthBook, the experiment assignment query defines how units are counted, so start there.

• Your own count doesn't match the platform's: Counting units per variation directly from the raw events table checks every filter, join, and date window your experiment platform's queries apply. If your count is balanced and the platform's isn't, the bug is in those queries. If your count is imbalanced too, the loss likely happened before the warehouse, which points back to exposure logging.
• The split is balanced entering a step and skewed after it: The imbalance is entering at that point in the pipeline, so inspect the step's logic directly. For example, if the raw exposure events are balanced but the counts skew after a join that stitches anonymous IDs to user IDs, that join is dropping or duplicating rows for one variation. This is the login-status pattern from Step 5, seen at the query level.
• Records that failed to join skew toward one arm: An inner join drops rows with no match on the other side, and nothing in the output indicates it happened. If the dropped rows come disproportionately from one variation, that asymmetric loss is likely the cause of your SRM. Filters deserve the same scrutiny. Count how many units each filter excludes from each variation and check whether the exclusions are balanced.
• The identifier in your queries doesn't match the hash attribute your SDK randomizes on: The units being counted are not the units that were assigned. This is the same mismatch from metric check in Step 1, seen from the warehouse side instead. GrowthBook's Multiple Exposures warning will usually appear alongside the SRM warning when it happens.

How to decide whether to restart your experiment

Once you've identified the cause of your SRM, there are 3 possible outcomes: recover the result, restart the experiment, or salvage a directional insight. Which outcome applies depends on whether the correct data exists and whether the treatment itself changed which units were counted.

Recover the result

If the raw events were logged correctly and a processing step created the SRM, you can fix that step, rerun the analysis, and keep the result. A bot filter that removed real users, a faulty join, or a misconfigured analysis filter all fall into this category. The 2019 study describes a case just like this where an SRM was traced to a bot filter that had removed the most-engaged treatment users. Once the filter was corrected, the experiment showed a statistically significant win, and no rerun was needed.

Restart the experiment

If the correct data was never captured, or the treatment changed the underlying population, no reprocessing brings the lost units back, and you have to restart. An exposure event that never fired for one arm, or a slower variation that lost users from one arm faster than the other, both leave you with groups that aren't comparable. Before you restart, fix the root cause first and re-randomize with a new salt (the value that seeds the hashing function). In GrowthBook, that means creating a new phase with re-randomization or starting a fresh experiment, so users from the first run are independently reassigned instead of carrying their old assignments forward, which would introduce carryover bias. You’ll also want to restart when the imbalance is severe and the cause remains unidentified because with an unknown bias direction, neither a win nor a loss can be trusted.

Salvage a directional insight

If the cause is unrelated to user characteristics (like an issue with how units were allocated to buckets), or external and time-bounded (like a one-day bot spike that doesn't coincide with the movement in your metrics), you can sometimes treat the result as directional evidence. Document your reasoning before making decisions on any SRM-affected number so readers know it’s directional.

SRM diagnosis table

Every row below maps to a step or decision above, so you can either follow the investigation in order or jump to the observation that matches what you're seeing.

How GrowthBook helps you find and fix SRMs

GrowthBook's experimentation platform automates several of the steps in this investigation. Every experiment gets an automatic SRM check, so you learn about an imbalance without having to run the test yourself. The Health Tab covers 2 of the quickest steps directly: traffic over time to see when the imbalance started, and traffic by dimension to identify which segment is affected. A Pre-Exposure Bias Check flags when the groups already differed before the experiment started, and Multiple Exposures detection surfaces the hash-attribute mismatches that often cause SRMs. These run alongside a larger set of data quality checks that run on every analysis. When you've found the cause of an SRM, the troubleshooting guide provides guidance on how to address the most common ones, and because the stats engine is open source, you can read exactly how any check is computed.

If you want these health checks running on your experiments by default, try GrowthBook for free.

‍

Running an experiment is the easy part. The harder part is knowing which experiment is worth running at all, and understanding why it wins or loses when it does. That distinction sits at the center of a recent episode of The Experimentation Edge, where host Ashley Stirrup spoke with Fabian Hans, founder and behavioral psychologist at Cogniteer, a consultancy that helps enterprises build in-house experimentation programs and raise both their testing velocity and their win rate.

Hans is unusual among the show's guests. Most are practitioners inside a single company. He works across many, which gives him a wide view of where experimentation programs go right and where they quietly go wrong. The pattern he keeps returning to is a warning to anyone scaling a testing program: it is easy to produce more tests, and much harder to produce more understanding.

🎧 Listen to the full episode →

Who is Fabian Hans?

Hans got into conversion rate optimization the way a lot of strong practitioners do, by accident and by being honest. Fifteen years ago he applied for an online marketing role and was asked, as a small test, what he thought of the company's website. He told them plainly that he did not like it, and gave specific reasons why. They hired him as a CRO manager before he had the vocabulary for the title.

From there he built an agency, then moved in-house for the depth that agency work rarely allows, and four years ago founded Cogniteer. Today the company helps enterprises stand up their own experimentation programs, increase test velocity, and improve win rate, drawing on Hans's background in behavioral psychology to understand not just what users do, but why.

The trap of mass-produced tests

The agency years taught Hans what not to do. With many clients and little time, his team fell into a familiar habit: if a test had won somewhere, they assumed it would win somewhere else, and rolled it out again. "It ended up that all of the clients got the same tests," he said. "It's a mass production of tests. And I did not like this approach."

The clearest example was a single line his team added to the basket: your items are not reserved. "It's one line. It's super easy to develop. It's psychology. It sells well to the clients. And we had a 50% win rate with that," Hans said. On paper, that is a great result. In practice, running the same idea for the hundredth time stopped teaching him anything. "You're losing the passion for it if you always do the same things over and over," he said.

That is the quiet cost of copy-paste experimentation. The win rate can look healthy while the learning goes to zero. Hans went in-house, and later built Cogniteer, precisely to trade breadth for depth: fewer clients, deeper understanding of each one's users, and tests designed to answer a specific question rather than repeat a familiar template. Mass-produced tests can raise velocity. Only deep understanding reliably raises win rate.

Most ecommerce drop-offs are structural

One reason the same tests keep reappearing is that the same problems keep reappearing. Across most ecommerce shops, Hans sees identical leaks: a high drop-off in the basket, and another on the product detail page. His counterintuitive point is that these are usually not your fault. "That is not because of the company or of the product," he said of basket abandonment. "It's just shopping." In roughly 80% of the shops he sees, the same issues show up simply because it is ecommerce.

The deeper cause is a mismatch between the product and the channel. Online shopping, as Hans notes, effectively started with Amazon selling books, and books are easy to sell on a screen because you can judge the content and the cover. Most other products are harder. A washing machine looks identical to every other washing machine in a photo, so the image barely influences the decision, and buyers care about features like capacity and water use instead. Fashion is the opposite, where style and design carry the decision and the picture is the point. And some products resist the screen entirely. "Perfume, you cannot sell online, because you need to smell it first," he said. Other products need to be touched.

For experimentation teams, the lesson is to diagnose before optimizing. Before testing a new basket layout, it is worth asking whether the drop-off is a genuine interface problem or the structural reality of selling something the screen cannot fully represent. That framing changes what you test and what you expect a test to fix.

Match the interface to how people actually buy

If the first job is understanding the product, the second is understanding the buyer, and Hans is emphatic that not all buyers are the same. One client asked Cogniteer to help win more new customers. The business was built on habit buying, where customers return regularly to restock, much like buying the same groceries every week. The page was perfect for that returning user, who wants to come in, press a button, and leave. It was almost useless for a first-time buyer, who needs information to decide whether the product is even right for them. "All the information is not accessible," Hans said. "New users don't convert because they are confronted with the transactional element way too early."

The fix was not a better page but a personalized one. New users are given the context they need before buying, while returning users keep the fast, transactional path they prefer. Same product, two interfaces, matched to two very different states of knowledge.

The same principle reshaped a B2B client selling water dispensers. In B2B, Hans explained, the buying behavior is different: someone is handed a budget and asked to find a fitting solution, then gather offers to bring to a manager. They do not want to browse a catalog and guess which product suits a thousand-person office or a factory. So Cogniteer removed the shop interface and replaced it with a finder, a short survey that ends by proposing a solution and inviting the buyer to request an offer. It reframed the page from a product catalog into a lead-generation tool, and conversion rose sharply. Sometimes the highest-impact change is deleting the thing everyone assumed the page needed.

Why this matters for experimentation teams

Underneath all three stories is the same argument. The biggest obstacle to a strong experimentation program is rarely tooling. Hans notes that AI is already dissolving the old developer-resource constraint through prompt-based experimentation. The stubborn obstacle is understanding: understanding the product, the buyer, and the specific problem worth solving. Ask five people what is wrong with a website and you get five answers, and in-house teams often lose the outside perspective that a first-time user brings.

That is why deep dives beat mass-produced tests. A program built on genuine understanding of user behavior does not just ship more experiments, it ships better questions, protects the metrics that actually drive decisions, and learns something whether a test wins or loses. Velocity without understanding produces motion. Understanding paired with velocity produces a program that compounds.

For product, data, and engineering teams building that kind of program, the tooling should get out of the way so the thinking can take center stage. That is the problem GrowthBook is built to handle, connecting experimentation to your existing data warehouse with a statistics engine your team can actually trust. Explore the open-source platform and start for free at growthbook.io.

Listen to the full conversation with Fabian Hans on The Experimentation Edge, and consider where repetition may have quietly replaced understanding in your own testing program.

Big experimental wins are rare. On a mature product, most of what you test produces a small effect. Across 1,450 experiments on Microsoft's Bing, most changes moved their target metric by a fraction of a percent. An effect that small is hard for a standard A/B test to detect. The treatment may be moving the metric, but the effect is small relative to its natural variation, and the sample isn't large enough to separate signal from noise, especially when the change reaches only a fraction of the user base. Without variance reduction, experiments like these often end up underpowered, with no significant result and a decision left to judgment rather than evidence.

To detect an effect that small, the experiment has to be more sensitive. Sensitivity is its ability to detect an effect when one exists, and short of waiting for a larger sample to accrue, the only input you can change is the variance of the metric itself. When that variance comes down, the same sample detects a smaller effect, or the same effect sooner.

This is a practitioner's guide to 5 variance reduction techniques in experimentation: metric choice, winsorization, CUPED, post-stratification, and triggered analysis. Each section covers how the technique works and the trade-off that comes with it. They're ordered from the simplest to the most complex, and, at the end, there's a decision framework for which to try first.

Why variance is slowing your tests down

Variance slows your tests down because a noisier metric needs a larger sample to separate a real effect from chance, and a larger sample takes longer to collect. Required sample size scales linearly with the variance of the metric, so halving the variance roughly halves the time to significance and tightens the confidence interval you end up with.

Most of that variance has nothing to do with your experiment. The signal you're after is the effect of your change. The noise is everything else. A user who was always going to spend $400 this month sits in the same dataset as one who was always going to spend $4, and that variation exists before your treatment does anything. Required sample size scales with that variance, so a noisier metric costs you directly. It also scales with the inverse square of the smallest effect you want to detect, so detecting an effect half as large needs four times the sample, not twice. A noisy metric and a small target effect compound each other, and both push your runtime up.

Lower variance also helps in the other direction. At a fixed sample size, it shrinks the minimum detectable effect (MDE), the smallest effect your test can reliably detect, so a 1% change that was previously below that threshold now comes within range. That leaves 2 ways to detect a smaller effect: collect more data, or remove variance. You usually can't generate more units on demand, whatever your experiment's grain (users, sessions, requests), so the 5 techniques below all take the second path. 

1. Fix your metrics before you fix your statistics

The largest change to your experiment's sensitivity is the metric you choose to measure, not the statistical method you apply to it. No amount of adjustment can rescue a primary metric that is intrinsically noisy, while a cleaner choice of metric can shorten the test's duration before you change anything in the analysis.

Move the metric closer to the treatment

A proximate metric measures the action your feature directly affects. A distal metric sits far downstream, where your change is one of hundreds of influences. If you're testing a new checkout button, the proximate metric is checkout starts or add-to-cart rate. The distal metric is revenue, which is moved by pricing, inventory, seasonality, and everything else your team ships that month.

Proximate metrics carry less of that unrelated variation, so they respond faster and detect smaller effects. Moving your primary metric closer to the treatment is often the largest single sensitivity gain available, and the easiest to make.

The caveat is that proximity can cost you business relevance. The reason you care about checkout starts is that they are supposed to lead to revenue. But if checkout starts come back significant and positive while revenue stays flat, you've shipped a win that delivered none of the value you were after. A proximate metric is only worth using if it tracks your Overall Evaluation Criterion (OEC), the metric or weighted blend of metrics that defines a successful experiment. A good OEC is measurable within the experiment but still predicts the long-term value you ultimately care about. That lets you decide now instead of waiting weeks or months for lagging metrics like revenue or retention. Pick the closer metric, but confirm it actually moves with the downstream outcome it stands in for.

Committing to a single primary metric also preserves statistical power. Each additional metric you test raises the probability of a spurious significant result, which forces a multiple-comparison correction that reduces sensitivity across every metric, including your primary. One well-chosen primary outperforms a battery of hedged ones.

Use rates and binary metrics over totals

The shape of a metric drives its variance. A binary metric like conversion rate is naturally bounded. Each user contributes only a 0 or a 1, so no single user can dominate the variance. (Formally, for a conversion rate p the per-user variance is p(1 − p), which tops out at 0.25.) A total like revenue per user has a long right tail, where a handful of high-spending users sit far above the rest. Those users move the mean on their own, and the variance they add is what makes a small treatment effect hard to detect.

When possible, make a rate or a binary metric your primary. For example, instead of average order value or revenue per user, where a few big spenders dominate, measure the share of users who spend more than some amount, say $50. Each user counts once, above or below that line, so no single $5,000 order can dominate the result. That's a proportion metric, the rate form of a yes-or-no outcome, rather than a mean metric. The tradeoff is that you lose magnitude, since it tells you whether more users crossed the threshold, not how much they spent past it. Keep revenue as a secondary or guardrail metric so you still track it.

2. Handle outliers with winsorization

Winsorization, also known as outlier capping, limits each user's aggregated value to a chosen threshold so a few extreme outliers can't dominate the result. If revenue per user normally runs around $40 and one customer places a $5,000 order, that single order can make whichever variation the customer was assigned to look like the winner. Set the cap at the 99th percentile, say around $200. The $5,000 order then counts as $200, so the customer still contributes to the result but can no longer make their variation look like the winner on their own.

Extreme values don't just move the mean. They inflate variance, which widens your confidence intervals and extends the experiment duration. Capping trades a small, deliberate bias for a reduction in variance.

Two practical notes matter here. First, the cap has to be pre-specified. Decide the rule before launch, whether that's an absolute cap or a percentile like the 99th, not after you've seen which users are outliers and which variation they helped. Choosing the threshold once results are in, to move the result in your favor, is p-hacking. GrowthBook's percentile capping computes the actual cutoff from experiment data, but you commit to the rule up front.

Second, and less often stated, capping changes the estimand, the quantity you're actually estimating. Once you cap, you're no longer estimating the treatment's effect on mean revenue. You're estimating its effect on capped revenue, and those aren't the same number. The deeper issue is that you can't see how the capped users would have responded, because you've flattened exactly the part of their behavior the feature might move. If the real value of your change lives in the tail, like a feature meant to get your highest spenders to spend more, capping can hide the effect you're testing for. Before trusting a capped result on a revenue metric, check the tail directly with a quantile metric to confirm the outliers aren't carrying the effect.

3. CUPED experiments: the most powerful general-purpose technique

CUPED (Controlled-experiment Using Pre-Experiment Data) is a form of covariate adjustment that reduces variance by adjusting each user's outcome for how they behaved before the experiment started. It removes the part of the metric you could have predicted in advance, leaving behind the part your treatment is responsible for. 

How CUPED works

CUPED uses each user's pre-experiment value of the metric to estimate what they would have done without the treatment, then subtracts it. Those baseline differences have nothing to do with your treatment, so subtracting them lowers the variance without biasing the estimate. For example, a frequent buyer who keeps buying at the same rate in treatment no longer looks like a treatment effect, because the adjustment already accounted for their baseline.

CUPED comes from a 2013 paper by Deng, Xu, Kohavi, and Walker. It has one requirement: for each user, you need a pre-experiment measurement that predicts their behavior during the experiment. Usually that's the same metric measured beforehand—a user's revenue in the weeks before the test predicts their revenue during it. The stronger that relationship, the more variance CUPED removes, because that predictable, pre-existing variation is exactly what it subtracts out.

When CUPED works (and when it doesn't)

CUPED's variance reduction is proportional to how strongly a user's pre-experiment behavior correlates with their in-experiment behavior. When that correlation is high, the gains are large. Netflix reported roughly a 40% variance reduction on key engagement metrics, and Microsoft found CUPED equivalent to adding about 20% more traffic to an experiment.

CUPED's limitation is that it does little when the correlation is weak or missing. Brand-new users have no pre-experiment history to adjust against. Rare conversions, like a first purchase, often have no usable pre-period signal either. GrowthBook applies CUPED across your metrics as an organization-wide default, and skips it automatically where there's no pre-experiment data to use.

4. Post-stratification: stacking on top of CUPED

Post-stratification splits users into groups, measures the treatment effect within each group, and combines those into a weighted average. Where CUPED removes variance using each user's own history, post-stratification removes the variance between groups, like the gap between countries or devices, which your treatment didn't cause.

For example, comparing US users only to US users, and Indian users only to Indian users, removes the large between-country differences from the comparison. Because it targets variance that CUPED leaves behind, it stacks on top of CUPED rather than competing with it. GrowthBook combines the two into a single setting called CUPEDps, which in the right conditions can be equivalent to running your experiment with 20%+ more traffic. Beyond cutting variance, it also corrects for imbalance. If the treatment-control split in a country comes out 48/52 instead of 50/50, post-stratification reweights the two groups to the split you set.

Post-stratification only helps when the groups you stratify by actually differ on the metric. If they don't, there's little between-group variance for it to remove. A related technique, stratified randomization, balances the groups before assignment rather than after.

5. Use triggered analysis for low-coverage features

Triggering restricts your experiment to the users actually exposed to the treatment, not everyone assigned to it. It's most useful for low-coverage features, where the unexposed users dilute your effect the most.

The dilution problem

When your analysis includes users who were assigned but never exposed, their treatment effect is exactly zero, and they pull your measured effect toward zero. The treatment effect concentrated among the exposed users gets averaged down across everyone who wasn't. If your feature adds $4 of revenue per exposed user but only 20% of assigned users are ever exposed, that $4 effect shows up as $0.80 across everyone assigned, because the other 80% contribute nothing.

Triggered analysis fixes this by filtering the unexposed users out of the analysis, leaving only those who were exposed. GrowthBook logs exposure from feature flag evaluation, so you can tell exactly who was exposed and filter the analysis to only include them.

Filtering relies on the treatment not changing who gets exposed. When it does, like a banner that drives more users to scroll or a variant that loads slower so fewer reach the feature, you get a sample ratio mismatch (SRM) where the realized split no longer matches the one you set. The two groups are no longer truly random, and a measurable difference between them could be caused by that mismatch rather than the treatment. Run an SRM test on the triggered split before trusting a filtered result.

When the treatment might change who gets exposed, move randomization to the trigger point instead, assigning each user the moment they're first exposed rather than up front. If you're testing a redesigned cart page, assign users the first time they open the cart, not before. Only users who reach the cart enter the experiment, and because the redesign appears only once they're there, it can't have changed who reached the cart.

The estimand caveat

A triggered result describes the triggered users only, not everyone. A 10% lift among users who experienced the change is not a 10% lift for your business, because most of your users never see the feature. Before you make a launch decision or a revenue projection, translate the triggered effect back to the full population.

Which technique should you use? A decision framework

Each technique can be stacked, so all 5 can run on a single experiment when relevant.

How GrowthBook implements variance reduction

GrowthBook implements most of these variance reduction techniques directly, so you don't have to build them yourself. Capping is a metric setting, either an absolute threshold or a percentile. CUPED runs as an organization-wide default on Pro and Enterprise plans, and post-stratification on Enterprise combines with it into CUPEDps.

Variance reduction is one half of running experiments faster. Safe peeking is the other, and GrowthBook makes this possible with sequential testing, which lets you check results repeatedly without inflating your false positive rate. The stats engine behind all of this is open source, so you can read exactly how any adjustment is computed.

The metric Debug tab added in GrowthBook 4.3 shows how CUPED, post-stratification, and capping each move a given metric's variance. Variance reduction is otherwise easy to treat as a black box, where the intervals come out tighter, and you trust that the adjustment was applied correctly. Seeing each layer's contribution makes that adjustment auditable.

If you want variance reduction by default, you can try GrowthBook for free or book a demo with the team.

A customer opens a support chat with a simple question about a refund. The AI agent, trying to be helpful, replies: "Of course, we're going to process your refund." It sounds perfect. It is also a problem, because no human ever authorized that promise, and the agent essentially made it up.

That moment is at the heart of how Fin, the AI support agent formerly known as Intercom, actually gets better. Pedro Tabacof, Principal Machine Learning Scientist at Fin, has spent three years running the experiments that turn a promising demo into a product companies trust. Fin now crosses $100M in ARR, serves more than 10,000 customers, and is Anthropic's own first line of customer support. None of that came from shipping features that looked good. It came from testing everything, especially the things that failed.

🎧 Listen to the full episode →

Why you cannot unit test an AI agent

Most software teams have a comforting ritual: write the test cases, watch them go green, ship with confidence. Tabacof's first argument is that this ritual quietly breaks the moment you put a large language model in the loop.

"Unit tests, like traditional software unit tests don't work with AI," he says. The reason is that the system is non-deterministic and the inputs are effectively infinite. "Even if you try to come up with all sorts of use cases, your end user is always gonna surprise you, and there's always gonna be something new, like a change in the product or the world that's gonna make users ask different questions."

If you cannot enumerate the cases, you cannot pin the behavior down with examples. What you can do is measure it at scale. That is why A/B testing, not offline evaluation, is Fin's gold standard for every decision. Offline evals run on maybe one thousand to ten thousand examples. Fin's live experiments pull millions of samples in a few days, which gives them the statistical power to detect very small effects with real confidence, sometimes in a single day.

The result is a culture where testing is the default reflex, not a special occasion. Fin runs one to two dozen experiments concurrently, has run thousands over three years, and A/B tests changes most teams would never bother to check, down to a single comma or period in a prompt. "Whenever we have any kind of question dilemma, we just put it to the test," Tabacof says. They even hired a full-time AI analyst whose only job is improving the experimentation framework itself.

The experiment that made Fin slower and better

The clearest example of why this matters is an experiment nobody expected to run. Leadership wanted to reduce Fin's latency. The instinct was universal: faster is better, and the literature on e-commerce and search backs it, where even 100 milliseconds moves revenue. But no one at Fin actually knew what latency cost their business.

There was a catch. You cannot easily test reducing latency, because if you could make the agent faster, you would have already done it. So the team did the counterintuitive thing and tested increasing it. They designed the experiment carefully, raising latency roughly in proportion to the natural distribution so that no single user would feel something was obviously wrong and contaminate the result.

Then the data came back inverted. "Higher latency only saw good stuff happening," Tabacof says. Some users bounced, which was expected. What was not expected was that positive feedback went up. The result was so counterintuitive that his manager forced him to run a confirmatory experiment, check for selection bias, and rule out any metric miscalculation before anyone would believe it.

The best explanation is psychological. When an AI answers a hard question instantly, it reads as a canned macro rather than real thought. A human never replies that fast. A small delay makes the agent look like it is doing the work, more human-like, and people reward that with more positive feedback. Fin still reduced latency in the end, because a slow agent looks bad in a sales demo. But they learned something durable: latency is a confounder. Now, when they run experiments that reduce it, they add a third arm that holds latency constant, so they can isolate the effect they actually care about.

Turning a hallucinating loser into a shipped winner

The article's title comes from the experiment that best captures Fin's whole philosophy. The team wanted to give the agent more conversation history when it generated answers. It is one of the most obvious ideas in the book: more context, better understanding, better replies.

The first result was genuinely mixed. Positive feedback improved massively. But hallucinations, which Fin measures as a hard guardrail metric using LLM judges, went up too. Worse, some of that new positive feedback appeared to be caused by the hallucinations, like the fake refund promise. As Tabacof puts it, "Fin could say things such as, yeah, of course we're gonna process your refund. This can be a very serious fake promise." It was not common, but it was common enough to register in the metric, and that made the first experiment a failure despite the upside.

Here is where most teams choose between two bad options: ship the feature and accept the hallucinations, or scrap it and lose the gain. Tabacof's team refused both. They went back to the drawing board, used offline analysis to map every new pattern of hallucination that the added context was creating, and addressed those specifically in the prompt. Then they reran the experiment to measure the real effect of the context with the new instructions in place.

The fix cut the positive-feedback gain roughly in half. But that remaining gain was real, and it now came with no increase in hallucinations at all. That was shippable, so they shipped it. Fin has carried a much stronger conversation-history context ever since, without the trust cost.

The lesson generalizes. "For most changes which fail, almost always with the right prompting you can overcome most of the challenges, not all," Tabacof says. A losing experiment is frequently a winning one with a single broken component. The skill is diagnosing which component, then fixing only that.

The uncomfortable math of a real experimentation program

If all of this sounds like a lot of work to ship one feature, that is the point. Tabacof estimates Fin's experiment win rate at roughly 20 to 30 percent, and he is quick to note they do not even track it formally because defining the denominator is genuinely hard. Most people, he observes, implicitly assume that 90 to 100 percent of their new features are winners. Learning the real number is humbling, and it is exactly what pushes a serious team to test more and claim less.

What makes it work at Fin is that leadership treats experimentation as the cost of quality, not a tax on speed. The team's manager, now Chief AI Officer, came from an experimentation background and once sold a company to Optimizely. That protection means the scientists can run losing experiments without fear, because everyone understands that "AI development is inherently very uncertain, very experimental, because you never know how much ground you have covered."

Looking forward, Tabacof expects AI to drive more of the experimentation itself. Claude already launches and analyzes many of Fin's experiments, dredging through data and flagging problems. But he draws a sharp line about where judgment still lives: "We humans this year, we own the strategy but Claude's owning the tactics." The tools will keep getting better at writing the prompt and running the test. Knowing which question is worth asking remains the human's job, and the way you answer it does not change. As he puts it, "decision-making can only essentially be done through experiments."

For product, data, and engineering leaders building on top of non-deterministic models, that is the whole playbook in one line. Ship less on intuition, measure more at scale, and treat every loser as a diagnosis waiting to happen. To see how teams run experiments like these end to end, learn more at growthbook.io.

In January 2026, DownDetector reported roughly 4,000 user reports about Monzo. Its mobile banking app stopped working for 2 hours, leaving thousands of customers unable to log in or make payments. For a fintech company serving 14 million customers, even a brief outage could put years of trust at risk.

When you’re running digital infrastructure for financial services, you can’t afford to continue using legacy processes.

Every software deployment you do carries a unit of risk with it, especially if you’re using all-or-nothing deployments. If something goes wrong, it can trigger a domino effect leading to significant financial consequences for you and your customers.

That’s why fintech organizations have started using feature flags to have more control over the process.

In this guide, we’ll explain why fintech organizations need feature flags and how they can consider using them.

Why fintech companies need to adopt feature flags for deployment

Here are a few reasons why fintech companies need feature flags:

Every failed deployment is a financial event

Every time your engineering team deploys a new feature without proper testing and observability, you’re increasing the surface area of risk. That’s because a problematic feature in your app prevents customers from accessing their funds or using them for other purposes.

Even one incident could result in heavy regulatory scrutiny. Cross-border payments fail at an average rate of 11%. And 82% of merchants don’t even know why payments fail in the first place.

That’s not a risk you should take when your customers trust you to know what’s happening under the hood.

Compliance is the basis of everything you do

Because of the nature of the work you’re doing, you’re also answerable to regulatory bodies across borders. You’re essentially operating in an environment that very few organizations do.

For instance, the EU’s Digital Operational Resilience Act (DORA) took effect in January 2025. So now you’re required by law to ensure your digital infrastructure can recover quickly when something goes wrong. That’s why banks like Monzo had a failsafe called “Monzo Stand-In” that took over when it experienced an outage in January 2026.

Feature flags are another way to make sure you can automate the fallback path.

Deployment safety requires data, but you can’t use it

Typically, to enable safe deployments, you need a continuous feedback loop. You ship a change and see how real users interact with it. The rollout behavior changes based on the metrics you choose to observe.

But in fintech, those signals are regulated financial data. You can’t export it without having the proper privacy protections in place.

Feature flags can bring rollout control in the same trusted environment as the financial data, as long as you’re using a compliant platform.

How are fintech teams using feature flags to ship safely?

The most basic use case and benefit of feature flags is decoupling deployment from release. It reduces the risk of deploying code, but also gives you more flexibility to track and manage new features without rushing to deploy everything at once.

Here are a few other use cases of feature flags for fintech companies:

1. Progressive rollouts for payment and onboarding features

Let’s say you’re rolling out a redesigned checkout flow. If you push it to 100% of users on day one and the conversion rate drops, you’ve just broken the revenue path for your entire user base. That’s the problem progressive rollouts fix.

Here, you start small by exposing the new flow to 1% of traffic and defining what “safe to expand” looks like before you launch. It could be that the conversion rate improves or holds steady, or that the error rates stay flat. You only move on when those conditions are met.

GrowthBook’s Monitored Ramp-Up automates the progression. You configure each stage with a target percentage and time window, and the platform moves traffic forward on a schedule. A fintech rollout might look like:

• Internal QA team first
• 5% of free-tier users
• 25% of free-tier users
• Approval gate before paid users enter the mix
• 25% paid to full rollout

2. Automated rollbacks with guardrail metrics

Progressive rollouts limit your blast radius. But they still assume someone on your team is keeping an eye on things. That’s not the reality your engineers deal with.

GrowthBook’s Safe Rollouts remove that dependency. You define guardrail metrics at flag setup time, such as payment conversion rate or transaction error rate, and the platform continuously monitors them as traffic ramps up.

If any guardrail metric shows a statistically significant regression, the platform automatically reverts the rollout.

3. Compliance gating and jurisdiction-based targeting

A World Economic Forum report found that 60% of fintech companies operate in multiple jurisdictions, while 31% operate in multiple regions. This means that if you’re releasing a new feature or an update, you need the ability to control where it’s released first.

For example, if you’re releasing a new lending feature and it’s approved in the US but not in Singapore, you have three choices:

1. Maintain separate code branches per region
2. Delay the entire launch till approval
3. Release in one region through granular targeting

GrowthBook’s Forced Value rule lets you implement option 3 with feature flags. You can define the segment as US-only, deploy the feature to all environments, and make it live only for those accessing it from the US.

When you get the approval for Singapore, just update the targeting condition.

4. AI model rollouts for fraud detection and credit underwriting

World Economic Forum’s report also found that 80% of fintechs are now implementing AI across multiple business lines. They’re either adopting it for software development or shipping AI features.

Let’s say you’re releasing a new fraud detection model. If the model misfires, it might give you too many false positives and reject legitimate ones for no reason.

Instead of rolling out a bug to your entire user base, just wrap it in a feature flag. Roll it out to 5% of traffic and monitor your guardrail metrics to decide whether to scale up or reverse the rollout.

Note: Within GrowthBook, you can even experiment on prompts and AI model versions to see what works best. And all of this works with feature flags being the main delivery mechanism. 

How feature flags satisfy fintech compliance requirements

Here are some of the features your feature flagging platform needs to help you stay compliant with fintech regulations:

Audit trails and revision history

Every time you go through an audit, your auditor might ask you to reconstruct every change to your payment processing flag over the last six months. Things like who enabled it, when, what the previous state was, and who approved it.

You should be able to pull that up in a minute. In fact, if you’re going through SOC 2 Type II audits, this is a baseline requirement.

GrowthBook’s audit logs capture every flag event, with timestamped records available across all plans. You can pull up the version history for any flag and compare revisions side by side to see exactly what changed.

Approval workflows and the four-eyes principle

You don’t want to be in a situation where an engineer pushes a change that hits production immediately. It could result in smaller bugs or full-blown incidents. That’s one of the reasons the “four-eyes principle” is a baseline expectation these days.

Essentially, one person creates the flag with specific rules, and another person approves it before it goes live.

In GrowthBook, approval workflows serve as configurable gates. You can draft the change and submit it for review. Your peer or manager can approve it before it goes live. Irrespective of whether you’re creating a kill switch or a saved group, you can require approval for everything.

Stale flag management

As you create more flags, you’re also creating more technical debt. Because feature flags are typically temporary, once a feature ships, engineering teams forget to remove them. The problem is that it’s still active in your codebase and could cause incidents later.

Make sure your feature flagging platform can detect these stale flags. For instance, GrowthBook automatically marks feature flags as stale if:

• They haven’t been active for two weeks.
• They aren’t in any active environments (the flag is disabled everywhere).
• Have one-sided rules that send 100% of traffic to a single variation.

You can also use its MCP server to find feature flags and pinpoint where they exist (via Code References). And then use the cleanup skill to remove them. 

Keeping financial data in your infrastructure

Most feature flag vendors need your data on their servers to evaluate experiment metrics. Very few of them offer self-hosted deployments or a warehouse-native architecture. But in the fintech industry, that could result in heavy penalties and reputational risk.

So choose a platform that:

• Offers a warehouse-native architecture that queries existing data warehouses for anything.
• Offers self-hosted deployment with air-gapped installations.
• Has compliance certifications such as SOC 2 Type II, GDPR, COPPA, or CCPA.

How Upstart cut experimentation time from days to hours with GrowthBook

Upstart is an AI-powered lending marketplace that connects consumers with over 100 banks and credit unions. Its engineering team runs experiments against real credit models and transaction data.

Before GrowthBook, Upstart’s setup was fragmented. They ran a third-party tool for feature flags alongside two separate in-house experimentation platforms. As a result, engineers kept switching between tools, and analysis depended on manual work.

Since it was a financial services company, they couldn’t let sensitive financial data leave their infrastructure. That’s where GrowthBook’s self-hosted deployment model solved the compliance problem, as all experimental data remained within its infrastructure.

Upstart’s team consolidated three platforms into one, GrowthBook, and cut experimentation time from days to hours. But they were also able to clean up legacy feature flags, which also helped them reduce technical debt in the long run.

“We had to rally our teams for this project and get our old tools cleaned up, but the process is significantly improving the quality of our systems and reinforcing best practices around experiments and feature-flagging hygiene. GrowthBook allowed us to uplevel our code, speed up decision-making, and focus on what we do best—building a world-class AI lending marketplace.”
‍
— Diego Accame, Director of Engineering, Growth, Upstart

What your feature flag platform needs to handle in fintech

If you’re actively evaluating feature flagging tools, these are the questions you should ask before choosing one:

Note: GrowthBook does all of this and more. Learn more about how we help fintech organizations.

Use feature flags to ship faster with confidence without risking your reputation

Monzo’s two-hour outage reminded the industry what fintech teams already know: when you’re handling other people’s money, there’s no such thing as a minor production incident. And at least in the bank’s case, they had a fallback path in place.

But the reality is that most fintech organizations don’t. This is why you should consider adopting feature flags. They give you the infrastructure to ship fast without taking on that risk. Whether you want to test in production or create fallback paths, you can do so without building excessive tooling.

If you’d like to see how feature flags can improve your deployment process, try GrowthBook for free or book a demo.

‍

On July 19, 2024, CrowdStrike pushed a configuration update to every one of its Falcon sensors at once. A single logic error crashed 8.5 million Windows systems and cost Fortune 500 companies an estimated $5.4 billion.

It wasn’t a silly mistake that brought down global systems. CrowdStrike tested the update, but it didn’t use a staged rollout, so a bug that could’ve been contained to 1% of users became a global outage instead. The problem is that using a delivery process like this just opens up your surface area of risk.

In fact, deploying software change is one of the three top causes of an IT outage. And downtime costs average at $14,056 per minute.

The question is: How do you ship changes into production without turning every release into an all-or-nothing bet?

Using canary releases with feature flags is one way to do that.

In this article, we’ll walk you through the concept of canary releases and how to implement them.

What is a canary release?

A canary release exposes a change to a small group of users before you roll it out to everyone. The name “canary” comes from a time when coal miners brought canaries underground to detect toxic gases in the mine. If the bird showed any signs of distress, miners left before anything bad happened.

In software development, the same logic applies. A small group of users absorbs the initial deployment risk so the rest of your users can stay protected. This helps even if you have staging and test environments because production traffic behaves differently with real data under real load. Edge cases only show up when you deploy the changes to production.

Pros and cons of canary releases

What’s the difference between infrastructure canary vs. feature flag canary?

You can run a canary at two different layers of your stack. The layer you choose determines how fast you can roll back.

With an infrastructure canary, you route a percentage of traffic at the load balancer or service mesh level to a new version of your service. You’re releasing the entire service as one unit. And if something breaks, you shift all the traffic back to the older version, which can take several minutes.

A feature flag canary is different. Here, you deploy the code to 100% of your servers but wrap the new behavior/change behind a flag. Only users who match your rollout criteria see it. If something breaks, you flip the flag. That takes seconds.

In short: an infrastructure canary controls which building your users walk into. A feature flag canary controls which rooms are unlocked inside the same building.

Mature engineering teams use both models.

For instance, Facebook deploys code continuously to all production servers but gates every feature through Gatekeeper, its internal feature flag system. Its employees see the new features first and assess their performance. Only when everything looks good do they roll out the change to 2% of production traffic and test it there.

How to do a canary release with feature flags?

Let’s say you’re launching an AI tutor product within your edtech platform. It’ll call an LLM, so there are new risks to consider, like timeout errors or unpredictable token costs.

Here’s how you can do using GrowthBook’s feature flagging platform by using feature flags for the canary release:

Step 1: Wrap the change in a flag

Create a new boolean flag called ai-assistant with a default value of false. It’ll decouple deployment from release, so nobody sees it unless somebody should.

Here’s what it looks like in GrowthBook:

if (gb.isOn("ai-assistant")) {

  return renderAIAssistant();   // new LLM-backed path

}

return renderManualFlow();       // existing path

gb.isOn() evaluates the flag locally without a network request, so it adds zero latency to your code path.

Deploy this to 100% of your servers with the flag set to false in your production environment. This step helps you deploy code to development, staging, and test environments, but it won’t appear in production unless you enable it.

Step 2. Define your rollout ladder

A rollout ladder is the sequence of stages your flag moves through before reaching 100%. Typically, you monitor specific metrics at each stage before moving to the next.

If you’re using GrowthBook, you can use the following features to do this:

• Internal team only: In GrowthBook, create a Forced Value rule targeted at a Saved Group of employees or design partners. You can also target by attribute (email contains yourdomain.com or plan = “internal”). Everyone else falls through to false.
• 1–5% of users: Add a Percentage Rollout rule below the forced value. GrowthBook uses deterministic hashing based on user ID, so the same user always gets the same experience—no server-side session storage required.
• 10–25%: Bump the rollout percentage if metrics look healthy.
• 50% → 100%: Full rollout.

Note: The riskier the change, the smaller the interval should be. But make sure you add an approval step based on parameters like error rates or similar criteria to gain more control over the process. 

Step 3. Define success and guardrail metrics before you ship

Before you enable the flag, set up two categories of metrics: success metrics and guardrail metrics. While success metrics tell you whether the feature is doing what you built it to do, Guardrail metrics tell you whether it’s breaking something else.

In GrowthBook, you define these as Fact Tables and Metrics that are directly connected to your data warehouse. Each metric has a type:

1. Proportion for rates (error rate, timeout rate)
2. Mean for averages (cost per request)
3. Quantile for percentiles (p99 latency)
4. Ratio for derived measures like tokens per resolved query

You can then mark each metric as either a guardrail (a hard gate that can trigger a rollback) or a signal (a leading indicator you watch but don’t gate on).

Step 4. Enable the flag, monitor, and ramp up accordingly

In GrowthBook, every flag change goes through a draft-to-review-to-publish workflow. The goal is to make sure changes don’t reach real users unless you want them to. Once you’ve defined all the changes and metrics, roll out the change and monitor your metrics.

For example, if you’re launching the AI tutor, start with the internal dev team and monitor error rates. Are the AI responses rendering correctly? Is the fallback path working when the flag is off? Is the UI responding well?

Once you’re confident, roll it to real users and ramp up slowly.

Step 5. Automate the rollout with guardrail monitoring

You can use Monitored Ramp-Up in GrowthBook to automate this whole process. You define a ramp schedule, attach guardrail metrics, and the platform moves on, or rolls back the changes based on what the data shows.

GrowthBook ships with a built-in preset:

• Linear ramp
• Fast ramp (three steps)
• Approval-gated milestones (define any ladder and any hold time)

For the AI tutor, your ramp config might look like this:

You can use the following metrics to monitor the rollout:

• Guardrail metrics (hard gates which trigger a rollback): llm_error_rate, p99_latency, cost_per_request 
• Signal metrics (watch, don't gate): answer_acceptance_rate

Note: If you prefer to automate this process, just use GrowthBook’s flag-monitoring AI agent skill via your MCP and create the ramp up schedule in minutes. 

The first step ramps to 10% and monitors for 24 hours. The next one ramps it up to 50% and waits for manual approval before proceeding. The guardrail failures trigger an automatic rollback.

Tip: When you need to pull the plug completely, GrowthBook has a built-in environment toggle on every flag so if you disable it in production, the SDK stops evaluating all rules simultaneously. It acts as an instant kill switch. It’ll turn off the previous flag and stop evaluating every rule immediately. The SDK returns the default value for all users, regardless of the ramp schedule.

Step 6. Clean up the flag

Once you’ve confirmed that the AI tutor feature works well, it’s time to clean up the flag. A 2022 study on feature toggles found that 33% of toggles interact with other code expressions—and those interactions grow 22% each year. So, the longer a flag stays in your codebase, the more entangled it becomes.

Use GrowthBook’s stale flag detection to automatically surface flags that have been fully rolled out for more than two weeks. You can even use Code References to show you where the flag is within your codebase.

When you’re ready, archive it and check if there’s a change or regression in your app. If not, delete it.

Tip: Schedule the cleanup the minute you create the flag. You can add a note in your calendar and use GrowthBook’s MCP to clean it up later. In fact, you can use the flag-cleanup skill to find these flags and archive them from your AI app.

Which metrics should you monitor during a canary release?

Here are three categories of metrics you should monitor with every canary release:

Infrastructure metrics

These tell you whether the servers are handling the new code path.

Application metrics

These tell you whether the feature itself is working.

Business metrics

These tell you whether the change is moving the needle in the right direction.

Tip: Always set your thresholds before you ship so you can act in real time rather than decide whether to act. Once your data team defines a metric in SQL, GrowthBook references it directly in your data warehouse and notifies you accordingly.

6 mistakes to avoid while using canary releases

Even with canary releases, you can still experience failure modes. Here’s what to avoid:

1. Treating the canary as your test suite: A canary is supposed to catch edge cases that pre-production testing can’t. So, if you’re using it to find issues that unit and test integrations should’ve caught, you’re just testing in production without real guardrails. Run your pre-canary gates (contract tests, integration tests, performance smoke tests) before the actual release to avoid this.
2. Not defining rollback criteria before shipping: You don’t want to spend time figuring out which metrics are “bad enough” to trigger an incident response. It’ll cost you time, money, and resources you’d rather spend elsewhere. Define your thresholds before the flag goes live. For example, which metrics you’ll track, when you’ll decide to act on them, and how you’ll act (your incident or remediation response).
3. Rolling out the feature too quickly: If you need robust test results, it doesn’t make sense to jump from 1% to 10% in a minute. Ideally, you should test the initial rollout for 30 minutes to a few hours or longer if you’re doing an infrastructure canary release. Over time, you can set your own thresholds based on previous experimental releases.
4. Using a canary cohort that doesn’t represent your users: If your canary group is only employees, only one region, or only power or enterprise users, you can’t generalize your results. Unless it’s something you know applies only to a specific segment, it’s best to test with a broader segment first and then drill down over time. Alternatively, run release experiments with different cohorts to see how they respond, then generalize the results.
5. Letting a canary sit at partial rollout for weeks: If your rollout is stuck at 10% for weeks at a time, you have a zombie canary. It creates a forked production environment with continuously diverging behavior. Avoid this by setting a maximum dwell time for each stage, especially when automating the process. Once it crosses that threshold, either promote it to the next stage or roll it back.
6. Skipping session stickiness: Without sticky assignment, a user might see the new version on one page load and the old experience on the next. It confuses them, and you won’t be able to realistically measure how the new version is performing. Platforms like GrowthBook handle this by using deterministic hashing based on user ID, so the same user sees the same flag state as long as the release is in progress.

Use canary releases to stop treating deployment as an all-or-nothing bet

After the July 2024 outage, CrowdStrike has adopted the “phased release” approach, which is essentially a canary release. Now, updates roll out in concentric rings, with monitoring at each stage, and customers can choose to adopt the change early or opt out altogether.

And this is the exact approach engineering teams should adopt.

With the pace of software development increasing by the day, especially with AI, you don’t need to add unnecessary complexity around every release. A canary release simplifies this process and removes any anxiety around it.

If you’re ready to start adopting a canary release approach with feature flags, try GrowthBook for free or book a demo.

‍

Every team that runs experiments wants to run more, faster. The difference between teams is where friction lives.

Consider two teams. The first team is finishing its first dozen experiments. Their process works, technically, but they run all experiments through two or three technical staff they trust. Experiments feel complex and high-stakes, and they worry that a bad customer experience could set the company back. The second team runs hundreds of experiments every day, against a warehouse with petabytes of data. They have the opportunity of fast, high-scale experimentation, but every analysis run has compute costs that equal real time and money. They need infrastructure to match their opportunity.

Different stages produce different friction. 

GrowthBook 5.0 meets friction at every stage, making it easier to experiment when you’re building the habit, faster and more efficient at the frontier, and safer all along.

Starting an experiment should be easy

One of the biggest barriers to early teams is an empty experiment creation screen, especially when it’s long and involves more complex, technical options.

In 5.0, we cut the experiment creation flow from 23 fields to 5. The rest of the options move to the experiment overview page. You can write a hypothesis, set up experiment assignment, and share the draft to your team in the time it used to take to scroll through the previous version.

We ran an experiment as we rolled this out, and there’s a surprisingly large effect. More people can create an experiment without feeling like they need to be an expert, and with sensible defaults, starting doesn’t mean shipping something accidentally.

Making experiment creation “cheaper,” with less time and expertise invested, makes it easier to imagine running more experiments and building a broader culture of experimentation. A thin experiment, with just the minimum fields, becomes a shared surface your team can edit and learn in together. Experimentation begins as collaboration rather than a handoff.

More hands, same standards

As the team grows and experimentation velocity increases, more and more people kick off experiments. You have an experimentation process that works, and you want more of it. More teams want to experiment, but a small handful of experts is still a bottleneck. Ideas for experiments might abound, but the organization is short on the people needed to push experiments from beginning to end.

Scaling experimentation means handing more people, more operations, the keys without giving up control of what ships. GrowthBook 5.0 helps.

‍More people can create. AI agents can now operate GrowthBook, which means anyone, through their code editor or with our new in-app AI Assistant, can brainstorm and create experiments, launch them, and check results with natural language. The new AI Visual Editor opens another door for teams, like growth and marketing, who prefer a point, click, and prompt experience. With the slim experiment creation flow and reusable templates, it’s so much easier to set up an experiment now and build on an existing experimentation program.

Guardrails travel with tests. Opening the door for more experimentation only helps your org if your best practices and processes stay in place for everyone. Our new custom hooks let teams codify their own rules and run checks for every experiment before it starts, automatically. Mandatory custom fields make sure every experiment is categorized and filled before it’s ever created. Linked feature flags mean a teammate still reviews production changes before launch. These checks live with the experiment, and are present and documented in GrowthBook.

Shared understanding builds shared commitment. Experiments that finish and features that roll out without others knowing can help a business, but they don’t help build shared knowledge and experimentation culture. We now have new meta-analysis experiment blocks for dashboards, so your team can have better oversight of what’s running, win percentages, lift, and scaled impact. Everyone in the org can learn, and, hopefully, be inspired to experiment also.

You don’t have to be at your desk to launch. GrowthBook 5.0 supports scheduled starts for experiments. You can kick off a test without being live at your computer. It sounds like a small thing, but as you scale and coordinate more and more experiments, control like this is what keeps a program manageable, right up until the next constraint takes over: compute.

Solve for people and trust, though, and a new limit appears: the volume of experiments you can now run starts to outgrow the infrastructure running them, and the bottleneck shifts from who can experiment to what your compute can afford.

Where compute is the constraint

At the frontier of experimentation scale, the bottleneck becomes compute. Data-rich environments give you the raw material for fast, accurate results, but can come with a bill to match. Teams that run hundreds of experiments a day against high-volume warehouses need infrastructure that meets their opportunity without equally large bills or compromises on velocity.

We think two techniques are highly relevant for these teams: quantile treatment effects (like the effect of a change on your p99 latency) and CUPED, which uses pre-experiment behavior to cut noise and bring results faster. Neither technique is new, but there’s a challenge to running them efficiently with a warehouse-native platform when data is large and quickly changing.

GrowthBook 5.0 includes major performance improvements with both approaches. For quantiles, we use KLL sketches to gather an event-level histogram into a single column that doesn’t grow as events pile up, which keeps quantiles cheaper at volume and even allows for pre-aggregations in the warehouse to be consumed by GrowthBook. For CUPED, we can now run a nightly pre-aggregation of each experimentation unit’s history, and since GrowthBook holds metric definition in our semantic layer, it refreshes only summaries that go stale when a definition changes.

We’ve seen >80% reduction in wall time for an experiment update, faster individual runtimes for experiments, and overall reduction in compute, which leads to lower warehouse bills. We’ll write about this in more detail later, but if you’re experimenting and analyzing at petabyte scale, come talk to us.

A platform you never outgrow

Everything in this post moved as you grew: the friction of starting, the bottleneck of people, the ceiling of compute. Safety is the one thing that didn't.

With GrowthBook 5.0, you can run experiments a little more easily, a little faster, with as much safety as ever, so you can make more chances to really learn about your product and users. That’s the real goal of experimentation: knowing instead of guessing, and building a better product for the people who use it. 

‍

Want to hear from our co-founders and engineers how we’ve made feature flagging and experimentation better? Come to our live Office Hours on July 30, 9am PT, and we’ll answer any questions you have. 

Governance, for those unaccustomed to this buzzy word, is simply referring to guardrails. With feature flags, it can be really easy to work yourself into a pickle. Our latest 5.0 release focused on many cool things, but one of the big ones was protecting you and your co-workers from yourselves. So you can work faster while still shipping safely.

As AI speeds up how fast code can be written and large organizations release more changes across more teams, the challenge is no longer simply getting work out the door. It is vital to make sure every change is valid, behaves as expected alongside everything else, and reaches production with the right context.

With this in mind, there are three areas in which we focused our feature flag governance in 5.0:

1. Catch problems while someone is making the change
2. Catch problems created by the full rule set
3. Make the final decision with context

Catch problems while someone is making the change

For this category of features, GrowthBook is checking whether the change itself is valid, sensible, and no larger than it needs to be. The best time to catch a bad flag change is before it becomes a review comment, a Slack thread, or a production scavenger hunt.

Schema validation for string and number flags

GrowthBook could already validate the shape of JSON flag values. Now you can also set rules for string and number flags.

For example, you can now limit a checkout layout flag to classic, compact, or express. GrowthBook will reject anything else, including typos.

["classic", "compact", "express"]

For number flags, now you can set a minimum and maximum, such as allowing results-per-page to be anywhere from 10 to 100.

It is basically type safety for remote config, because TypeScript cannot save you from someone entering "expres" in a dashboard. Invalid values can now be caught before they are saved or published.

Feature-scoped Custom Hooks

Custom Hooks let your team write its own rules for what counts as a valid flag change.

For example, you could require a description, prevent targeting by email address, or make sure every rollout uses userId.

Custom Hooks already existed, but they previously applied to every feature in an organization or project. Now you can attach one to a single feature that needs its own rules.

• Organization-wide: Every feature flag must have an owner and at least one tag.
• Project-wide: Every rollout in the checkout project must use userId for hashing.
• Single feature: Only the checkout-config flag must include a Jira ticket before it can be changed.

Soft warnings

Not every questionable choice deserves the full red-screen treatment.

Custom Hooks can now raise a warning instead of blocking the save:

addWarning("Consider adding a ticket number");

The person editing can review it and choose Save anyway.

So now you have two volumes:

• addWarning() says, “Are you sure about this?”
• throw says, “Absolutely not.”

Sparse patches for JSON rules

Suppose a JSON flag has ten fields, but a rule only needs to change one.

Instead of copying the entire object, you can save just the keys and values you want to update:

{  
"theme": "dark"
}

‍
GrowthBook merges that onto the default value, which means smaller changes, cleaner diffs, and fewer stale copied fields.

One important caveat: the merge is shallow. It only merges top-level keys, not deeply nested objects.

Catch problems while someone is making the change

Together, these features make the editor feel less like a blank text box with consequences and more like a development environment that knows what “valid” means.

The value can be checked against an expected shape, custom rules can catch requirements unique to your team, warnings can flag questionable choices without blocking every save, and sparse patches keep changes focused on the fields someone actually meant to touch. By the time the change reaches review, many of the easiest mistakes have already been caught, while they are still cheap to fix.

Catch problems created by the full rule set

A rule can be perfectly valid on its own and still cause trouble once it joins the group chat.

GrowthBook evaluates rules from top to bottom. An earlier rule can serve a user before a later rule ever gets the chance.

For example:

1. Force a value for everyone in the US.
2. Force a different value for Pro users in the US.

The second rule will never run. Pro users in the US already matched the first rule and left the building.

GrowthBook already warned when a rule was completely unreachable. In 5.0, that detection expands to catch more ways rules can compete for the same traffic.

The warnings now have three levels:

• Unreachable: No matching traffic will reach the rule.
• Will not reach: Some matching traffic will definitely be served by an earlier rule.
• May not reach: An earlier rule might serve some of the same traffic, but we cannot say exactly how much.

That last distinction matters for partial rollouts. A 90% rollout above another rule does not completely block the rule below it, but it does take a large bite out of the traffic before it gets there.

The warning also points to the earlier rule causing the conflict, so you can reorder the rules, narrow the targeting, or confirm that the overlap is intentional.

This catches a different class of problem than schema validation or Custom Hooks. Each rule may be valid, but the full rule set may not behave the way you pictured it.

Make the final decision with context

Automated checks can catch a lot, but eventually a human still has to decide: Is this change ready to go live?

GrowthBook 5.0 adds a dedicated Review & Publish tab that brings the scattered pieces of that decision into one place. It replaces separate publishing, review, and conflict-resolution flows with one shared workspace.

The Conversation view gives you the readable version of the story: what changed, why it changed, who contributed, any comments or reviews, and what needs to happen next.

When you need the technical receipts, the Changes view shows the detailed diff and lets people comment on the specific changes.

If Approval Flows are enabled, reviewers can leave a comment, request changes, or approve the draft before it is published. Without required approvals, someone with permission can still review the same context and publish directly. Approval Flows are a commercial feature.

The goal is not to make every flag change pass through a courtroom. It is to put the diff, discussion, reviewers, conflicts, and final action in the same room, so the person clicking Publish knows exactly what they are sending out the door.

Guardrails not stop signs

Good governance is not about slowing creators down. It is about making the work they publish quality and catching problems early before they hit users.

By adding governance while changes are created, when rules interact, and before publishing, teams will catch mistakes earlier and ship with more confidence. I'm so excited for what 5.0 at GrowthBook has brought to feature flags. If you'd like to learn more or hear from the engineers behind all of our newly released features, you can join us for Office Hours Live on Thursday, July 30 at 9 a.m. PT. See you online and happy experimenting!

GrowthBook’s Product Analytics is now generally available. Monitor your KPIs, explore trends, build dashboards, and analyze funnels, using the same trusted metric definitions your team already relies on for experimentation and feature flags.

If you're running experiments in GrowthBook, you've already done the hard work. You've connected your data warehouse, defined your metrics, and built a shared language for how your team measures success. 

Product Analytics puts those metrics to work beyond experiments. Keep your north star KPIs front and center so your team can identify new opportunities to test, and track how past experiment wins are translating into long-term impact. Build dashboards that show product trends and experiment impact side by side, giving stakeholders the full picture in one place. 

This post covers everything that is now generally available within Product Analytics.

AI-powered analytics

A native AI chat agent is built directly into Product Analytics. This opens up analytics to any user, turning a few prompts into actionable insights without needing to know the schema or rely on a data analyst for answers.

Ask questions in plain language, such as “Chart the number of daily users and conversions for the past 30 days” and the agent will build the exploration, render the chart, and surface insights behind the data.

In-app AI assistant

GrowthBook 5.0 also includes a platform-wide AI assistant available from any page in the app. In addition to taking action and supporting workflows across feature flags and experiments, it can also pull up analytics and surface trends from wherever you're working. This assistant pulls from 25+ open-source skills spanning feature flag management, experiment design, and product analytics.

‍

‍

The Metric Explorer: visualize and explore your data

The Metric Explorer lets you visualize any metric your team has defined in GrowthBook without needing to write SQL. These are the same metric definitions powering your experiments, so there's no custom SQL to write and no risk of metric drift between your analytics and your experiment results. This lets teams go quickly from question to chart in seconds.

The Metric Explorer supports a range of chart types (line, area, bar, stacked bar, horizontal bar, stacked horizontal bar, timeseries table, table, and big number) and the ability to segment by dates, dimensions, and filters.

If your team hasn't fully defined metrics yet, the Explorer also supports querying fact tables directly or even raw data source tables in your warehouse, so you can start exploring immediately and promote what you find to reusable metrics over time.

Because GrowthBook generates the SQL behind the scenes, it works with GrowthBook's Managed Warehouse, BigQuery, Snowflake, Databricks, Redshift, ClickHouse, Postgres, and more. Your data never leaves the warehouse.

‍

SQL Reports

The SQL Explorer lets you write, save, and share queries directly within GrowthBook. No external BI tool needed. Pin results to any dashboard alongside your metric explorations and experiment data.

Write a query in the built-in editor with schema browsing and autocomplete, or describe what you want in plain language and let AI generate the SQL from your warehouse schema. Run the query, then layer on the types of visualizations you’d like to see: bar, line, area, scatter, pivot table, or big value (KPI). Configure axes, aggregations, dimensions, and filters visually without touching the SQL again.

Save reports, refresh them on demand, and pin individual visualizations to any dashboard alongside your metric explorations.

Funnel analysis

Define step sequences to visualize conversion rates between each step and understand where users fall out of the flow. Break funnels down by any dimension to find where specific segments convert differently.

For example, you might define a checkout funnel and break it down by device type. You notice mobile users drop off at the payment step at twice the rate of desktop. That's a clear signal. From the same platform, you can launch an experiment to test a simplified mobile payment flow and measure whether it closes the gap.

Whether it's your onboarding flow, checkout, or activation sequence, funnel analysis turns drop-off points into experiment hypotheses you can act on immediately.

Dashboards

Dashboards let you tell a story with your data. Create custom dashboards tailored to your north star metrics so they remain front and center. Arrange the KPIs that matter most to you on a flexible grid with drag-and-drop, and set a refresh schedule to keep everything current, and share with your team.

GrowthBook dashboards support a range of block types, from metric explorations and saved SQL queries to rich text, so you can build exactly the narrative your team or stakeholders need to get the full picture.

A typical dashboard might include your north star metric trend, a product funnel showing where users drop off, and the scaled impact from all your experiments pertaining to the relevant metrics.

One source of truth

Product Analytics is an extension of the work your data team has already done. The metrics are defined. The warehouse is connected. Now those same definitions power dashboards, funnels, and ad-hoc exploration, not just experiment results.

For teams that want programmatic access, the REST API exposes endpoints for running explorations, and every exploration gets a shareable URL encoding its complete configuration.

Product Analytics is available now. Get started or book a demo to see it in action.

‍

TL;DR: GrowthBook 5.0 ships with an AI Visual Editor that lets anyone build and launch a live experiment on their website from a plain-language prompt, no engineering ticket required. 

Growth and marketing teams rarely suffer from a shortage of ideas. 

There is always another headline to test, another landing page to improve, another audience that may respond to different messaging, or another campaign that could convert more effectively.

The problem is getting those ideas into production. The reality for many marketing and growth teams is that testing capacity is capped by engineering bandwidth. 

The GrowthBook AI Visual Editor unlocks this entire process. For the first time, anyone can vibe code a new home page or product page and deploy a rigorous experiment in minutes without engineering expertise. The AI Visual Editor allows users to move sections, change text and colors, and even swap out images with just a series of prompts. Once a new variation is built, these same users can kick off rigorous experiments using the same metrics and templates created by your data science team.  So anyone can run experiments you can trust.

See the visual editor in action here:

‍

Run experiments without waiting for engineering

The most immediate benefit of a visual experimentation system is straightforward: growth and marketing teams can move from an idea to a live experiment without waiting for an engineer to implement every variation.

Using GrowthBook’s AI Visual Editor, teams can describe a change in plain English or make it directly through a WYSIWYG interface. They can update text, modify styles, rearrange content, replace imagery, hide elements, or create more substantial page variations.

The editor then turns those changes into an experiment that can be previewed, reviewed, and launched through GrowthBook.

This removes a significant source of friction from the experimentation process.

Engineering teams no longer need to spend time implementing every headline test, campaign-specific landing page, image variation, or call-to-action adjustment. Growth teams no longer need to wait for an open sprint before learning whether an idea works.

And GrowthBook’s AI Visual Editor keeps engineering teams happy. The AI Visual Editor, like all of GrowthBook, is built on transparency so the technical teams can still review exactly what the experiment is doing, audit the results, look at the metrics, etc. Visual experiments also run on the same SDK used for other parts of GrowthBook, and even be flicker-free. 

The difference is that engineering is no longer required for every step of every experiment.

‍

What you can do with the AI Visual Editor

Many visual editors fall short and break on modern sites, and quietly push you back into writing CSS or HTML. We fully rebuilt our new AI Visual Editor from scratch to fix these issues. 

The AI Visual Editor lets anyone describe the change they want in plain language and get a working variation without writing code. Use manual mode or enter a prompt to do things like:

• Generate images with AI: Use AI to generate and modify hero images, product photos, background visuals, and more.
  

• Change headlines and copy: Manually update copy to test different messaging.
• Generate new copy ideas: Prompt the AI to write different headlines, CTAs, or copy variations to test.
• Run multi-arm bandits: Use the AI Visual Editor to make changes to your site and run it as a multi-arm bandit to dynamically allocate traffic to the highest performing variant.
• Update designs and layouts: Adjust fonts, padding, button styles, restructure layouts, and more.

• Import Figma frames or mockups: Bring in a design straight from Figma to test actual user interaction without rebuilding it from scratch, not just as a static image.

• ‍Import image files: Pull in your own images or brand assets.

The variation is built directly in the editor so you can see exactly how all your changes will look to your end users.

‍

Test more ideas and learn faster

Reducing implementation work changes more than test speed. It changes which ideas are worth testing at all.

When every experiment requires engineering time, teams naturally reserve experimentation for larger ideas. Smaller questions remain unanswered because the expected value of the result does not justify the cost of implementation. A visual editor lowers the incremental cost of answering them, resulting in teams running more experiments.

The real advantage is that teams can explore more ideas, test smaller assumptions, iterate on promising concepts, and build a clearer understanding of what customers respond to.

Instead of spending weeks debating which message should become the new default, teams can put several credible alternatives in front of real users and measure the result. You can even run a multi-arm bandit directly from the AI Visual Editor. The new workflow becomes:

1. Identify an opportunity.
2. Create a high-quality variation.
3. Launch it safely.
4. Measure its effect.
5. Use the result to inform the next decision.

The faster that loop becomes, the faster a team can improve.

‍

Personalize messaging for different audiences

Most websites present a single version of the company to every visitor.

But not every visitor arrives with the same problem, the same level of familiarity, or the same reason for evaluating the product.

Someone arriving from an AI-focused campaign may care about evaluating nondeterministic product experiences. An enterprise buyer may care more about security, governance, and deployment flexibility. A developer may care about SDK performance and implementation details. A marketing leader may care about conversion rates and how quickly their team can launch tests.

Sending all of these visitors to the same generic page often means presenting each of them with a diluted version of the message they actually need.

Client-side experimentation gives growth and marketing teams a practical way to test more relevant experiences for different audiences. The objective is not personalization for its own sake. Every additional experience creates complexity, and not every audience needs its own version of a website.

The value comes from being able to test whether a more relevant message actually improves the outcome.

‍

Validate ideas before investing in permanent development

Some ideas require substantial engineering work to implement correctly.

A redesigned pricing page may need new components. A different onboarding flow may require backend changes. A personalized experience may eventually need to be integrated deeply into the application.

But teams do not always need to build the complete version before learning whether the underlying idea has value.

A visual experiment can serve as a lightweight production prototype.

A growth team can create a realistic variation, expose it to a controlled audience, and measure whether it changes customer behavior. If the experiment performs poorly, the company avoids investing in a larger implementation. If it performs well, the team has evidence that can justify and guide the permanent build.

Not every experiment can or should be implemented through a visual editor. Changes involving application logic, backend systems, authentication, pricing calculations, or complex product behavior will still require engineering.

But even in those cases, a client-side test may help validate the customer-facing premise before the company commits to the full investment.

‍

Turn experimentation into a fun part of your work

Growth and marketing teams should not need to choose between moving quickly and running rigorous experiments. But more than that, GrowthBook’s AI Visual Editor allows your team to express their creativity while measuring quantifiable results. 

Because the goal is not simply to change the website faster.

It is to learn what works, and have fun while doing it. The new AI Visual Editor is a Chrome extension. You can add it to your browser, open up the extension, connect to your GrowthBook account, and create experiments. Read the Visual Editor docs to see how it fits your setup.

‍

The Fyxer team ran 541 experiments last year with a small team. They didn’t just get faster at using their tools. They put agents in the loop to run the workflow itself.

This is the workflow they shared with us: someone fills out a form with a hypothesis and target metrics. This triggers an agent to create a flag and experiment in GrowthBook and publish to a company-wide experiment index. Someone starts the experiment with a “launch” message in Slack. Only 25% of their experiments won, but that fast iteration and learning loop helped drive ARR growth from $1M to $35M.

Fyxer wired that together themselves. GrowthBook 5.0 makes this agentic experience first-class and accessible for more users and teams natively. Now, agents can operate GrowthBook: create flags, set targeting rules, brainstorm and create and launch experiments, with a consistent experience across interfaces.

How agents operate GrowthBook

Agents can now run GrowthBook from the places you work. Picture one experiment over the course of a week.

You start a morning in Claude Desktop brainstorming your next experiment. The GrowthBook skills ground your ideas in your own history: what you’ve run, what succeeded, and how metrics moved.

You pick a direction, switch to Cursor and build it. Your agent creates the feature flag, sets up the variations, and configures the experiment alongside the code changes. Our skills carry guardrails, so the experiment lands as a draft for a teammate to review rather than a live test. Review, launch, and ship it.

Later in the week, you’ve got results. You open the GrowthBook app and ask the new AI Assistant how the test did and whether you should ship. You want to dig deeper into how feature usage differs across segments, so you work with the AI analyst to build out a dashboard to share with the team.

You change tools, but the platform and the experience stay the same.

We built this consistency by building reusable foundation elements:

• 25 open-source skills that teach an agent how to use GrowthBook effectively across flags, experiments, and product analytics. 
• A CLI rebuilt with Speakeasy that covers 100% of our REST API and regenerates automatically from our OpenAPI spec. It won’t drift, and it returns typed output with errors an agent can use effectively.
• A comprehensive REST API underneath that fuels the skills and the CLI. If you’d rather see it than read about it, watch here.

This foundation powers new ways you and your agents can run GrowthBook: the in-app AI Assistant, a reworked MCP server that integrates the skills, and a native Slack app (coming soon!).

How to trust your agents

When agents are part of your team, they should follow your team’s standards and processes, just like anyone else shipping flags or experiments. Agents can make mistakes, but we’ve built our skills and Assistant so that agent-driven changes land as drafts and a person reviews before anything publishes. The same approval policies, audit trails, and guardrails to catch bad flags before they ship apply to both humans and agents. Your team maintains judgment and governance.

We’ll dive more into the improvements we’ve made to governance, especially for features flags, on Thursday.

One platform for humans and agents

Whether you and your agents are in the terminal, the editor, or the app, you’re working on the same GrowthBook platform, with your flags, experiments, and warehouse data. We’ll keep evolving interfaces; you should never feel like you’ve outgrown the platform.

Connect your agent and run your first skill.

‍

Sign up and join us for a live Office Hours Thursday, July 30, at 9am PT and tell us what to build next.

‍Tomorrow: the new AI Visual Editor, rebuilt from scratch to make experimentation easier in the browser.

GrowthBook 5.0 ships today

It’s not enough to write code and ship at the speed of AI. Companies that win consistently make their products better. GrowthBook 5.0 enables companies to build, ship, and improve their product at any scale.

Our last major release (v4) shipped a year ago. Since then, we’ve rewritten feature flags from scratch, built Product Analytics, made experimentation significantly faster, and integrated AI throughout.

GrowthBook 5.0 crystallizes the last year of work, and it’s big enough that a minor version number just wouldn’t do. We've touched nearly every corner of the platform, from how experiments are built to how flags are governed. And we’ve opened experimentation to your whole team, not just engineers. More people can ship flags, run tests, and make decisions from shared data, wherever they work: in the app, terminal, code editor, or browser.

There’s far more to cover than we can fit into one post, so we’re covering something new every day, all week:

• Monday: Agents. Official skills, a new in-app assistant, and a CLI that covers our entire API. Your agents can operate GrowthBook directly: ship flags, launch experiments, analyze results, and clean up after.
• Tuesday: The new AI Visual Editor. Rebuilt from scratch, AI-native experimentation from your browser, with image generation and Figma support. Your growth team can ship tests without waiting on an engineer.
• Wednesday: Product Analytics hits GA. Warehouse-native product analytics for engineers and experimenters, out of beta. Built on the same metrics your experiments already use, now with funnels, composable dashboards, and experiment meta-analysis.‍
• Thursday: Governance for feature flags. Customizable guardrails that catch bad flags before they ship, protecting rollouts whether they come from humans or AI agents.‍
• Friday: Faster experiments. Streamlined experiment setup, scheduled starts, and faster, cheaper warehouse queries. Designed for running experiments at scale.

GrowthBook 5.0 is live on Cloud now and available for self-hosted deployments. Check out the full release notes.

Sign up and join us for a live Office Hours on Thursday, July 30, at 9am PT. Bring your questions and tell us what we should build next. 

GrowthBook recently hosted a webinar with two people who have spent their careers on one hard problem: how do you know an A/B test result is real? Ronny Kohavi is the co-author of Trustworthy Online Controlled Experiments and previously led experimentation at Amazon, Microsoft (where his teams eventually ran about 300 new treatments every workday), and Airbnb. Luke Sonnet is head of experimentation at GrowthBook, where he leads the team building the statistics, metrics, and analysis behind the platform.

The theme was simple and a little uncomfortable: getting numbers is easy, getting numbers you can trust is hard. That line comes from an article Ronny co-wrote in 2010, and it still describes most published test results. Running an A/B test looks trivial. You ship the change, read the p-value, celebrate the lift. Underneath sit a dozen gotchas that quietly turn a "win" into noise.

This post distills what we took away: how easy it is to run an untrustworthy experiment, the steps that can improve trust during setup, the checks to run after it finishes, and the best questions from the live Q&A. You can watch the full webinar here or download the slides here.

1. It's surprisingly easy to run an untrustworthy experiment

Ronny opened with a John F. Kennedy line: we choose to go to the moon "not because it is easy, but because it is hard." His twist on it describes what he sees constantly: too many teams run experiments not because it is easy, but because they thought it would be easy.

To make it concrete, he dissected a real, recently published experiment. The claims were impressive. A 44.8% lift, with a conversion rate jumping from about 55.7% to roughly 80%. "99% confidence," which the author defined as being 99% certain the result reflected a genuine difference and not a statistical accident. A 50/50 split with 54,000 users per variant. And an AI-powered treatment that, in the write-up, "smashed it out of the park." It was, as Ronny put it, buzzword compliant.

Then he took it apart. Four things were wrong, and every one of them is common:

• The real sample was tiny. The 54,000 "users per variant" counted everyone who entered the site. Only a small fraction actually reached the page that changed. That triggered population, the only one that matters for the test, was about 300 users total, split 176 and 124.
• No power calculation. With those numbers, the test had roughly 7% power instead of the standard 80%. In a low-power test, a significant result is almost guaranteed to exaggerate the true effect, here by an expected five times, and there was nearly a 10% chance the result pointed in the wrong direction entirely.
• The p-value was misread. "99% confidence" was treated as "99% chance the result is real." It is not. Accounting for a realistic win rate, the false positive risk on this result was about 87%.
• Sample ratio mismatch. A 50/50 design that produces 176 vs 124 has a p-value of about 0.003. The split being wrong was more statistically significant than the "win" itself.

The uncomfortable takeaway: this was not a uniquely bad experiment. It looks like a large share of the A/B test results published today. The rest of this post is about not being that example.

2. Steps to take during setup to make your test more trustworthy

Most trust problems are decided before a single user is bucketed. Here is what Ronny and Luke recommend building into setup.

Run a power calculation, and respect what it tells you

Power analysis gives you the minimum sample size you need. It takes four inputs: your baseline variance (or conversion rate), alpha (industry standard 0.05), power (industry standard 80%), and the minimum detectable effect, or MDE. The first three are easy. The MDE is where teams go wrong.

The MDE is the smallest effect you want to be able to detect, and reality is humbling. Real average treatment effects are small. At Bing, across tens of thousands of experiments, the average effect was rarely above 0.3%. Airbnb Search’s successful experiments improved conversion by about 0.3%. One widely cited toolkit analysis found a median lift of 0.1% across more than a thousand experiments. Ronny’s rule of thumb: never set an MDE above 5%. Anything higher is unreasonably optimistic, because results above 5% almost never happen unless the product is broken.

The catch is that small MDEs demand large samples, and many teams do not have them. That is a real trade-off, not a moral failing, but you have to make it consciously. Ronny’s own community project, Trustworthy A/B Patterns, settled on a 2% MDE because that matched the sample sizes available.

To ground it, here is roughly how many users you need to detect a 5% relative change, by baseline conversion rate:

Ronny’s blunt summary: if you only have around 10,000 users, you are nowhere unless your metric converts at around 50%.

Pick your significance threshold on purpose

Luke’s framing is that trust in an experiment is built on transparency and reliability. Statistical significance is the signal most teams lean on, and you control the threshold, often without realizing it. The default alpha of 0.05 says that if there is no real effect, 5% of experiments will look significant anyway. But why 0.05 for everything?

The better question is how costly a wrong call is. A feature that will dictate your roadmap for three years, or that needs ongoing support, deserves a stricter threshold like 0.01 or 0.001. A small, low-consequence change can tolerate a more relaxed one. The number should reflect the risk you are actually willing to take, not habit.

Plan against peeking and metric-shopping before the data can tempt you

Two of the most damaging habits are either baked in during setup or prevented there. Peeking, which means checking results and stopping the moment something looks significant, or simply running longer until it does, inflates your false positive rate. Running two weeks and then checking daily for another two weeks pushes a 5% Type 1 error rate up to about 17%, and it inflates the estimated effect size on top of that. Testing many metrics has the same effect: with one decision metric your error rate is 5%, with two it is 10%, with three about 14%.

If you know you will want to look early, plan for it. Use an alpha-spending approach that only looks at pre-set intervals, or sequential testing if you have robust enough data infrastructure to check continuously. If you genuinely need multiple decision metrics, apply a multiple-comparisons correction. The goal is not to make experimentation impossible; it is to decide the rules before the data can tempt you.

Setup checklist

• Define your OEC (overall evaluation criterion) and guardrail metrics first; they set your MDE and required sample size.
• Run a power calculation. Do not skip it because the MDE is hard to pick.
• Set your MDE at or below 5%, and lower if your sample allows.
• Size for the triggered population that actually sees the change, not total site traffic.
• Choose alpha based on how costly a wrong decision would be.
• Commit to a fixed runtime or sample size up front, and write it down.
• Aim for a single decision metric; if you need several, plan a correction now.
• If you are underpowered, try variance reduction such as CUPED before concluding you cannot test.

3. What to check after you've run an experiment

A clean setup is not enough. Before you trust a result, run these checks.

Check for sample ratio mismatch first

This was Ronny’s number one test. If you ran a 50/50 split, you should see roughly equal counts in each variant. When you do not, something is usually broken. He showed a Microsoft experiment split 50.2 / 49.8, which sounds close enough, but with large samples that deviation should occur only about 1 in 500,000 times. Since you did not run 500,000 experiments, you have a problem.

And SRM is not rare. Microsoft found 6% of its experiments had one, years into a mature program, measured against a strict 0.001 threshold. Convert.com found 6.5% after adding the check. One company Ronny consulted for had SRM in 20% of its experiments.

Why it matters so much: an SRM usually means a skewed population snuck in. He showed a Bing experiment with gorgeous results, five key metrics all up, some p-values as small as 2e-10. But the split was 0.497 instead of 0.5. The cause turned out to be a bot. Once they excluded it, none of the five metrics was significant anymore.

Back to the published example: 176 vs 124 on a 50/50 design gives a p-value of about 0.003, so the mismatch was more significant than the reported result. If your tool does not run a sample ratio mismatch check automatically, it is a quick p-value calculation, and it is worth doing at the end of every experiment.

Compute false positive risk, not just the p-value

"99% confidence" does not mean a 99% chance the result is real. A p-value is a conditional probability: it assumes the null hypothesis is true. What you actually want is the false positive risk, the chance that a statistically significant result is a false positive. That needs one extra input: the prior probability that an idea succeeds. Historical win rates give you that prior.

At a median success rate and alpha 0.05, you do not have 95% confidence, you have about 22% false positive risk, so roughly 78% confidence. Ship at a looser 0.10, as some tools default to, and more than a third (36%) of your "significant" results are false positives. If you do not know your win rate, Ronny suggests assuming 10%. Applied to the published example, the false positive risk was about 87%.

Apply Twyman’s Law to anything that looks great

"Any figure that looks interesting or different is usually wrong." When you see a beautiful lift, the instinct is to email the whole company. Do not. Double- and triple-check it first, because extreme results are usually bugs. Ronny has never seen a trustworthy experiment move a real OEC by 44%. It does not happen unless the product is broken, like a checkout that literally cannot complete.

Run the background sanity checks

• A/A tests. Ronny’s top recommendation. Split users into two identical groups with no difference between them. If the system is healthy, a given metric should be significant only about 5% of the time. Better still, run many A/A tests and confirm the p-values are roughly uniform; deviations expose bugs in randomization, variance estimation, or the pipeline.
• Bot traffic. Most teams underestimate it. At Bing, 50% of US traffic was bot-generated, and 90% in Russia and China. Yours is probably lower, but bots create skew and SRMs, so filter them.
• Novelty and primacy effects. If the treatment effect drifts up or down over time, it may be users reacting to newness rather than a durable effect. It is less common than people fear, since early trends are often just noise, but it is worth watching.

Post-run checklist

• Run the SRM check and treat a failing one as a red card, not a footnote.
• Compute false positive risk using your real win rate, not just the p-value.
• Do not peek or extend runtime just to reach significance.
• Do not cherry-pick whichever metric happened to turn out significant.
• Apply Twyman’s Law to any unusually large result.
• Confirm A/A tests, bot filtering, and novelty/primacy checks are in place.

4. Frequently asked questions from the webinar

We are a B2B product with low user counts and are always underpowered. Should we just not A/B test?

No. B2B is genuinely harder; even Microsoft struggled to power tests on products like Office when randomizing by company. But you have levers. Use CUPED variance reduction, which exploits historical data and pays off especially when customers are repeat visitors: it cut required sample size by roughly 50% at Bing and 5 to 10 times at Airbnb. Consider larger MDEs, since B2B changes are often bigger, and accept a longer runtime. Luke’s higher-level point is to fit experiments into your broader information ecosystem. Sometimes the honest goal is just to rule out that you are cratering a key flow, which you can do with a one-tailed non-inferiority test instead of pretending you have precision you do not.

How do we keep the winner’s curse from burning us?

Use holdouts. Combine a batch of shipped tests and compare their summed predicted effect against a long-running holdout. If your experiments claimed +10% conversion for the quarter but the holdout shows +5%, that gap tells you how much you are fooling yourself.

Is Bayesian A/B testing affected by low power? Would it change the 300-user example?

Not really. You still only have 300 users. Bayesian methods let you bring in prior information, but with uninformative priors the results align closely with frequentist ones. Informative priors only help if you genuinely understand the domain, and in online experiments, believing you have strong, correct priors is usually misleading. Notably, GrowthBook uses weak informative priors over the lift, which actually widens the uncertainty bounds and makes a 44% lift on 300 users harder to claim, exactly the Twyman’s-Law guardrail you want.

How do we handle ratio metrics (like views per user) when randomization is at the user level?

Use the Delta method to correct the variance for metrics whose denominator is not the randomization unit. Good tools handle this automatically in both the analysis and the power calculation; GrowthBook, for example, uses your historical data to size ratio metrics correctly.

If one arm wins on every single day of the week, is that extra evidence it is a real winner?

A little, but do not invent new rules like "three winning days means ship." Trust the aggregate and its p-value. The more useful version of this habit is the reverse: if an experiment is positive every day and then sharply negative one day, investigate that day for an outage or bug, but only exclude it if you find a real, documented cause.

In low-power settings, can we use more sensitive proxy metrics, like add-to-cart instead of revenue?

Yes, if you have a sound mental model of how the proxy relates to the real goal. Both Ronny and Luke are fans of validated surrogate metrics; Ronny’s teams ran experiments specifically to confirm a surrogate’s causal link before trusting it. Just keep an eye on the downstream metric so you are not optimizing add-to-cart while customers fall off later.

5. Conclusion

The theme running through the webinar: A/B tests are the gold standard, but trust does not come free. It comes from effort, from the sanity checks, the power calculations, and the discipline to honor the threshold you set. The published "44.8% lift" failed four independent checks at once, and it is representative, not exceptional.

Run the setup checklist before you launch and the post-run checklist before you ship, and you will catch the overwhelming majority of untrustworthy results before they reach a roadmap. If you want a platform that runs SRM checks, CUPED, sequential testing, and proper ratio-metric handling out of the box, you can start experimenting in GrowthBook. And to hear the full discussion, including Ronny’s worked calculations, watch the webinar recording or explore his A/B testing courses on Maven.

‍

LaunchDarkly is the enterprise default for feature management, and for good reason. It offers mature governance, broad SDK coverage across 25+ languages, progressive delivery controls, and since the Highlight acquisition in 2025, an observability layer for release monitoring. 

The platform was built for release governance, and it excels there. But here are a few areas that make many users report an issue with:

• Usage-based pricing that increases as you grow. So, you pay per service connection and per client-side monthly active user, and teams regularly report costs doubling at renewal with little visibility into what drove the increase.
• The experimentation product is sold as a separate paid add-on with its own metering at $3 per 1,000 client-side MAUs, in addition to your existing feature management tier.
• There’s no self-hosting option for the control plane. This is a problem for engineering organizations with strict data-residency requirements or air-gapped environments.
• There are documented reliability concerns, including the October 2025 outage that affected flag evaluation for multiple customers.
• It’s closed source with no way to audit the stats engine or fork the code if the vendor relationship changes.

A recent Reddit thread drew dozens of engineers making this exact point. The tool that simplified releases has become its own source of vendor lock-in.

If you’re looking for a LaunchDarkly alternative, this guide walks you through 7 platforms that include a combination of feature management, experimentation, analytics, and deployment governance.

What is LaunchDarkly?

LaunchDarkly is a proprietary enterprise feature management platform that’s now specifically catering to companies adopting AI-native development processes. It was established in 2014 and was one of the first tools to give engineering teams a way to separate deployments from releases at scale. Since then, though, it has expanded well beyond that original scope through acquisitions.

The platform covers the core feature management workflow and, as of 2026, also spans observability and analytics. You get:

• Boolean and multivariate feature flags with percentage-based rollouts
• User targeting and segmentation by attributes and custom contexts
• Approval workflows, audit logs, role-based access controls, and environment-level permissions
• Progressive delivery with kill switches and flag scheduling
• Experimentation via a paid add-on with its own stats engine
• Warehouse-native analytics via the Houseware acquisition (February 2025), currently Snowflake-only
• Session replay and error monitoring via the Highlight acquisition (April 2025)

The Highlight acquisition was one of the reasons the company repositioned LaunchDarkly around what the company calls “Runtime control for AI-era software.” In practice, the platform now spans feature flags, experimentation, analytics, and observability. But all of those products are billed separately, which balloons the overall cost of ownership. It has four pricing plans:

• The free Developer tier caps you at 5 service connections and 1,000 monthly active users. 
• Paid plans start with Foundation, priced per service connection plus per client-side MAU, with experimentation billed separately at $3 per 1,000 client-side MAUs. 
• Enterprise and Guardian tiers use custom pricing for advanced security and governance features.

When it comes to deployment, it’s SaaS-only with no self-hosted control plane. The Relay Proxy lets you cache flag evaluations locally, but the management UI, targeting rules, and experiment configuration all depend on LaunchDarkly’s infrastructure.

As of July 2026, G2 users rate it 4.5/5 based on 740+ reviews.

It’s built for large engineering organizations with complex release pipelines and strict compliance requirements. But if you need experimentation, analytics, governance, and feature flags working together without separate billing for each, you’ll want a platform that integrates these capabilities from the beginning.

Why engineering and product teams look for LaunchDarkly alternatives

Even though LaunchDarkly does feature management well, engineering and product teams start evaluating alternatives when issues start arising after the flag goes live. Here’s what happens:

• The usage-based pricing model is unpredictable: LaunchDarkly bills per service connection on the server side and per monthly active user on the client side. Many LaunchDarkly users say that the costs frequently spike at renewal. In fact, one user said their annual contract would increase from $10,000 to $45,000 under the new pricing model. These billing surprises come too late, when you’re already locked in, and migrating can be a huge hassle. Many teams find LaunchDarkly to be too expensive and end up looking for a cheaper alternative. 
• Experimentation sold as a paid add-on: The experimentation module isn’t bundled with feature flags. It’s a separate product with separate billing. And it’s metered layer at $3 per 1,000 MAUs. The stats engine offers Bayesian and frequentist methods, but percentile analysis is still in beta and incompatible with CUPED. Also, funnel metrics are limited to average analysis only. If you want to run experiments with the rigor you’d apply to product analytics, you’re paying extra for a module that doesn’t fully deliver it.
• The stats engine is a black box: You can see experiment results within the platform, but you can’t reproduce the statistical calculations independently or validate them in your own warehouse. They don’t publish their methodology, so there’s no way to audit how results are computed. For data teams that need to verify outcomes before making product decisions, this is a significant gap.
• No self-hosting option: The control plane is SaaS-only. That means your targeting rules, experiment configuration, user segments, and the management dashboard all run on LaunchDarkly’s infrastructure. If you’re in a regulated industry or need an air-gapped environment, this won’t work because it poses a significant security and compliance risk.
• Reliability tied to vendor uptime: LaunchDarkly has logged over 800 tracked outages since November 2019. The October 2025 incident affected approximately 99% of server-side SDKs globally for 24 hours. Even though the Relay Proxy mitigates network dependency, it adds operational complexity, and you’re still dependent on the vendor for configuration updates. The new updates have also resulted in a more unstable version of the app, forcing users to look for more reliable alternatives.
• Warehouse-native capabilities limited to Snowflake: The Houseware acquisition added warehouse-native experimentation, but it’s currently restricted to Snowflake and requires high-level account permissions to set up. The platform-managed metrics can fall out of sync with your warehouse data, creating discrepancies between what LaunchDarkly reports and what your data team sees.
• Complex targeting that requires cross-team coordination: LaunchDarkly’s multi-context targeting model requires upfront schema design and SDK-level changes. If you’re adding a new targeting rule, you’ll need to coordinate across engineering teams, and you can run only one active experiment per feature flag without workarounds. This issue limits how quickly you can iterate.
• No native way to measure rollout impact: The platform controls how features ship, but doesn’t connect the rollout to an actual result. You’ll need additional tools to actually see if the feature flag and its associated rollout made a difference to business-related metrics.
• Closed source with high switching costs: LaunchDarkly SDKs, which are roughly twice the size of most competitors’, embed deep into your codebase across services. It takes months to migrate from LaunchDarkly, and there’s no way to run a self-hosted fallback during the transition. The code is proprietary, so you can’t fork it or audit the internals.

What to look for in a LaunchDarkly alternative

Here are a few things you need to look at before choosing an alternative to LaunchDarkly:

• Breadth versus depth: Do you need a single-purpose feature management platform, or an all-in-one tool that bundles experimentation, analytics, and session replay? A dedicated platform will go deeper into flag governance and release safety, but an all-in-one platform will consolidate your tool stack. However, the latter could treat any single capability as one product among many, so the product you need may not be as robust.
• Pricing model predictability: LaunchDarkly bills per service connection and per client-side MAU, with experimentation adding another metered layer. While per-seat pricing scales with your team size, per-MAU and per-event pricing scale with your traffic. You really need to look at your traffic patterns and team size to see which model makes the most sense for your organization.
• Built-in experimentation with a transparent stats engine: Some platforms include experimentation as a core capability with an open, auditable statistical methodology. If your data team needs to reproduce and verify results, the difference matters because it decides how credible your experiments are—especially if you’re making product-related decisions.
• Self-hosting and open source: A full self-hosted deployment gives you control over data residency, air-gapped environments, and vendor independence. But an open-source license (MIT or Apache 2.0) adds another layer of transparency and auditability. In this case, you can audit the code, fork it if the vendor relationship changes, and contribute to the product. This gives you more space to comply with regulatory requirements while building confidence in your infrastructure.
• Warehouse-native measurement: If your data already lives in Snowflake, BigQuery, or Redshift, a warehouse-native platform can run experiment analysis directly against your existing tables. This keeps metrics consistent across your analytics pipeline and eliminates the need to reconcile numbers between vendor dashboards and your own data warehouse. If you do use other platforms, check if the alternative actually integrates with them.
• Depth of governance and at what tier: These days, governance features like approval workflows, audit logs, RBAC, ramp schedules with guardrails, and stale flag detection are table stakes even for mid-market teams. But most feature flagging, experimentation, and analytics platforms gate it behind enterprise tiers. So cross-check that before signing up.
• SDK coverage and evaluation architecture: Choose a platform with SDKs for every language in your stack. Evaluate the initialization workflow because some SDKs require a network call on every evaluation, which adds latency to every request. Others evaluate locally from a cached payload and stay off your hot path entirely.
• Migration path from LaunchDarkly: Switching feature flag platforms touches every service that evaluates a flag. Look for a dedicated importer that can pull your existing flags, environments, targeting rules, and rollout configurations via API. This is the part that decides whether you’ll actually spend months migrating between platforms.

Best LaunchDarkly alternatives in 2026

Here’s how different alternatives to LaunchDarkly stack up against each other:

1. GrowthBook

GrowthBook is an open-source platform that pairs enterprise-grade feature flags with built-in, warehouse-native experimentation and product analytics. It also offers full self-hosting so you can stay compliant with different regulations in your industry.

You can think of GrowthBook as three products inside one MIT-licensed codebase: 

• A fast feature flag engine with 24+ SDKs and sub-millisecond local evaluation
• A production-grade experimentation engine that runs SQL against the warehouse you already use. 
• A strong product analytics product that acts as a managed warehouse for your product data.

All your feature flags evaluate from a locally cached payload, so your application never depends on GrowthBook being available. That’s how companies like Dropbox process over 3 billion flag evaluations daily on self-hosted GrowthBook. The platform is trusted by Khan Academy, Upstart, Breeze Airways, Mistral, and 3 of the 5 largest LLM companies in the world.

“GrowthBook gave us a modern experimentation and release platform that actually fits how Dropbox works. We can run analytics directly on our data lake, roll features out safely in stages, and support teams across different stacks without duplicating data or tooling.” 

— Alex Kalish, Engineering Manager, Dropbox

In fact, GrowthBook’s latest 4.4 release closed the governance gap that enterprise teams previously cited when comparing to LaunchDarkly. You get in-depth features like: 

• Ramp schedules to define staged rollouts with guardrail metrics at each stage and auto-rollback if those metrics degrade. 
• Approval workflows now cover environment kill switches, prerequisites, saved groups, and metadata changes. 
• The rebuilt API exposes over 250 Zod-driven endpoints
• The MCP server makes the full experiment and flag lifecycle programmable for tools like Claude Code and Cursor.

In short: GrowthBook doesn’t treat feature flagging, experimentation, or product analytics as three separate modules—but rather as one platform to make better product decisions.

GrowthBook key features

Pros of GrowthBook

• Create and manage feature flags, experiments, and rollout plans directly from Claude Code or Cursor through the MCP server and the rebuilt REST API.
• Smart feature flags with ramp schedules, per-stage guardrail metrics, and auto-rollback connect to any metric in your data warehouse. You can ship features at velocity and verify they’re working without switching tools or waiting for a separate analysis.
• Flags and experimentation live on a single platform with a single data source, so you don’t reconcile data across vendors. Several users say that these capabilities being available on a single platform make it a much more affordable option.
• The lightest-weight SDKs in the category evaluate locally with zero network calls, so your application never depends on GrowthBook’s availability.
• Per-seat pricing with unlimited traffic and experiments means your bill stays predictable as your product scales. GrowthBook costs roughly 1/5th the cost of LaunchDarkly for comparable deployments.
• The stats engine is fully open source, and every calculation is reproducible via SQL. Your data team can audit the math on GitHub rather than trusting a vendor’s black box.
• Even though it has its own experimentation product, you can still import data from other tools and analyze it in GrowthBook.

Drawbacks of GrowthBook

• Full experimentation value depends on having a data warehouse. GrowthBook Cloud now offers a Managed Warehouse, but self-hosters need to bring their own.
• Fewer federal-specific compliance certifications compared to LaunchDarkly. Even though it has certifications such as SOC 2 Type II and ISO 27001, it’s not FedRAMP or ISO 27701-certified.
• No Terraform provider for infrastructure-as-code workflows.
• Narrower niche SDK coverage. LaunchDarkly supports Haskell, Erlang, and Apex, in addition to the standard set. GrowthBook’s 24+ SDKs cover most platforms but don’t support these legacy runtimes.

How GrowthBook compares to LaunchDarkly

Both platforms handle enterprise feature management, but they’re built on different assumptions about what happens after the flag goes live:

• GrowthBook wins for teams that want enterprise-grade feature flags with predictable per-seat pricing, open source and full self-hosting, lighter SDKs with zero-network-call evaluation, and built-in experimentation measured in their own warehouse—at roughly 1/5th the cost. The dedicated LaunchDarkly importer means most teams complete the migration the same day.
• LaunchDarkly wins for large enterprises that need the most mature release-governance ecosystem with multi-stage approvals, the broadest integration catalog (including native Terraform and ServiceNow), specific federal compliance certifications like FedRAMP, and are comfortable with cloud-only, usage-based pricing.

See a full side by side comparison of GrowthBook vs LaunchDarkly.

Who is GrowthBook best for?

Engineering, product management, and data teams from any kind of company, whether it’s a startup or an enterprise. These companies typically want LaunchDarkly-grade feature flag management plus real experimentation without per-MAU pricing or vendor lock-in. It’s a particularly strong fit for companies in regulated industries like fintech, healthtech, and AI software, where data sovereignty and self-hosting are non-negotiable.

2. PostHog

PostHog is an all-in-one, MIT-licensed product platform that bundles product analytics, feature flags, A/B testing, session replay, error tracking, surveys, and heatmaps into a single codebase.

The platform’s main focus is on consolidation, so instead of stitching together LaunchDarkly for flags, Amplitude for analytics, and FullStory for session replays, you get all of it from one vendor with a shared event model. Considering this, the pricing model is quite generous and usage-based, and in fact, 90% of PostHog users don’t pay for the platform. 

But where it actually differs from LaunchDarkly is the scope of the product. While LaunchDarkly was built for feature management, PostHog was built analytics-first—and it shows. So, you might get robust analytics data, but you won’t get the feature-flagging depth that the former offers.

PostHog key features

Pros of PostHog

• Analytics, flags, experiments, session replay, error tracking, surveys, heatmaps, and a CDP in one tool with a shared event model. You don’t reconcile data across vendors because everything runs on the same data layer.
• The free tier includes 1M analytics events, 5K session recordings, and 1M feature flag requests per month with no credit card required. Most small teams never hit the paid threshold.
• The core codebase is MIT-licensed and self-hostable for full data control, though the infrastructure footprint is substantially larger than that of lighter-weight alternatives.
• You can go from a funnel drop-off to the exact session recording that shows what happened, without switching tools. In fact, even error tracking becomes easier because of the qualitative data at hand.
• Many users also say it’s quite easy to install and get started with, even if you’re a non-technical user.

Drawbacks of PostHog

• The experimentation engine is not as advanced as GrowthBook or LaunchDarkly. For example, it doesn’t offer sequential testing, CUPED variance reduction, post-stratification, or SRM detection. You can’t add metrics retroactively after an experiment starts.
• PostHog analyzes experiment results inside its own platform, not in your data warehouse. Results aren’t reproducible via SQL against your own data, and you can’t audit the calculations.
• Growth and marketing teams can’t run headline tests or CTA changes without a visual editor — every experiment requires code.
• Every experiment evaluation counts as a feature flag request, which means high-traffic experiments directly increase your bill. You’ll need to plan your event-tracking strategy carefully to avoid it.
• Self-hosting requires ClickHouse, Kafka, PostgreSQL, and Redis at a minimum, along with 4 vCPU, 16GB RAM, and 30 GB+ of storage. Kubernetes deployments are no longer officially supported for new installs. The company actively discourages self-hosting for that reason.
• The web app tends to use too much memory, which can slow down your browser/computer while you use it.

How PostHog compares to LaunchDarkly

PostHog and LaunchDarkly solve different problems. Here’s how they compare with each other:

• PostHog wins for product teams at startups and mid-stage companies that want one vendor for analytics, flags, session replay, and experimentation. And this is especially true if the free tier covers their volume and they value MIT-licensed source code.
• LaunchDarkly wins for mid-sized and enterprise teams that need multi-stage approval workflows, the broadest integration catalog, federal compliance certifications, and deep feature management. LaunchDarkly also has stronger enterprise governance controls that PostHog simply doesn’t offer right now.

Who is PostHog best for?

Product teams at startups and growth-stage companies want to consolidate their analytics, feature flag, session replay, and experimentation tools into a single platform. If you’re currently paying for Amplitude plus LaunchDarkly plus FullStory and want one vendor instead, PostHog is designed for that tradeoff. It’s less suited for teams that need deep, warehouse-native experimentation or enterprise-grade release governance.

3. Statsig

Statsig is a product experimentation platform founded by former Facebook VP of Engineering Vijaye Raji. The platform bundles feature flags, A/B testing with advanced statistics, product analytics, and session replay.

While it’s known for its excellent experimentation capabilities, its recent acquisitions have brought that into question. In September 2025, it was acquired by OpenAI, but by May 2026, Amplitude took over the Statsig brand. Essentially, Amplitude takes over the code but not the team behind it. It has not yet been clarified how Statsig’s warehouse-native architecture will coexist with Amplitude’s event-stream roots, or what the pricing will be at renewal.

Statsig key features

Pros of Statsig

• It offers various analysis methods, such as CUPED, sequential testing, stratified sampling, switchback experiments, mutual exclusion layers, and holdout groups. Before the acquisition, this was one of the most statistically rigorous experimentation platforms.
• Enterprise customers can run experiment analysis directly in their own BigQuery, Snowflake, Databricks, or Redshift instance through warehouse-native mode. Data never leaves your infrastructure.
• The free Developer tier includes 2M metered events per month, unlimited seats, and 50K session replays with no credit card required. Analytics and experimentation are both included.
• You can define customer user dimensions, which makes it easier to slice and dice data as you see fit.

Drawbacks of Statsig

• The engineering team that built Statsig is at OpenAI. Amplitude inherited the code, customers, and brand — but not the people. How quickly Amplitude can staff up, learn the codebase, and ship improvements is an open question that every buyer should pressure-test before signing a multi-year contract.
• The platform is entirely proprietary and cloud-only. You cannot audit the statistical engine or run the platform on your own infrastructure.
• Some users report that the platform can feel overwhelming at first, and it takes time to set up (typically days to weeks).
• Feature flagging capabilities are best suited to experimentation use cases rather than launching new features. Statsig’s bread and butter is A/B testing.
• Statsig’s feature flags auto-capture a lot of metrics, which is useful for product analytics, but can cause Statsig’s event-based pricing to increase dramatically, even if the customer isn’t using Statsig product analytics.
• Event-based metering means costs grow with traffic. So, higher-volume applications can see bills increase faster than expected, and you’ll need to jump between plans to keep up.

How Statsig compares to LaunchDarkly

Statsig and LaunchDarkly overlap on feature flags but differ in the depth of experimentation. Here’s how you can choose:

• Statsig wins (or won, pre-acquisition) for teams that want advanced experimentation bundled with feature flags and analytics at a lower entry price than LaunchDarkly’s experimentation add-on.
• LaunchDarkly wins for enterprise teams that prioritize platform stability, the broadest compliance certifications, and mature release governance. 

Who is Statsig best for?

Teams that already use Statsig and are evaluating whether to continue using it through the Amplitude transition. For new buyers, the calculus is different: the experimentation engine is strong, but signing a new contract with a platform mid-ownership change requires confidence that Amplitude will maintain the quality and roadmap that attracted customers in the first place. If you need warehouse-native experimentation from an independent vendor, GrowthBook is a much more stable long-term bet.

4. Optimizely

Optimizely is an enterprise digital experience platform (DXP) that bundles several features under the Optimizely One brand. Some of them include:

• A/B testing
• Feature flags
• Content Management System
• Commerce
• Content marketing
• AI-powered personalization 

It was founded in 2009 as a pure-play A/B testing tool. The company was acquired by Swedish CMS vendor Episerver in 2020, which rebranded itself as Optimizely in January 2021. That’s how it moves on from being a pure experimentation platform to “powering” digital experiences.  The pricing is quite comparable to LaunchDarkly, as they cater to enterprises. In fact, the average contract value is $82,894, according to public data.

But if you’re evaluating Optimizely specifically as a LaunchDarkly alternative for feature flags, you’re buying a full DXP to solve a feature management problem. 

Optimizely key features

Pros of Optimizely

• Several users say it’s not that hard to set up feature flagging for experimentation, even though implementing the full suite takes months.
• The Web Experimentation product gives marketing teams a no-code visual editor for client-side A/B tests, so you don’t have to write code.
• You don’t need extensive technical knowledge to set up and run experiments within the platform.
• Warehouse-native analytics connect experiment results to Snowflake, BigQuery, Databricks, and Redshift, enabling you to tie experiments to downstream metrics such as LTV and subscription renewals.
• For enterprises already using Optimizely’s CMS or Commerce Cloud, adding experimentation means having a single vendor for content, commerce, and testing, with shared audience definitions.

Drawbacks of Optimizely

• While you can use the feature experimentation platform to define experiments and run them, integrating in-house metrics can be particularly problematic. 
• There’s no free tier or even a trial version you can demo before using the product. There’s no way to evaluate the costs without talking to sales—and users report it’s too expensive for what it is.
• There’s no open-source option and no self-hosting. The platform is entirely proprietary and cloud-hosted, so if you have data-residency requirements, you can’t use it.
• When you add more audiences or handle a large audience in the experimentation platform, the front end suffers. The app slows down because it has to load too many elements and doesn’t render quickly. And it’s also difficult to track the specifics later.
• Its flag management tooling is weaker than dedicated feature management platforms. You don’t have capabilities like an overview of running versus stale flags, custom tagging or labeling, insights into flags running too long, or native Slack integration for flag notifications.
• SDK coverage is narrower at 12 SDKs compared to LaunchDarkly’s 25+ or GrowthBook’s 24+. There’s no Haskell, Erlang, Rust, Elixir, or edge runtime support.
• No automated rollback tied to health metrics. While LaunchDarkly offers Guarded Rollouts with circuit-breaker-style automatic rollback, GrowthBook offers ramp schedules with automatic rollback—but Optimizely offers neither.

How Optimizely compares to LaunchDarkly

Both these platforms were made for enterprise companies, but they serve different buyers. Let’s see how they compare:

• Optimizely wins for enterprises that need a no-code visual editor for marketing-led experiments, want CMS and commerce capabilities alongside experimentation, and have a mid five-figure USD budget for multi-year contracts. The experimentation stats engine is also more methodologically complete than LaunchDarkly’s add-on.
• LaunchDarkly wins on feature management depth, SDK breadth, automated rollback with Guarded Rollouts, enterprise governance maturity, and federal compliance certifications. It’s also substantially less expensive for teams that only need feature flags.

Who is Optimizely best for?

Large enterprises with huge annual budgets that need marketing-led web experimentation alongside server-side feature flags—especially if they already use Optimizely’s CMS or Commerce Cloud. If you’re evaluating it purely as a LaunchDarkly alternative for feature management, it’s dramatically over-scoped and overpriced for that single use case.

5. Split (Harness)

Split (by Harness) is a feature management and experimentation platform that Harness acquired in June 2024 and now sells as Harness Feature Management and Experimentation (FME). Its original pitch focused on being a “Feature Data Platform” in which every toggle corresponds to business and engineering metrics. But over time, the platform has changed completely.

Despite being a strong feature flagging and experimentation platform, Split is no longer a standalone platform. It sits inside Harness’s broader DevOps platform alongside CI/CD, security testing, cloud cost management, and chaos engineering. FME is only available on the Enterprise tier, and you need to contact sales to find out the actual cost. For teams that only need feature flags and experimentation, engaging with the full Harness platform feels like overkill.

Split (Harness) key features

Pros of Split (Harness)

• Automated impact measurement ties every flag toggle to business and engineering metrics without requiring manual experiment setup. You can toggle a flag and immediately see whether it moved conversion rates or error rates with statistical significance calculated automatically.
• Since it offers several built-in connectors and a broader set of CI/CD and security features, enterprises tend to get more value from a single platform.
• It offers extensive cost-optimization features powered by AI, so you can easily save on cloud costs.
• You also get detailed reports on feature management and rollouts so that you can communicate them to different stakeholders with ease.
• You can create an entire CI/CD pipeline without writing any code, which is useful as teams move to more AI-native development processes.

Drawbacks of Split (Harness)

• FME is only available on the Harness Enterprise tier. There’s no way to buy it standalone, no public pricing, and the sales motion is a full platform engagement. For teams that only need feature flags and experimentation, this is a heavy procurement process for a focused use case.
• There’s no open-source version or self-hosted version for FME. The Split Proxy can be deployed in your infrastructure for latency and caching, but the control plane and analytics remain in Harness’s cloud.
• The integration ecosystem is much narrower than LaunchDarkly’s. You get a few primary integrations like Datadog, Jira Cloud, New Relic, and Sumo Logic. But other connections require custom webhooks or Zapier.
• It doesn’t offer advanced feature management capabilities, such as CUPED variance reduction and scheduled rollouts. And there’s no easy way to revert flag changes to a previous state.
• There’s no built-in bandit optimization for automatically shifting traffic, and you have to do a custom build for similar optimization techniques.

How Split (Harness) compares to LaunchDarkly

These platforms were built on different product philosophies. While Split was built around automated metric attribution, LaunchDarkly was built around release governance. That’s why:

• Split (Harness) wins for teams already on the Harness platform that want feature flags integrated into their CI/CD pipelines, and for organizations that value automated impact measurement without manual experiment setup.
• LaunchDarkly wins on SDK breadth, governance maturity, integration ecosystem, federal compliance certifications, and better billing structure. You can buy LaunchDarkly without buying an entire DevOps platform.

Who is Split (Harness) best for?

Teams already invested in the Harness DevOps platform that want feature flags and experimentation tightly integrated with their CI/CD pipelines. If you’re not already a Harness customer, the enterprise procurement process and platform bundling make Split a difficult choice for teams that just need feature flags and experimentation.

6. Unleash

Unleash is an open-source feature flag platform, licensed under Apache 2.0 with 13,600+ GitHub stars and 500+ contributors. It only does feature flags and enterprise governance—and nothing else. It was founded in 2019 and built its reputation on deployment flexibility and enterprise compliance. 

The platform has a very simple architecture, which consists of a stateless Node.js API server backed by PostgreSQL, deployable on Docker, Kubernetes, bare metal, or fully air-gapped networks. And the recent version includes updates such as observability metrics and an MCP server in beta to support more granular feature rollouts. But the biggest gap shows up in experimentation and product analytics. It doesn’t have any capabilities for these use cases and you’ll need external tooling to make it possible.

Also, with its recent push to move away from the open-source version, it’s pushing users towards paid plans or building their own edge layer.

Unleash key features

Pros of Unleash

• Because of the way it’s set up, you can manage a large scale of feature flags with ease, even if you’re working with multiple teams.
• It uses an API-first approach and also lets you automate provisioning and configuration, which is ideal for complex microservice architectures.
• Server-side SDKs ensure that no end-user data ever leaves your infrastructure, as the privacy architecture is built into the platform.
• Unleash claims most customers cut their LaunchDarkly bill by 75% or more. So it might be a more cost-effective solution for feature flags and governance only.
• Many users say that the documentation is excellent, and so is the customer support, as they regularly gather customer feedback and incorporate it into the product.

Drawbacks of Unleash

• Unleash’s browser and mobile SDKs are thin clients as they don’t contain targeting logic, hashing, or bucketing. All of that runs on a separate Unleash Proxy or Frontend API server that you have to deploy and maintain. If that proxy goes down, your client-side flags stop evaluating.
• It uses a polling-based architecture so your SDKs periodically fetch the latest configuration rather than receiving changes via streaming.
• OSS Edge is sunsetting December 31, 2026. After that, self-hosted open-source users lose the edge evaluation layer entirely unless they upgrade to Enterprise Edge or build their own version.
• Given the price point, the fact that you only get feature flagging with basic experimentation makes it an expensive alternative. Even features like SSO and real-time streaming (beta) are only available on the Enterprise plan.
• It doesn’t offer the capability to monitor whether a feature rollout is degrading a metric and reverse it automatically. If you need guardrail-driven rollbacks, you’ll have to use another tool like GrowthBook.

How Unleash compares to LaunchDarkly

Both Unleash and LaunchDarkly were built for enterprises and have comparable SDK breadth and governance. That said, the differences are architectural and philosophical:

• Unleash wins on cost (claims 75%+ savings), full self-hosting including air-gapped deployments, FedRAMP-ready infrastructure, and data privacy by design. If feature management is your only use case, it makes sense to use it.
• LaunchDarkly wins on experimentation (Guarded Rollouts with metric analysis vs. Unleash’s zero experimentation), integration ecosystem depth, multi-stage approval workflows, and the breadth of enterprise compliance certifications. LaunchDarkly is also an independent, publicly traded company with a stable roadmap.

Who is Unleash best for?

It’s meant for engineering teams in regulated industries like finance or government that want full deployment flexibility and don’t need product analytics or experimentation. But if you want feature flags and experimentation on a single platform, GrowthBook covers both under the same open-source license.

7. VWO

VWO (Visual Website Optimizer) is a conversion rate optimization platform that bundles A/B testing, heatmaps, session recordings, on-site surveys, and personalization into a single dashboard. Founded in 2008, it’s one of the oldest experimentation tools on the market.

VWO merged with AB Tasty in January 2026. AB Tasty brings AI-powered personalization (including Evi, an agentic AI engine that automates A/B testing workflows), and the combined roadmap is still being defined. But the pricing plans have already changed as the free plan was discontinued and new sign-ups get a 30-day trial.

It’s essentially a marketing experimentation platform that happens to have feature flags, not a feature management platform. That’s why the flag governance features are basic compared to LaunchDarkly or GrowthBook.

VWO key features

Pros of VWO

• Since it’s made specifically for marketing and CRO teams, many users say it’s quite easy to set up and run experiments. It has a mature no-code visual editor for client-side A/B testing.
• Customer support consistently receives high praise across platforms, and their response times are quite fast compared to other platforms.
• The AB Tasty merger brings AI-powered personalization (Evi engine) and a deeper European enterprise presence. So, it has actually strengthened the platform’s positioning as an A/B testing and CRO tool.
• It bundles qualitative tools like heatmaps, session recordings, form analytics, and on-site surveys with quantitative data. So CRO teams get a complete behavioral analytics toolkit alongside experimentation.
• It has 12+ SDKs and server-side feature flags via FME, enabling progressive rollouts so your basic feature flagging use cases are covered.

Drawbacks of VWO

• It only offers MTU-based pricing, with modular add-ons that scale aggressively. So much so that, post-merger, there’s no public pricing available, and average contracts sit above $16,000 per year.
• It also doesn’t offer self-hosting options, open-source infrastructure, or warehouse-native analytics, so it might not be a fit for companies in regulated industries.
• The visual editor breaks down on complex or dynamic pages with heavy JavaScript. As you add more event-based goals, it becomes buggy, and you’ll need to involve a developer.
• VWO’s client-side snippet increases load time and can cause Cumulative Layout Shift (CLS) issues, which Google penalizes in Core Web Vitals. The anti-flicker technology helps but doesn’t eliminate the fundamental performance impact.
• Feature flagging and experimentation live on two different platforms and require separate billing.

How VWO compares to LaunchDarkly

Within VWO, feature flags are merely an addition that lets you run A/B tests and similar experiments. But in LaunchDarkly, that’s not the case. That’s why:

• VWO wins for marketing-led experimentation programs that need a visual A/B testing editor, heatmaps, session recordings, and surveys alongside basic feature flags, without engineering involvement for most workflows.
• LaunchDarkly wins for engineering-led feature management, with better SDK breadth, governance depth, compliance certifications, release automation, an integration ecosystem, and flag lifecycle management.

Who is VWO best for?

Marketing and CRO teams at mid-market companies that want a complete web optimization toolkit to understand how users experience the website and how to improve it. Feature flagging is essentially a secondary capability. If you’re evaluating VWO as a LaunchDarkly alternative for engineering-led feature management, it’s the wrong tool. They solve different issues, so you’d be better off using a platform like GrowthBook, which is made for developers and engineering teams.

How to choose the right LaunchDarkly alternative for your team?

Yes, LaunchDarkly built the category, and it remains the strongest option for teams whose primary need is enterprise release governance, with the broadest SDK coverage and compliance certifications. Depending on what you need and your primary use cases, there may be a better or more cost effective solution for you. 

Here’s what we recommend based on what’s driving the switch:

• If you want open-source feature flags with built-in, warehouse-native experimentation and product analytics at predictable pricing, GrowthBook is the best choice. It covers the full flag-to-experiment-to-analysis lifecycle in a single MIT-licensed platform that runs on your infrastructure or GrowthBook Cloud.
• If you want one vendor for analytics, flags, session replay, and experimentation, PostHog consolidates the entire product tool stack under a generous free tier. But you won’t get advanced experimentation capabilities.
• If you want advanced experimentation and are already using Amplitude, consider Statsig, which offers a robust stats engine and was recently acquired by Amplitude.
• If you want marketing-led web experimentation with a visual editor, heatmaps, and session recordings, Optimizely (enterprise budget) or VWO (mid-market budget) is the way to go. But if you’re an engineering-led team or are looking for a platform for those use cases, you’re better off with GrowthBook.
• If you want enterprise flag governance with full self-hosting and don’t need experimentation, Unleash offers the simplest self-hosting architecture in the category with FedRAMP-ready infrastructure.
• If you’re already on the Harness DevOps platform, Split (Harness FME) makes the most sense since it integrates feature flags directly into CI/CD pipelines with automated impact measurement.

If you already know that factors like open-source development, self-hosted deployment, and feature flagging/experimentation are a deal-breaker for you, why not give GrowthBook a shot?

We’ve made it easy with our LaunchDarkly importer. Just follow the steps and get started for free.

‍

Most companies say they value experimentation. Far fewer have built a culture where a failed test is treated as a win. James Falzone, Director of Product Management at Kargo, leads a team that does, and on The Experimentation Edge, he made a case for why that distinction is the whole game.

🎧 Listen to the full episode →

Kargo engineers technology that helps brands connect with consumers and grow their businesses. Every day, its teams build products across agentic AI, CTV, eCommerce, social, mobile, and its OpenAI integration, giving advertisers new ways to reach audiences across premium media. When you tap a link and a page loads, the ad you see didn't get there by accident. In the split second before the content renders, a real-time auction fires. The publisher signals an opportunity, companies like Kargo bid on behalf of advertisers, and the winner places the ad, all inside roughly a thousand milliseconds. Multiply that by up to 10 billion ad requests a day, across tens of thousands of advertisers and publishers, and you start to see why Falzone says experimentation at Kargo isn't a team you visit. It's embedded in the culture because the scale and constant change of the industry require it.

A bad result is not a bad experiment

The line that anchors Falzone's philosophy is simple: "There's a difference between a bad result and a bad experiment." A bad result is the market telling you an assumption was wrong. A bad experiment is the one you were too cautious to run.

"If you're not getting those bad results, if you're not failing, are you really trying anything new?" he asked. It's a reframe worth sitting with. Most teams instinctively measure themselves by hit rate, the percentage of experiments that succeed. But a high hit rate often means a team is only testing the safe, obvious bets. The genuinely new ideas, the ones that could move the business, are exactly the ones most likely to fail at first. A perfect record isn't a sign of great experimentation. It's a sign you stopped experimenting.

Falzone is candid that his team probably learns more when it fails than when it wins. That isn't a consolation prize. It's the operating model.

The experiment that failed, and why

The clearest example came from Kargo's own bidding strategy. The team built a click optimization model: predict the likelihood of a click within five milliseconds, then adjust the real-time bid accordingly. For Kargo's direct demand, the advertisers who come straight to the company, it worked beautifully. The improvements were real and substantial.

So the team did what looked like an obvious next step. They took the same model, the same tech, the same rationale, and applied it to their third-party demand, where another party taps Kargo's inventory on an advertiser's behalf. "And we failed," Falzone said. "It did not work. Bad results across the board."

The instructive part is the diagnosis. "The bad results didn't come from technical implementation," he explained. "They came from a lack of contextual implementation." The model was sound. The context was wrong. With first-party demand, Kargo has rich visibility: the geographies a customer wants to serve, the audiences they're targeting, the dates, the budget. Third-party demand is full of idiosyncrasies, and crucially, it isn't just customers reacting; it's their models interacting with Kargo's models. Different segments react to different data and different inventory. What a customer says they want and what their models actually optimize for are not always the same thing.

The fix wasn't a better algorithm in the abstract. It was going back to the drawing board to customize the model around the verticalization of those customers, and to make sure the metrics and signals being tested were business-driven rather than borrowed. Eventually, the team recovered the 25 to 40% performance gains they'd been chasing. But the takeaway outlived the project: a result that works in one context is a hypothesis everywhere else, not a conclusion.

Putting failure on the agenda

Knowing that failure is valuable is one thing. Building a team that acts on it is another. Kargo's mechanism is refreshingly concrete. In every biweekly retro, there's a dedicated section: "Where did you fail?" Everyone takes a turn. "I tried this, it didn't work." Out loud, on the agenda, by design.

The effect is cultural, not procedural. When failure is a scheduled topic instead of a quiet embarrassment, three things happen at once. Losses get normalized, which is the precondition for taking real swings. One person's dead end becomes the whole team's shortcut. And the psychological safety that every leadership book talks about stops being aspirational and becomes a recurring calendar item.

Falzone is also honest about a tension he hasn't fully resolved. He believes deeply that a single person should be able to run an experiment without needing a committee, that autonomy and speed and a startup mentality matter. But he equally believes the best ideas come from bouncing ideas off other people. "You're a person first," he said, not a job title, and the best ideas can come from outside your function. Holding both individual speed and collective creativity is an ongoing balancing act rather than a solved problem. That he names it openly is part of the same culture that schedules failure into the retro.

Better, not bigger

Asked where experimentation goes next, Falzone did what almost everyone does now and brought up AI. But his framing was specific. AI's real unlock, he argued, is access. People who previously couldn't read the code or understand the system can now form opinions and run experiments, whether that's generating a SQL query or interrogating data they couldn't reach before. More people at the table mean more ideas and more experimentation opportunities.

He's also clear-eyed about the limits. In ad tech's latency world, where everything has to resolve inside a second, large language models simply aren't fast enough to sit in the live auction path. But they can operate at an orchestration layer, and Kargo is already thinking about agents that run tests on the team's behalf. The caution underneath the optimism is the part worth keeping: AI can't be treated as a magic bullet, because everything still has to be built on solid ML and infrastructure engineering.

His closing phrase captured the whole conversation. "Better, not bigger." More compute, more data, and more agents don't matter if they're pointed at the wrong thing. What compounds is a team that runs new experiments, welcomes the bad results, talks about them openly, and rebuilds with context. In an industry where a single mistake can burn an entire campaign budget in under a minute, that discipline isn't a nicety. It's the edge.

‍

GrowthBook’s AI Visual Editor lets anyone on your team go from idea to live experiment in minutes, without touching code. It's a way to scale experimentation beyond one team and across your whole organization.

Just describe the change you want in plain language. The editor handles the rest. No code, no dev ticket, no flicker. The result is a fully configured A/B test ready to launch from your GrowthBook account.

The AI Visual Editor is available on GrowthBook Pro and Enterprise plans.

‍

How does GrowthBook’s new AI Visual Editor work?

The AI Visual Editor is a Google Chrome extension. Open it, connect to your GrowthBook account, and immediately get started. The extension stays in the sidebar and doesn’t overlap your page.

Describe the change you want, such as “make the headline larger and move the CTA button directly beneath,” and the editor will generate a variant in seconds.

There's no code to write. The editor understands the structure of your page, applies the change, and connects it directly to a GrowthBook experiment. You prompt, then preview your changes in real time without ever leaving the browser. If you want more control, you can use the manual editor to do fine adjustments.

‍

Key features

AI-generated image creation and visual testing

The AI Visual Editor is a powerful creative tool. Describe the image you want and the editor generates it in real time, along with options you can place into the right spot on your page and size to fit exactly. Test hero images, product photos, background visuals, and more, iterating until you get what you want without a designer or a design file.

It’s a quick way to immediately see changes in context and visually brainstorm live before you build anything. Once you’re aligned on variations, publish and A/B test new designs in minutes. 

Images are stored in GrowthBook’s image storage bucket and served from a CDN-fronted URL so SDKs can fetch them efficiently when delivering variations.

‍

‍

Design updates and layout changes

Beyond text and images, the AI Visual Editor also lets you adjust fonts, colors, padding, button styles, layouts, and more all through prompts or via the manual editor.

Make design decisions without mocking anything up or writing a line of code. Try it, see how it looks in context, and decide if it's worth testing before it goes live.

‍

‍Import from Figma or an image design

You can quickly turn a Figma frame or mockup image into a testable variation. Import the file into the AI Visual Editor and it will rebuild it as a new component, placing it on the page as a variation. You can also add additional context as to how the AI is to implement the design. This makes it easy to test designs your team has mocked up, such as a new hero, redesigned pricing card, modal window, or promo banner.

What use cases does this work well for?

Landing page optimization and headline testing

The AI Visual Editor is ideal for landing page optimization, such as testing headlines, messaging, copy order, and structure. Use AI to generate variant ideas, preview them directly on the page, then launch the test in minutes.

‍

Promotional banner and offer testing

Test things like the impact of a promotional banner on bookings to Hawaii. The AI Visual Editor lets you build and swap promotional messaging directly on the page. Run the experiment, read the results, and roll out the winner.

With GrowthBook’s feature targeting capabilities, you can deliver different promotions to different audiences based on the criteria you choose, personalizing as you see fit.

‍

Experimentation for every team

The AI Visual Editor expands experimentation across your organization by putting the ability to build and launch tests directly in the hands of the people with ideas, regardless of technical skillset.

• Marketing and growth teams can independently run their own tests on landing pages, campaigns, and offers.
• Teams that want to move fast can go from idea to live experiment in minutes without touching a line of code.
• Product and design teams can validate new layouts, messaging, and user flows before investing engineering time.
• Developers can stay focused on higher-complexity work instead of implementing every copy or design test.

By unlocking experimentation across your organization, you get more tests, more learning, and faster iteration for everyone. It also improves collaboration by giving marketing, growth, and product teams shared visibility into what's working, so no one operates in a silo.
‍

Your existing guardrails stay in place

The AI Visual Editor handles the creative execution by generating variants, applying changes, and wiring up the experiment, but everything still runs through the GrowthBook platform. Your approval and review workflows, as well as existing processes, remain in place, so your team maintains governance and control.
‍

Get more out of your experimentation platform

The ability to go from idea to experiment in minutes, without engineering support, means more people can run experiments than ever before. And when more people can run experiments, it results in more learnings, faster and more informed decisions, and improvements that compound over time.

‍

Try out the AI Visual Editor (beta)

The AI Visual Editor is currently in beta. Try it today, share your feedback directly with our team, and connect with other users in our community Slack channel.

Download the Chrome extension

Ideate on features, create feature flags, analyze product data, and test what you build, all from Claude Code, Cursor, or Codex against GrowthBook.

AI coding tools are reducing the cost of coding in ways no one ever imagined. Coding that used to take days now takes a prompt. But coding is just one step in the software development lifecycle. Deciding what to build, deploying safely, and tracking the impact of new features still take time. Finding ways to speed to up these processes unlocks even more of the benefits of agentic coding.

That is the vision behind GrowthBook’s 4.4. Your AI coding agent can now drive the entire experiment lifecycle against GrowthBook. From a single conversation in your editor, an agent can analyze your product data, ideate on a new feature, create the feature flag, build the variant, and test it. This post covers what that workflow looks like, how you can incorporate guardrails to create trust, and how to run it from any AI coding agent.

How AI agents can automate the entire lifecycle 

Automating the lifecycle is broader than running a single A/B test. AI coding agents can now cover the full product development lifecycle:

• Analyze product data. The agent pulls product metrics and experiment data to identify new product opportunities
• Ideate. Describe a problem in plain language and have the agent propose concrete changes to try, grounded in your actual codebase.
• Create feature flags. The agent creates the feature flag and wires it into your code, so the change is rolled out gradually and safely.
• Build and test. Create experiments using templates, standardized metrics and guardrails.
• Analyze experiments and make decisions. Use AI to analyze experiment data, extract learnings, and make go/no-go recommendations.

How templates and guardrails streamline using AI to create experiments 

There are many ways an experiment can go wrong, including flawed assignment rules, bad metrics, and unexpected side effects. Consistency and rigor become even more important as you scale.

GrowthBook addresses these needs in a couple of key ways. Experiment templates pre-define the metrics, randomization, and setup for a class of experiments, so the agent knows your best practices. A team can keep a template for, say, logged-out conversion tests, and every front-end experiment of that type inherits the right goal metrics and guardrails automatically. A built-in decision framework encodes when a result is genuinely shippable, rather than leaving that to the agent's guesswork. And GrowthBook stays the source of truth, so business context lives in one place instead of being re-derived in every prompt.

This is the real shift behind agentic experimentation. The agent handles the mechanics. The platform holds the judgment.

AI can apply a series of skills to automate the experimentation process

Take a simple example: testing new copy for a landing-page headline. You give your coding agent the idea in plain language and point it at the file. From there, the agent works the lifecycle:

• Ideate and build. Reads the file, proposes a few headline variants, and picks one to test.
• Configure. Finds the right experiment template in GrowthBook, confirms the metrics, and creates the experiment in draft.
• Wire it up. Creates the feature flag, links the experiment to it, and edits the component to read from the flag. Nothing is live yet.
• QA checkpoint (you). Confirm the flag is wired correctly in GrowthBook before anything ships.
• Launch. On your approval, the agent starts the experiment and tracking fires automatically as users are bucketed into variants.

Reading and shipping work the same way. Later, you ask the agent how the test is doing:

• Analyze. Pulls the latest results, checks the test is well-powered with clean data, and summarizes what happened.
• Recommend. Because the decision framework lives in GrowthBook, the recommendation reflects your team's shipping criteria, not a generic heuristic.
• Ship (your call). If the result clears the bar, tell the agent to roll out the winner. It ships the treatment and writes up the result, with flag cleanup as an optional next step.

This works from any AI coding agent

Whether you use Claude Code, Cursor, Codex or any other coding agent, you can apply this same workflow. All the key components are coding agent-independent, including the programmatic endpoints, the CLI, and a set of open-source skills that teach an agent how to drive GrowthBook correctly. All of this then combines with the context of your own repository.

The governance layer, the templates and the decision framework and your source of truth for experiments, all work the same, consistently and safely with any tool.

The skills are open source and available on GitHub. You can use them as-is or adapt them to how your team runs experiments.

Where humans stay in the loop

Automating the lifecycle does not mean removing yourself from it. The workflow is designed around checkpoints precisely because some decisions should not be fully delegated.

QA is the clearest example. Before an experiment goes live, a person should confirm the feature flag is wired correctly and behaves as expected. Ambiguous results are another. When a test is underpowered, or a goal metric moves while a guardrail slips, that is a judgment call about tradeoffs, not a mechanical decision. The agent can surface the situation and make a recommendation, but a human should own the call.

Think of the agent as a fast, tireless operator that handles the setup, the wiring, the data pulls, and the rollout. The strategy, the QA gate, and the final ship decision stay with you.

Key takeaways

• AI made building cheap, but the experiment lifecycle stayed manual. Agentic experimentation closes that gap.
• An AI coding agent connected to GrowthBook can run the full product development lifecycle: ideate, create feature flags, analyze product data, build, and test.
• Guardrails are what make it trustworthy. Experiment templates and a built-in decision framework move judgment into the platform so agents cannot improvise metrics or ship on noise.
• It is tool-agnostic. The workflow runs from Claude Code, Cursor, Codex, or any agent, because the logic lives in GrowthBook's skills, CLI, and endpoints.
• Humans still own QA and the ship decision. The agent handles mechanics, not judgment.

Get started

The build step and the test step are now both moving at the speed of AI. The way to keep that fast loop safe is to put your experimentation structure into the platform, then let your coding agent drive against it.

The GrowthBook skills are open source and ready to try on GitHub. To see the full workflow end-to-end, watch the walkthrough video. And if you are new to GrowthBook, you can start with feature flagging and experimentation in one platform.

‍

Running an experiment is the easy part. Knowing what it's telling you, and what to do when it tells you nothing clearly, is the hard part. That's the thread running through Dan Layfield's fifteen years in product management, from a mid-2010s startup to one of the largest consumer apps in the world to the boardrooms of the Fortune 1000.

Layfield is Director of Product Management at Diligent, a roughly 2,000-person company that serves around 70% of the Fortune 1000 on bespoke corporate governance problems. Before Diligent, he was head of growth at Codecademy, where he took the company from about $10M to $50M ARR, and a backend-focused PM on Uber Eats' home feed ranking system. Across all three, the constant has been experimentation, and lately, a new collaborator in the work: AI.

The losing experiment that became a 35% win

The story product teams should sit with is the one that didn't work right away. At Codecademy, Layfield's team set out to rebuild the trial model, the single biggest growth lever for most consumer subscription products. A predecessor had already shipped a successful reverse trial, where new users get the paid product automatically and then have to decide whether to keep it. Layfield's team went further, removing the auto-enroll step and asking people to choose the trial and enter a credit card upfront.

The first rounds did not produce a clean win. They produced the thing most experiments produce when they lose: inconclusive results. Not statistically significant positives, not statistically significant negatives, just noise that tells you very little.

This is where most teams move on. Layfield calls moving on too early one of the biggest mistakes he made in his career. A PM's life is a roadmap of things to ship, and every extra week on one project is a week stolen from the others. The pull toward the next thing is constant.

His team stayed. Over roughly four months and three to four rounds of tests, they kept refining the paywall structure, the question of how and where free users come into contact with the paid product, and whether they hit it at a moment that relieves real pain. The tool that broke it open was almost embarrassingly simple: they put every screen of the user experience onto a giant Figma board and overlaid the metrics at each decision point. With the entire funnel laid out, they could see how users flowed through the product and exactly where to move the gates. The payoff was a 35% increase in conversion, a massive result for a business where monetization compounds.

The lesson is not "never give up." It's that a losing test is often a map, not a verdict, and the discipline is knowing when a problem is big enough, and your read of the data deep enough, to take another shot.

Two flavors of experimentation, and one feature factory to escape

Layfield draws a clean line between two kinds of experimentation. One is high-volume conversion rate optimization: run many small tests, expect a quarter of them to deliver small wins, and let volume do the work. The other is using experiments to de-risk something big and genuinely uncertain. The trial-model rebuild was the second kind, which is exactly why taking multiple shots made sense. Email subject-line tests are the first kind; no single one matters much by the time it trickles down to a purchase.

That distinction gets sharper in B2B, where many teams aren't really experimenting at all. Layfield's description of the failure mode is precise: the feature factory. You have a thousand clients, the top 5% request things in every QBR, and the PM's job becomes shipping that list. It keeps the biggest accounts happy in the short term and produces a disjointed product over the long term.

The alternative is a foundation of disciplined, top-down product management: a top-down OKR system where leadership sets meaningful business goals, each team owns a thoughtfully chosen North Star metric that ladders up to those goals, and feature pick metrics that ladder up to the North Star. Layfield's most useful observation is about how this breaks. When a planning process goes wrong, it's usually not that any single layer is broken. Leadership picks good goals. Teams pick reasonable metrics. The failure lives in the connections between the layers, when a team's North Star only loosely relates to what the business actually cares about. Every layer can look healthy while the whole structure quietly drifts.

B2B makes this harder than B2C because of the feedback loop. In consumer products, an unhappy user leaves immediately, which is painful but fast. In B2B, clients sit on long-term contracts, so there's a long delay between weak product usage and the dollar retention hit at renewal. If you aren't watching usage and adoption closely, the bill arrives long after you could have done anything about it.

Anchor the North Star to the natural use case

At Diligent, Layfield's team thinks about North Star metrics by product. The flagship is a board-of-directors collaboration suite used by, in his estimate, nearly every famous board director you could name. The core value isn't receiving a document; if all you do is print a board deck, Diligent is an expensive way to move a file. The value shows up when directors actually collaborate, comment, take edits, and prepare on the plane. So the team anchors the board product on director-side usage.

The catch is rhythm. A director on one board has a meeting roughly once a quarter, so they use the product the week before and a few days after, and then nothing. Push for more engagement than that and it starts to feel spammy, because unless a board deck is waiting, there's genuinely nothing to do. Layfield's rule is worth posting on a wall: retention and engagement should always ride whatever the natural use case is, not fight it. Guardrail metrics like session length and abandonment matter, but the headline number has to respect how the product is actually meant to be used, especially in a regulated space where Diligent deliberately tracks far less than a consumer app like Uber ever would.

AI as the data scientist in the room

Which brings us back to the title. Ask Layfield where AI has earned its place so far, and the answer is research and data synthesis. The old workflow was weeks of labor: find a hundred users, email all of them, spend two to three weeks scheduling calls, write up every interview, then synthesize by hand. Now he points AI at the raw material. Gong's MCP delivers reasonably good synthesis from sales calls in an hour or two, and for simple A/B test analysis, in his words, Claude is a pretty good data scientist.

The point isn't that AI replaces product judgment. It's that AI collapses the weeks of grunt work between having a question and having an answer. For experimentation teams, that's the part that compounds: the faster you can synthesize what users are telling you and what a test actually did, the more shots you get at the problems that matter, and the less tempting it is to move on too early from the one that's about to pay off.

Listen to the full episode of The Experimentation Edge with Ashley Stirrup. How does your team decide when a test is worth one more round? Share below.

‍

When Danielle Olean describes her job, she does not start with a feature roadmap. She starts with a number. As Director of E-commerce at Box, she is not measured by whether she shipped something on time. She is measured by revenue. That single distinction, she argues, changes everything about how a product team should work, and it is the thread running through one of the most practical conversations The Experimentation Edge has hosted.

🎧 Listen to the full episode →

Olean has spent more than 15 years in e-commerce. She came up in the B2C world at companies like Wayfair and Drizly, then spent nearly six years scaling online sales at Zoom through the chaos of the pandemic, watching a one-person operation balloon into a 100-person department almost overnight. Now she is at Box, helping reposition the company from a cloud storage business into an AI platform for managing unstructured content. Across all of it, one tool has been her constant: the A/B test.

Experimentation is not an e-commerce luxury

The most important idea Olean offers is also the one most product teams resist: experimentation is not just for people who own a checkout flow. It belongs to everyone who builds.

Her reasoning is hard to argue with. Time is finite. Engineering capacity is finite. Every feature you choose to build is an opportunity cost for one you chose not to. So the question that matters is never simply "did we launch it." It is "did this actually move the business, and was it a net positive."

You do not need a shopping cart to ask that question. As Olean points out, every product has a flow. Even an AI chatbot has one. A user asks something, gets a response, and either abandons the interaction or comes back. Someone trying to create a document has to find the right place, start the action, and complete it with the formatting they wanted. All of that is a funnel, and a funnel can be measured.

That measurement reveals things intuition never would. Olean describes a recurring pattern: a feature that very few people discover, but that the small group who find it use at a high rate. The instinct is to call the feature a failure. The data says the opposite. The feature is valuable. The placement is wrong. Catching that early turns what would have become a customer complaint or a pile of technical debt into a quick fix.

The humility underneath this is what makes it powerful. Host Ashley Stirrup framed it as the hard truth of experimentation: only two or three out of ten ideas tend to land the way you expected. That is not a sign that the other ideas were bad. It usually means a piece was missing, the wording was off, or the timing was wrong. For a B2B product manager, that is not discouraging. It is an enormous opportunity to double their impact by building what customers actually want to use.

The simplification tightrope

If the first lesson is that you should test, the second is that testing will humble you, even when you are winning. Olean's pricing page saga is the clearest example.

Box's pricing page was, in her words, robust. Lots of colors, lots of elements, eight self-service plans, and a tangle of security, compliance, and AI features competing for attention. Her hypothesis was straightforward: simplify the page, reduce the cognitive overload, and more customers will be able to choose a plan and buy. The team did a large overhaul, even setting rules to keep feature descriptions to one line with tooltips for anything longer. The result beat expectations. A genuine, measurable win.

And here is where most teams go wrong. A win feels like a direction. "This is the right hypothesis, let's do more of it." So the team kept simplifying. They tested removing the slash-out pricing, the crossed-out higher number next to the discounted one, assuming it was just more clutter. It failed. That crossed-out price was quietly reinforcing the 25% annual discount, and customers valued the cue more than anyone realized.

Undeterred, they tried once more, trimming each plan's feature list to show only what differentiated it from the plan below. That failed too. Box's plans are complex, and customers wanted the full detail to feel confident about what they were buying. Stripping it away did not reduce overload. It removed information people needed.

One win, two losses. Olean calls it the tightrope. There is a diminishing return where simplification stops helping and starts hurting, and the only way to find that edge is to test until you cross it. Tellingly, the team's next move was not more simplification but an addition: a third tab to serve a new persona, on the theory that the page was now simple enough to absorb it. And of course, they are testing that too.

The wine effect

The third lesson is Olean's favorite kind of result: the test that wins for a reason you never hypothesized.

In a prior role, the team faced a churn problem. So they created a cheaper plan called Basic Plus and showed it only to customers in the cancellation flow, never on the public pricing page. The hypothesis was logical. These people think we are too expensive, so offer them something cheaper and convert a full churn into a partial one.

The test won. But not because people downgraded. The Basic Plus plan had roughly 70% fewer features, and simply seeing it made customers' current plan look far more valuable by comparison. So instead of switching down, more of them stayed exactly where they were.

Olean calls it the wine effect. On a wine list, you rarely order the cheapest bottle. You order the second-cheapest because the cheapest one makes the next option up look like a deal. The experiment succeeded, the hypothesis was wrong, and both things were true at once. That gap between what you predicted and what actually happened is not noise. It is the entire reason to run the test.

Building a culture that can handle losses

None of this works if a team is afraid to lose. Olean is deliberate about that. She presents every win and every loss to leadership, with a biweekly impact report to the COO and a monthly one to the CEO. The wins build credibility and prove the team knows what it is doing. The losses, presented just as openly, create the psychological safety that a testing culture depends on. Every loss comes with what was learned and what the team will try next.

The payoff has been cultural. A year into her role at Box, Olean now has executives who proactively say "let's make sure we test that" before shipping a new feature, and a growing line of colleagues asking how to run experiments of their own. She thinks about it as a pyramid: start with the widest-impact experiences like the pricing page and checkout, then work up into smaller cohorts, like desktop versus mobile or high-context international markets such as Japan. And she is already looking ahead to monetization tests and AI agents that scan competitors and help ideate new experiments.

The throughline is simple. You will be wrong more often than you are right; you cannot reason your way to the balance point, and your most valuable insights will arrive disguised as surprises. The teams that win are not the ones with the best guesses. They are the ones willing to find out.

‍

If you run A/B tests long enough, you'll eventually ship a false positive. Your test calls a winner, you ship it, but the metric that should have moved stays flat. This happens far more often than the 5% significance threshold promises, even at big companies with mature experimentation programs like Microsoft, Bing, Netflix, and Airbnb. What's worse is that the faster you test, the more false positives you ship. This is the technical playbook for keeping that number down without slowing your program to a crawl.

How bad is the false positive problem in A/B testing?

False positive rates in real-world A/B testing run far higher than the 5% most teams assume. Even mature experimentation programs carry 6% to 26% risk that a given statistically significant win is a false positive (a Type I error), and at the industry-median success rate, roughly 1 in 5 experiments called winners are a false positives.

Below is a list of false-positive risks by company. False positive risk is the probability that a statistically significant winner has no real effect, and it follows directly from how often a team's experiments succeed. The lower the success rate, the higher the risk.

The more experiments you run, the more false positives you ship. A team running 100 experiments a month at the median 10% success rate sees about 10 winners, and roughly 2 of those are false positives with no real effect. Each one costs engineering time and gets counted as a win, so the team keeps building on a feature that did nothing. Over a year, that's around 24 of them, and the faster a program runs, the faster they accumulate.

What causes false positives in experimentation? 

Most false positives trace back to 4 causes: peeking at results early, testing too many things at once, assignments that are not truly random, and underpowered tests. Each one pushes your true false positive rate higher.

Peeking

Peeking happens when you check results while a test is still running and stop as soon as you see a significant positive result. The bias comes from an asymmetry in how the test ends. A positive result that crosses significance stops it, while a flat or negative one keeps running to give it time to resolve. A test with no measurable effect will still cross into positive significance from time to time through sampling variation, and stopping at one of those crossings mistakes that fluctuation for a real win.

A p-value assumes a sample size fixed in advance and examined once, so repeated looks invalidate them. Each additional look raises your false positive rate. With no correction, it climbs from 5% at a single look to roughly 14% by 5 looks and 19% by 10.

Multiple comparisons

With 20 metrics in a single test, your odds of at least one false positive are about 64%, even if nothing you changed had any effect. Each metric you add is another test with its own 5% chance of a false positive, so the family-wise error (FWER) rate keeps climbing as the list grows (FWER is the chance that any one of your metrics shows a false positive). Teams that track a wide metric dashboard on every experiment are especially exposed here. Learn more about how to choose the right metrics and KPIs for A/B testing.

Sample ratio mismatch (SRM)

Sample Ratio Mismatch (SRM) happens when the traffic split you actually get doesn't match the one you configured. A 50/50 test that comes out at 53/47 because of a bot exclusion, or SDK bug, or targeting error no longer has comparable groups, so you can't separate a real effect from bias. SRM is common and easy to miss, and the only way to catch it is to test the observed split against the one you configured, usually with a chi-squared test, which GrowthBook runs automatically. 

Underpowered tests under pressure

An underpowered test without the sample size to reliably detect the effect you care about is usually framed as a false negative (Type II error) risk, but it feeds false positives too. It produces wide confidence intervals and unstable point estimates. When an underpowered experiment does cross significance, that win is disproportionately likely to be spurious, and the winner's curse inflates its measured effect, since only the largest swings reach significance when power is low. Deadline pressure makes this worse, as teams peek, stop the first time the result crosses significance, and ship it. Underpowering and peeking reinforce each other, and a high-velocity program tends to run into both at once.

7 ways to limit false positives in experimentation

Some false positives are unavoidable, but the inflated rates described above are not. Every cause has a practical control. Here are 7 actions you can take to lower your false positive rate.

1. Pre-commit to sample size and duration

Do your power analysis before launch, not after the early numbers start looking good. Define your minimum detectable effect (MDE), set power to at least 80%, and design your experiment around the required sample size. Then leave it alone. "It's trending positive" is not a stopping criterion. GrowthBook's power analysis computes the required sample size before you launch.

2. Calibrate with A/A tests

Run an A/A test whenever you stand up a new experiment surface or change the data pipeline behind it. An A/A test is an A/B test where both arms serve the same variation to validate your instrumentation and assignments before a real experiment. With no true effect by construction, a sound setup should return roughly identical results for both arms.

A significant A/A test result for a single metric doesn't necessarily mean something is broken. Even when there is no real difference, some metrics end up significantly different by chance alone. At GrowthBook's default 95% chance-to-win threshold, that's about 10% for a single metric, or 41% across 5 independent metrics. 1 metric out of 5 with a statistically significant difference is usually fine, especially if no SRM or other experiment issues are detected. A genuine problem produces a different pattern, with many metrics reaching significance simultaneously, often at extreme values such as 99%+ chance-to-win. To tell them apart, re-randomize and rerun the A/A test. Random noise will affect different metrics each time, while a real bug will consistently affect the same ones.

3. Use sequential testing to solve the peeking problem in A/B testing

Sequential testing lets you watch results as often as you want without inflating your false positive rate. It works by using confidence intervals that are wide enough to remain valid no matter how many times you look. GrowthBook implements asymptotic confidence sequences from Waudby-Smith et al. (2023), with a tuning parameter (N*) that you can set to the sample size you expect to reach when you make a decision. Since sequential testing produces wider intervals than fixed-sample testing, you trade a little power for the freedom to peek. At any serious velocity, catching a bad variation early and shutting it down is worth the small loss in power.

Learn more about common A/B testing methodologies. 

4. Run SRM checks before interpreting results

Each time you conclude an experiment, check for a sample ratio mismatch before you read the results. An SRM means the realized traffic split doesn't match what you configured, like a 50/50 test that comes out 53/47. When you detect an SRM, the comparison is invalid, and no metric can be trusted, even the ones that look unaffected. Discard the results, identify and fix the cause, and rerun the experiment. GrowthBook automatically flags SRM in your experiments, so you can catch it before acting on the results.

5. Apply multiple testing corrections

The more metrics or variants you test, the more likely it is that at least one will appear significant by chance. That's the multiple comparisons problem. A correction counters it by tightening your significance threshold as the number of comparisons grows. The two corrections, Holm-Bonferroni and Benjamini-Hochberg, are explained below. Decide on the correction and your primary metrics before launch, not after seeing which ones moved. Choosing after the fact introduces bias from p-hacking.

Which correction to use depends on how costly a single false positive is.

Holm-Bonferroni is the more conservative option. It raises the bar for calling any result significant, so it's unlikely that even one of your metrics is a false alarm. In statistical terms, it controls the family-wise error rate (FWER), the chance of even one false positive across all your comparisons. The tradeoff is power. A higher bar also means missing some real effects, so use it when a single wrong call is unacceptable, as in pricing, legal, or safety decisions.

Benjamini-Hochberg is the more lenient option. Rather than guard against every false alarm, it keeps the share of false winners among your significant results low. In statistical terms, it controls the false discovery rate (FDR), the expected proportion of your significant results that are wrong. You catch more real effects in exchange for tolerating a few false positives, so use it when you're monitoring many secondary metrics and can absorb some noise.

Microsoft's Windows experimentation team evaluated both in production across 200+ metrics and found Benjamini-Hochberg won on the combined scorecard of precision, recall, and A/A false positive rate, in exactly the many-metric setting where you'd reach for it. GrowthBook supports both corrections as an organization-wide default, applied across your goal metrics.

6. Validate winners with the causal chain

An increase in your primary metric is strong evidence for a trustworthy win, but not sufficient on its own. A real effect should also show up in the metrics downstream of your primary metric, not just in the metric itself. For example, if the session-to-free-trial rate rose, the session-to-paid-plan rate should rise too, since some of those incremental free trials could reasonably be expected to convert to paid.

Fanatics, which runs close to 100 experiments a month and credits experimentation with about 8% of its annual growth, caught a false positive this way. The team tested removing ads from its product grid pages, and the results showed a positive effect at the 95% confidence level, with higher revenue. But the supporting micro-metrics like products viewed and grid-to-cart all stayed flat, and nothing explained the revenue lift. 

The team stopped the experiment, reran it, and the second run found no measurable effect. They have run the same change 6 to 8 times over the years, and it comes back flat every time. It turns out, shoppers simply overlook the ads, so removing them changes neither how people browse nor what they spend. 

A win that can't be traced through the behavior that should have produced it is usually a false positive.

7. Reduce false positives through variance reduction with CUPED 

CUPED (Controlled-experiment Using Pre-Experiment Data) reduces variance in your results by using each user's pre-experiment behavior to account for noise unrelated to the treatment. Lower variance lets you reach a sound conclusion faster, removing the temptation to peek and stop a slow test early.

Netflix reported a roughly 40% reduction in variance for key engagement metrics, and Microsoft found CUPED to be equivalent to adding roughly 20% more traffic to an experiment. CUPED has one important limitation. It only helps when you have pre-experiment data correlated with your metric, so it does little for brand-new users or rarely observed events. Where it applies, faster experiments with lower variance mean fewer false positives. GrowthBook automatically applies CUPED to every metric as an organization-wide default, so variance reduction is built in.

How GrowthBook handles false positives by design

The best experimentation platforms offer multiple features to prevent false positives. They let you monitor results mid-test without inflating your false positive rate, they correct for multiple comparisons, they run SRM checks automatically on every experiment, and they reduce variance so experiments reach significance sooner and there's less reason to peek in the first place.

GrowthBook's experimentation platform supports all of these features. Sequential testing, included in Pro and Enterprise plans, handles safe peeking, with the confidence sequences and configurable tuning parameter described earlier. Multiple testing corrections (Holm-Bonferroni and Benjamini-Hochberg) ship as organization-level defaults. SRM detection runs automatically on your experiments. CUPED, available on Pro and Enterprise plans, cuts variance and experiment duration.

GrowthBook's stats engine is also open source, so you can read the exact code that produced any result instead of trusting a black box.

The false positive prevention checklist

Use this checklist to reduce your false positive rate in every experiment, especially as your experimentation program scales.

• Pre-commit to your sample size and stopping date before launch.

• Calibrate your setup with an A/A test on new surfaces and after tracking changes.

• Turn on sequential testing if you need to watch results while the test runs.

• Check for SRM before you interpret any metric.

• Apply multiple testing corrections: Benjamini-Hochberg for many metrics, and Holm-Bonferroni when a single false positive is unacceptable.

• Confirm every winner against its full causal chain.

• Reduce variance with CUPED to take the pressure off peeking in the first place.

False positives can slip in at any stage of an experiment, from experimental design through implementation and analysis. The 7 controls above each close a different gap, and they compound, so teams that run all of them get far more trustworthy results than teams that run only 1 or 2.

Want these controls on by default? GrowthBook’s experimentation platform runs SRM checks automatically and lets you enable multiple corrections, sequential testing, and CUPED as organization defaults.

If you’re looking to run high-velocity experimentation, try GrowthBook for free or book a demo with our team.

‍Sum your significant experiment wins, and you almost certainly overstate the true impact. The reason is the winner's curse: you're adding estimates that were selected for looking good. Corrections help a little, but a holdout is the real fix.

You ran forty experiments this quarter. A dozen of them had significant results, so you rolled them out. You summed up the effects, and the slide going to the leadership team says your team drove a 10% gain. Everyone celebrates. The only problem is that the number overstates the true impact.

Sure, every one of those experiments was run cleanly and read correctly. And they did pass the significance threshold. The significance threshold is what's causing the bias. The estimates you're summing are already skewed upward because lucky draws are more likely to pass than unlucky ones. So the total comes out larger than the value you actually generated.

How big of a problem can it be? Airbnb measured it: a set of winning experiments summed to 7.2%. Their de-biasing formula pulled that back to 5.3%, and a holdout put the real number near 4%. Nearly half the reported impact wasn't there, and the holdout caught more of it than the formula did.

A holdout is a simple idea: hold back a small group of users from all your new changes and measure the gap directly. There are different ways of configuring a holdout, but any of them is better than trying to infer the total impact from individual estimates.

The leadership slide is only one purpose you might have. Any time you sum individual significant results up into a single number, the same bias applies. A meta-analysis of a program, the quarter's total impact, and one team's contribution next to another's. Aggregating results that way is still useful and worth doing. This post is about doing it without overstating what you actually found. We walk through the exact mechanics of the selection-on-significance bias to give you a solid understanding. That is how you can judge, for your setting and purpose, just how big a problem you might have. And what you should do about it.

What is the winner's curse?

Every experiment hands you a measured effect, and that number is the true effect plus some noise. Run the same test next week, and you'd get a slightly different figure.

The problem arises when you look only at the significant, or shipped, results. Counterintuitive, I know. If you summed all forty estimates, winners and losers, you'd be fine. The lucky-high and the unlucky-low roughly cancel, and the total should come close to the truth. The bias enters only when selecting on statistical significance.

It is because the significance threshold isn't a random gate. An experiment is more likely to pass when the noise component pushes its estimate up than when it pulls it down. So the winners you keep aren't a fair sample of the effects you tested. On the contrary, they over-represent the ones that got a lucky draw. Gelman calls this the statistical significance filter, and the amount by which it inflates the survivors is his Type M (magnitude) error.¹

To fully understand, let's walk through the mechanics of hypothesis testing visually. Figure 1 shows the distribution of estimates you would draw from under the null, and the distribution you would draw from at a given true effect size. The scale of the x-axis is in standard errors to make it more general.

The null distribution sets the significance threshold by the chosen false positive rate (that's your α).² We can use it to visualize what part of the true-effect distribution would fall below. That's the hatched grey chunk, and these features don't get shipped. Those are your unlucky false negatives, which are more likely if your power is low (imagine the distributions widening).

The blue area illustrates the results of the significance filter. It shows the estimates that pass the significance threshold, with some luck, and get shipped. And the difference in luck is the problem. You cut away the low draws and keep the high ones, even though they have the same underlying true effect. Therefore, the average of what's shipped jumps to the right, while the center is the true effect. The average of what you shipped is higher, and it goes that way, whatever the true effect is.

This figure is constructed to show the case with 80% power, and it leads to an overestimate of 13%. The next part shows how it gets even worse with lower power.

Figure 2 shows the relationship between power and overestimated winners. At 80% power, where you'd like to be, the overstatement is the 13% we just saw. It's real, but maybe you can live with it. If the true effect comes in smaller than expected, however, the same metric and sample size give you less power. At 40% power, the survivors overstate by more than half. The reason is mechanical. A smaller true effect makes the distributions overlap more. The same thing happens if your metric is noisier than expected, or your sample is smaller than planned. Kohavi and colleagues recently replicated four widely cited 'win' patterns across eight large experiments, and the published effects came back far smaller than first reported.³

80% power is the good case, but even there, the bias is 13%. And in reality, you're probably running lower power than that on most of your experiments without knowing it. Actual power depends on the true effect you never see. Imagine the true effect actually being 40% lower than your powered MDE. Then, your realized power is 40%. Even a miss of 20% in true effect size drops your power to 60%, where you overstate your true effect by 30% when selecting on significance.

Unfortunately, you can't read the bias off a single result. A win that barely cleared could be a lucky draw from a small effect or an unlucky draw from a large one, and the number itself can't tell you which. What you can judge are the conditions you run under. If your metrics are noisy, your samples are thin, and your experiments often end inconclusive, power is scarce across the board. Your significant wins then mostly sit on the steep part of the curve. If you are generally well powered, the bias is milder. Understanding your general power situation can give you a hunch, but not the exact bias.

How to correct for the winner's curse

Can't you just correct for it? Partly. The crude way is a flat discount: Ronny Kohavi's rule of thumb is to knock at least 20% off your reported wins, the figure they used at Bing. The more careful way is to let a model do it per estimate. Switch on a proper prior in GrowthBook's Bayesian engine, and the number you read becomes the posterior mean. The raw estimate is pulled toward zero, hardest when the estimate is noisy. It shrinks on precision, not luck, which lands right on average because the noisy wins are the ones most likely inflated.

Both are more sound than estimating power from the effect you just measured and then deflating with that. Because that estimate is already inflated, it makes your power look better than it was, so the correction comes out too small. And it misses by the most, exactly when power is low. Post-hoc power like that is a known trap, not a fix.³ It's also a closed loop. The only input is the number you're trying to correct, so no new information ever enters.

What every correction shares is that it re-reads the same selected sample. To undo the selection you need a measurement that wasn't selected on. A fresh draw.

The textbook way to get one is replication. Re-run a winner and read it once, without filtering on significance, and the lucky draw that pushed it over the line the first time has no reason to repeat. That removes the bias, one experiment at a time. But replicating every winner is expensive. You would spend next quarter re-proving last quarter's, and you would still read each feature alone, with no view of what they add up to. If the purpose is to get the aggregate effect, you might as well replicate them all together in one go.

Replicating them all together is the holdout. Hold back one group from all your new changes and measure the gap once against the rollout. You give up knowing which individual winner was exaggerated, but you get the thing you were aggregating for: one honest number for the whole batch. Whether that's the total on the leadership's slide, a meta-analysis of the program, or one team's output against another's, the holdout is the complete solution for ambitious experimentation teams.

It also solves other problems for you. Some of those wins were driven by novelty effects that fade out over time. Depending on how you configure your holdout, you can solve this problem at the same time. In other experiments, your outcome metric was a proxy because your key metric was too noisy for the effect size you expected. Literally, the low power problem we have iterated here. Pooled in the holdout, these experiments either finally show an effect on the key metric, or you find out that the proxy approach doesn't work.

Because the holdout is solving multiple problems at once, there are different ways to configure it. The differences may seem subtle, and how they matter isn't obvious. That is why we cover those carefully in a separate post: what each holdout configuration actually measures.

You may have heard another reason to hold out: interactions between experiments. Several platforms list this as a key motivation. But it probably isn't moving your sum much against the holdout. When two winners interact, the sum already captures it, the same way the holdout does. That holds as long as they ran concurrently, or you shipped one before testing the next. What's left is a winner interacting with a loser, and that only biases the winner if the loser was still live during the winner's test. Because losers get switched off when they lose, often early, they should interfere less. Winners get shipped and stay on, which is the harmless case the sum handles. Interactions are worth understanding properly, and that's why we will cover them in a separate post.

Should you trust the sum of your wins?

The sum overstates what you generated, and the winner's curse is the main reason. Every winner you kept leans high because you selected it for looking good. Discounting the total or shrinking each estimate toward a prior helps, but those corrections don't solve the underlying problem. The holdout does.

Add up your wins if you like. Just think twice before you put the total on a slide.

¹ The Type M (magnitude) error, or exaggeration ratio, and the design-analysis framing the figures use are from Gelman, A., & Carlin, J. (2014). "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors." Perspectives on Psychological Science, 9(6), 641–651. PDF.

² The diagram uses a two-sided 5% test, and counts only the positive, significant part as shipped, which is what you'd actually roll out. A true-zero feature lands there 2.5% of the time, not 5%. Going one-sided wouldn't change the story. It only slides the threshold to the left, to about 1.64 standard errors. That raises power and trims each winner's overstatement a little. But it buys that by doubling the rate at which pure noise ships, and the selection is still there. Drop the threshold and more marginal winners crowd into the sum, not fewer.

³ Kohavi, R., Linowski, J., Vermeer, L., Andreev, A., Dodin, M., & Furuseth, J. "Trustworthy A/B Patterns and the Winner's Curse: Lessons from Eight Large-Scale Replications." KDD 2026 (forthcoming). Preprint; doi:10.1145/3770855.3818498. Across eight high-powered replications, only two effects were significant in the expected direction and one was significant in the opposite direction. The paper quotes the same exaggeration-by-power factors used here: a relative 13% at 80% power, 40% at 50%, and 130% at 20%.

⁴ Hoenig, J. M., & Heisey, D. M. (2001). "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis." The American Statistician, 55(1), 19–24. The companion practitioner argument is McKenzie & Ozier, "Why ex-post power using estimated effect sizes is bad, but an ex-post MDE is not," World Bank Development Impact (2019).

Teams use holdouts differently but call them by the same name. This post is about what each one actually estimates, what it's for, and when the configuration matters.

‍TL;DR: A holdout measures what everything you shipped over a period was worth. Holdout estimates are usually less than the sum of your individual A/B test wins, mostly because the wins are inflated by the winner's curse. This piece explains four different configurations that all get you slightly different versions of the "effect of everything over the last period". The right pick depends on how much your features interact, how much novelty effects worry you, and which version you find most relevant.

What is holdout testing?

Most experiments answer a single-feature question: what is the value of shipping this change, versus staying with the current product? A holdout experiment asks the bigger question about the effect of every feature you shipped recently. You hold a group of users back from everything your team shipped (or tested) over a given period and compare them against users who got all of it. The result tells you whether the cumulative impact of those releases actually moved the needle for the business.

You hear people talk about holdouts, and it sounds like a good idea. But why exactly would you do it, and what do you do if it comes back flat? Roll back a quarter of the features? Probably not. So why bother?

Part of the trouble is the word. "Holdout" covers a few different things, and people reach for it to mean whichever one they have in mind. Two are worth separating before we go further.

One is about a single feature. You shipped something, it won, and you keep a small group on the old version for a few months to see whether the short-term win holds up in the long run. That's a question about the long-term effect of one change, and we'll cover that separately someday, but that's not this piece.

The focus of this post is testing everything at once. You hold a group of users back from the whole set of shipped work and compare them against everyone who got the changes. That's the kind of holdout this piece is about: the one you run to answer "how much did all of this really help the business?"

The idea is simple. What's confusing is that many companies are doing this slightly differently, and each tends to present its way as the obvious one. Same word, different machinery, different answers. To see what's really going on, and whether the differences matter, you have to be precise about what each one estimates. Let's start with that.

What a holdout actually measures

A combined holdout is just an experiment. The only unusual thing is the treatment you assigned. Normally, the treatment is one feature, and you hold everything else still. Here, the treatment is the whole batch of changes you shipped this quarter, and the control group stays on the product as it was before any of them.

That can sound like a clumsy way to get a number you could back out for free. You ran a bunch of experiments, kept the winners, and you can add up their effects. Why hold a group out from everything to learn what summing the wins would tell you?

Because summing fools you. The wins you kept are inflated: you only ship the experiments with significant effects, and those oversample the ones whose noise happened to push them up (the winner's curse). A companion piece, Why summing your experiment wins overstates impact, works through how much and the mechanics more clearly. Interactions are the softer second worry: two changes that each win in isolation can partly cancel when both ship, so the sum overcounts them. But that only happens when they were genuinely tested apart. If they ran concurrently, or you shipped one before testing the next, the sum already captures the interaction, the same way the holdout does. Whether it alters the sum at all depends entirely on how your experiments overlapped, and under a clean rollout, it often doesn't. We’ll cover this in a separate piece too. Either way, a holdout sidesteps both: it turns everything on and reads the joint effect as a whole. Here we take that as given and ask: what, precisely, does the holdout measure?

What holdout testing is actually for (and what it isn't)

If a holdout comes back flat, you're not going to roll back the quarter, so what was the point?

The point is rarely a ship-or-rollback decision. A holdout is an accounting instrument and an alarm. It tells you whether the wins are real once they're added up, and what to expect going forward (with some faith). The holdout result answers, "Did all the work we did really move the needle?" Hold out per team instead, and the question changes to "which teams are driving more impact," which is a funding-and-headcount call as much as a measurement one. Etsy, a marketplace, runs a quarter-long holdout for exactly this kind of company-level read: the collective impact of everything its teams shipped, which is rarely the sum of the individual wins.

That alarm is worth taking seriously. A holdout is the safety net against naive claims: it measures the whole configuration at once, so the net of every interaction, detectable or not, is already in the number. It won't tell you which culprit is at work, the inflated wins or the features stepping on each other, but it tells you there's something to find, and roughly how much of your reported progress went missing on the way. Airbnb reported a case where a run of individual wins summed to a 7.2% effect, but the holdout put the real number near 4%, almost half the sum gone.

Four holdout testing configurations: what each one estimates

Now, the part that confuses everybody: implementation. The configurations really do differ, and we will now unpack exactly how. Whichever one you run, you get an answer to the same bottom-line question of what the quarter was worth. But each answers a slightly different version of it, and two choices set them apart. First, what you compare the held-out group against. It is either a sample of the full population that lived through the rollout, losers and all, or a clean winners-only arm carved out of the holdout itself, a split. Second, when you read the result. Either incrementally, as the winners accrue, or in a terminal window at the end with everything on. Those two choices make four configurations, one per cell of the grid below. A fifth, the Reverse holdout, sits off the grid. It is the after-the-fact option for when you never set one up.

Two things separate the configs below. Which features are on when you measure: today's adopted winners, or also the losers users lived through on the way. And how fresh those features are when you read: just-shipped, worn-in, or averaged across the whole rollout. Each configuration is described in plain terms here. The corresponding potential-outcomes estimand for each lives in The estimands, formally at the end.

Full-terminal (GrowthBook)

GrowthBook reads a single held-out group two ways from the same setup, switched by a toggle on the analysis window: the terminal read here, the incremental one in the next configuration. One holdout group, both reads. Hold a group on the old product and compare it against a sample of everyone else, deliberately the same size as the holdout, so the comparison stays balanced. The held-out group gets no new features after the holdout starts. The comparison group lives through the full product experience over the time period, features arriving as they ship, and the losers are the losers switched on for a while before getting dropped.

What makes this read terminal is the analysis period: after the active period, the set of features freezes, and the number is read over a few weeks. By then, it is the adopted winners all on and the losers all off. So the transient cost of weeding out the losers is not part of the measurement, only the cost that really persists. What's left to measure is the impact of the winning features, net of most novelty effects and the persistent cost of finding the winners. One thing to keep in mind: the comparison group lived through the losers earlier in the period, so any permanent damage the losers did still pulls the number down. The transient cost of running the losers is taken out, but the lasting cost is not.

Full-incremental (Statsig/GrowthBook)

Statsig keeps the same structure. A single held-out group against an equal-size sample of everyone else. However, it is measured continuously from the start. It thus captures the winners as they accrue and the losers as they get tested. No terminal measurement window. So the number is the net effect on your users as they actually lived through the full product experience. The winners and the running cost of weeding out the losers, each interaction weighted by how long the features overlapped. This is the full account of what happened.

Read this way, the held-out gap is not one number but a moving one: small early, when only a few winners are on, and growing as they accumulate. You add the gap up as it builds, rather than reading a single end-of-quarter snapshot.

For GrowthBook, this is the same held-out group as the terminal read, now read continuously, the other setting of that toggle.

Split-incremental (Eppo)

Take a holdout of any size, say 10%, and split it in half: half see no new features for the whole period, the other half get each winning feature the moment it's adopted. Both are shielded from the live experiments. Because the winners were picked from the population outside the holdout, the split re-measures them on a fresh sample. That is a replication on a fresh sample: winners that got through on noise tend not to survive it, and the number can come in lower than the individual tests promised.

Both arms come from the holdout and sit out every live experiment, so the contrast runs between two halves: a winner's arm that picks up each winner the moment it is adopted, against a status-quo arm still on the old product. Read incrementally, like the full version, but over winners only. Two things set it apart from the full version. The losers never reach either arm, so no loser cost enters. And because the winners were selected from the population outside the holdout, this re-reads them on units that took no part in picking them, the cleanest form of replication that every holdout does.

Split-terminal (Etsy)

Splits the holdout in half, too, but delivers everything at once. Hold a group out from everything for a quarter. At quarter's end, turn on all the adopted winners together for half the holdout for several weeks. Compare that half against the other half still on the old product. Because every feature goes live at the same moment, they all have the same age when measured. No ambiguity about which feature has been live the longest and which was turned on only at the end. Like split-incremental, this is a split, so the winners reach held-out users who had no hand in picking them, the same clean replication on out-of-sample users.

In the other configurations, features arrive across the holdout period. So at the measurement, each carries a different age. Turning everything on at once for half the holdout group gives every feature the same starting line instead. All of them fresh.

Every winner shares the same age, and that age is young. It aims at the same winners-all-on contrast as GrowthBook's full-terminal read, but with the features fresh instead of worn in: every feature at the same young age, measured the first time users meet it.

Reverse holdout

The one you probably didn't plan. Every design above has to exist before launch, because you can't hold a group out from something you already shipped. If you didn't set one up, you can still take a random group and revert them, turning the quarter's changes off again. That measures something genuinely different: not the effect of never having the features, but the effect of losing them. Take away something people have settled into, and you measure their annoyance along with the feature's value. Useful when it's the only option, but "had it and lost it" is its own estimand.

The same comparison shows the asymmetry. The kept group is on the worn-in winners. The reverted group is back on the old product, but reached by removal: its baseline carries the history of having had the features and lost them, not the clean never-had-it baseline the other four compare against. Losing something you have settled into provokes a reaction of its own that a never-had-it baseline never contains. Strip out features that were doing real work, and the reverted group drops below the kept one, so the number comes out positive; the bigger it is, the more the quarter is worth.

One practical thing cuts across all four, but not the Reverse holdout. Something has to keep the held-out users out of every experiment. On a platform that's handled for you: a feature-flag prerequisite in GrowthBook, the holdout config in Eppo or Statsig. The coordination cost mainly shows up if you build it yourself, which is why Etsy's writeup is half about the infrastructure it takes to keep a holdout clean across every team for a whole quarter.

Does your holdout testing configuration actually matter?

Start with what you actually want to measure. Read continuously against the full population, like Full-incremental, and you get the quarter as your users lived it, losers and all. The other four lean toward what to expect going forward: they focus on the winners, and the terminal reads let novelty settle before they measure.

How much the choice between configs then bites sharpens along the grid's two axes. The read axis, terminal versus incremental, decides how interactions and novelty land. The comparison axis, full versus split, trades the winner's curse against power.

On the read axis, start with interactions. If your teams ship to separate corners that don't touch, features barely interact, and the choice hardly matters. When many fight over the same surface, it does. The two terminal reads have every winner on when they are measured. So each interaction enters at full weight. Clean interaction accounting. The two incremental reads absorb interactions as they build over the rollout, each pair weighted by how long the two features happened to coexist. That is a more faithful picture of what users lived through. It is also a less clean account of the period's all-on state. The larger the interactions, the more the two reads diverge, so the choice matters. If you want the joint effect of everything turned on together, a terminal read measures it at full weight. If you want what users actually lived through the rollout, the incremental read is more appropriate.

Novelty is the other read-axis driver, and it sets the two terminal reads apart. They look like the same measurement taken at different times, and novelty is the difference you notice first. Split-terminal turns everything on fresh, so its window is novelty-heavy, while Full-terminal reads the winners worn in. With strong novelty effects, the two genuinely disagree, one catching the first reaction, the other closer to the mature state. A longer read settles the novelty but leaves the held-out group further behind. Age is not all that separates them, though: one is a full comparison and the other a split, so they part on the compare axis too.

On the compare axis, hold the read axis fixed and line up two terminal reads: GrowthBook's full-terminal against Etsy's split-terminal. Going from full to split changes two things at once, and they are worth separating.

The first is who you measure on. A full comparison measures the winners using a new sample of the same population from which they were selected. A split comparison measures them on held-out users who took no part in the selection. The winner's curse isn't creeping back in either case: both compare against a freshly randomized held-out baseline, so a false-positive winner regresses toward its true effect anyway. The split is the cleaner replication, but a full comparison is mostly fine on this count, too, since it still draws a fresh group to compare against, independent of the one that picked the winners.

The second is whether the losers are in the number. A full comparison's baseline lived through the losing experiments, too, so whatever lasting damage they did still sits in the comparison. A split's winners reach users who never met a loser. So a split also isolates the winners from the loser residue. Put the two together, and a split gives a cleaner, selection-independent read of the winners on their own. The more underpowered your experiments, the more lucky draws you ship, and the more that a cleaner read is worth.

That cleaner read costs something, though. A split keeps two arms out of your experiments, where a full comparison keeps only one, so it takes more traffic, and so more power, from the feature tests you're running. Whether a cleaner winner number is worth thinner experiments is a real trade-off, so it matters how much traffic you have to spare.

One caveat cuts across all of them. Every configuration assumes the held-out group shows you the world without your changes. In a marketplace, or any product with network effects, that breaks. The held-out users shop in the same market as the treated majority, who just moved. So the holdout's baseline is not really the old world. It is the old product in a moving market. This is not unique to holdouts. But a holdout makes it bite harder. The holdout group is deliberately tiny, so the market has shifted almost completely to the new equilibrium under the shipped features. And a batch of winners all pushing demand the same way moves that market more than any single test would. So read a holdout in a marketplace as a partial-equilibrium number, not the grand total.

The estimands, formally

For readers who want the potential-outcomes version, here is each configuration written as an estimand. Everything above stands without it, but here is a more technical description that can add some clarity.

Take every change you shipped as a switch, on or off. The product is a vector d, one entry per feature, and a customer's outcome under it, say revenue per user, is Y(d). The held-out group stays on the start-of-quarter product, every switch off: Y(0). Today's product is the winners you kept with the losers switched back off; call that configuration W.

A feature's effect the week it ships is not its effect once users settle in, so tag the treated outcome by age:

• Y(·; novel): just shipped, the first reaction
• Y(·; mature): worn in, the settled effect
• Y(·; global): the whole lived experience, novel and mature together, averaged across the rollout

The age tag, not a day-by-day index, is what separates the reads. A terminal read lands on a single settled state, mature if novelty has likely worn off, novel if everything goes live at once. An incremental read integrates the gap over the whole rollout, which is exactly the global outcome: young and mature in the proportions users actually lived them. One more tag, Y(0; loss), marks a baseline reached by taking features away rather than never having them.

The five estimands, side by side:

• Full-terminal (GrowthBook): E[ Y(W; mature) − Y(0) ]
• Full-incremental (Statsig/GrowthBook): E[ Y(d; global) − Y(0) ]
• Split-incremental (Eppo): E[ Y(W; global) − Y(0) ]
• Split-terminal (Etsy): E[ Y(W; novel) − Y(0) ]
• Reverse holdout: E[ Y(W; mature) − Y(0; loss) ]

A couple of things to read off the list. The treated state is W, the adopted winners with losers off, everywhere except full-incremental, which keeps the full lived configuration d because it never drops the losers from the measurement. The age tag carries the terminal-versus-incremental distinction: mature or novel for a settled window, global for the running read. And the reverse holdout is the only one whose baseline moves, from Y(0) to Y(0; loss).

Note that W = W(S) is a function of the selection time window data — the winners are whichever features cleared the bar during the rollout period. The formal estimands above condition on the realized W. Whether the estimator that produces them is independent of S is the argument in the compare-axis section of the body.

Know what you're measuring with holdouts

Simple idea, but messy in practice. The choice between configs comes down to three things. Whether you want the quarter as your users lived it or the settled value going forward — that's the read axis. Whether novelty effects are strong enough to matter, which separates the two terminal reads. And whether the power cost of a split is worth it for your traffic situation. Get those three straight, and the config mostly picks itself.

Know what you're holding out for.

For the GrowthBook implementation specifically, see the mechanics and the business case.

Visual editors are a staple of experimentation platforms. The promise is simple: let anyone change a website and launch an A/B test without writing code or waiting on engineering. In practice, most visual editors fall short. They break on modern sites, cause flicker, and quietly push you back into writing CSS or HTML the moment anything gets tricky.

We designed the GrowthBook Visual Editor from a blank slate to fix these issues. The result is an AI-first editor that lives in your browser's side panel, and it finally lives up to the full potential of a visual editor. This post walks through what is new and why each piece matters for running A/B tests without engineering overhead.

AI first

Most visual editors are visual right up until they are not. The moment you need to nudge spacing or fix a layout, you are back in a CSS box. GrowthBook’s visual editor moves the editing into an AI prompt by default, so any change you can describe, you can build.

Ask it to change the background color, rewrite a headline, generate a few alternative CTAs, or restyle a section. The editor applies the change on the page instantly, and you approve it, reject it, or ask for something different. Nothing is saved to your variation until you accept it.

The AI also reads the live page, including the real elements and their computed styles, so its suggestions are grounded in your actual site rather than guesses. If it is unsure which section you mean, click to select one or more elements and the prompt uses them as context.

‍

AI image editing and generation

Images are often the highest-impact thing to test and the hardest to change without a designer. GrowthBook builds image editing and generation directly into the editor.

Ask the AI to generate a brand new image from a description, replace an existing image, or produce variations based on the image already on the page. Generated images are automatically cropped to the exact dimensions of the slot they replace, so nothing stretches or distorts. You can generate several options at once and pick the one that fits.

‍

Import from Figma or a mockup

Designs usually start in Figma or as a static mockup, and getting them into an experiment has meant a manual rebuild. GrowthBook can import directly from either.

Connect your Figma account and point the editor at a frame, or hand it an image of the design you want. It analyzes the design and turns it into a variation you can preview and refine, so a mockup can become a running test without rebuilding it by hand. You can also add additional context about how you want the design implemented. 

‍

Built for modern web platforms

This is where most visual editors quietly fail. Modern frameworks and site builders generate class names like css-1x9f3k that change on every deploy. A visual editor that targets those names will silently break the next time your site ships, and the experiment stops running with no warning.

GrowthBook builds durable selectors instead. It ignores hashed and build-generated class names, prefers stable attributes, and anchors to elements in a way that survives redeploys. Your variations keep working after the next release.

‍

Manual mode and global code

When you want precise control, manual mode gives you a full WYSIWYG editor for any element you pick. Edit text, typography, layout, background, visibility, and classes, with each control pre-filled to the element's current values so you are adjusting from where the page actually is. This is also where the most advanced image editor lives. Select any img element to see the options. For changes that span the whole variation, the global CSS and JavaScript editors let you drop in custom code without leaving the panel (or let the AI generate this for you).

‍

No flicker

Flicker is the flash of original content that appears before a variation loads, and it is one of the fastest ways to bias an experiment. GrowthBook avoids it by applying changes before the page renders. Serve the GrowthBook SDK and your feature definitions from your own CDN or edge, and the variation is applied on the initial render rather than after it. Visitors see the variant immediately, with no flash of the control. If you don’t use a CDN, you can still enable the usual no-flicker scripts. See the docs on avoiding flicker for setup.

‍

Edit in 4 languages, and even in dark mode

The editor speaks more than English. Switch the interface into German, Spanish, or Portuguese so teams outside English-first organizations can work in their own language. And if you prefer a darker workspace, dark mode is one click away.

‍

Complete transparency

Every change the editor makes is listed in one place. You can see each edit, preview it on or off, jump straight to the affected element on the page, and edit or delete anything you no longer want. Nothing happens in a black box. You can also test your targeting before launch: the built-in URL tester tells you whether a specific URL would be included in the experiment, using the exact same logic the SDK uses in production.

‍

Try it out!

The rebuilt GrowthBook Visual Editor is AI-first, works on modern sites, and keeps you in control of every change. It makes launching an A/B test faster for everyone on the team, not just the people who write CSS. Get started with GrowthBook to create your first visual experiment, or read the visual editor docs to see how it fits your setup.

‍

When Kevin Yang puts a number on what experimentation has been worth to JPMorgan Chase, it stops the conversation. Over the years, his team estimates the value driven by experimentation and innovation at more than a billion dollars. For most companies, that figure alone would be the headline. For Kevin, who has spent the last six years building experimentation across Chase's digital platforms, the more interesting story is hiding inside it.

"That's only from the winners of the experiments," he told Ashley Stirrup on The Experimentation Edge. "Not even the losers." And then the line that reframes the whole discipline: "A lot of the value, if you think about it, the losers are probably where the value is really coming from."

That idea runs against the instinct of almost every team that ships software. We are trained to celebrate wins and quietly bury losses. Kevin's argument is that the losses are the point. They are the cheapest insurance a business can buy, and the only reliable engine of learning it has.

Listen to the full episode →

Why the losers matter most

The logic is simple once you sit with it. When an experiment wins, you ship it and move on. You rarely interrogate it, because success doesn't demand an explanation. When an experiment loses, you are forced to ask the question that actually makes a team smarter: why did this lose? Was the assumption wrong, or was it the execution? Should we refine the idea, or abandon it?

A losing test also does something the win never does. It stops you from rolling out a change that would have quietly degraded the experience for millions of customers. At Chase's scale, where Kevin says nearly every line of business could be its own company, that protection is enormous. The billion-dollar figure counts the upside of the winners. It doesn't even count the losses avoided by catching bad ideas before they scaled.

That scale is real. Kevin's team supports roughly 100 product teams across Chase's consumer bank, spanning credit cards, checking and savings accounts, mortgages, a sizable travel business, and even call-center and backend systems. They run about 300 experiments a year now, up from just eight in the first year after the infrastructure was in place. The team is built on two pillars: a platform that lets product teams self-serve experiments without needing a data scientist embedded on every team, and a practice arm that embeds experts into strategic initiatives to instill the right culture and decision frameworks.

The chart nobody can read

The cultural work is harder than the technical work, and Kevin has a favorite exercise for it. He puts up a 90-day time series of an app completion rate, a metric where a small relative change is worth millions to the business. Somewhere in that chart, a real and significant change occurred. He asks the room to point to it.

"So far, every time I try this, I don't think I've ever had anybody get it right," he said.

People pick the spikes. They pick the drops. They pattern-match to whatever looks dramatic. But the dramatic movements are just noise and seasonality, the same up-on-Monday, down-on-Sunday rhythm that anyone who has stared at marketing data will recognize. The genuinely valuable change is invisible to the naked eye, buried under variance.

Then Kevin reveals the trick. Add a control group, and a second line appears. The orange control line and the shaded gap between it and the treatment are the impact, finally made visible. "Most of the stuff you're releasing to market is not going to create huge spikes that you can really observe," he explained, "especially with seasonality and everything. So having a control group is super important."

This is the first myth he busts in every training: the belief that you can simply monitor a metric and know whether your change worked. At Chase's scale, where a single percentage point can be worth millions and is almost impossible to see by eye, monitoring without a control is just trusting noise. As Kevin put it, once you start measuring everything, you don't just learn more about individual features. You start to see your whole portfolio at a higher level, and that comprehensive view is what moves an organization in the right direction.

Measuring what's easy to count and hard to interpret

Not every metric behaves. Kevin is careful about engagement in particular, calling it one of those things that is easy to measure and hard to interpret. For a bank, more time in the app is not automatically good. The goal is trust and fast task completion, not screen time.

He gave a concrete example. On the mobile home screen, there's a tile that leads to your credit score. The team debated whether to bring the score forward or tuck it away, worried that surfacing it might stop people from clicking deeper into the feature. They brought it forward, and what they saw was repeat engagement, the healthy kind, alongside satisfaction and retention holding up over time. That's the difference between real engagement and vanity engagement.

Sometimes the right move is to deliberately slow users down. Chase introduces speed bumps in payment flows, including on Zelle, to protect customers from being tricked into sending money to the wrong person. Raw engagement would say to remove the friction. Customer trust says add it. Only a balanced decision framework, agreed on before the results come in, keeps a team honest about that trade-off.

Plan for failure before you run the test

That phrase, "before the results come in," is where Kevin spends a lot of his energy. Confirmation bias, he warns, creeps in precisely when a team never expected to lose. When the loss arrives unplanned, people start hunting for evidence to support what they wanted to do, questioning the metric or the methodology after the fact.

His antidote is to plan for failure up front. "One of the things we want to preach when we work with teams is to plan for failure," he said. Build the playbook for a loss before you run the test. Decide in advance what a loss would mean for your assumptions and what you'll do next. When losing is just the trigger for an already-planned next step, it stops feeling like a defeat, and teams stop gaming the outcome. He points to the iPod, which didn't truly become a winner until its third version, as a reminder that expecting V1 to win is rarely realistic.

The golden era, and the measurement it demands

All of this becomes more urgent in the AI era, which is where Kevin sees experimentation entering a golden age. The reasoning is direct. AI lets everyone ship faster, and soon lets people build and customize their own features. But speed without measurement is dangerous. "If you don't measure them right, your mistakes are going to compound," he said. More output multiplies the cost of being wrong.

AI also pushes teams into a non-deterministic world, where the same input doesn't always produce the same output, and the most important outcomes are qualitative. Ashley offered the example of Khan Academy, a GrowthBook customer whose AI tutor optimizes for cognitive engagement, whether a student is genuinely trying to learn rather than just extracting an answer. They lifted it by 6%, an outcome no automated eval could grade. Evals are strong at QA, at confirming you got the expected answer. They are weak at predicting how millions of real people, asking the same question nine million different ways, will actually behave and whether they leave satisfied or frustrated.

That gap is exactly where experimentation earns its place. As more software gets built faster, by more people, the teams that win won't be the ones shipping the most. They'll be the ones who can still tell which of those ships actually worked. "Everybody becomes a builder," Kevin said. "It's a fun time." It will be a fun time for the teams that remember to measure.

‍

You’re deploying more often than you were a year ago. And there’s a good chance your team has already shipped AI-assisted code into production. Despite these changes, your deployment pipeline still works the same way it always has. It either goes out, or it doesn’t. That’s where the risk is.

The 2025 DORA report found that AI adoption increases throughput and stability. As you ship more code, the surface area for mistakes increases, too. And recovery just takes longer when you don’t have fallback systems or derisking mechanisms in place. 

That is the same operating problem behind JPMorgan Chase's experimentation in the age of AI: AI changes shipping speed, but teams still need measurement systems that catch bad changes before they scale.

However, feature flags change the unit of risk. Instead of every deployment being an all-or-nothing bet, you control who sees what, when, and how fast—and reverse it if needed.

In this article, we’ll walk you through different ways how feature flags can reduce risk during deployment.

1. Decouple deployment from release

Instead of merging code and exposing it to users in the same step, you can use feature flags to decouple those steps. Just wrap the feature with a flag, and deploy code to production with the flag off. Once you confirm everything looks good, you can release the feature independently. The operational impact is on your recovery time. A rollback means flipping the flag off, not undoing your work entirely.

If something breaks, you can actually investigate the specific issue and reverse the deployment immediately. Even companies like Amazon Web Services (AWS) operate this way. They deploy features weeks in advance but hide them behind flags. When executives announce the features on stage, their engineers flip a flag to make them available. As a result, nothing risky happens during a keynote.

This use case requires your flag platform to maintain an independent state per environment, so a flag stays off in production while it’s enabled in staging. 

Platforms like GrowthBook let you use feature flags for all of this and more, free with unlimited environments on every plan. Each environment gets its own API key and independent flag states, so the separation between ‘deployed’ and ‘released’ is enforced at the infrastructure level.

2. Kill switches

Typically, if you have to roll back a deployment, you need to reverse the entire deployment. It takes more time, more coordination, and adds another layer of risk to the issue you’re dealing with.

In fact, DORA’s report found that 56.5% of engineering teams need between 1 and 7 days to recover from failed deployments. Only 21.2% recover within an hour. If you want to be a part of the latter group, you need to use kill switches.

Create a kill switch by wiring a pre-configured feature flag into a feature before it goes live. And when something goes wrong, you just have to flip it off to kill the feature. In 2024, CrowdStrike experienced a global outage when a faulty Falcon sensor update shipped directly to endpoints. It bricked 8.5 million Windows machines, and the recovery had to be done manually for each device.

If CrowdStrike used a staged rollout process and had a kill switch in place, it could’ve dramatically reduced the blast radius of this update.

The speed of a kill switch depends on how fast your flag platform evaluates state changes. If evaluation requires a server round-trip, you’re adding latency to your recovery window. So, choose a feature flagging platform that evaluates flags in milliseconds to enable almost immediate responses. In GrowthBook, kill switches work instantly because flag evaluation happens locally in the SDK.

3. Progressive rollouts

An all-or-nothing release means that a single bad deployment immediately affects 100% of your user base. And the CrowdStrike incident is just one example. That’s why you should use feature flags for progressive or staged rollouts instead.

Instead of releasing to everyone at once, you start with a small percentage of users and expand in increments based on what you observe. This directly reduces the severity of your change failures over time.

The rollout sequence typically looks something like this:

• 5–10%: Validate core functionality and watch for error rate spikes
• 25%: Confirm performance holds under moderate load
• 50–75%: Monitor business metrics like conversion and latency at scale
• 100%: Full release once confidence is established at each stage

Automating the ramp requires two things: deterministic evaluation so users don’t flip between variants mid-rollout, and metric-aware expansion so the rollout doesn’t advance past a stage it shouldn’t. GrowthBook offers both through Safe Rollouts where it automates this process. So, the same user always sees the same variant throughout the rollout. You can monitor results at each stage—and only ramp it up if the observability thresholds have been met.

4. Attribute-based targeting

Progressive rollouts control how many users see a new feature. But it distributes the exposure randomly. That means your highest-value customer could experience the buggy version of your release. You can use attribute-based targeting to avoid this.

Instead of random assignment, you control which users see the feature based on specific properties. For example:

• Organization
• Subscription tier
• Geography
• Device type
• Internal employee status

The impact is similar to progressive rollouts, where you’re reducing the severity of your change failure rate.

Also, this kind of targeting requires flexible rule logic where you can combine attributes with AND/OR conditions and reuse audience definitions across flags without recreating them every time. You can use GrowthBook to do this, as it supports AND/OR rule logic for targeting and Saved Groups for reusable segments like “beta users” or “enterprise tier.” Since it uses deterministic hashing, you can keep the experience consistent based on the attributes you choose.

5. Dark launches

You deploy in staging so that you can dot all your I’s and cross all your T’s. But in reality, you can’t accurately replicate production because the data volume or traffic shapes aren’t the same.

Dark launches resolve this issue by routing real production traffic through a new code path while discarding the output. None of your users sees the feature or experiences any change. But your engineers do see how the new code behaves under real-world conditions. From an operational perspective, dark launches improve your change failure rate by catching regressions before release. You’ll still see the failures, but your users won’t.

Let’s say you’re launching a new AI feature. You can route it to a small percentage of your traffic. Then compare guardrail metrics, such as latency or error rate, against the existing path. You can discard the output and just use the results to iterate from there.

It’s useful for database migrations, infrastructure swaps, algorithm changes, and new API integrations.

6. Guardrail metrics and automated rollback

Even with progressive rollouts, someone still has to watch the dashboard—unless you’re using a platform that alerts you automatically. But that’s not the case for most engineering teams. By the time you notice something, the damage is done.

Guardrail metrics and automated rollback remove that bottleneck. Within GrowthBook, you can define thresholds that are continuously monitored during the rollout. If the threshold is breached, you can expect the rollout to reverse automatically. It reduces the severity of change failures and your mean time to recovery for each incident.

GrowthBook uses sequential testing and one-sided confidence intervals to monitor guardrails continuously. And it’s warehouse-native, so you’re checking against the same data definitions your analytics team already trusts.

The metrics you choose depend on the feature or deployment, but here are a few examples:

• Error rate on affected endpoints (e.g., rollback if error rate exceeds 2% in the treatment group)
• p99 latency on critical paths (e.g., rollback if checkout latency exceeds 800ms)
• Conversion rate on revenue-critical flows (e.g., rollback if purchase conversion drops below baseline)