The Uplift - Modern Product Development Blog

The Uplift Blog

Experiments

Designing A/B Testing Experiments for Long-Term Growth

Elyse Fox

Apr 6, 2026

min read

Ronny Kohavi — Stanford PhD, Ex-VP and Technical Fellow at Airbnb, formerly Microsoft and Amazon — is one of the top cited researchers in Computer Science and a leading voice in experimentation. He recently joined Luke Sonnet, Head of Experimentation at GrowthBook, for a webinar sharing best practices, mistakes to avoid, and surprising insights into how often experiments actually succeed. Watch Designing Experiments for Long-Term Growth on demand.

This article covers the key principles Ronny and Luke shared for designing experiments that drive long-term growth — from understanding the importance of experimentation, why you shouldn’t ship on flat results, the key metrics you should track, and how to create a shipping criteria framework. Whether you're just getting started with experimentation or looking to sharpen how your team makes decisions, these are the foundational concepts that separate programs that deliver real impact.

In science, randomized controlled experiments are the gold standard, sitting at the top of the hierarchy of evidence. A/B tests are the online equivalent and the most reliable tools teams have for determining whether a change actually has an effect — whether that's a new feature, a UI change, a pricing change, or a backend optimization.

The problem is that most teams haven't done the harder work first: agreeing on what success actually looks like before the data comes in. Without that foundation, even a well-run experiment produces a result nobody knows how to act on.

Experimentation is How You Stop Guessing: Embrace the High Failure Rate

Humans are systematically bad at predicting what will work and assessing the value of ideas. You cannot reliably judge which ideas are valuable before testing and will be wrong far more often than most teams expect. An effective experimentation program is critical for focusing effort toward what actually works.

Here is some surprising success rate data from across the industry:

Company	Experiment Success Rate	False Positive Risk — Probability that a statistically significant result is a false positive	Reference
Microsoft	33%	5.9%	Kohavi, Crook and Longbotham 2009
Avinash Kaushik	20%	11.1%	Kaushik 2006
Bing	15%	15%	Kohavi, Deng and Longbotham, et al. 2014
Booking.com	10%	22%	Manzi 2012, Thomke 2020, Moran 2007
Google Ads	10%	22%	Manzi 2012, Thomke 2020, Moran 2007
Netflix	10%	22%	Manzi 2012, Thomke 2020, Moran 2007
Airbnb Search	8%	26.4%	Kohavi

Microsoft's 33% success rate stands out, but this came at a cost. Significant upfront work went into scoping and refining ideas before they ever entered an experiment, which directly impacted that number.

The median organization sees roughly 10% of experiments move the metrics they were designed to improve. Given this success rate, we can compute the False Positive Risk (FPR) — the probability that a statistically significant result is actually a false positive. At a 10% success rate with standard thresholds (𝛼=0.05, 80% power), that risk is around 22%, meaning roughly 1 in 5 'successful' experiments are actually false positives. Most teams assume p < 0.05 means they will rarely make mistakes, but the math shows otherwise.

The most impactful teams are the ones with the infrastructure to test fast and realign priorities based on evidence. A $120M improvement at Bing sat in the backlog for months because nobody thought it was worth testing. At Airbnb, the biggest win was a one-line code change. Neither of these could have been predicted. Both required running the experiment.

The Importance of Building–and Aligning on–A/B Testing Key Metrics

An experimentation program is only as good as the metrics it optimizes for. These metrics include:

Success or goal metrics: Defines why an organization or product exists and what success looks like (stock price, revenue, market share, etc.) These are the real objectives, but are not easy to move or measure in the short-term.
Driver metrics: Short-term metrics that are the signals believed to predict movement in success metrics. These are what you actually measure to signal success.
Overall Evaluation Criterion (OEC): The weighted combination the organization agrees to optimize for, typically composed of a few success and driver metrics. Defining a good OEC is one of the hardest and most important things an experimentation program does.

What Goes Wrong Without a Good OEC

Real-world scenarios from search engines (Bing, Google) and booking sites (Airbnb, VRBO) illustrate how badly things can go wrong with poor OECs, despite well-meaning intentions.

The Search Engine Example

At Bing, naively using queries per user as the OEC would have led to very poor decisions. The example he gave was a ranking bug that returned terrible search results. This increased queries by 10% due to users reformulating queries several times and increased ad revenue by 30%. The short-term metrics look great, but the product is broken.

More optimal metrics to track here would be to minimize queries per session (users should be able to find answers quickly) and maximize sessions per user (repeat usage indicates high value). Bing has a suite of metrics they actually track, including sessions per user, queries per user, time to success, revenue per user and more. We'll cover a framework for identifying and aligning on good OECs later in the article.

A Booking Site Example

Similarly, a booking platform such as Airbnb that ignores satisfaction signals like user rating and instead optimizes purely for conversion rate is optimizing for the wrong thing. If users book listings they end up hating, they don't return.

A better OEC would also include a measure of satisfaction, such as the user's star rating, so you can build machine learning models that predict whether this user will book a listing they love and rate five stars. Deciding on the trade-off between multiple metrics, such as revenue and user satisfaction, is a key business decision.

The Flat Result Trap: The Most Expensive Mistake in Product

Getting your OECs right is important, but only if you're willing to act on what the data actually tells you. A flat result means an experiment didn't produce a statistically significant improvement in the OEC. Shipping flat means deploying that feature anyway. It was discussed that this is a decision error in nearly every case.

One example from Bing was a major effort with ~100 engineers to introduce a third pane to the search window. The experiments failed to show value, but it shipped to all users anyway because it was determined to be a strategic business move. A year later after countless additional experiments failed to show value, the 3^rd pane was rolled back at significant costs to Bing.

Had Bing acted on what the data told them to begin with, they could have failed much faster, avoiding months of sunk cost and instead redirected their engineering resources toward something that actually moved the needle.

Debunked: Common Justifications for Shipping Flat

Ronny shared the four primary reasons he has seen teams use to rationalize the decision to ship flat and dives into the real implications of each.

Justification #1: It’s flat, we’re not hurting the users or business

A flat result doesn't mean no effect exists. All it tells you is “we didn’t find enough evidence of an effect.” The experiment could simply be underpowered. "Not statistically significantly worse" is not the same as safe to ship. The true effect could still be negative.

Justification #2: Team morale depends on shipping:

Shipping a flat feature to protect morale means celebrating shipping rather than actually moving goal metrics, which can also complicate the codebase and require maintenance costs. The culture should be results-oriented and simply recognize that many ideas fail. Hold a learning review, share what was discovered, and move on. Failures that generate learning are worth celebrating.

Justification #3: It’s an enabler for future work:

You can cut through this justification with one question: if we ship this and deprecate the old version, would we ever roll it back? At Bing, the answer was yes. Every flat enabler that ships becomes code that must be maintained and a foundation you'll keep building on even when the follow-on value never arrives.

Justification #4: It’s strategic:

Strategic conviction is not a substitute for evidence, and as the data shows, even small changes are hard to predict correctly. Set a vision, but move toward it in small, testable steps. Test a meaningful component first, get data, then adjust.

A Framework for Making Better Experimentation Decisions

With the importance of good metrics and understanding of what can go wrong without them clearly laid out, the conversation then shifts to a practical approach for building a decision framework that connects short-term measurements to long-term goals without overcomplicating the process.

Bridging Short-Term Experimentation Metrics to Long-Term Goals

The messy reality is that most measurable short-term metrics don’t align 1:1 with business goals, so we must instead build frameworks to do so.

Start by identifying your long-term goals and what you can actually measure. From there, identify the short-term metrics that are the strongest indicators of those long-term goals. These are the signals that move in the right direction when the product or business is genuinely improving.

Once you've identified the right metrics, put guardrails in place. Guardrails are secondary metrics you monitor to ensure that improving your primary metric isn't coming at the expense of something else that matters, such as revenue, retention, or user satisfaction. They don't have to move, but they can't go backwards.

A word of caution: overcomplicating things and tracking too many metrics can make it difficult to act. Before running an experiment, think critically about what you would do if your metrics told conflicting stories afterward. This exercise forces clarity around prioritization and how you make business decisions around tradeoffs. The goal is to identify the key signals you can build a decision framework around so you know exactly how you'll act on them.

A Real-World Shipping Example: LLM Chatbots

An example that highlights this concept is an AI chatbot company. They can't measure customer lifetime value in a two-week experiment. Instead, they’ll need to look at the short-term metrics that signal value, such as distinct sessions per user, topic breadth, short-term subscription conversion, and how often responses are copied externally. Build a framework connecting these to the long-term goal, validate against historical data, and you have an OEC you can actually experiment on.

But throwing all of these metrics into your results dashboard can complicate the picture. If some results are flat or vaguely negative, while others are statsig negative, and others are statsig positive, then how do you make a shipping decision?

Most metrics in a typical experiment cluster near zero. A single strong signal stands out, illustrating why clear shipping criteria matter before results come in.

This is exactly where clearly defined shipping criteria earns its value.

Shipping Criteria: Enabling Independent Shipping Decisions at Scale

Translate your metrics into explicit shipping criteria that are determined prior to an experiment launching. This is a decision framework that enables independent shipping decisions and eliminates bias from decision-making during the evaluation phase.

Some decisions are very straightforward, such as the example below. With the revenue change being equal, you would choose the latter with higher Daily Active Users.

When revenue is equal, the choice is clear: ship the variation with higher DAU. Defined criteria make this decision automatic.

However, a clearly defined framework for shipping criteria becomes increasingly necessary in situations where metrics conflict, such as in the example below, where DAU is higher in the first experiment, but revenue is higher in the second. In this situation, you need to understand the tradeoff between these metrics that you’re willing to accept when shipping.

This approach encodes your decision-makers' preferences into a repeatable framework so shipping decisions are consistent, defensible, and free from bias.

When DAU and revenue point in different directions, you need a pre-agreed framework to make the call. This is exactly where shipping criteria earn their value.

Luke’s Twitter Example

An example from Twitter highlights how this works in practice. Daily Active Users (DAU) was a key metric for Twitter, but they wanted to make sure that people were using the product repeatedly and over time to see that they're getting value out of it in a wide variety of applications. Some of the measured indicators included tweets created, likes, and other forms of engagement. They used the decision framework below to determine when to ship:

If DAU is up and stat sig → ship
If DAU is negative → rollback
If DAU is up, not stat sig and no guardrails are negative:
- If engagement metrics are up (tweets created, likes, etc.) → ship
- Otherwise → experiment review
Murky results → rollback

This type of framework scales. It forces tradeoffs to be agreed on before you're under pressure from a live result.

A key note to remember is that your metric models will likely drift over time. This is something teams need to revisit regularly as their product and business evolves. The metrics that predicted success a few months ago may not be the right ones today.

Closing: Shift the Experimentation Culture

Ronny and Luke close with a shared belief: the teams that win at experimentation aren’t always the ones with the most resources or sophisticated tools, but the ones that have built a culture around learning.

The most important piece of advice is to shift the organizational mindset from celebrating shipping to celebrating learning. Most ideas will fail. The teams that internalize this stop treating failed experiments as something to hide and start treating them as the mechanism by which they get smarter and faster over time.

That cultural shift is supported by the practical framework Luke outlined. When you have clearly defined metrics, explicit shipping criteria, and a shared understanding of your tradeoffs, experimentation becomes the foundation for confident, independent decision-making at scale.

Key Takeaways

Most experiments fail. The median industry success rate is ~10%, meaning you will be wrong far more often than you expect. An effective experimentation program is how you find what actually works.
False positive risk is higher than most teams realize. At a 10% success rate, roughly 1 in 5 "winning" experiments are actually false positives, even when running at p < 0.05.
Your experimentation program is only as good as the metrics it optimizes for. Poorly defined OECs lead to decisions that look good on paper, but break the product.
Shipping flat is a decision error in nearly every case. "Not statistically significantly worse" is not the same as safe to ship. The true effect could be negative and the code will have maintenance costs.
Short-term metrics rarely align 1:1 with long-term business goals. Build an explicit framework connecting the two and put guardrails in place to protect what actually matters.
Define your shipping criteria before the experiment runs, not after. This eliminates bias, enables independent decision-making, and forces tradeoffs to be agreed on in advance.
Shift the culture from celebrating shipping to celebrating learning. The teams that win at experimentation are the ones that treat failed experiments as the mechanism by which they get smarter.

Want to go deeper? Ronny teaches two online courses on Maven

Accelerating Innovation with A/B Testing: Ronny’s flagship course and recommended starting point for most practitioners

Advanced Topics in A/B Testing: A follow-on to Accelerating Innovation with A/B Testing for practitioners with a solid foundation in p-values, statistical power, and OEC design

‍

Releases

Product Updates

1.9

GrowthBook Version 1.9 🚀

Jan 24, 2023

min read

We’ve been hard at work on GrowthBook 1.9 for the past two months, and are excited to release one of our biggest updates ever! This release includes a Frequentist statistics engine, our GrowthBook Proxy server, scheduled feature flags, and event-based webhooks.

Frequentist Stats Engine

Does your team have a strong opinion about Frequentist vs Bayesian statistics? You can now select which statistics engine you want to use when analyzing experiment results. By default, we will continue to use Bayesian statistics, but you can change this on our general settings page. We have a lot planned for our Frequentist engine in the near future — variance reduction with CUPED, sequential analysis, and more, so stay tuned!

GrowthBook Proxy Server

The GrowthBook Proxy server sits between your application and GrowthBook. It turbocharges your GrowthBook implementation by providing extra speed, scalability, security, and real-time feature rollouts. We’ve also made substantial improvements to our JavaScript and React SDKs to better take advantage of these new capabilities. Check out the docs here — https://docs.growthbook.io/self-host/proxy

Feature Flag Scheduling

You can now schedule feature rules to be enabled or disabled at specific times. No more waking up at midnight to turn on a sales banner or setting calendar reminders to turn off experiments in two weeks. This feature is available on Pro and Enterprise plans.

Event-based Webhooks

One of our goals is for GrowthBook to integrate with all of the existing tools you use — Slack, Jira, DataDog, you name it. We started this effort in the last release with our REST API and are continuing it now with Event-based Webhooks. As we expand both of these systems in the coming months, you will be able to build complex custom workflows — for example, “post in Slack every time an experiment reaches significance” or “turn off feature flag X if DataDog detects an increased error rate.”

Other Features and Improvements

Override metric settings on a per-experiment basis
Limit metrics and data sources to specific projects
Added the ability to easily duplicate features
Archivable targeting attributes
SQL tester for segments, dimensions, and metrics

Plus many more changes and bug fixes, which you can read about here: https://github.com/growthbook/growthbook/releases

News

4000 Stars!

Jan 14, 2023

min read

GrowthBook now has over 4000 stars on GitHub! A big thank you to all the community members who contributed to building GrowthBook and helped make it the best open-source feature flagging and experimentation platform. (https://github.com/growthbook/growthbook)

Releases

Product Updates

1.8

GrowthBook Version 1.8

Nov 15, 2022

min read

GrowthBook continues to improve, and in this version, we’ve launched some highly requested features, including a new REST API, a Java SDK, and advanced permissions. The highlights of the release are below.

We’re doing a live event to demo all the new features in this release and answer your questions. You can join us here.

REST API

You can now list and toggle feature flags programmatically with our new REST API. Use this to integrate GrowthBook into your CI/CD pipelines and internal admin tools. This is just the start; we have many more endpoints planned, which we will release over the coming weeks and months to enable even more integrations and use cases. Stay tuned!

Java SDK ☕

We are proud to release an official Java SDK for GrowthBook! Check out the docs and examples at https://docs.growthbook.io/lib/java

Fine-grained permissions by environment and project 🔑

Environment and project-scoped permissions UI showing per-environment role assignments for Pro and Enterprise users

We added advanced permissions to our Pro and Enterprise plans, available both on GrowthBook Cloud and when self-hosting. You can now restrict access by environment and assign project-specific roles to users. This opens up a ton of use cases. For example, let engineers make changes to feature flags in dev, but require admin approval before publishing to production.

New documentation site

-We launched a completely new documentation site, now with built-in search powered by Algolia. We also added a table of contents to every page to make navigation easier.

Encrypted Feature Flags

Encrypted SDK endpoint settings hiding feature flag definitions from client-side network inspection

When you use feature flags in a client-side application, it’s possible for technically savvy users to inspect your network requests and see which features you have defined and how you’re rolling them out to users. We added an option to encrypt the SDK endpoint so features won’t be exposed in plain text anymore. Read more about this feature and its limitations in our docs. Encryption is only available with our Pro and Enterprise plans.

VS Code extension

We launched our first ever VS Code extension! Configure the extension with your GrowthBook account info, and it will list all of your feature flags directly in the UI. Developers can now easily use our SDKs without constantly switching back to the GrowthBook application. We’re always trying to improve the developer experience, so let us know what else you’d like to see added to this extension!

Other Features and Improvements

“Test Query” button to debug experiment assignment queries
Improved sorting and searching throughout the app
Auto-authentication and default catalog support for AWS Athena
Better formatting of SQL queries

Plus many more changes and bug fixes, which you can read about here: https://github.com/growthbook/growthbook/releases

Releases

Product Updates

1.7

GrowthBook Version 1.7

Oct 11, 2022

min read

One of our biggest releases yet! This version includes over 50 highly requested features and bug fixes. We are super excited to share the highlights of the release. As always, please let us know if you have any feedback.

🌙 Dark Mode

At GrowthBook, we always strive to build a great developer experience, and sometimes that means embracing the dark side. Our new Dark Mode is more friendly for the eyes, extends laptop battery life, and just plain looks cool. We’ll try to detect your OS preferences automatically, but you can switch themes at any time in the top nav bar.

➗ Ratio Metrics V2 with the Delta Method

Back in version 1.4, we added initial support for Ratio Metrics, but there was a big restriction — the randomization unit and analysis unit had to be the same. If you split traffic per user, all of your metrics had to also be per-user. That means you couldn’t do things like “Pages per Session” or “Average Order Value”, which, instead of users, are based on sessions and orders, respectively.

In this latest release, we lifted this restriction. You can now have metrics based on any unit — orders, sessions, page views, whatever you want. Behind the scenes, we use the Delta Method for variance correction to make sure our statistics engine can give you accurate results, no matter what units you pick.

🎯 Reusable Targeting Groups

Saved Groups UI showing a reusable list of user IDs for targeting features across multiple experiments

One of our most requested changes is now live! Instead of copy/pasting lists of user IDs between feature targeting conditions, you can now define that list once in a Saved Group and easily reference it from multiple places.

For example, you could make a Saved Group called “Enterprise Customer Ids” and use it to release all of your new enterprise features. If you later add or remove a customer from the group, it will automatically update all of the features. We have a lot planned for this section in the future, including the ability to populate groups from a SQL query, so stay tuned!

Different Attribution Models

Attribution model selector showing First Exposure vs All Exposures options for experiment metric analysis

The attribution model is how GrowthBook decides which metric conversions to include in an experiment analysis. Until now, we have always used a “First Exposure” model, where only a user’s first time viewing the experiment is considered. If they came back and viewed the experiment again a week later, we ignored it.

With this release, you can now choose to use an “All Exposures” attribution model instead, in which these subsequent experiment views are also considered. Depending on the experiment, this can dramatically reduce the amount of time needed to reach significance, so we highly recommend trying it out to see if it makes sense for your use case. If we get enough positive feedback, we may decide to make this the new default model in the future, so let us know your thoughts!

Self-hosted Enterprise SSO

Enterprise SSO configuration supporting self-hosted GrowthBook deployments with custom identity providers

We overhauled our authentication system to support Enterprise SSO, no matter where GrowthBook is deployed — whether it’s on our managed GrowthBook Cloud or in your own infrastructure. Reach out to sales@growthbook.io if you’d like to learn more about our Enterprise offerings.

Microsoft SQL Server Support

Microsoft SQL Server added as a supported data source for experiment analysis in GrowthBook

We added Microsoft SQL Server as an officially supported data source. Our goal at GrowthBook is to let you experiment on top of your existing data infrastructure, no matter where it lives. This brings us one step closer to that vision. Let us know what other data sources you’d like to see us add next!

Other Features and Improvements

Support for Snowflake authenticated proxies when self-hosting
Add documentation and examples for Swift and C# SDK
Invite multiple team members at once
Added SSL support for MySQL and Trino/Presto data sources
Validate SQL and feature values before saving
Added experimentId as a SQL template variable
Allow OR in Mixpanel metric event names to match against multiple events

And many more changes and bug fixes, which you can read about here: https://github.com/growthbook/growthbook/releases

News

Platform

GrowthBook is Now SOC 2 Compliant!

Oct 6, 2022

min read

From the start, we built GrowthBook with privacy and security as core principles. We recognize that building this trust is critical to our success as a feature-flagging and A/B testing platform. This focus has affected many aspects of how we have architected our product, and now by completing the System and Organization Controls (SOC) 2 Type 1 audit, we demonstrate how GrowthBook safeguards customer data and ensures good security practices.

SOC 2 is an audit conducted by certified third-party auditors who check an organization against trust criteria. The official audit report provides a thorough review of the GrowthBooks systems, including the suitability of the design and the operating effectiveness of controls in achieving the Trust Services Criteria: security, availability, confidentiality, and processing integrity.

You can learn more about GrowthBook’s security and privacy philosophy, and request a copy of our SOC 2 report, on our site.

Releases

Product Updates

1.6

GrowthBook Version 1.6

Sep 19, 2022

min read

Version 1.6 of GrowthBook includes big UI improvements when installing and configuring the platform, plus lots of time-saving improvements for existing users.

New data source form and pre-built schemas!

We’ve added built-in support for tons of additional event sources, including Firebase, Matomo, Heap, Jitsu, and Freshpaint! When adding a new data source, you’ll be presented with the above screen, and based on your event source, we will pre-fill SQL queries and settings for you to help you get started quickly.

New Getting Started flow

New GrowthBook onboarding flow walking through SDK install, feature flag setup, data source connection, and metric definition

We added a brand new onboarding flow when first installing GrowthBook. We also added a persistent “Get Started” section to the left nav so you can easily get back to where you left off. This new setup process will walk you through installing our SDK, adding a feature flag, connecting to a data source, and defining metrics.

Always show user counts in experiment results

When dealing with new experiments or experiments with low traffic, it’s not uncommon to have 0 conversion events on your goal metrics.

This change allows you to see the total traffic exposed to the experiment even when the number of conversion events is zero.

Support optional experiment/variation name columns

Experiment exposure query settings showing optional variation_name and experiment_name columns for more readable results

You can now select variation_name and experiment_name in the experiment exposure query to make the results more human-readable. This setting can be adjusted in the ‘edit data sources’ modal.

Miscellaneous other features and fixes

Added the ability to change roles on invites (thanks reecenil!)
Added dev containers support for easy developer setup
Added the ability to rename and delete namespaces
Update how we display suspicious results

And many more changes and bug fixes, which you can read about here: https://github.com/growthbook/growthbook/releases

Releases

Product Updates

1.5

GrowthBook Version 1.5

Aug 8, 2022

min read

We are pleased to announce the release of GrowthBook 1.5! In this version, we redesigned the experiment page, added a CSV export, and a lot more. As always, please let us know if you have any feedback!

New UI for experiments

The redesigned experiment page now features everything in a single view instead of needing to switch back and forth between different tabs and menus. Also, you can now quickly see whether or not an experiment has an activation metric or segment applied, as well as a history of all the experiment phases. We’d love to know what you think!

New “How To” guides

Step-by-step setup guides for Mixpanel, RudderStack, BigQuery, GA, and Next.js in the GrowthBook docs

We have created step-by-step guides for setting up GrowthBook with Mixpanel, RudderStack, BigQuery, GA, and Next.js, with more coming soon. You can check out the guides at https://docs.growthbook.io/guide and let us know what other ones you think we should have!

Export experiment results as CSV

Experiment results CSV export option alongside the existing Jupyter Notebook download for deeper data analysis

Way back in version 0.5, we added the ability to download results as a Jupyter Notebook. This was great for data teams to dive in and perform deeper analyses, but less technical users were out of luck. Now you can also download the results as a CSV file, which you can open in any spreadsheet tool. This furthers our quest for data transparency and lowers the barrier to truly owning your own data.

Support Google Cloud Storage

Google Cloud Storage configuration for self-hosted GrowthBook deployments on GCP

Screenshots and images are an important part of documenting features and experiments. Almost exactly 1 year ago, we added the ability for self-hosted deployments to use Amazon S3 to securely store uploaded images. In this release, we added support for Google Cloud Storage for all of you hosting GrowthBook in a GCP environment.

Activity log

You can now watch specific features for changes by clicking on the Eye icons on the feature list. Then, you can see a list of all their recent activity by clicking the Bell in the top navigation bar. Coming soon, you’ll be able to set up email notifications for all your watched features, so you can easily stay up to date on what’s happening.

Owners

Owner field on features, metrics, segments, and dimensions for tracking team responsibility in GrowthBook

We added an “Owner” field for features, metrics, segments, and dimensions. Use this to keep track of which team or individual is responsible. This is great when onboarding new employees to your GrowthBook account since it’s an easy way for them to know who to ask if they have any questions about how something is defined or implemented.

Miscellaneous other features and fixes

Clickhouse error caused by missing SQL aliases #446
Control the default environment toggle states for new features #428
Metric tooltips on experiment results with helpful info #417
Sort projects dropdown alphabetically #456
Update dependencies — React 18, Next.js 12, Tailwind 3, Typescript 4.7 #459

And many more changes and bug fixes, which you can read about here: https://github.com/growthbook/growthbook/releases

Releases

Product Updates

1.4

GrowthBook Version 1.4

Jun 29, 2022

min read

Our latest version is here with some highly requested updates, including ratio metrics, and a new UI for showing experiment traffic splits. Check out the changes below. As always, please let us know if you have any feedback or ideas.

Ratio metrics! 🎉

Ratio metric configuration showing denominator selection for metrics like Revenue per Signup or Checkout Completion Rate

By default, metrics are evaluated against all users in an experiment. Now, we let you customize this by setting a denominator. For example, you can now create ratio metrics like “Revenue per Signup” or “Checkout Completion Rate”. Stay tuned for more updates to ratio metrics to handle even more advanced use cases!

New experiment assignment UI

We improved the UI when creating experiment rules for features. We now let you control the exposure percent independently from the traffic split. The preview at the bottom also now accurately represents how our bucketing algorithm works and shows how you can safely scale the exposure percent without causing users to switch variations.

New permissions and roles for feature flags

Revamped permissions and roles UI showing feature flag access controls as part of the role system

We revamped our permission system to incorporate feature flags into the roles. This change lays the groundwork for future improvements, such as per-environment and per-project permissions.

Syntax highlighting for SQL and Python input

SQL and Python code editor with syntax highlighting, line numbers, and indentation support replacing plain text inputs

There are several places in the GrowthBook UI where we ask users to write code — usually either SQL or Python. We switched from using a plain text box to a full-featured code editor — the same one used by Mode analytics and others. This brings syntax highlighting, easier indenting, line numbers, and more highly requested UX improvements!

Ability to archive features

Archive feature option removing a flag from API and webhook payloads while preserving full revision history

You can now archive features and prevent them from appearing in API and webhook payloads without deleting them. Archived features are also hidden by default in the feature list. This is a great way to remove a feature while keeping the full revision history as areference in case you need it later.

Support for feature flags in Ruby SDK

Ruby SDK release completing feature flag support across all GrowthBook SDKs with a shared 300+ test case suite

The past couple of months, we’ve been updating all of our SDKs to support feature flags, and we just completed the final one — Ruby! In addition to feature flag support, all of our SDKs now use the exact same test suite (300+ test cases) to ensure we maintain full cross-platform compatibility. You can find the Ruby SDK here.

Miscellaneous other features and fixes

Search/Filter by tag improvements
Feature discussion threads #402
Admins can reset users’ passwords when self-hosting #378
Search by toggled environment on feature list #366
Support string, number, and boolean comparisons in Mixpanel metric conditions #355

And many more changes and bug fixes, which you can read about here: https://github.com/growthbook/growthbook/releases

News

GrowthBook hits 3,000 stars on GitHub! 🎉

May 25, 2022

min read

In under 1 year since we released the first open source version of GrowthBook on GitHub, we’ve passed over 3,000 stargazers. It has been an amazing journey building GrowthBook with our community.

Ready to ship faster?

No credit card required. Start with feature flags, experimentation, and product analytics—free.

Get Started

Book a Demo

Simplified white illustration of a right angle ruler or carpenter's square tool.

White checkmark symbol with a scattered pixelated effect around its edges on a transparent background.