Version 1.3.0 is here, adding much-needed flexibility for SQL data sources, revision history for features, and a brand-new Flutter SDK! Below is a quick summary of some of the changes. As always, please let us know if you have any feedback or ideas.

Custom Identifier Types (Randomization Units)

Previously, GrowthBook had two hard-coded user identifier types (or randomization units) — user_id and anonymous_id. This worked for some websites, but was not very flexible. For example, B2B products might want to split users by their company instead of an individual user_id. Or an app might only have logged-in users and have no use for an anonymous_id.

Now, you have complete control over the identifier types and can pick the exact ones that make sense for your business. Identifier types can be set from the data source page by clicking on “Edit Query Settings”. This has been one of our most requested features, and we’re excited to finally bring it to you!

Feature versioning and audit logs

When editing a feature, we now queue up changes in a draft version and let you review and publish them all at once, along with an optional commit message. You can see the full revision history and easily revert to a previous state if needed.

You can also now view a detailed audit log that shows exactly what changes were made to a feature and who made them.

Multiple experiment assignment queries

Previously, data sources had a single “experiments’’ query that needed to return all exposure events for all experiments. If you needed different variations of that query for whatever reason, you had to make a bunch of duplicate data sources.

Now, GrowthBook lets you define multiple assignment queries for a single data source. When analyzing an experiment, you choose which assignment query to use.

New Flutter SDK!

We are pleased to announce the release of the GrowthBook Flutter SDK! This SDK was contributed to our open source project by Dhruvin Vainsh and Prince Bansal, Software Engineers at Alippo. You can find the repo here: https://github.com/alippo-com/GrowthBook-SDK-Flutter.

Miscellaneous other features and fixes

• Add the ability to sort goal metrics in an experiment
• Add line numbers and a “copy SQL” button to the “View Queries” modal
• Added SSO/SAML support for GrowthBook Cloud (requires enterprise contract)
• Added support for dimensional analysis when using Mixpanel

And many more changes and bug fixes, which you can read about here: https://github.com/growthbook/growthbook/releases

Our latest version is packed full of usability updates and some of our most requested additions. Check out the changes below. As always, please let us know if you have any feedback or ideas via Slack.

Multiple customizable environments

Instead of just the pre-defined ‘production’ and ‘dev’ you can now define an unlimited number of environments for your feature flags to match your infrastructure (eg: adding staging, and QA). Each environment gets its own API endpoint that only includes features enabled for that environment.

Built-in SQL schemas for GA4, Segment, Snowplow, RudderStack, and Amplitude

Many companies use an event-tracking library to load data into their SQL warehouse. GrowthBook now supports many of these database schemas out-of-the-box — Segment, Snowplow, RudderStack, Google Analytics 4, and Amplitude. If you use one of these systems, you no longer need to write SQL to start analyzing experiment results! Let us know if there are additional ones you think we should support.

Mutually exclusive experiments

We added the concept of Namespaces to the GrowthBook UI to enable mutually exclusive experiments! If two experiments are in the same namespace and their ranges do not overlap, users will only be included in at most one of them. To get started, add your first namespace under Settings.

Per-environment rules

Rules are now defined separately for each environment and can easily be copied and moved between them. Use this to test a complicated targeting rule or experiment on dev before enabling it in production.

Tag manager and customizable colors

There’s a new dedicated page under Settings to manage all of the tags you’re using to organize features, metrics, and experiments. You can also now set custom colors for tags to make it even easier to differentiate them visually.

Quick tag-based filtering

When your list of feature flags and metrics starts to get large, organization becomes key. In addition to our powerful search, you can now quickly filter the list by tags.

Redesigned left navigation

We reorganized the left navigation to make it more intuitive, especially for new users. Everything having to do with analyzing experiment results is now grouped under an “Analysis” section. Project management tools like our Ideas board and Presentation generator now live under “Management”.

Miscellaneous other features and fixes

• Custom aggregations support for Mixpanel metrics
• Ability to delete ad-hoc reports
• Tons of bug fixes and documentation updates

You can read about the full list of changes here.

Version 1.1.0 of our open-source feature flagging and experimentation platform is here, which includes new SDKs and numerous other improvements. Below is a quick summary of some of the changes we released. We’re also live on ProductHunt, and we would really appreciate an upvote if you’re a fan of what we’re building.

Android and Go SDK

We released our Go and Android SDK. You can find the Go SDK here: https://github.com/growthbook/growthbook-golang. The Android version is supported through Kotlin; you can find the repo here: https://github.com/growthbook/growthbook-kotlin . The full docs for all the SDKs is here: https://docs.growthbook.io/lib.

Streamlined feature creation modals

We simplified and restructured the feature creation process so you can now add an initial rule in fewer steps.

Project-scoped APIs and WebHooks

You can now filter webhooks by environment and project. This limits which features trigger the webhook and filters the list of features in the payload. We also added the same filters to the features API endpoint for those of you not using webhooks.

No more ‘pageviews’ data source query

We’ve restructured our SQL query generator and no longer require a ‘pageview’ query, which was a common source of confusion and not applicable to backend or mobile applications.

Miscellaneous other features and fixes

• React Native support in the React SDK
• Support both Datetime and Timestamp data types in BigQuery
• Added Mixpanel tracking instructions, FAQ, and How it works section to the docs.

You can read about the full list of changes here.

We’ve recently released an SDK for our feature flagging and experimentation platform in Go. The library is extremely lightweight, fast and has no external dependenciees- you can use it entirely stands- you can use it entirely stand alone or integrated with our platform. It allows you to easily do feature flagging, variation assignment, and user targeting

Installation is easy:

go get github.com/growthbook/growthbook-golang

Here are some simple examples of how you can use it:

// Parse feature definitions JSON (from GrowthBook API)
features := growthbook.ParseFeatureMap(jsonBody)

// Create context and main GrowthBook object.
context := growthbook.NewContext().WithFeatures(features)
gb := growthbook.New(context)

// Simple boolean (on/off) feature flag
if gb.Feature("my-feature").On {
  // show the feature
}

// Get the value of a non-boolean feature with a fallback
color := gb.Feature("signup-button-color").GetValueWithDefault("blue")
fmt.Println(color)

There are many more use cases that you can read about in our docs. As with all our SDKs, you can use them by themselves to help with feature flag rule evaluation or variation assignment, but the real value comes when you pair them with the GrowthBook API so that you can adjust the values in the JSON payload directly from the UI. For more information, please check out the Go library on GitHub or our full platform.

Experimentation has many parts: choosing the hypothesis, implementing the variations, assigning, results analysis, documentation, etc, and each of these can have their own nuances that make it complicated. Usually, choosing the metric (or metrics) that determines if your hypothesis is correct is straightforward. However, there are some situations where choosing metrics can be problematic, and surface classic issues of Goodhart’s law. Goodhart’s Law says that when a measure becomes a goal, it ceases to be a good measure. It is a close relative of Campbell’s Law, the Cobra Effect, and, more generally, Perverse Incentives. Plainly put, these laws describe conditions where there are unintended negative effects of choosing metrics.

The classic example and etymology behind “Cobra Effect” comes from colonial India, where, as the story goes, they had a problem with too many cobras. To solve the issue, the British government placed a bounty on dead cobras. This caused the locals to start breeding cobras to turn in for the bounty.

A more recent example comes from U.S. News & World Report's Best Colleges rankings. One factor in the ranking is selectivity — how many students applied vs. were accepted. This measure encouraged some schools to game the number of applicants by including every postcard expressing interest as an “application”, and rejecting more students from the Fall admission to increase their ranking.¹

These same unintended negative effects can apply to A/B testing. If you tell your teams to improve one metric, teams can be incentivized to game these metrics. As an example, consider the case where you want to increase pages per visit. One idea that would absolutely work is to paginate your content into smaller and smaller pages (and you can see real examples on many sites). You risk destroying your user experience and annoying your users, but that one metric will almost certainly increase.

Facebook is a classic example of this phenomenon. Facebook is famous for hyper-focusing on growth metrics (eg, active users and user engagement). Teams have these growth metrics as their primary performance indicators, and have bonuses that are dependent on improving them. They therefore have no incentive to improve unrelated metrics that give a more complete picture. Facebook does have teams that measure other impacts of their work, like their Integrity teams or internal researchers, but due to these conflicts of interest, they routinely fail to meaningfully address the concerns. Even when decisions are escalated, growth is put above all. This has led to them building a product that certainly has high engagement, but alienates large segments of the population, and spreads misinformation that has caused real harm in the world (see Jan 6th² or Myanmar³). This relentless focus on a single metric without regard for user sentiment is a prime example of how short-term optimization leads to long-term product degradation, effectively turning your experimentation program into a generator for dark patterns."

Assuming that you have chosen your metrics for your teams carefully to avoid the above problems, and your internal teams are good at avoiding gaming the metrics, there are still other aspects of unintended negative effects to watch out for when choosing metrics with A/B testing.

Correlation, Causation, and Proxy Metrics

A few years ago, we ran an interesting experiment on a freemium content site, where we had a north star metric of improving revenue. The hypothesis for the experiment came from an insight from our data team, who noticed that users who used multiple types of content converted to paid at a much higher rate than normal. With this correlation in mind, our product teams set out to push more users to consume multiple kinds of content with the hope of increasing revenue. For these tests, we used metrics of both the proxy (multiple content use) and the goal (revenue). The results were fascinating and counterintuitive.

What we found after many experiments was that not only did they not increase revenue, but by pushing users to do actions they weren’t naturally inclined to do, we decreased pages per visit and actually reduced revenue. We did succeed, however, in increasing the metric of multiple content use. What we actually proved through these experiments was that this correlation was not causal. If we had just used the proxy metric of multiple content and trusted its causal relationship to revenue, we might have declared these tests a success.

Pushing on metrics can also destroy the original correlation. During the experiments, we looked back at the original correlation, but it had disappeared. We had pushed so many more unqualified users into using multiple content types without the increase in revenue that the original correlation was no longer significant.

Another common reason for using proxy metrics is that that’s all you have. You want to make sure you are testing against the metrics you need, rather than the metrics you have. Think about what the right metrics are to prove the hypothesis, and if you find yourself not being able to test against them, then you have a problem. Using proxy metrics without knowing the effect on the goal metric will mean you are not able to determine causal relationships, and it will limit the effectiveness of your experiments. Make sure your experimentation platform is not the limiting factor in running correct experiments, and run tests with the right metrics.

Conclusion

The lesson for metric selection from Goodhart’s Law is to be thoughtful with your metrics. Try not to use proxy metrics unless you are sure they are causal. Be careful with the implications of your goal metrics and consider what other metrics might also need to be measured to determine what the negative effects could be. Finally, not all experiment results are straightforward to analyze, and careful consideration is needed to determine the right conclusion for your product which is quite often nuanced. With these thoughts in mind, hopefully you can avoid some of the pitfalls of metric-driven decision-making.

[1] G. S. Morson and M. Schapiro, Oh what a tangled web schools weave: The college rankings game (2017), Chicago Tribune

[2] C. Timberg, E. Dwoskin and R. Albergotti, Inside Facebook, Jan. 6 violence fueled anger, regret over missed warning signs (2021), Washington Post

[3] J. Clayton, Rohingya sue Facebook for $150bn over Myanmar hate speech (2021), BBC News

We are thrilled to announce the release of version 1.0 of GrowthBook. We want to thank everyone who has provided feedback or made contributions over the past year and across the previous releases. We are excited for the future of GrowthBook in 2022 and beyond!

Release highlights

The major highlight of this release is the addition of feature flagging to our platform, which fits in naturally with experimentation. We are also releasing our Chrome DevTools extension, which is a huge productivity boost for engineers working with our front-end SDKs. Read about all the improvements below.

Feature Flagging 🔥

Feature flags are now a core part of the GrowthBook platform. Feature flags let engineers wrap their code with conditional checks that are remotely controlled. For example, turning a sales banner on or off, or rendering one of three different versions of a registration form. Instead of just setting the feature flag to the same value for everyone, GrowthBook lets you do advanced segmentation, gradual rollouts, and, of course, run A/B tests.

Feature flags are available today for our JavaScript and React SDKs, with the rest coming soon. Give it a try and let us know what you think!

SChrome DevTools Extensionous 20

As part of our developer focus, we’ve published a Chrome DevTools extension. For those using our front-end SDKs (Javascript/React), this extension adds a new panel to Chrome’s DevTools that makes it easy to see what features and experiments are being used on the page and lets you toggle between states to test all the different variations of your site, all without leaving the browser. Install the GrowthBook Chrome extension here.

My Reports Page

Back in version 0.9.0, we launched Ad-hoc Reports, which let you dig into experiment results in a sandboxed environment. Now, we’re making those reports easier to find and manage. You’ll be able to see all reports for an experiment directly beneath the results. Plus, there’s a dedicated “My Reports” page, accessible from the user menu in the top nav, that shows all of your reports across all experiments

Import Experiment Improvements

You’ve been able to import experiments from your datasource to GrowthBook for a long time, but this action was hidden deep within the settings section. Now, you can import directly from the Experiments page. Just click the “Add Experiment” button, and you’ll see a list of tests to choose from.

Multiple Variation Detection

When analyzing experimental results, GrowthBook will now automatically detect and remove users who were exposed to more than one variation. This can happen for various reasons, such as someone logging in and out of multiple accounts. Usually, this affects only a tiny fraction of users, but if it’s happening more often than expected in an experiment, we will show a warning message above the results with more information.

Other improvements:

• Added custom SSL certificate support for Postgres/Redshift
• Support for custom delimiters for Google Analytics

Plus numerous other enhancements and bug fixes. You can read about the full list of changes here.

Just in time for the holidays, this release of the GrowthBook open source experimentation platform includes a number of the most requested features, as well as some huge performance improvements. We cut the number of executed SQL queries in half, and the ones that remain are now up to 10x more efficient! Here are some highlights:

North Star Metrics

It’s common for a company to set one or two North Star metrics to rally the team behind and drive experimentation efforts. Now you can specify these important metrics in GrowthBook and see how they are improving over time and which experiments had a measurable impact on them.

Ad-hoc experiment reports

You can now create ad-hoc reports of experiment results, and adjust parameters such as dates, metrics, and even add custom SQL filters. This allows data teams to really dive into things and explore without affecting the main experiment results for anyone else. And if you need to move beyond what a report can do, you can easily export to a Jupyter notebook at any time to keep going deeper.

Experiment Update Frequency

We now support adjusting the update frequency for experiment results. You can choose to refresh automatically based on the age of results, a fixed Cron schedule, or even turn off automatic updates entirely. This gives you complete control over how and when queries hit your data source.

Custom metric aggregations

We’re continuing to make metrics more flexible and customizable so they can support all of your use cases. The latest feature lets you control how metrics are aggregated for each user in an experiment when they have multiple conversions. Previously, the only option was to sum the values of the conversions, but now you can do anything supported by your SQL engine (avg, median, min, max, etc).

Other improvements:

• Improved metric query performance, plus added support for SQL template variables for more advanced optimizations
• Added a cache layer for faster experiment queries
• Automatic CDN invalidation for experiment changes on GrowthBook Cloud.

Plus numerous other enhancements and bug fixes. You can read about the full list of changes here.

As always, if you have any questions or feedback, we would love to hear from you- you can reach us on Sslack.

We wish you all the best for the holiday season and a happy New Year!

We are excited to announce the release of GrowthBook version 0.8.0 which includes some large improvements and features:

All new design!

The design of GrowthBook got an overhaul from top to bottom, with new navigation, colors, and layout to improve the user experience. Try it out on our cloud version.

Up to 10x faster statistical analyses

We greatly reduced the number of round-trip requests between our TypeScript API and our Python statistics engine and optimized the data processing steps. This resulted in a huge performance gain, especially noticeable in experiments with many metrics or a high-cardinality dimension selected. We’ve seen some analyses go from 50+ seconds of processing time down to under 5 seconds.

Dimensional Analysis Improvements

We launched several new improvements to the dimensional analysis features of our experiment results:

Automatic dimensionality reduction

If you have a dimension with high cardinality (e.g., a “country” dimension with 200 possible values), GrowthBook now automatically groups values to reduce the number of statistical analyses that are performed. It will leave the top 20 values (by number of users) as-is so you can still draw useful inferences, but it will group everything else into “(other)”. This reduces the chance of seeing a false positive and makes the report easier to read.

Jupyter notebooks of dimensional breakdowns

We reworked and improved our popular Jupyter Notebook Export feature. Now you can export any experiment results page as a notebook, including dimension breakdowns. Plus, we’ve added many new columns to our Pandas dataframes to give you maximum flexibility when playing around with the data. Make sure to install the latest version of gbstats (0.2.0) from PyPi to support this new notebook format.

Mutually exclusive experiment support

As you scale up experimentation at your company, you will inevitably run into situations where you want to run two conflicting experiments at the same time. With the introduction of namespaces in our JS and React SDKs (others coming soon!), you can now safely achieve this. Namespaces work by splitting users into 1000 buckets and letting you specify the bucket range each experiment within that namespace should apply to. As long as the bucket ranges of the two experiments don’t overlap, they will be mutually exclusive. You can segment your product into as many namespaces as you want and greatly improve your test velocity!

Plus numerous other enhancements and bug fixes. You can read about the full list of changes here.

We recently dropped version 0.7.0 of the GrowthBook experimentation platform. Here are some of the cool new features:

Experiment results over time

One of our most requested features is now live! There’s a new built-in Date dimension you can use to view how your experiment performed over time. The line shows the uplift with a shaded 95% confidence interval. You can toggle between viewing each day independently (the default) and cumulatively. Performance over time can help detect novelty or primacy effects.

Full statistics when exploring by dimensions

Now, when you do a dimensional analysis on the results of an experiment, you can see the full statistical results, with the chance to beat control, risk, and a percent change graph. This is enabled by default when there are fewer than 5 unique dimension values and can be toggled on manually for dimensions with more values.

Experiment Segments and SQL Filters

You can now apply a segment to an experiment to limit who is included in the analysis. For more control, you can also now add a custom WHERE clause to the experiment query. These new settings, along with existing ones that control the analysis, are now all easily accessible in a single place on the results page.

Added per metric minimum percent change

We added support for a customizable minimum percent change per metric. If an experiment fails to move a metric by this amount, we consider the change insignificant. This is related to the max percent change, where if an experiment changes a metric too much, it’s considered suspicious. These settings help encourage best practices when it comes to interpreting results.

Other improvements:

• Easily clone/duplicate a metric
• Added filtering and sorting to the metrics pages
• Added filtering and sorting to past experiments
• Added the ability to delete segments and dimensions

Plus numerous other enhancements and bug fixes. You can read about the full list of changes here.

Have a great week!