GrowthBook is a warehouse-native feature flagging and experimentation platform - which means, instead of having to send data to us, GrowthBook integrates with your existing data infrastructure. Once you connect GrowthBook to your Data Source, you can write SQL to configure Metrics, Segments, and more.

But writing SQL can be challenging. Nearly 36% of the engineers surveyed in Stack Overflow’s 2022 Developer Survey dreaded writing SQL*. To make it easier to define Segments, Metrics, and Dimensions, we built the GrowthBook Schema Browser.

Before, unless you were intimately familiar with a particular data source, you may not have known exactly what schemas, tables, and columns were available, requiring you to switch back and forth between GrowthBook and your data source. Now, with the Schema Browser, you can easily see and search through all of the schemas, tables, and columns available within a data source.

As of GrowthBook 2.0, all new BigQuery and Postgres data source connections will automatically support the new Schema Browser. If you have an existing Postgres or BigQuery data source defined within GrowthBook, simply go anywhere you can write SQL, and you’ll see a call-to-action to generate an Information Schema View for the data source.

By default, the Information Schema is refreshed every 30 days; however, if you make a change to your data source, you can always click the “Refresh” button to force an update.

We will be expanding the list of supported data source types in the coming weeks, so be sure to keep an eye out for follow-up announcements if you use a data source other than BigQuery or Postgres.

• Stack Overflow's 2022 Developer Survey

This is the last day of feature announcements for our upcoming GrowthBook 2.0 release. Today we’re highlighting some of the improvements we’ve made for product managers and other non-technical GrowthBook users.

New Visual Editor 🔥

We’re proud to announce the upcoming release of our brand new Visual Editor! Soon, you’ll be able to design A/B tests on your website directly in your browser, ship the tests to production, and analyze the results, all without writing a single line of code. We’ve made huge improvements since our initial Beta release last year. Read more about the new features on our blog.

Slack Integration

We’re excited to announce our much-requested Slack integration! Now you can configure GrowthBook to send alerts to a Slack channel of your choosing every time something you care about happens in GrowthBook — new features created, experiments stopped, etc. Read more about this feature and see instructions for setting it up in our docs.

A/B Testing Best Practices Guide

The team here at GrowthBook has put together a best practices guide for A/B testing. This guide outlines everything you need to know as you scale up experimentation at your company. It covers everything from foundational knowledge (“what is an A/B test?”) to advanced topics and common mistakes. This is intended to be a living, continually updated, and fully open-source document. You can find the first version of it here.

This is the second day of feature announcements for our upcoming GrowthBook 2.0 release. Today, we’re focusing on changes related to data and statistics.

CUPED (Variance Reduction) for Faster Experimentation

Waiting to collect enough data can be a significant roadblock in increasing experimentation velocity. CUPED, a form of variance reduction, is one of the simplest and most effective ways to remove those roadblocks and get answers faster. For example, Microsoft estimates that using CUPED was equivalent to getting 20% or more of traffic for most of the metrics used by one of its product teams.

CUPED is now available in the Frequentist engine in GrowthBook 2.0. Read more about it on our engineering blog.

Improved SQL Editing Experience

Getting your metric queries exactly right can require a lot of switching between your database environment and GrowthBook. With the schema browser, you can now browse the database schemas directly from GrowthBook, making adding exposure and metric queries easier to build. Read more about this feature on our engineering blog: https://blog.growthbook.io/introducing-the-growthbook-schema-browser

Faster SQL and Simpler Configuration

GrowthBook 2.0 can execute SQL queries up to 2X faster than before. By introducing a new attribution model and applying other internal optimizations, the median query runtime is consistently better across test queries and up to twice as fast in certain configurations.

Furthermore, the new “Experiment Duration” attribution model makes it easier to analyze metrics from experiment exposure until the end of the experiment, rather than requiring arbitrarily large conversion windows. The following figure shows the two Attribution Models you can select from in GrowthBook 2.0.

You can read more about the faster SQL queries on our engineering blog.

Databricks Integration

GrowthBook now natively supports Databricks as a data source for experiment data. Simply add a new data source, choose the event tracker you use, then select “Databricks” and add your connection details. (This is the 14th data connector for GrowthBook — let us know if you have a request for any more!)

Experiment analysis queries in GrowthBook now run up to 2X faster!

This performance improvement is due to 3 main changes.

One Dimension and Variation per User

When users are exposed to multiple variations in an experiment (due to bugs) or have multiple dimension values (e.g., by using multiple devices), we have to figure out what to do.

The previous behavior in GrowthBook was both counterintuitive and bad for performance, so fixing this was a win-win.

Now, when a user is exposed to multiple variations, we remove them completely from the analysis. We also keep track of how many users fall into this bucket. If it's above a critical threshold (1% of experiment users), we show a big warning on the experiment results. This is a sign that something went seriously wrong in your experiment.

When a user has multiple dimension values, we now pick the earliest dimension they had when viewing the experiment. So if someone first viewed an experiment on a phone and then later on a desktop, their "device" dimension will be set to "phone".

You can read more about how we treat dimensions in our documentation here.

New Attribution Models

Back in GrowthBook 1.7, we introduced a new attribution model called "Multiple Exposures". This model was great for increasing the number of conversions included in the analysis, but it had one main drawback - performance.

We are replacing this model with a new one - "Experiment Duration". This new model also increases the number of conversions in the analysis but in a much more performant way.

‍

‍

The following figure is another representation of the different attribution models, and which data they use, for an example metric with a 6-hour conversion window and 0 conversion delay.

How can I use the new model?

You can change the model on a per-experiment basis under the Experiment Settings.

You can also choose the default attribution model for new experiments under your general organization settings.

Removing OINs and GROUP BYs

Due to the above changes, we eliminated multiple JOINs and GROUP BYs, resulting in significant performance improvements, especially for ratio metrics.

To give you a sense of how much we were able to simplify, some of our test queries went from 130 lines of SQL down to only 80!

Thanks to our extensive testing infrastructure, we were able to safely make these big sweeping changes to our SQL structure while ensuring the results remain accurate.

This is just the beginning, and we have big plans for further performance improvements. Stay tuned for more!

GrowthBook 2.0 is here! Learn about the developer-focused improvements in 2.0, including Typesafe SDK, REST API, and more!

GrowthBook 2.0 is finally here, and this new version is so packed full of feature announcements that we had to split it up over 3 days. Today, we’re focusing on class developer experience.

Typesafe SDK

GrowthBook is pleased to offer the first-ever fully typesafe feature flag SDK for Typescript. Now you can detect typos and bugs in your feature flag checks during compile time. To make this possible, we built a GrowthBook CLI that can auto-generate type definitions based on features defined in your GrowthBook account. You can read more about this feature and get detailed instructions for setting it up on our engineering blog — https://blog.growthbook.io/fully-typesafe-sdk

Instant Feature Releases

After the success of our self-hosted GrowthBook Proxy release in version 1.9, we’re excited to bring the same benefits to our Pro and Enterprise customers on GrowthBook Cloud. Over the next several weeks, we will enable instant feature rollouts on these accounts. When enabled, feature changes you make in the GrowthBook UI will be released to all of your users in under a second. Read more about this feature and how we’re scaling it to billions of feature requests on our blog — https://blog.growthbook.io/faster-feature-releases-on-growthbook-cloud

REST API

We have been hard at work expanding our REST API, adding over 20 new endpoints and a brand new documentation site: https://docs.growthbook.io/api. We’re super excited about all of the use cases this unlocks, including the ability to sync experiment results to your data warehouse, one of our most requested features! Read more about the new endpoints and changes on our blog — https://blog.growthbook.io/new-rest-apis

More Code Examples

We’ve improved our example repository to include even more examples on how you can integrate GrowthBook with your code. You can check out the examples repo here: https://github.com/growthbook/examples

Other Developer Focused Features and Improvements

• Improved GoLang documentation and examples
• Add variation IDs to experiment feature API payloads on Developers and the work we’ve been doing behind the scenes to enable a world-
• Add support for Experiment CRUD events in webhooks

Tomorrow, we’ll turn our attention towards the many data scientists and analysts using GrowthBook every day. Stay tuned!

In GrowthBook 1.9, we launched the GrowthBook Proxy server to enable faster feature rollouts for self-hosted instances. With the Proxy, changes you make in the GrowthBook UI (e.g., disabling a feature) are released to all of your users in production in under a second.

We've gotten great feedback from our self-hosted community, and we're excited to bring the benefits of the GrowthBook Proxy server to more people. Over the next several weeks, we're going to enable these same instant feature releases on all Pro and Enterprise accounts on GrowthBook Cloud!

No need to deploy and scale your own Proxy servers, everything will just work automatically.

How does it work?

GrowthBook Cloud serves billions of feature flags every month. To do this reliably at scale, we rely on a globally distributed CDN, short TTLs, and stale-while-revalidate headers.

If a user requests feature flags from our Cloud, it will get routed to an edge location closest to them. Most of the time, the edge location will already have a cached copy and can return instantly. If that cached copy is more than 30 seconds old, the CDN will refresh it in the background. In the rare event a cached copy does not exist, we forward the request to our Origin server and cache the response for future requests.

This is all well and good, but it means features could be 30-60s out-of-date. If a feature is causing your site to break, you want to be able to turn it off immediately, not in 30-60 seconds.

To solve this issue at scale, we use Redis Pub/Sub and Server-Sent Events. After receiving a cached copy of features, SDK clients connect to one of our globally distributed proxy servers, replay recent updates, and subscribe to future changes. When you toggle a feature in GrowthBook Cloud, we use Redis Pub/Sub to notify all proxy servers, which in turn notify all connected clients via Server-Sent Events. All of this happens quickly - typically within 1 second.

You get the best-in-class uptime and speed of a global CDN combined with the responsiveness of a real-time system.

SDK Support

Right now, these instant feature releases only support our JavaScript and React SDKs. We plan to add support to all of our SDKs over the coming months.

Interested in contributing and helping us implement this in your language? Join our community Slack

Back in October, we launched our very first REST API endpoint (GET /features) to list all of the feature flags in your GrowthBook account. Since then, we've been adding new endpoints and improving documentation. There's still a lot of work to do, but we wanted to highlight some of the cool new use cases unlocked by our recent work.

OpenAPI and New Documentation

Our REST APIs are now fully documented using the latest OpenAPI 3.1 standard. For those unfamiliar with OpenAPI (formerly Swagger), it's a standardized way to describe API endpoints, their inputs and outputs, and how everything ties together.

The coolest part about this is our new documentation site - https://docs.growthbook.io/api. Everything you see there is 100% auto-generated. When we add a new API endpoint or change the source code, the docs will always stay completely up to date. This lets us iterate quickly and is the main reason we've been able to add so many new endpoints in such a short amount of time.

Syncing Experiment Results to a Data Warehouse

When you run an experiment in GrowthBook, we query the raw data in your data warehouse, do some fancy aggregations, and pass it through our statistics engine. This final processed data is what powers the GrowthBook UI. But what if you want these processed results to be available back in your data warehouse alongside the raw data?

This is something we want to support natively in GrowthBook eventually, but now that we have a REST API, you no longer need to wait for us to implement this functionality. Now, you can use the /experiments/{id}/results endpoint and a cron job to build out this functionality yourself.

Automatically Disable a Feature Based on Guardrails

If you launch a feature as an A/B test, you can specify Guardrail Metrics in GrowthBook. These are things you are not specifically trying to improve but want to keep an eye on. For example, if you're testing a new signup form, you might add "bounce rate" as a guardrail.

To use this guardrail today, you have to manually log into GrowthBook, refresh experiment results, and if you notice a guardrail metric is failing, manually disable the feature flag.

Using the REST API, you can now fully automate this process. Fetch experiment results and use the /features/{id}/toggle endpoint to disable a feature programmatically if you detect a failing guardrail.

And lots more...

Check out the docs at https://docs.growthbook.io/api for all of the currently supported endpoints.

Most endpoints are read-only today, but we're adding support for additional methods shortly to unlock even more advanced use cases.

GrowthBook client-side SDKs have always been written in 100% Typescript, which provides a great baseline level of type safety. Trying to use a numeric feature flag value as a string? Get a compile-time error.

const value: string = gb.getFeatureValue("my-feature", 5.0);
// Error: Type 'number' is not assignable to type 'string'

‍
This was done entirely through type inference. In the above example, we noticed you passed a number (5.0) as the fallback value, so we inferred the feature value was numeric. When you try to assign that to a string variable, TypeScript complains.

The Missing Link

However, there were two glaring holes in our approach:

1. What if you made a typo in the feature name itself? (e.g. my-faeture )
2. What if you used the wrong fallback value? (e.g. my-feature really is a string, the 5.0 is the real mistake)

With the latest release of our SDK, we are now able to catch both of the above during compile-time type checks:

gb.isOn("my-faeture");
// Argument of type '"my-faeture"' is not assignable to parameter...

gb.getFeatureValue("my-feature", 5.0);
// Argument of type 'number' is not assignable to parameter of type 'string'

How does it work?

The magic happens when creating the GrowthBook instance. You pass in an object describing all of your features and their data types. From that point forward, all of the methods on the GrowthBook instance will be strictly typed.

type AppFeatures = {
  "my-feature": string;
  "other-feature": boolean;
}

const gb = new GrowthBook<AppFeatures>(...);

The GrowthBook CLI

Creating AppFeatures manually and keeping it up-to-date can be tedious and error-prone. Luckily, we also released a handy command-line tool to generate these for you automatically.

First, install our CLI.

 yarn add growthbook

Then, authenticate the CLI to your GrowthBook account using a Secret Access Key, which you can generate under Settings > API Keys.

yarn growthbook auth login --apiKey XXX

Lastly, add a script to your package.json to generate feature types and store them in a file:

{
  "scripts": {
    "type-gen": "growthbook features generate-types --output ./types"
  }
}

Now you can run your script anytime features change within GrowthBook and it will generate a new TypeScript file that defines an AppFeatures type:

yarn type-gen

To use the generated types in your application, import them as follows:

import { AppFeatures } from "./types/app-features";

const gb = new GrowthBook<AppFeatures>(...);

‍‍
What's Next?

We're just getting started with type safety. In the future, we want to support strongly typed targeting attributes and integration with all of our SDK languages - client-side, back-end, and mobile.

We also have a lot more planned for our GrowthBook CLI. Imagine being able to create and toggle features as part of a CI/CD pipeline. Or quickly spinning up a local webhook listener for development.

Growth Book now natively supports Databricks as a data source for experiment data. Databricks users can now easily run A/B tests with their data using GrowthBook. Simply add a new data source, choose the event tracker you use, then select “Databricks” and add your connection details.

This is the 14th data connector for GrowthBook. Let us know if you have a request for any more!