In-house built A/B platform Compared to Growth Book
Having a custom home built A/B testing platform is great. It can exactly meet your needs if you have the expertise and resources to build it. However, building one is full of more pitfalls than you might expect. For instance, do you know if you want to implement Frequentist or Bayesian statistics? Do you want to build a front end editor for easy front end testing? Have you planned for ongoing maintenance costs? Let's take a look at Growth Book vs your home built platform.
Homebuilt A/B testing solutions are not cheap. Engineering time and resources are required. Depending on your team, costs can range from $10k for a very basic system using manual statistics, to $500,000+ for something more advanced. Don't forget maintenance costs and feature expansion, which can be about 25%+ of a data engineer's time or more per year (depending on how it's built). All together, in-house engineered A/B testing platforms are very expensive no matter how you slice it.
Growth Book's pricing model makes it much cheaper to have a customized A/B testing platform. Growth Book is open core, so you can self host it entirely for free. If you want a hosted version, we charge per seat, not by traffic or MTUs. This makes us significantly cheaper than the competition.
If you build it it's going to be from weeks to months until you can be up and experimenting. This assumes that your platform doesn't have any bugs.
With Growth Book you can be up and running experiments in under an hour, and within a day if you require customization of metrics or data sources.
When you are running a lot of tests, and some of those tests come back with counterintuitive results, you will need to trust your numbers are right. In house built platforms have many more places for mistakes than something that has been tested for years. It could be in event collection, bucketing errors, incorrect distribution modeling, or just statistical errors. You can always run A/A tests to validate your framework, but you should be spending that time testing.
Growth Book uses Bayaisian statistics to make the results more intuitive to understand. Our statistical model is validated by experts, and we also use other statistical checks to make sure there aren't errors with implementation.
In-house platforms can be deeply integrated with your code base. This helps you avoid some problems with integrating other platforms, such as working with React or caching. You also unlock some really interesting experiments that are easier to run when you can change the back end code.
Growth Book allows you to integrate deeply into your code, exactly as you would if you were building it yourself. We also offer a front-end visual A/B test editor, that almost no one builds in-house because they're a pain to make.
Home built solutions can be very fast. Since they are changing variations on the back end, they don't have flickering, flashing, or performance issues that Optimizely and some of the other solutions have.
Growth Book is built for high performance sites. All calls are cacheable so you don't create any blocking dependencies. Our front end testing causes no flickering or performance impacts.
Let's be honest, if you are building it yourself for your internal team to use, you're probably not going to spend too much time making it look pretty. Like with a lot of software, the functionality is a smaller portion of the effort required, the admin interface can take longer.
Growth Book includes all the touches that take you from an A/B test to an A/B testing platform. It includes Slack integration, insights, tagging, searchable results, etc, that you're unlikely to build yourself.
Front End Testing
To create an interface that allows non engineers to edit the DOM and create variations is hard to build internally. This means that engineering will be a limiting factor for your A/B testing program.
Growth Book also includes a front end editor to let non-engineers build experiment variations.
With your team focused on building the testing platform, you'll likely forget to build some of the features required to create a successful testing program. Often forgotten are ways to go back to see old tests and their results, ways to see exactly what the test looked like, and what the variations were specifically. Furthermore there is usually no systematic way to abstract out learnings.
Growth Book is designed to help you be successful with your AB testing program. To this end we are designed to help companies integrate an experimentation culture. Namely this comes from the ability to have discussions around any aspect of your experiment, easily share results of an experiment, summarize the conclusions and results, and create insights with larger themes and the evidence of them. Experiments are hypothesis driven, and hierarchical, so if one test sparked another idea to test, you can track that chain of thought. Particularly important is capturing the discussion and context around why an experiment was created, and what happened next so that you are capturing this tribal knowledge, which accelerates onboarding and de-risking departures.
To build vs buy can be a hard decision. A/B testing platforms have come a long way over the recent years, and are adding enough features that you should think really hard about building yourself. The costs of building and maintenance are probably much higher than you expect. We built Growth Book based on our experiences with exactly this decision. We tie into your existing data infrastructure, are built to make a seamless developer experience, and have tools to encourage a culture of experimentation. We have years of experience doing experimentation and building these tools. We offer all the advantages of building your own system, without the negatives of buying. Spend your time on products that add value for your users.
|Home Built||Growth Book|
|Integration||Deep code integration||Deep code integration or front end|
|Review past tests||limited||yes|
|Capture discussions & context||no||yes|