Optimizely Compared to GrowthBook

Optimizely is one of the largest experimentation platforms, and also one of the most expensive. Built primarily for marketing teams, optimizely used to be the default for AB testing before Google Optimize came along. Both GrowthBook and Optimizely offer solid AB testing statistics and results. Here we’ll highlight some of the major differences between Optimizely and GrowthBook, and why we think Growth Book is the smarter choice.

Pricing

Optimizely does not disclose their exact pricing, but billing is done per user per test, so the more you test, the more you pay. For example, a product getting 200,000 visitors a month can expect quotes of around $36,000 a year for a normal cadence of 1 test a week.

GrowthBook believes that you should test as often as you can, and you only pay per seat, not based on traffic. We also have an entirely free open source version if you want to host it yourself. This makes GrowthBook significantly cheaper than Optimizely.

Data Sources

Optimizely requires you to use their customer tracking, which presents a few problems. One of them is that you now have yet another place where personal information on your users is stored, which presents compliance issues, particularly with GDPR, HIPAA, COPPA, etc. Another is that you cannot tie into existing metrics and calculations you might already have on your users (such as custom LTV calculations, or engagement metrics). Finally, many ad blockers and do not track browser extensions will block Optimizely from working completely.

GrowthBook works with your existing data sources, whatever they are. In this way we are compliant with most regulations. Our AB tests are tightly integrated with your code, which makes them immune to ad block problems.

Growth culture

Optimizely primarily focuses on short term tests. You can go back to see old tests and their results, but if your site has changed significantly, seeing exactly what the test looked like, or what the variations were specifically is not possible. Furthermore there is no way to abstract out learnings.

GrowthBook is designed to help you be successful with your AB testing program. To this end we are designed to help companies integrate an experimentation culture. Namely this comes from the ability to have discussions around any aspect of your experiment, easily share results of an experiment, summarize the conclusions and results, and create insights with larger themes and the evidence of them. Experiments are hypothesis driven, and hierarchical, so if one test sparked another idea to test, you can track that chain of thought. Particularly important is capturing the discussion and context around why an experiment was created, and what happened next so that you are capturing this tribal knowledge, which accelerates onboarding and de-risking departures.

Test Exposure

Optimizely's payload is loaded onto the front end through a javascript call. For your average user this poses no real problem. However, your competitors can inspect the page and easily see what you’re testing, and even toggle tests on and off to see what each case looks like. They don’t get to see the results, but they could keep track of the changes and tests and infer based on what version is implemented in the end what the results were.

GrowthBook encourages deeply integrated tests which do not expose the tests cases to the end users.

Summary

Optimizely is a good A/B testing tool built for marketing teams. Once you want to test beyond these simple tests, you'll hit limits with what Optimizely can do. Futhermore Optimizely charges more the more you test, which is counter what you need to do to ensure a successful A/B testing program. A/B tests usually have a low success rate, so you should be running lots of tests to increase your chances of successful results. Optimizely is not built to encourage a growth culture, it's tools make it hard to revisit context around old tests, or leverage insights for the future. GrowthBook a cheaper and better option.

Comparison Table

OptimizelyGrowthBook
Pricing$$$free/$
AB Testingunlimited, but you pay moreunlimited
Data SourcesOptimizelyAny, (including optimizely)
AB test focusfront end (back end integraion avalialbe)deep integration & front end
Growth culturenoyes
Track insightsnoyes
Review past testslimitedyes
Capture discussions & contextnoyes