Challenge

Khan Academy built an internal feature flag and A/B testing system due to the high risk and strict compliance requirements associated with student data. Over time, the system became difficult to maintain and experienced increased latency.

“The feature flag logic that we built wasn’t very good from a performance perspective, and the complexity of maintaining that was very high. We wanted something that was more maintainable, easier to use.”
— John Resig, Chief Software Architect, Khan Academy

However, most off-the-shelf testing systems did not meet Khan Academy’s complex roll-out requirements for educational settings. Answering research questions about learning outcomes and meeting specific district and school needs requires fine-grained control and tagging capabilities.

“We were looking to customize attributes on which we could toggle a roll-out, instead of using a percentage roll-out. Having tags for the classroom or the district a student is in, and then actually rolling out based on those, gives us a lot more power.”
— John Resig, Chief Software Architect, Khan Academy

‍

‍

Solution

Resig began researching options for feature flags and A/B testing that combined performance and customization with the data protections that were so crucial to the organization. That’s when they found GrowthBook.

GrowthBook’s warehouse-native architecture meant Khan Academy could retain ownership of its data without passing user data through a third-party service.

Khan Academy integrated GrowthBook across the website, backend, and mobile apps. The team toggles features on and off based on individual attributes, like classroom tags or student districts. This functionality alone proved to be a powerful tool for streamlining their deployment of product updates.

The team can see if making a change on the site or the mobile app results in more learning.

“This model gives us a lot more power to release mobile features faster. The more that can happen, the better the overall experience.”
— John Resig, Chief Software Architect, Khan Academy

Tailoring attributes beyond percentage-based rollouts enables Khan Academy to run more conclusive experiments. This is key to optimizing the product for students and teachers worldwide.

Results

With GrowthBook’s customization options, they could measure learning outcomes effectively and ensure data privacy. The static settings object model simplified their processes, improving the user experience without affecting performance.

• Warehouse-native architecture met strict data privacy requirements
• No more performance issues or flickering when loading variations
• Increased confidence in data accuracy across the testing landscape
• More control to measure unique metrics for learning outcomes
• Aggregated data sources provide a complete picture of overall impact
• Ease of use has increased excitement about running experiments

“We can really look at putting out a small feature, seeing how people respond to it and then committing to it based on that data.”
— John Resig, Chief Software Architect, Khan Academy

Next Steps

GrowthBook is helping Khan Academy expand a culture of experimentation with large-scale A/B testing. By tracking changes across Khan Academy’s aggregated data resources, they can evaluate educational outcomes for millions of learners.

"People [are] running more experiments with more confidence. GrowthBook is going to help us do a lot more testing."
— John Resig, Chief Software Architect, Khan Academy

‍

‍