Khan Academy’s Journey to Experimentation
How Khan Academy scaled experimentation across product teams and now uses AI-driven evaluation and A/B testing to improve Khanmigo, its generative AI tutor.
Experimentation rarely starts as a mature system. Most organizations begin with a few A/B tests and gradually learn what it takes to build a real culture of evidence.
In this session, Dr. Kelli Hill shares how Khan Academy evolved experimentation from early testing efforts into a cross-functional discipline spanning research, analytics, and engineering.
She’ll walk through the lessons learned while scaling experimentation across product teams, including how they established governance, built trust in metrics, and operationalized testing with GrowthBook.
Kelli will also explore Khan Academy’s newest frontier: experimenting with generative AI.
Her team now uses a combination of AI-driven evaluation and traditional A/B testing to improve Khanmigo, their AI tutor. These experiments go beyond typical product metrics, focusing on learning quality, student outcomes, and responsible AI behavior.
You’ll learn how their team approaches:
In this session, Dr. Kelli Hill shares how Khan Academy evolved experimentation from early testing efforts into a cross-functional discipline spanning research, analytics, and engineering.
She’ll walk through the lessons learned while scaling experimentation across product teams, including how they established governance, built trust in metrics, and operationalized testing with GrowthBook.
Kelli will also explore Khan Academy’s newest frontier: experimenting with generative AI.
Her team now uses a combination of AI-driven evaluation and traditional A/B testing to improve Khanmigo, their AI tutor. These experiments go beyond typical product metrics, focusing on learning quality, student outcomes, and responsible AI behavior.
You’ll learn how their team approaches:
- Designing experiments for AI systems, prompts, and model behavior
- Evaluating learning quality, not just engagement metrics
- Combining automated AI evaluation with production A/B tests
- Scaling experimentation across product, engineering, and data science teams
Khan Academy
DR. KELLI HILL, Senior Director of Data Insights, Khan Academy
GrowthBook
LUKE SONNET, Head of Experimentation, GrowthBook
Register Today
If you're building AI-powered products or trying to turn experimentation into an engineering discipline, this session offers a practical look at the journey.
We do not sell your information for any purpose. Please see our Privacy Notice for details.
Trusted by 2,700+ companies worldwide
"The fact that we could retain ownership of our data was very, very important. Almost
no solutions out there allow you to do that. Most of them you're passing the user data to a third-party service and that's something we really wanted to avoid."
no solutions out there allow you to do that. Most of them you're passing the user data to a third-party service and that's something we really wanted to avoid."
JOHN RESIG, Chief Software Architect, Khan Academy
