How the Wikimedia Foundation scaled experimentation for 15B+ monthly page views

Executive summary

The Wikimedia Foundation operates at a scale most product teams never have to consider: over 15 billion monthly visits and over 300 language communities, making it one of the largest, open-source knowledge platforms on the web. This scale, along with their absolute commitments to protecting user privacy, maintaining community trust, and upholding the norms of a global volunteer movement creates a high bar for experimentation infrastructure.

As the demands have continued to evolve, the Foundation needed an open-source platform that could run in its own environment, connect to its existing data lake, support rigorous analysis, and fit into a privacy architecture where user identifiers stop at the CDN boundary. It also needed to reduce the operational burden of a capable but increasingly expensive homegrown system.

GrowthBook met those requirements by integrating directly with the Foundation’s existing stack: self-hosted deployment, Presto/Trino connectivity, Kerberos authentication, OIDC/SSO support, warehouse-native analysis, and an open-source collaboration model.

Today, the Wikimedia Foundation is using GrowthBook to bring more consistency, context, and automated analysis to experimentation across its product teams. Most experiments are being analyzed in GrowthBook, and teams are moving toward configuring experiments there as well.

The result is a more scalable experimentation foundation for one of the web’s largest and most technically complex environments.

The challenge: scaling experimentation without breaking trust

The Wikimedia Foundation had already built a serious experimentation system. Its legacy stack included a homegrown Node app for experiment configuration, CDN-layer enrollment in Varnish (homegrown feature flagging), custom JavaScript and PHP SDKs, Kafka pipelines, a data lake, Airflow aggregations, Bayesian analysis, Superset dashboards, and a YAML metrics catalog. The system had been shaped over years to match the Foundation’s unusual scale, privacy expectations, and technical architecture.

But the maintenance burden was growing. Launching an experiment could require touching multiple config files. Analysis was pipeline-based and bare-bones. Metrics were defined in YAML. Results could not easily be sliced across important dimensions, such as wiki, language, platform, or user type. Information about a single experiment could be spread across a Jira-style board, Google Docs, Google Sheets, the bespoke configuration tool, and a separate Superset dashboard. That made experimentation possible, but not easy to scale.

“It all generally works, but it’s a lot of maintenance and it requires our guru statisticians to continue building and maintaining the existing software.”
— Adam Baso, Principal Engineer, Wikimedia Foundation

The team estimated that building the capabilities GrowthBook already had, including UI-based metric definitions, fact tables, and flexible dimension breakdowns, would take one to two years. The Wikimedia Foundation decided that time would be better spent helping more teams run better experiments.

“We decided it was a better use of our time to get teams to run experiments, as opposed to continuing to build a thing that would allow them to run experiments.”
— Mikhail Popov, Data Scientist, Wikimedia Foundation

The objective was not experimentation for its own sake. The objective was better evidence, produced faster, with less specialist effort and more consistency across teams.

Privacy shaped every architecture decision

The Foundation’s experimentation approach is deeply influenced by its privacy commitments. The team uses separate hashing and encryption routines per experiment to avoid unnecessary cross-experiment user correlation. For many commercial companies, the assumption is that there is one customer profile to optimize against. The Wikimedia Foundation’s model is different.

The organization is not trying to maximize revenue. It is trying to do the right thing without causing harm.

That changes the meaning of experimentation. A successful platform has to support better product decisions while respecting privacy, community expectations, and the broader public mission of the organization.

GrowthBook fit because the Foundation could keep experimentation analysis inside its own infrastructure, using its own data, without sending a copy of that data to GrowthBook.

Why experimentation matters to the Wikimedia Foundation’s mission

For the Foundation, experimentation is not about extracting more value from users. It is about helping people participate in a shared knowledge project.

Many experiments focus on contribution. Accounts unlock features that help newcomers get started, including suggested edits and other small on-ramps to editing and article creation. Improving account creation is not just a growth metric. It is a way to help more people become contributors.

“We want to meet people where they are and meet their intention.”
— Julie van der Hoop, Product Manager, Wikimedia Foundation

That mission context changes how experiments are designed and evaluated. A test that increases activity but harms trust, overwhelms volunteer moderators, or creates friction for a specific community is not a win.

The Wikimedia Foundation needs experimentation infrastructure that helps teams move faster, but also helps them move carefully.

Experimentation in the open

The organization’s experimentation culture is unusually transparent. Product changes are not only tested internally; they are documented, discussed, and shared with the broader Wikimedia community.

A recent example is their public write-up on account creation experiments for mobile readers. The team explained why account creation matters to long-term contributor growth, what changes were tested, and how the results were evaluated. One experiment added a visible account button to the mobile web header and produced approximately 20% more mobile account creations, with no statistically significant decline in constructive edit rates. Another redesigned the logged-out editing prompt across ten Wikipedias, making users 27% more likely to create a permanent account while also reducing temporary account creation by 16%.

That level of public reporting reflects the way the Foundation approaches experimentation. They are part of an open decision-making process that helps teams, contributors, and communities understand what changed, why it changed, and what the organization learned.

Small experiments, outsized impact

Some of the Foundation’s highest-impact experiments have been surprisingly small.

In one experiment, the growth team added a simple banner prompting people to confirm their email address. Email confirmations rose by roughly 40%.

In another, the team questioned why the first thing a new user saw during account creation was a warning message. Removing that warning increased account creation by about 30%.

“These really, really small things can have outsized impact.”
— Julie van der Hoop, Product Manager, Wikimedia Foundation

Those examples show why scalable experimentation matters. Small product details can shape whether someone confirms an email address, creates an account, starts editing, or becomes part of the contributor community.

The Wikimedia Foundation also runs larger, mission-level experiments. One explores whether search can better support semantic queries rather than exact match searches. Another rethinks article creation by giving contributors structure instead of a blank page, such as common section headers for an article about a person, to see whether it helps new articles survive longer.

The range is broad: from small UI changes to core product experiences that affect how people discover and contribute knowledge.

Why the integration requirements were non-negotiable

The Foundation could not use a platform that required a conventional SaaS architecture, copied data into a vendor-controlled system, or forced teams to abandon the technical decisions that protect user privacy. GrowthBook had to fit into their stack.

Self-hosted, airgapped deployment

The Wikimedia Foundation runs GrowthBook on its own infrastructure, including production and staging instances. The team evaluated outbound network behavior and controls to prevent unwanted egress.

Self-hosting was not a preference. It was a privacy requirement.

Warehouse-native analysis

The organization’s data already lived in its Hive/Iceberg data lake. GrowthBook needed to connect into that environment through Presto/Trino rather than forcing data into a separate vendor-controlled analytics store.

This was central to the architecture. GrowthBook had to meet the Foundation where its data already lived.

Kerberos authentication for Presto

The Foundation’s data lake sits behind Kerberos. GrowthBook did not already support the exact connection layer Wikimedia needed, so the GrowthBook team worked with their SRE team to build it.

The collaboration happened through a massive Slack thread: a practical, engineer-to-engineer effort to get Kerberos authentication working inside GrowthBook.

GrowthBook then merged the capability upstream, turning the organization’s integration requirement into part of the broader open-source product.

OIDC/SSO validation

Identity management was another key requirement. The Wikimedia Foundation used its trial period largely to validate OIDC login with its data platform SRE team before moving forward.

Open-source alignment

Open source was a hard requirement and a values match. The Wikimedia Foundation wanted a platform it could inspect, self-host, and collaborate on through GitHub, Slack, and direct technical conversations.

That mattered throughout the evaluation. GrowthBook was not a black-box vendor. It was a product the Foundation could work with in the open.

One place for the full experiment story

Before GrowthBook, the context around an experiment could be scattered across several systems.

A team might use one tool for planning, another for measurement specs, another for configuration, and another for results. That made it harder to understand what happened later, especially when an experiment moved through multiple phases or an implementation issue changed the setup.

GrowthBook gives the Foundation one place to bring that context together, which helps preserve the institutional memory around each experiment. Teams can see comments, understand why a phase changed, review the setup, and connect results back to the decision that was made.

“I have one place to go for all the information: how it was set up, what it looks like, who to talk to, what the results were, what the decision was.”
— Julie van der Hoop, Product Manager, Wikimedia Foundation

For the organization, that is part of building a durable culture of experimentation. The fewer places people have to search, the more likely they are to use the system consistently.

Making rigorous experimentation more self-service

Analyst time was one of the clearest ROI drivers. Before GrowthBook, defining metrics often required a data scientist to write SQL. Product managers and engineers depended on specialists to set up analysis, define schemas, and get the experiment data into a form where decisions could be made.

GrowthBook changes that dynamic. Product managers and engineers can now define metrics and fact tables through the UI, then use them directly in experiments. Data scientists still play an important role, but they are not stuck as the bottleneck for every repeatable setup task.

“GrowthBook has been really great for getting engineers on their teams to do the kinds of things that normally a data scientist would be doing.”
— Mikhail Popov, Data Scientist, Wikimedia Foundation

The goal is not to remove data scientists from the process. The goal is to let them focus on higher-value work: improving measurement, interpreting results, and helping teams make better decisions.

The team estimated that analysis could move from roughly four hours per experiment toward one hour, which is significant across a growing experimentation program.

Analysis across hundreds of communities

Wikipedia exists in hundreds of languages, alongside Wikimedia Commons and other projects. Teams may run an experiment across multiple wikis, such as Portuguese, Chinese, and English Wikipedia, and then need to break results down by wiki.

That breakdown matters because different communities can behave differently for cultural, linguistic, or product reasons. An aggregate result can hide important differences.

The same is true for user type. Logged-out readers, account holders, and temporary-account users can respond differently to the same change. The Foundation needs to evaluate those differences before deciding what to ship.

“These communities are very distinct sometimes. These are nuances in behavior that are really important when we’re trying to make a decision.”
— Katherine Reid, Engineering Manager, Wikimedia Foundation

GrowthBook’s support for dimensions and analysis breakdowns gives The Foundation a clearer way to evaluate results across those differences. For a global, multilingual platform, that is how teams avoid averaging away key insights.

Experiment assignment at Wikipedia scale

One of the most distinctive technical details in the organization’s setup is experiment assignment. For anonymous traffic, the Foundation performs assignment at the CDN layer in Varnish. This reflects both scale and privacy. 

“We’ve made a commitment to the Wikimedia communities that identifiers will never enter our servers. They stop at the CDN boundary.”
— Katherine Reid, Engineering Manager, Wikimedia Foundation

That commitment is one reason why they use their own SDKs and perform enrollment without passing identifiers through to an enrollment authority.

Performance was another constraint. MediaWiki has strict performance requirements, and loading third-party code into the page is a concern on its own.

GrowthBook needed to fit into that environment rather than prescribe a standard SDK-only approach. That flexibility is one of the strongest parts of the story: GrowthBook did not ask the Foundation to rip out the architecture that made sense for its scale and privacy model, but integrated around it.

Adoption is underway

The Wikimedia Foundation’s GrowthBook adoption is a significant platform transition that takes time and effort. Most experiments are now being analyzed in GrowthBook. The team has also completed its first end-to-end test of users configuring experiments in GrowthBook, with more teams being moved over.

Because the organization uses its own SDKs and CDN-layer enrollment under the hood, the rollout has surfaced edge cases. The teams have worked through those issues through GitHub and direct collaboration.

The Foundation did not need a generic experimentation tool. It needed a platform and partner that could handle the messy, specific, high-stakes details of its environment.

A partnership built through engineering responsiveness

The Wikimedia Foundation’s evaluation was rigorous. The journey ran from first contact through technical evaluation, security review, airgapped licensing, legal and procurement, trial validation, and contract close. The team evaluated not only whether GrowthBook had the right feature set, but whether GrowthBook could work the way the organization needed to work with its set of values and commitment towards its readers, contributors and to the open web

What stood out was GrowthBook’s responsiveness. Our teams collaborated to implement Kerberos support for Presto, supported security review findings, gave proactive guidance on deployment changes, and worked directly with the Foundation through Slack and GitHub. The Wikimedia Foundation’s team members also called out GrowthBook’s documentation and appreciated the transparency of the sales and technical process.

That responsiveness reinforced the broader fit. The organization was not just buying a platform. It was choosing a partner whose engineering model aligned with the way the Wikimedia Foundation builds software: open, practical, transparent, and collaborative.