Compare ranking algorithms with interleaving experiments (private beta)

Interleaving experiments compare two ranking algorithms by merging both rankers' results into one list for each user. Each user sees results from both rankers, so their interactions give you a direct comparison. In published examples, interleaving reached a ranking decision with about 1–2% of the traffic of a traditional A/B test, a 50–100× reduction. Interleaving is in private beta for Enterprise customers.

How the merged list works

Changes to a mature search or recommendation ranker tend to produce small effects, and small effects need large samples in an A/B test. Interleaving shows each user items from both rankers instead of splitting users between them. GrowthBook uses team-draft interleaving to build that list.

• Each round, a coin flip decides which ranker picks first.
• Each ranker then adds its highest-ranked item that isn't already in the list.
• When both rankers want the same item in the same round, the item is shared and credits neither ranker.

Two ways to pick a winner

• Paired analysis compares the rankers within each merged list and credits each engagement to the ranker that drafted the item. It needs a ranking_id on exposures and engagement events, and suits in-list actions such as clicks or add-to-cart.
• Preference analysis gives each user one vote for the ranker whose items they engaged with more. It works from item_id alone, so it fits outcomes logged elsewhere, such as orders or bookings.

Use the results to see which ranker users prefer as they interact with a single list.

Getting started

Contact your GrowthBook account team to request private beta access on an Enterprise plan. The JavaScript, React, and Node.js SDKs support interleaving from version 2.0.0.

• In your app, register interleavePlugin and replace your ranking fetch with interleave(). Send its exposure payload to your warehouse, and add item_id to engagement events, plus ranking_id for paired analysis.
• In your data source settings, add a query under Interleaving Exposure Queries.
• Go to Experiments → Interleaving and select Add Interleaving Experiment. The first ranker you add is the control.

Follow the interleaving setup guide for metric requirements and SDK options.