What are contextual bandits (and when should you use them)?

A contextual bandit does not ask which experience wins for everyone. It asks which available action is best for this context, right now.

That makes contextual bandits appealing for recommendations, ranking, messaging, and other repeated decisions where different users may prefer different actions. The algorithm observes context, chooses one action, receives feedback for that action, and updates its policy. It must keep learning without wasting too much traffic on poor choices.

The adaptation is also what makes bandits easy to misuse. If your real question is whether a product change caused an outcome, a conventional randomized A/B test usually gives a cleaner answer. If today's action changes tomorrow's state, a contextual bandit may be too short-sighted. If you cannot log decision probabilities and delayed rewards, you may not be able to evaluate the system you deployed.

Use contextual bandits when the decision repeats often, context meaningfully predicts which action works, rewards arrive soon enough to learn, and maximizing reward during learning matters more than estimating one stable average effect.

A contextual bandit learns a policy from partial feedback

The word bandit comes from the multi-armed bandit problem: a learner repeatedly chooses among actions, historically imagined as slot-machine arms, and tries to maximize cumulative reward. It faces an exploration-exploitation tradeoff. Exploitation chooses the action that currently looks best; exploration tries uncertain actions so the system can learn.

Context changes which action looks best

A standard multi-armed bandit learns one reward estimate per action. A contextual bandit also observes features available before the decision. These might include device type, locale, subscription tier, recent behavior, query features, item attributes, time of day, or system load.

The policy maps context to a probability distribution over eligible actions. A documentation site might choose among 3 help panels based on the reader's language, entry page, and product area. A recommender might rank a limited set of items from user and item features. A cloud controller might choose a retry delay from machine type and failure history.

The official Vowpal Wabbit contextual-bandit tutorial defines the loop with 4 parts: context, action, probability, and observed cost or reward. That probability is not implementation trivia. It records how likely the logging policy was to choose the action and enables later policy evaluation.

The learner sees only the chosen action's reward

If the system shows message B and the user converts, it observes B's reward. It does not observe what that user would have done under messages A or C. This partial feedback separates contextual bandits from ordinary supervised learning, where a training row normally has a known label independent of the model's prediction.

The data is also policy-dependent. As the learner favors an action, it collects more rewards for that action and fewer for alternatives. Training a standard predictor on these observations without accounting for action probabilities can reinforce selection bias.

Regret measures the cost of learning

Bandit algorithms are often evaluated by cumulative regret: the reward lost relative to an oracle policy that always chooses the best action for each context. Exploration creates short-term regret because the learner deliberately tries uncertain choices. Too little exploration can create larger long-term regret by locking into a plausible but suboptimal policy.

Microsoft Research's production overview of contextual-bandit learning highlights why the model is simpler than general reinforcement learning: the relevant feedback is summarized in the immediate reward, and the preceding action is assumed not to change the next context. That assumption defines an important boundary.

Pick the right experiment design

See how experimentation leaders separate optimization questions from causal questions and choose metrics that support the decision.

Watch the Webinar

Contextual bandits solve a different problem than A/B tests

Both methods randomize actions and learn from outcomes, but they optimize different objectives.

A/B tests estimate a stable treatment effect

A conventional A/B test assigns units to fixed variants according to a predeclared allocation. Stable assignment and concurrent control make the comparison interpretable as a causal effect under the experiment's assumptions.

Use an A/B test when the question is “Should we ship this change?” or “What is the average effect of this policy on activation and retention?” GrowthBook's A/B test design guide centers the work on a causal hypothesis, eligible population, randomization unit, exposure, metrics, power, and decision rule.

A contextual bandit changes assignment probabilities as it learns. Users with different contexts may receive different actions, and the distribution of traffic over actions changes over time. A naive comparison of average rewards by arm becomes biased because the groups were generated by an adaptive policy.

Bandits optimize reward during the learning period

Use a bandit when showing a known poor action has opportunity cost and the system makes the choice repeatedly. Rather than sending a fixed share to every option until a test ends, the policy increasingly exploits promising actions while retaining enough exploration to learn.

This can reduce cumulative regret, but it changes the output. The result is a decision policy, not merely a global winner. The policy may learn that action A works for new mobile users, B works for returning desktop users, and C is rarely appropriate.

GrowthBook's guide to experiments, rollouts, and bandits is a useful framing: choose the method from the operational goal, not from which algorithm sounds more advanced.

Reinforcement learning models longer consequences

A contextual bandit assumes each action's relevant consequence can be represented by its reward and does not need to model how that action changes future state. Full reinforcement learning handles sequential credit assignment: today's action affects tomorrow's context, options, and rewards.

If a notification changes a user's future engagement state, or a pricing action changes inventory and later choices, a one-step reward may optimize the wrong behavior. You can sometimes design a delayed or composite reward, but a longer horizon also increases variance and slows feedback. Do not call a sequential control problem a bandit to avoid the harder modeling work.

Use contextual bandits when five conditions hold

The use case should fit the learning loop before you choose an algorithm.

Decisions repeat at meaningful volume

The policy needs repeated opportunities to learn. A homepage hero, recommended article, notification template, support response, or ranking choice may appear thousands of times. A once-a-quarter pricing architecture change does not.

Volume must be considered per relevant context and action, not in aggregate. Ten million requests do not help a rare locale-action combination if it receives only 20 eligible decisions. Start with a small action set and context features that have a plausible mechanism.

Context predicts heterogeneous rewards

A contextual bandit earns its complexity when the best action genuinely differs by observable context. If one variant is best for nearly everyone, a non-contextual bandit or A/B test is easier to operate and explain.

Use only features available at decision time. Including future behavior leaks outcome information into the policy. High-cardinality identifiers can let the model memorize noise. Sensitive attributes and proxies require fairness, privacy, and policy review even when they improve short-term reward.

Rewards are timely, attributable, and hard to game

Clicks arrive quickly but may be weak proxies for satisfaction. Retention is valuable but delayed. Revenue can be sparse and skewed. A bandit needs a reward that arrives soon enough to update the policy and remains aligned with the product outcome.

Define:

• The reward event and value range.
• The attribution window.
• How multiple events aggregate to one decision.
• Treatment of missing or late rewards.
• Guardrails that can stop harmful exploration.
• A longer-term holdout metric that checks proxy alignment.

GrowthBook's metric documentation is useful for defining outcome logic consistently when the bandit's decisions and rewards land in the warehouse.

Exploration is safe enough to permit

Learning requires nonzero probability for alternatives. Do not use unconstrained exploration for medical dosing, fraud decisions, security controls, or actions with irreversible harm. Even lower-stakes products need eligibility filters, action constraints, default behavior, rate limits, and kill switches.

Microsoft Research has shown that average-case optimization can miss tail risk in contextual-bandit systems. Guardrails should reflect the distribution of harm, not only the mean reward.

You can log the complete decision record

At minimum, persist this event:

{
  "decision_id": "<DECISION_ID>",
  "policy_id": "help-panel",
  "policy_version": "2026-09-09.3",
  "model_version": "linucb-184",
  "decided_at": "<ISO_TIMESTAMP>",
  "context": {
    "device": "mobile",
    "locale": "en-US",
    "entry_surface": "billing"
  },
  "eligible_actions": ["search", "chat", "article"],
  "chosen_action": "article",
  "chosen_probability": 0.27
}

Log the reward separately with the same decision_id, reward definition version, value, and timestamp. Preserve the exact decision-time context instead of reconstructing mutable profile fields later.

If the chosen probability is missing, inverse-propensity and doubly robust estimators cannot properly account for the logging policy. The PMLR research on offline contextual-bandit evaluation explains why evaluating adaptive recommendation policies from historical data is not a normal train-test split.

Build the system around evaluation, not only serving

The online decision endpoint is the visible part. A production system also needs feature computation, eligibility, exploration, event logging, reward joins, training, model registry, deployment, monitoring, and rollback.

Separate policy scoring from exploration

The model estimates expected rewards. The exploration strategy turns those scores and uncertainties into action probabilities. Common strategies include:

• Epsilon-greedy: Choose a random action with probability epsilon and exploit otherwise.
• Thompson sampling: Sample from uncertainty distributions and select the best sampled action.
• Upper confidence bound: Favor actions with strong estimates or high uncertainty.
• Explore-first: Randomize for an initial period, then use the learned policy.
• Cover or bagging methods: Maintain diverse policies or models to drive exploration.

Vowpal Wabbit's current documentation lists exploration strategies and propensity-aware formats. Choose the strategy from risk tolerance, nonstationarity, action count, and model assumptions rather than treating epsilon as a default constant.

Evaluate candidate policies off-policy

Historical logs came from the incumbent policy, so they overrepresent actions it favored. Offline policy evaluation reweights or models observations to estimate how another policy might have performed.

Inverse propensity scoring weights observed rewards by the ratio between target and logging probabilities. Doubly robust methods combine propensity weighting with a reward model. Both depend on support: the logging policy must have assigned nonzero probability to actions the candidate policy may choose in the relevant contexts.

The Vowpal Wabbit offline policy evaluation guide warns that a conventional holdout estimate is biased under partial feedback and describes IPS and doubly robust alternatives. Treat offline evaluation as a risk filter, not a final production verdict.

Run an online randomized policy test

Before replacing the incumbent, randomize eligible units between the candidate policy and a stable baseline. Keep that meta-assignment sticky even though each policy makes adaptive decisions inside its arm. Measure primary reward, longer-term outcomes, system cost, latency, fairness slices, and operational guardrails.

This is where standard experimentation remains essential. Put the policy choice behind a feature flag, record policy exposure, and use a warehouse-native test to compare downstream outcomes. Maintain a small holdout after rollout if ongoing adaptation could degrade unnoticed.

Do not compare “before the bandit” with “after the bandit” and call the difference causal. Traffic mix, inventory, seasonality, and the reward model may all change.

Know the failure modes before traffic becomes adaptive

Contextual bandits create feedback loops. Mistakes can become self-reinforcing because the deployed policy controls the data used to train its successor.

Insufficient exploration creates blind spots

If an action receives near-zero probability for a context, its estimate stops improving. A transient early loss can permanently hide a better option. Monitor action probability distributions, effective sample size, context coverage, and the overlap required for offline evaluation.

The practitioner discussion on bandits in production highlights a recurring reality: the statistical gain may not justify the infrastructure and change-management complexity for many products. A fixed A/B test is often the stronger baseline.

Nonstationarity invalidates yesterday's policy

User preferences, inventory, page design, campaigns, and competitors change. Add drift monitors, recency weighting or explicit retraining schedules, and policy-level experiments. Keep model version, feature definitions, and action eligibility in the decision log so performance changes can be diagnosed.

Reward hacking optimizes the proxy

A notification bandit can increase clicks by becoming intrusive. A recommender can improve watch time while narrowing diversity. Define guardrails and evaluate delayed outcomes. Periodically test the policy against a simple baseline, not only against its immediate predecessor.

Context can encode unfair treatment

Audit feature provenance, protected attributes, proxies, missingness, and outcome quality across groups. Constrain ineligible actions before scoring. A high average reward does not justify exposing a subgroup to persistently poor or risky choices.

Adaptive data breaks naive analysis

Arm averages, ordinary confidence intervals, and standard supervised evaluation can be misleading when action probabilities depend on prior rewards. Preserve propensities and use estimators designed for adaptive data. For important product claims, use a separately randomized evaluation whose analysis matches its design.

Start with a policy-versus-baseline experiment

Do not begin by turning on fully adaptive traffic for everyone. Build the decision log, replay it, and confirm that context, action, propensity, and reward join correctly. Simulate delayed and missing rewards. Run offline evaluation with uncertainty and support diagnostics.

Then place the candidate policy behind a randomized flag. Compare it with the current deterministic or rules-based policy using the same primary metric, guardrails, and cost accounting. GrowthBook's experiment analysis can evaluate that outer test using warehouse data while the bandit system retains its own policy-learning telemetry.

Choose contextual bandits only when the personalized policy is the product decision you want. If you need a clean answer about one feature, run an A/B test. If decisions alter future states, investigate reinforcement learning or a deliberately simplified policy. Sophistication is not the goal. A decision loop you can audit, constrain, and evaluate is.

Evaluate the adaptive policy

Use GrowthBook to compare a contextual policy with a stable baseline and measure product outcomes beyond the immediate reward.

Get Started