Contextual Bandits

Contextual bandits learn optimal actions in real-time by balancing exploration (trying new options) with exploitation (using known best options). Unlike A/B tests that require fixed assignments, bandits adapt continuously. Unlike full RL, bandits don't consider long-term consequences - just immediate rewards. This makes them practical for personalization, recommendations, and optimization.

Where It Is Used

Contextual bandits power adaptive systems across industries:

• Recommendations: Personalized content, product suggestions
• Advertising: Ad selection, creative optimization
• Pricing: Dynamic pricing decisions
• Healthcare: Treatment selection, clinical trials
• News: Article personalization, headline testing
• UX: Layout optimization, feature rollout

Products: Netflix (artwork personalization), Spotify (playlist generation), news feeds, ad platforms.

The Bandit Problem

Multi-armed bandit: Choose between K arms (options), observe reward, maximize cumulative reward.

Contextual bandit: Same, but you see context (user features) before choosing.

Context x → Agent selects action a → Observe reward r
         ↓
   Update policy to maximize future rewards

Key trade-off: Exploration vs Exploitation - Exploitation: Choose the action that seems best now - Exploration: Try actions to learn more about them

Too much exploitation → miss better options Too much exploration → waste opportunities on bad options

Typical Data You Will See

Data structure:

timestamp | context_features | action_taken | reward
10:00:01  | [0.5, 1.2, ...]  | action_3     | 1
10:00:02  | [0.8, 0.3, ...]  | action_1     | 0

What You Do in Practice

1. Define the problem - What actions, what reward, what context?
2. Choose algorithm - Thompson Sampling, UCB, or neural
3. Implement exploration - Ensure sufficient exploration
4. Handle delayed feedback - Rewards may come later
5. Build serving infrastructure - Low-latency decisions
6. Log everything - Actions, contexts, rewards for analysis
7. Evaluate offline - Counterfactual evaluation
8. Deploy carefully - Start with high exploration
9. Monitor performance - Regret, reward, arm pulls
10. Iterate - Add features, tune exploration

Baseline: Epsilon-Greedy

Simplest exploration strategy:

import numpy as np

class EpsilonGreedy:
    def __init__(self, n_actions, epsilon=0.1):
        self.n_actions = n_actions
        self.epsilon = epsilon
        self.counts = np.zeros(n_actions)
        self.values = np.zeros(n_actions)

    def select_action(self, context=None):
        if np.random.random() < self.epsilon:
            return np.random.randint(self.n_actions)  # Explore
        return np.argmax(self.values)  # Exploit

    def update(self, action, reward):
        self.counts[action] += 1
        # Incremental mean update
        self.values[action] += (reward - self.values[action]) / self.counts[action]

Why epsilon-greedy baseline: - Simple to implement - Easy to understand - Predictable exploration rate - Good starting point

Bandit Algorithms

Thompson Sampling (recommended):

import numpy as np
from scipy.stats import beta

class ThompsonSampling:
    def __init__(self, n_actions):
        self.n_actions = n_actions
        # Beta distribution parameters (successes, failures)
        self.alpha = np.ones(n_actions)
        self.beta = np.ones(n_actions)

    def select_action(self, context=None):
        # Sample from posterior for each arm
        samples = [beta.rvs(self.alpha[a], self.beta[a])
                   for a in range(self.n_actions)]
        return np.argmax(samples)

    def update(self, action, reward):
        # Update Beta distribution
        if reward == 1:
            self.alpha[action] += 1
        else:
            self.beta[action] += 1

LinUCB (contextual):

import numpy as np

class LinUCB:
    def __init__(self, n_actions, n_features, alpha=1.0):
        self.n_actions = n_actions
        self.alpha = alpha
        # Per-action parameters
        self.A = [np.eye(n_features) for _ in range(n_actions)]
        self.b = [np.zeros(n_features) for _ in range(n_actions)]

    def select_action(self, context):
        ucb_values = []
        for a in range(self.n_actions):
            A_inv = np.linalg.inv(self.A[a])
            theta = A_inv @ self.b[a]
            # UCB = expected reward + exploration bonus
            ucb = context @ theta + self.alpha * np.sqrt(context @ A_inv @ context)
            ucb_values.append(ucb)
        return np.argmax(ucb_values)

    def update(self, action, context, reward):
        self.A[action] += np.outer(context, context)
        self.b[action] += reward * context

Evaluation and Metrics

Online evaluation is gold standard, but risky. Use offline evaluation first.

Offline evaluation (counterfactual):

def inverse_propensity_scoring(logs, new_policy):
    """
    Estimate new policy performance from logged data.
    logs: [(context, action, reward, propensity), ...]
    """
    total_reward = 0
    total_weight = 0

    for context, logged_action, reward, propensity in logs:
        # What would new policy choose?
        new_action = new_policy.select_action(context)

        if new_action == logged_action:
            # Weight by inverse propensity
            weight = 1.0 / propensity
            total_reward += weight * reward
            total_weight += weight

    return total_reward / total_weight if total_weight > 0 else 0

Failure Modes and Debugging Checklist

• [ ] Insufficient exploration: Stuck on suboptimal arm
• [ ] Delayed feedback: Rewards arrive after decisions
• [ ] Non-stationary rewards: Best arm changes over time
• [ ] Context leakage: Using post-decision information
• [ ] Selection bias: Logged data biased by old policy
• [ ] Cold start: No data for new actions
• [ ] Propensity issues: Logging propensities incorrectly
• [ ] Simpson's paradox: Aggregate metrics misleading

Production Checklist

Before deployment: - [ ] Offline evaluation on logged data - [ ] Exploration rate appropriate - [ ] Logging infrastructure ready - [ ] Propensities recorded correctly - [ ] Latency requirements met - [ ] Fallback policy defined

After deployment: - [ ] Cumulative reward tracked - [ ] Arm pull distribution monitored - [ ] Exploration rate verified - [ ] Delayed rewards handled - [ ] Model updates scheduled - [ ] A/B test for algorithm comparison

Logging requirements: - [ ] Context features logged - [ ] Action taken logged - [ ] Propensity (probability) logged - [ ] Reward logged when observed - [ ] Timestamp logged

Tools and Stack

Example Mini-Case

Scenario: News website wants to personalize article recommendations for homepage.

Setup: - 50 article slots to fill - 1000 articles in pool - Context: User features (history, time, device) - Reward: Click (binary) - Traffic: 1M homepage views/day

Approach: 1. Baseline: Random selection - CTR: 2.1% 2. Epsilon-greedy (ε=0.1): CTR: 2.8% 3. Thompson Sampling: CTR: 3.2% 4. LinUCB with user features: CTR: 3.8% 5. Neural bandit: CTR: 4.1%

Architecture:

User request → Feature extraction → Bandit model →
Article selection → Serve → Log (context, action, propensity) →
Click event → Log reward → Model update (batch)

Results: - CTR improvement: +95% over random - Exploration: 8% of traffic exploring - Model update: Every 4 hours - Cold start: New articles get minimum 100 impressions

Key learnings: - Thompson Sampling best balance of simplicity and performance - User context features doubled improvement - Proper logging critical for offline evaluation - Batch updates sufficient vs real-time

Related Pages

• Index: ML Field Guide
• A/B Testing for ML
• Collaborative Filtering
• Session-Based Recommendations

Further Reading

• Vowpal Wabbit
• Introduction to Multi-Armed Bandits
• Contextual Bandits Tutorial
• Azure Personalizer

Try it live

Everything above runs in your browser — open A/B Test Optimizer — UCB1 Multi-Armed Bandit Live and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.