A/B Testing for ML Models

A/B testing is how you validate that a new ML model actually improves business outcomes. Offline metrics are necessary but not sufficient - models that win offline often lose online. Production experimentation is essential for confident deployment decisions.

Where It Is Used

A/B testing for ML is critical across all ML applications:

• Recommendations: Testing new algorithms and features
• Search ranking: Validating ranker improvements
• Personalization: Testing targeting models
• Pricing: Testing dynamic pricing models
• Fraud detection: Validating new rules without increasing losses
• Ad tech: Testing bidding and targeting models

Companies with mature ML: Netflix, Spotify, Google, Meta, Amazon all run thousands of experiments continuously.

Typical Data You Will See

Data sources: - User assignment logs - Event streams (clicks, conversions, revenue) - Session and user features - Model predictions and versions

What You Do in Practice

1. Define hypothesis - What improvement do you expect and why?
2. Choose metrics - Primary, secondary, and guardrail metrics
3. Design experiment - Unit, duration, sample size, allocation
4. Implement safely - Ramping, kill switches, monitoring
5. Run experiment - Collect data, monitor health
6. Analyze results - Statistical significance, segment analysis
7. Make decision - Ship, iterate, or kill
8. Document learnings - What worked, what didn't, why

Baseline First

Before testing a new model, ensure you have:

# Baseline requirements
baseline_metrics = {
    'primary': 'conversion_rate',
    'secondary': ['revenue_per_user', 'session_duration'],
    'guardrails': ['latency_p99', 'error_rate', 'user_complaints']
}

# Historical variance for sample size calculation
historical_std = calculate_metric_std(baseline_metrics['primary'])
baseline_mean = calculate_metric_mean(baseline_metrics['primary'])

Why baseline matters: - Variance determines sample size needed - Establishes what "normal" looks like - Identifies seasonality and trends - Defines guardrail thresholds

Experiment Design

Key decisions:

Sample size calculation:

from scipy import stats
import numpy as np

def sample_size_per_variant(
    baseline_rate: float,
    mde: float,  # Minimum detectable effect (relative)
    alpha: float = 0.05,
    power: float = 0.80
) -> int:
    """Calculate sample size for two-proportion test."""
    p1 = baseline_rate
    p2 = baseline_rate * (1 + mde)

    effect_size = abs(p2 - p1) / np.sqrt((p1*(1-p1) + p2*(1-p2)) / 2)

    z_alpha = stats.norm.ppf(1 - alpha/2)
    z_beta = stats.norm.ppf(power)

    n = 2 * ((z_alpha + z_beta) / effect_size) ** 2
    return int(np.ceil(n))

# Example: 2% baseline CTR, want to detect 5% relative lift
n = sample_size_per_variant(0.02, 0.05)  # ~63,000 per variant

Metric Selection

Metric categories:

Good ML experiment metrics: - Downstream business outcomes (not just model accuracy) - Sensitive enough to detect expected effect - Stable enough for reasonable sample sizes - Observable within experiment duration

Common mistakes: - Using model metrics (AUC) instead of business metrics - Too many primary metrics (multiple testing problem) - Ignoring long-term effects - Not including guardrails

Analysis and Statistics

Standard analysis:

from scipy import stats

def analyze_experiment(control, treatment):
    """Two-proportion z-test for conversion rate."""
    n1, conversions1 = len(control), sum(control)
    n2, conversions2 = len(treatment), sum(treatment)

    p1 = conversions1 / n1
    p2 = conversions2 / n2

    p_pooled = (conversions1 + conversions2) / (n1 + n2)
    se = np.sqrt(p_pooled * (1 - p_pooled) * (1/n1 + 1/n2))

    z_stat = (p2 - p1) / se
    p_value = 2 * (1 - stats.norm.cdf(abs(z_stat)))

    return {
        'control_rate': p1,
        'treatment_rate': p2,
        'relative_lift': (p2 - p1) / p1,
        'p_value': p_value,
        'significant': p_value < 0.05
    }

Variance reduction techniques: - CUPED: Use pre-experiment data as covariate - Stratification: Balance on known confounders - Ratio metrics: Normalize by exposure

Multiple testing correction: - Bonferroni: α / number of tests - Holm-Bonferroni: Sequential rejection - FDR control: Benjamini-Hochberg

Failure Modes and Debugging Checklist

• [ ] Sample ratio mismatch (SRM): Unequal split indicates bug
• [ ] Network effects: Treatment affects control users
• [ ] Novelty/primacy effects: Results change over time
• [ ] Segment differences: Overall neutral, segments differ
• [ ] Metric dilution: Effect lost in non-triggered users
• [ ] Flicker: Users see multiple variants
• [ ] Cookie/ID churn: Same user counted multiple times

SRM check:

def check_srm(control_n, treatment_n, expected_ratio=0.5):
    """Check for sample ratio mismatch."""
    total = control_n + treatment_n
    expected_control = total * expected_ratio
    expected_treatment = total * (1 - expected_ratio)

    chi2 = ((control_n - expected_control)**2 / expected_control + 
            (treatment_n - expected_treatment)**2 / expected_treatment)

    p_value = 1 - stats.chi2.cdf(chi2, df=1)
    return p_value > 0.001  # No SRM if p > 0.001

ML-Specific Considerations

ML experiments differ from typical A/B tests:

Handling interference: - Cluster randomization: Randomize at group level - Ego-cluster: User + their network - Switchback: Alternate treatment over time - Holdout: Long-term user holdout groups

Model-specific metrics to track: - Prediction distribution shifts - Coverage (what fraction of requests use model) - Latency impact - Feature health

Production Checklist

Before experiment: - [ ] Hypothesis documented - [ ] Metrics defined and tracked - [ ] Sample size calculated - [ ] Randomization verified (A/A test) - [ ] Guardrails and alerting set up - [ ] Kill switch ready

During experiment: - [ ] SRM check daily - [ ] Guardrails monitored - [ ] No peeking at results (except for safety) - [ ] Document any issues or changes

After experiment: - [ ] Full analysis completed - [ ] Segment analysis done - [ ] Novelty effects checked - [ ] Decision documented - [ ] Learnings shared

Tools and Stack

Example Mini-Case

Scenario: E-commerce site testing new recommendation model that should increase purchase rate.

Setup: - Primary metric: Purchase rate (baseline: 3.2%) - Guardrails: Page latency p99 < 500ms, add-to-cart rate ≥ baseline - MDE: 3% relative lift - Sample size: 85K users per variant - Duration: 2 weeks

Execution:

Week 1:
- SRM check: Passed (50.1% vs 49.9%)
- Guardrails: All green
- Preliminary: +4.2% purchase rate (p=0.08)

Week 2:
- Final analysis: +3.8% purchase rate (p=0.02)
- Segment analysis: Effect stronger for returning users (+5.1%)
- Novelty check: Effect stable across weeks

Result: - Decision: Ship new model - Impact: +$2.4M annual revenue - Follow-up: Run holdout for long-term effects

Key learnings: - Waited full duration despite promising early results - Segment analysis revealed opportunity for further personalization - Latency impact was minimal (2ms increase)

Related Pages

• Index: ML Field Guide
• MLOps Foundations
• Learning to Rank
• Model Monitoring

Further Reading

• Trustworthy Online Controlled Experiments (Kohavi et al.)
• Statsig Best Practices
• Eppo Experiment Design Guide
• Microsoft Experimentation Platform

Try it live

Everything above runs in your browser — open A/B Test Optimizer — UCB1 Multi-Armed Bandit Live and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.