Bayesian Optimization for Hyperparameter Tuning

Fundamentals

Surrogate Models

• Gaussian Processes: smooth, small-dimensional spaces
• TPE: tree-structured spaces, categorical variables
• Random Forests/ExtraTrees: robust, mixed-type spaces

Acquisition Functions

• Expected Improvement (EI)
• Upper Confidence Bound (UCB)
• Probability of Improvement (PI)
• Thompson Sampling

Batching and Parallelism

Use q-EI or fantasizing for batch selection; prefer asynchronous schedulers to avoid stragglers.

Warm Starts and Space Design

Seed with random/Sobol samples, use log scales for rates, and encode conditionals explicitly.

How the Algorithm Works

Algorithm

1. Initialize with random/Sobol samples
2. Fit surrogate model to observed (x, y)
3. Optimize acquisition to propose new x
4. Evaluate objective; update dataset
5. Repeat until budget or convergence

Acquisition Choices

• EI: balances exploration and exploitation via expected improvement
• UCB: optimism-in-the-face-of-uncertainty trade-off
• PI: probability of improvement

Batching and Async

Use q-EI or fantasizing; asynchronous BO avoids idle workers by proposing without waiting for all evaluations.

Real-World Applications

Domains

• AutoML pipelines for tabular data
• Hyperparameter tuning for deep learning
• Recommender systems parameter optimization
• Robotics and control parameter tuning

Constraints and KPIs

Latency, memory, and fairness constraints; KPIs include accuracy, calibration, and cost.

Best Practices

Checklist

• Seed with random/Sobol samples
• Use appropriate surrogates for the space
• Monitor acquisition values and uncertainty
• Deduplicate suggestions; enforce min distance
• Validate with nested CV when needed
• Track all runs with artifacts and seeds

Anti-Patterns

• One-shot BO without warm start
• Ignoring uncertainty in selection
• Overfitting to a single validation split
• No experiment tracking
• Unbounded spaces causing instability

Evaluation

Protocols

• Nested CV or fixed hold-out test for final evaluation
• Track uncertainty via CIs or bootstrapping
• Mitigate selection bias by limiting peeks

Worked Examples

Optuna + TPE

study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler())
study.optimize(objective, n_trials=200)

BoTorch + q-EI

# Build GP, use q-EI to propose batch candidates

Implementation

Optuna

study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler())
study.optimize(objective, n_trials=100)

BoTorch

# GP surrogate with q-EI acquisition and batch suggestions

SMAC

# Tree-based surrogates for categorical-heavy spaces

Patterns

• Warm start with random/Sobol seeds
• Batch or async proposals
• De-duplicate and enforce min distance
• Track acquisition values and surrogate fits

The Math Behind It

Gaussian Process Posterior

Given prior f ~ GP(m, k) and noisy observations y = f(x)+ε, the posterior mean and variance admit closed forms; kernels (Matern, RBF) control smoothness.

TPE

TPE models p(x|y) via l(x) and g(x) densities over good and bad observations, choosing x to maximize l(x)/g(x).

Acquisitions

EI integrates expected improvement over the posterior; UCB uses mean+β·std; PI integrates tail probability beyond best f*.

Batch Selection

q-EI and fantasizing enable parallel proposals by integrating over pending evaluations.

Training Strategy

Budgets

• Calibrate per-trial budget via learning curves
• Increase budget as promising regions emerge
• Use early stopping with patience

Stability

• Repeat noisy trials and average
• Fix seeds, splits, and environment
• Checkpoint and resume reliably