Probability Theory in Hyperparameter Optimization

Learn about probability theory in hyperparameter optimization. Understand Bayesian inference, Gaussian processes, and probabilistic models.

Introduction

Probability theory plays a central role in hyperparameter optimization, especially in Bayesian methods. Understanding probabilistic models enables better algorithm selection and interpretation of results.

Bayesian Framework

Bayes' Theorem

Foundation of Bayesian inference:

P(λ|D) = P(D|λ) × P(λ) / P(D)

Where:

P(λ|D): Posterior probability of hyperparameters
P(D|λ): Likelihood of data given hyperparameters
P(λ): Prior probability of hyperparameters
P(D): Evidence (normalization constant)

Posterior Distribution

The posterior P(λ|D) represents our updated beliefs about hyperparameters after observing data. It balances prior knowledge with observed evidence.

Gaussian Processes

GP Definition

Gaussian Process is a collection of random variables where any finite subset has joint Gaussian distribution:

f(λ) ~ GP(m(λ), k(λ, λ'))

Where m(λ) is mean function and k(λ, λ') is covariance kernel.

Kernel Functions

Common kernels:

RBF: k(λ, λ') = exp(-||λ - λ'||²/2l²)
Matern: Flexible smoothness
Linear: k(λ, λ') = λᵀλ'

Expected Improvement

Mathematical Formulation

Expected Improvement acquisition function:

EI(λ) = E[max(0, f* - f(λ))]

Where f* is best observed value.

Closed-Form Solution

For Gaussian Process:

EI(λ) = σ(λ)[Φ(z) + zφ(z)]

Where z = (μ(λ) - f*)/σ(λ), Φ and φ are CDF and PDF of standard normal.

Upper Confidence Bound

Mathematical Formulation

UCB(λ) = μ(λ) + βσ(λ)

Where β balances exploration and exploitation.

Probability of Improvement

Mathematical Formulation

PI(λ) = P(f(λ) < f*) = Φ((f* - μ(λ))/σ(λ))

Noise Modeling

Observation Model

Observations are noisy:

y = f(λ) + ε, where ε ~ N(0, σ²)

Heteroscedastic Noise

Noise variance may depend on hyperparameters, requiring more sophisticated models.

Key Insight

Probability theory enables Bayesian Optimization to model uncertainty explicitly, balance exploration and exploitation, and provide principled acquisition functions for hyperparameter selection.

Prior Distributions

Common Priors

Uniform: Equal probability across range
Log-Uniform: For hyperparameters spanning orders
Normal: Central values more likely
Beta: Bounded, flexible shape

Posterior Inference

GP Posterior

After observing data D = {(λᵢ, yᵢ)}:

P(f(λ*)|D) ~ N(μ(λ*), σ²(λ*))

Mean and variance computed using kernel matrix.

Frequently Asked Questions

What is Bayes' theorem in hyperparameter optimization?

Bayes' theorem updates beliefs about hyperparameters: P(λ|D) = P(D|λ)P(λ)/P(D). It combines prior knowledge P(λ) with observed evidence P(D|λ) to form posterior P(λ|D).

What is a Gaussian Process?

A Gaussian Process is a probabilistic model f(λ) ~ GP(m(λ), k(λ,λ')) where m is mean function and k is covariance kernel. It provides uncertainty estimates for hyperparameter optimization.

How does Expected Improvement work mathematically?

Expected Improvement computes EI(λ) = E[max(0, f* - f(λ))], the expected improvement over best observed value. For GP, it has closed-form: σ(λ)[Φ(z) + zφ(z)] where z = (μ(λ) - f*)/σ(λ).

What are kernel functions in Gaussian Processes?

Kernel functions k(λ, λ') define covariance between hyperparameters. Common kernels include RBF (smooth), Matern (flexible smoothness), and Linear. They determine GP behavior.

How is noise modeled in Bayesian Optimization?

Noise is modeled as y = f(λ) + ε where ε ~ N(0, σ²). This accounts for stochasticity in evaluations from random initialization, data sampling, or optimization.