Approximation Theory in Hyperparameter Optimization

Learn about approximation theory in hyperparameter optimization. Understand surrogate models, function approximation, and approximation error bounds.

Introduction

Approximation theory studies how complex functions can be approximated by simpler ones. In hyperparameter optimization, surrogate models approximate the expensive objective function, enabling efficient optimization.

Surrogate Models

Purpose

Approximate expensive objective:

f(λ) ≈ ŝ(λ)

Where ŝ is surrogate model.

Desired Properties

Cheap to evaluate
Accurate approximation
Uncertainty quantification
Easy to optimize

Approximation Error

Pointwise Error

ε(λ) = |f(λ) - ŝ(λ)|

Uniform Error

||f - ŝ||_∞ = sup_λ |f(λ) - ŝ(λ)|

Mean Squared Error

MSE = E[(f(λ) - ŝ(λ))²]

Gaussian Process Approximation

Universal Approximation

GP can approximate any continuous function:

Flexible non-parametric model
Uncertainty estimates
Data-dependent complexity

Approximation Quality

Depends on:

Number of observations
Kernel choice
Hyperparameter tuning

Convergence Rates

GP Convergence

For smooth functions:

||f - ŝ||_∞ = O(n^(-α))

Where α depends on smoothness.

Neural Network Approximation

Universal approximation theorem:

Single hidden layer sufficient
Approximation error → 0
May require many neurons

Bias-Variance in Approximation

Approximation Bias

Bias = E[ŝ(λ)] - f(λ)

Approximation Variance

Variance = Var[ŝ(λ)]

Total Error

MSE = Bias² + Variance

Key Insight

Approximation theory provides theoretical foundations for surrogate models. Understanding approximation error helps select appropriate models and balance accuracy with computational cost.

Model Selection

Trade-offs

Complexity vs accuracy
Computational cost
Uncertainty quantification
Optimization ease

Adaptive Approximation

Progressive Refinement

Improve approximation with more data:

Start with simple model
Refine with observations
Adapt complexity

Frequently Asked Questions

What is approximation theory in hyperparameter optimization?

Approximation theory studies how surrogate models approximate expensive objective functions. It provides theoretical foundations for understanding approximation error and model selection.

What are surrogate models?

Surrogate models approximate expensive objective functions f(λ) ≈ ŝ(λ). They're cheap to evaluate, enabling efficient optimization. Examples include Gaussian Processes, neural networks, random forests.

How do I measure approximation error?

Measure pointwise error ε(λ) = |f(λ) - ŝ(λ)|, uniform error ||f - ŝ||_∞, or mean squared error MSE = E[(f(λ) - ŝ(λ))²].

Can Gaussian Processes approximate any function?

Gaussian Processes can approximate continuous functions (universal approximation). Approximation quality depends on number of observations, kernel choice, and hyperparameters.

What's the convergence rate of surrogate models?

For smooth functions, GP approximation error ||f - ŝ||_∞ = O(n^(-α)) where α depends on smoothness. More observations improve approximation.