A Gaussian process (GP) surrogate models the unknown objective f(x) — e.g. validation accuracy as a function of a hyperparameter. After n trials, the posterior mean μ(x) and variance σ²(x) at any candidate x have closed forms:
μ(x) = k(x)ᵀ K⁻¹ y
σ²(x) = k(x,x) − k(x)ᵀ K⁻¹ k(x)
k(a,b) = σf²·exp(−(a−b)²/2ℓ²) (RBF kernel)
The Expected Improvement acquisition function scores every candidate x by how much it is likely to beat the current best observation f⁺, then the optimizer evaluates the argmax next:
z = (μ(x) − f⁺ − ξ) / σ(x)
EI(x) = (μ(x) − f⁺ − ξ)·Φ(z) + σ(x)·φ(z)
- GP lengthscale — how smooth we assume the objective is; small ℓ trusts data only very locally (spiky, uncertain mean), large ℓ generalizes broadly from few trials.
- Exploration ξ — added to the improvement threshold; higher ξ favors sampling uncertain, unexplored regions over exploiting the current best.
- Sample next (EI) — evaluates the noisy objective at the point that currently maximizes EI (the pulsing radar marker on the curve), exactly like a real hyperparameter-tuning loop.
- The glowing teal ribbon is the ±2σ credible band (brighter = more uncertain); the amber-to-white bars below are the EI acquisition surface guiding the next trial.