The bumpy surface is a stand-in for a model's validation-score landscape over two hyperparameters (e.g. learning rate on one axis, model complexity/depth on the other) — every point on it is one full train+evaluate run. AutoML's job is to find the peak using as few expensive trials as possible.
Random search: x_next ~ Uniform(space)
Grid search: x_next = next unvisited cell on a fixed lattice
Bayesian (UCB): x_next = argmax μ̂(x) + κ·σ̂(x)
μ̂(x): estimated score near x (exploit)
σ̂(x): distance to nearest trial (explore)
κ: exploration weight — higher favours unexplored area
Regret: r = score(best known) − score(true optimum)
- Random — simple, embarrassingly parallel, a strong baseline for small/cheap search spaces.
- Grid — systematic and reproducible, but wastes trials on a fixed lattice that ignores what earlier trials revealed.
- Bayesian — fits a cheap surrogate of the landscape after every trial and spends the next trial where it's most likely to pay off, converging to a low regret in far fewer evaluations.
Real AutoML systems add early stopping (Hyperband/ASHA), constraints (latency, memory, fairness) and nested cross-validation on top of this same explore/exploit loop.