Loading…
🔬 Hyperparameter search
The science

The task. Choose two hyperparameters (think log learning rate and log regularisation, scaled to [0,1]) to minimise a validation loss f(u,v). Each evaluation is one full training run, so you can afford only n of them. The surface here is a synthetic stand-in: a narrow curved valley plus a shallower basin plus a small ripple.

Grid search tests a regular √n × √n lattice of cell centres. Random search draws n uniform points. A fixed lattice can hit or miss a narrow valley depending on how the optimum happens to align with it; random search finds a region holding fraction q of the space with probability 1 − (1 − q)n, and tries n distinct values of every parameter instead of √n.

Bayesian optimisation. After 5 random starts, a Gaussian process with an RBF kernel (length 0.18, fixed, not tuned) models the loss. The next point maximises expected improvement EI = (fmin − μ)Φ(z) + sφ(z), z = (fmin − μ)/s, searched over 220 candidates (40 near the incumbent). The magenta mesh is the GP mean; the magenta ring is the next point.

Regret. Simple regret is f(xbest) − f*, using the true loss at the point whose observed value was lowest, so noise can mislead the search. f* is found on a fine grid. "Run 40 landscapes" repeats everything on 40 shifted landscapes and averages, which removes the luck of one fixed layout.

Simplifications. Two dimensions only; real searches have many parameters of mixed type. The GP hyperparameters are fixed, noise is Gaussian and homoscedastic, and the landscape is synthetic. Which method wins depends on the budget, the noise and the landscape; the page reports what it measures and does not assume a winner.

Running numerical checks…

Search results

Evaluations–
True optimum f*–
Grid best–
  regret–
Random best–
  regret–
Bayesian best–
  regret–
40-landscape mean regretnot run
Wins (lowest final)–
Drag to orbit. Orange cubes: grid. Blue spheres: random. Magenta diamonds: Bayesian.
Made with 🟠 Blender · Three.js