Search Space Analysis — the science

What you are looking at

The surface is the validation loss over two hyperparameters, x₁ (log₁₀ learning rate) and x₂ (log₁₀ weight decay). When the space has d > 2 dimensions, every other hyperparameter is held at its own optimum on the surface, so the surface is the best you could do with x₁, x₂. A dot is a configuration that was really evaluated with all d values; its height is its own loss, and the thin stem shows how much worse it was than the surface — the cost of the other d − 2 hyperparameters not being at their optimum. The landscape is a synthetic model, not a trained network: it is built to have a broad good basin, a deceptive second basin and a narrow "needle". Branin–Hoo and Rastrigin are the standard optimisation benchmarks.

The six strategies

The mathematics of the space

Model limits

Experiments to try

Independent checks

The same functions were tested outside the page against SciPy (scipy.stats.qmc.Sobol, qmc.discrepancy, scipy.special.gamma, scipy.stats.norm), Monte-Carlo simulation and the Hyperband paper's Table 1. Press Self-check to rerun four of them in your browser.

Sources

Bergstra & Bengio (2012) Random search for hyper-parameter optimization, JMLR 13. · Bergstra, Bardenet, Bengio & Kégl (2011) Algorithms for hyper-parameter optimization, NeurIPS. · Jones, Schonlau & Welch (1998) Efficient global optimization, J. Global Optim. 13. · Snoek, Larochelle & Adams (2012) Practical Bayesian optimization, NeurIPS. · Rasmussen & Williams (2006) Gaussian Processes for Machine Learning. · McKay, Beckman & Conover (1979) Technometrics 21. · Joe & Kuo (2008) SIAM J. Sci. Comput. 30. · Hickernell (1998) Math. Comp. 67. · Li et al. (2018) Hyperband, JMLR 18. · Akiba et al. (2019) Optuna, KDD. · Dixon & Szegő (1978) Towards Global Optimisation 2. Not affiliated with Optuna, Hyperopt or scikit-learn.

🗺️ Search Space Analysis: Mathematics of Hyperparameter Spaces

A 3D loss landscape shaped by two hyperparameters, explored live by grid search, random search, Latin hypercube sampling, or a Bayesian-guided sampler that homes in on the best region found so far.