Search Space Analysis — the science
What you are looking at
The surface is the validation loss over two hyperparameters, x₁ (log₁₀ learning rate) and x₂ (log₁₀ weight decay). When the space has d > 2 dimensions, every other hyperparameter is held at its own optimum on the surface, so the surface is the best you could do with x₁, x₂. A dot is a configuration that was really evaluated with all d values; its height is its own loss, and the thin stem shows how much worse it was than the surface — the cost of the other d − 2 hyperparameters not being at their optimum. The landscape is a synthetic model, not a trained network: it is built to have a broad good basin, a deceptive second basin and a narrow "needle". Branin–Hoo and Rastrigin are the standard optimisation benchmarks.
The six strategies
- Grid: m = ⌊B1/d⌋ levels per axis, md points; the rest of the budget is unused. Only m distinct values of each hyperparameter are ever tried.
- Random: independent uniform draws (seeded). Every draw tries a new value on every axis.
- Latin hypercube (McKay, Beckman & Conover 1979): each axis is cut into B strata and every stratum is used exactly once.
- Sobol (Joe & Kuo 2008 direction numbers, random digital shift): a low-discrepancy sequence that fills the cube more evenly than random points.
- TPE (Bergstra et al. 2011): after 10 random trials, split the trials into the best γ(n) = min(⌈0.1 n⌉, 25) and the rest, fit a Parzen density l(x) to the good ones and g(x) to the others, and pick the candidate (of 24 drawn from l) that maximises l(x)/g(x).
- Bayesian optimisation (Jones et al. 1998; Snoek et al. 2012): a Gaussian process with a Matérn-5/2 kernel (length-scale chosen by marginal likelihood) predicts mean μ(x) and uncertainty σ(x); the next point maximises the Expected Improvement
EI = (f⁺ − ξ − μ)Φ(z) + σφ(z), z = (f⁺ − ξ − μ)/σ. Cost grows as n³, so the budget is capped at 200.
The mathematics of the space
- Hit probability: if the good region is a fraction p of the space, n independent random trials hit it at least once with probability
1 − (1 − p)ⁿ. For p = 5 % that is 95 % after n = ⌈ln 0.05 / ln 0.95⌉ = 59 trials — independent of the dimension (Bergstra & Bengio 2012).
- Low effective dimensionality: when only one axis matters, a 4 × 4 grid tests 4 values of it and random search tests 16. The expected distance from an unknown optimum to the nearest tested value is 1/12 ≈ 0.083 for the grid and ≈ 0.031 for random points.
- Volume: the inscribed ball fills
π^(d/2) / (Γ(d/2 + 1) · 2^d) of the cube (78.5 % at d = 2, 0.25 % at d = 10); a shell of width ε at the faces holds 1 − (1 − 2ε)^d of it (65 % for ε = 5 %, d = 10). A 10-level grid needs 10d points.
- Discrepancy: the centred L2 discrepancy CD² (Hickernell 1998) measures how unevenly points fill the cube; smaller is more even.
- Hyperband (Li et al. 2018): brackets of successive halving; with R = 81 and η = 3 the brackets start 81, 34, 15, 8 and 5 configurations.
Model limits
- The loss is deterministic: no validation noise, no cross-validation, every evaluation costs the same.
- Only continuous hyperparameters; no categorical or conditional ones (e.g. "optimizer = Adam → β₂").
- The GP is isotropic (one length-scale for all axes) and EI is maximised over 384 candidates, not exactly.
- TPE treats each axis independently; Hyperband here is budget arithmetic only, not a learning-curve model.
- Hyperparameter names and ranges are illustrative labels for the synthetic surface.
Experiments to try
- E1 — Student, Important axes = "only learning rate", B = 16: compare Grid and Random with the rug ticks on. Then run the ensemble (Researcher) with "new landscape seed for every repeat": random's median regret is lower. On one fixed landscape the grid can be lucky or unlucky — the advantage is an average over where the optimum happens to lie.
- E2 — p = 5 %, B = 60, Random: the hit probability card says 95.4 %; the ensemble's "hit top-p" column should be close to 0.95.
- E3 — d = 4, B = 128: compute CD² for Random, Latin hypercube and Sobol. Expect Sobol < LHS < Random.
- E4 — Rastrigin, B = 121, Grid: an 11 × 11 grid lands exactly on the optimum at the centre (regret 0) — luck of alignment, not skill.
- E5 — Grid with B = 100 while increasing d from 2 to 5: the grid collapses from 10 to 2 levels per axis.
- E6 — Branin, B = 40: TPE and Bayesian optimisation usually end with regret below 0.05; Grid does not.
Independent checks
The same functions were tested outside the page against SciPy (scipy.stats.qmc.Sobol, qmc.discrepancy, scipy.special.gamma, scipy.stats.norm), Monte-Carlo simulation and the Hyperband paper's Table 1. Press Self-check to rerun four of them in your browser.
Sources
Bergstra & Bengio (2012) Random search for hyper-parameter optimization, JMLR 13. · Bergstra, Bardenet, Bengio & Kégl (2011) Algorithms for hyper-parameter optimization, NeurIPS. · Jones, Schonlau & Welch (1998) Efficient global optimization, J. Global Optim. 13. · Snoek, Larochelle & Adams (2012) Practical Bayesian optimization, NeurIPS. · Rasmussen & Williams (2006) Gaussian Processes for Machine Learning. · McKay, Beckman & Conover (1979) Technometrics 21. · Joe & Kuo (2008) SIAM J. Sci. Comput. 30. · Hickernell (1998) Math. Comp. 67. · Li et al. (2018) Hyperband, JMLR 18. · Akiba et al. (2019) Optuna, KDD. · Dixon & Szegő (1978) Towards Global Optimisation 2. Not affiliated with Optuna, Hyperopt or scikit-learn.