Top-down map of a validation-accuracy landscape (Blender-rendered). Click the map to add your own try. Blue: random search. Orange: Bayesian optimisation.
The science
Setting. Two hyperparameters (think log learning rate and log weight decay, scaled to [0,1]) give a validation score f(u,v): a smooth sum of four Gaussian bumps plus a small ripple. The maximum is known so everything can be checked exactly. It is a teaching surface, not the measured loss of a real network.
Random search. If a fraction q of the space is “good” (top q of f by area), each uniform try hits it with probability q, so P(at least one hit in n tries) = 1 - (1-q)^n. For q=5% about 14 tries give 50% and 59 give 95%. The expected best score after n tries is E[max] = ∫ y d(F(y)^n) with F the distribution of f over the square (240x240 grid).
Bayesian optimisation. A Gaussian process with RBF kernel exp(-r²/2ℓ²) models f from the tries so far; the next try maximises expected improvement EI = (μ-f*-ξ)Φ(z) + σφ(z) over 300 random candidates. Noise 1e-4, prior sd 0.5, first three tries random. The GP mean and EI maps show what the optimiser believes.
Simplifications. Noise-free objective, two dimensions (random search keeps the same formula in any dimension; GP methods degrade), fixed kernel length (not fitted), candidate-set maximisation instead of a continuous optimiser, only 60 repetitions for the Bayesian curve (visible wobble). Which method wins depends on how smooth the real objective is; the lesson is the probability argument, not a ranking of tools.