A Blender-built validation-accuracy landscape over two hyperparameters. Blue flags: random search. Orange: Bayesian optimisation. Drag to orbit.
The science
Setting. Two hyperparameters (think log learning rate and log weight decay, scaled to [0,1]) give a validation score f(u,v). Here f is a smooth sum of four Gaussian bumps plus a small ripple, so the maximum is known and the page can be checked exactly. It is a teaching surface, not the measured loss of a real network.
Random search. If a fraction q of the search space is “good” (top q of f by area), each independent uniform try lands there with probability q, so P(at least one hit in n tries) = 1 - (1-q)^n. For q=5% about 14 tries give 50% and 59 give 95%. The expected best score after n tries is E[max] = ∫ y d(F(y)^n), where F is the cumulative distribution of f over the square (computed numerically on a 240x240 grid).
Bayesian optimisation. A Gaussian process with RBF kernel k = exp(-r²/2ℓ²) models f from the tries so far; the next point maximises expected improvement EI = (μ-f*-ξ)Φ(z) + σφ(z), z=(μ-f*-ξ)/σ, over 300 random candidates. Observation noise 1e-4, prior sd 0.5, first three tries random.
Simplifications. Noise-free objective, two dimensions (random search does not degrade with dimension in this formula, GP methods do), fixed kernel length (not fitted), candidate-set maximisation instead of a continuous optimiser, 60 repetitions for the Bayesian curve (visible Monte-Carlo wobble). Which method wins depends on how smooth the real objective is; the lesson is the probability argument, not a ranking of tools.