Tuning a machine-learning model's hyperparameters (learning rate, regularization strength, etc.) means searching an expensive, unknown function: each trial requires training a model to see how well it validates. Bayesian optimization treats that unknown function as a random process. A Gaussian process (GP) surrogate places a probability distribution over every possible landscape consistent with the trials observed so far, giving both a predicted mean score and a calibrated uncertainty everywhere it hasn't tried yet.
Libraries such as Optuna, Ax and scikit-optimize use this exact probabilistic machinery to tune real neural networks in far fewer trials than grid search — because the model knows, in a precise probabilistic sense, where it is still unsure.
A Gaussian-process surrogate model builds a live probability distribution over an unknown validation-accuracy landscape, then an acquisition function decides — using both predicted performance and uncertainty — exactly which hyperparameters to try next.
Each trial (learning rate, regularization pair) updates a Bayesian posterior mean and variance over the whole 2D hyperparameter grid. Colour encodes uncertainty; the Upper Confidence Bound acquisition function trades off exploiting known peaks against exploring uncertain regions.
Adjust the kernel length-scale, exploration weight κ, and observation noise, then click "Sample suggested point" to run a Bayesian-optimization trial, or "Random trial" to compare against blind search. Toggle the true landscape and acquisition surface to see how the surrogate compares.
Because a Gaussian process gives a full predictive distribution — not just a point estimate — Bayesian optimizers like the ones in Optuna and Ax can find near-optimal hyperparameters in a fraction of the trials that grid or random search would need.