🎯 Hyperparameter Optimization Landscape

About this simulation

Tuning a model means searching a loss landscape — one axis per hyperparameter, height equals validation loss — for the lowest point. The surface here is shaped by two hyperparameters, a log-scale learning_rate and a log-scale regularization strength, and it can be smooth (one clear basin) or rugged (several deceptive local minima), exactly like real loss surfaces.

How it works

• Grid Search — evaluates every point on a fixed lattice. Systematic but wasteful in high dimensions, and easy to step over a narrow optimum.
• Random Search — samples uniformly at random. Often out-performs grid search per-trial because it never wastes budget repeating a bad coordinate on every row.
• Bayesian Optimization — fits a cheap surrogate model of loss vs. already-seen trials, then picks each next point to balance exploiting the best region found so far against exploring uncertain ones — converging to the true minimum with far fewer evaluations.
• Marker color moves from cool blue (low loss) to hot red (high loss); the pulsing accent sphere marks the best trial found so far.

Controls

• Search strategy — switch between Grid, Random and Bayesian search.
• Trial budget — total number of model trainings the search is allowed to run.
• Landscape ruggedness — how many deceptive local minima the loss surface has.
• Search speed — how many trials are evaluated per second.
• ↺ Restart search — resets the run with the current settings. Drag to rotate, scroll to zoom.

Did you know?

In practice, random search often reaches a near-optimal region faster than grid search for the same budget, because most hyperparameters have far less effect on the loss than a couple of dominant ones — a grid wastes trials varying the unimportant ones. Bayesian methods like Gaussian-process or tree-based (TPE) optimizers go further, actively deciding where to look next.