📉 Mathematical Foundations of Hyperparameter Optimization

About this simulation

Every hyperparameter search is, mathematically, a search for the minimum of an objective function L(θ) over a space of hyperparameter vectors θ — learning rate, regularization strength, layer width, and so on. Because this function is expensive to evaluate (it usually means training a whole model) and rarely convex, optimizers must balance exploration of the space against exploitation of promising regions. The purple surface below is a synthetic stand-in for a validation-loss surface over two hyperparameters; watch how each strategy searches it differently.

How it works

• Gradient Descent — follows the local slope −∇L(θ) downhill from a random start. Fast on smooth bowls, but can get trapped in a local minimum on rugged surfaces.
• Random Search — draws a fresh batch of random points every step. No exploitation of gradient information, but immune to local traps and trivially parallelizable.
• Grid Search — sweeps a regular lattice of points exhaustively. Systematic and reproducible, but its cost grows exponentially with the number of hyperparameters.
• The gold marker always shows the best point found so far — the running estimate of θ* = argmin L(θ).

Controls

• Optimizer — switch between gradient descent, random search and grid search.
• Step size — the learning rate used by gradient descent's update rule θ ← θ − η∇L(θ).
• Landscape ruggedness — adds more local minima to the surface, simulating a harder non-convex objective.
• Search points per batch — how many candidates random/grid search evaluate each step.
• ↺ Reset search — restarts the optimizer from scratch. Drag to rotate, scroll to zoom.

Did you know?

Bergstra & Bengio (2012) showed that for most real hyperparameter spaces, random search finds comparably good configurations to grid search using far fewer trials — because only a handful of hyperparameters actually matter for a given problem, so a grid wastes most of its evaluations along unimportant axes.