Hyperparameter optimization is, at its mathematical core, a search over a non-convex surface: every point on the surface is a choice of hyperparameters (or model weights) and its height is the resulting loss. This scene renders that surface directly and drops a marker onto it that performs real gradient-descent updates, exactly as an optimizer would while tuning a model. The surface is intentionally rugged — full of local minima and saddle points — because that is what makes advanced optimization theory (momentum, adaptive learning rates, second-order curvature) necessary in practice.
Real hyperparameter-tuning loss surfaces live in dozens of dimensions, not two — which is exactly why meta-learning and Bayesian optimization methods exist: they build a cheap surrogate model of the surface so an expensive full training run doesn't have to be repeated at every candidate point.
A rugged 3D loss surface stands in for a hyperparameter search space, and a live optimizer marker descends it in real time using the actual update rules of SGD, Momentum, and Adam.
The surface's height is a non-convex loss function; the marker takes real gradient-based steps computed from the analytic gradient at its current position, showing why optimizer choice and learning-rate schedules matter on rugged landscapes.
Adjust learning rate, momentum/β₁, and landscape ruggedness, and switch between SGD, Momentum, and Adam. Watch how each optimizer handles the same terrain, then start a fresh descent from a new point.
Real hyperparameter loss surfaces have dozens of dimensions; meta-learning and Bayesian optimization exist precisely to build a cheap surrogate of a landscape like this one without training the full model at every candidate point.