📉 Convergence Analysis in Hyperparameter Optimization

About this simulation

Training a model, or tuning its hyperparameters, is a search over a loss landscape: a surface whose height at each point is how badly a given configuration performs. An optimizer walks that surface, always stepping roughly "downhill" using the local gradient. Convergence analysis studies how quickly and how reliably that walk approaches a minimum, and how we decide, in practice, when to stop.

How it works

• The purple surface is a synthetic non-convex loss function with one broad global basin and several shallower local minima — a stand-in for a real hyperparameter search space.
• The glowing marker performs gradient-based updates each frame, tracing a live trajectory (the amber trail) toward lower loss.
• Learning rate sets the step size: too small converges slowly, too large can overshoot or diverge.
• Momentum accumulates a velocity term across steps, smoothing the path through narrow valleys and helping escape shallow local minima.
• Gradient noise mimics the stochasticity of mini-batch gradients (as in SGD on real data) — some noise helps escape poor local minima, too much prevents settling.
• Stopping threshold (ε) is the convergence criterion: once the gradient norm stays below ε, the search is declared converged and the marker turns green.
• The side chart plots loss versus iteration on a log scale, the standard way convergence rate is visualized in the optimization literature.

Controls

• Learning rate — step size of each update (0.005–0.4).
• Momentum — how much of the previous velocity carries over (0–0.95).
• Gradient noise — magnitude of stochastic noise added to each gradient estimate.
• Stopping threshold (ε) — gradient-norm level below which the run is considered converged.
• Optimizer — switch between plain SGD, SGD with momentum, and an Adam-style adaptive method.
• ↺ New start point — restarts from a fresh random location on the surface. Drag to rotate, scroll to zoom.

Did you know?

Adam (Kingma & Ba, 2015) adapts its effective learning rate per-parameter using running estimates of the gradient's mean and variance, which is why it often converges faster and more robustly on noisy, non-convex objectives than plain gradient descent — though it can converge to a different, sometimes worse, minimum than SGD.