Machine Learning Optimization Simulator

Loss Surface Optimizers

Learning rate η 0.12

Momentum β 0.85

SGD loss

–

Momentum loss

–

Adam loss

–

Steps taken

–

How it works

The bumpy sheet is a loss surface L(w₁, w₂) with one deep global minimum and two shallower local traps. Every optimizer starts from the same point and repeatedly estimates the local gradient ∇L(w), then takes a step against it. Plain SGD reacts only to the current gradient, so it usually stalls in the first dip it falls into. Momentum keeps a running velocity, letting it carry through shallow dips it would otherwise get stuck in. Adam rescales each step by the recent size of the gradient in that direction, so it slows down automatically near the bottom instead of overshooting.

SGD:       w = w − η·∇L(w)
Momentum:  v = β·v − η·∇L(w);  w = w + v
Adam:      m = β₁m+(1−β₁)g;  v = β₂v+(1−β₂)g²
           w = w − η·m̂/(√v̂+ε)

Learning rate η — the step size; too high and a ball overshoots or oscillates off the surface, too low and it barely moves.
Momentum β — how much of the previous velocity carries forward for Momentum and Adam's first moment; higher β powers through shallow local minima but can overshoot the true minimum.
Gradient noise — injects random jitter into every gradient estimate, mimicking the mini-batch noise real stochastic gradient descent sees when it only looks at a handful of training examples per step.
New random start — repositions all three balls to a fresh point on the surface and resets each optimizer's internal state (velocity, Adam moments, step count).

Real-world relevance: this is exactly the trade-off behind choosing an optimizer for training a neural network — SGD is cheap and simple but can stall on flat or bumpy regions of a real loss landscape, while Momentum and adaptive methods like Adam usually converge faster and are far less likely to give up in a shallow local minimum or saddle point.

SGD Momentum Adam

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.