🎛️ Optimizer Hyperparameters Lab

About this simulation

Training a neural network means repeatedly nudging its weights downhill on a loss surface — a landscape whose height is the error for a given set of weights. Which way you step, and how far, is entirely controlled by the optimizer and its hyperparameters. This lab drops four balls, one per optimizer, on the same 3D loss surface and lets you watch how learning rate and momentum change the path each one takes to the bottom.

How it works

• SGD (red) — steps straight along the negative gradient. Too large a learning rate on a steep ravine makes it overshoot and oscillate.
• Momentum (orange) — accumulates a velocity from past gradients, smoothing zig-zags and coasting through shallow dips.
• RMSProp (blue) — divides each step by a running average of recent squared gradients, automatically shrinking steps on steep axes.
• Adam (purple) — combines momentum and RMSProp's adaptive scaling, which is why it's the default choice in most deep learning training loops.

Controls

• Learning rate — the base step size η used by every optimizer. Push it high on the ravine to see SGD diverge while Adam stays stable.
• Momentum / β1 — how much of the previous velocity/gradient average carries into the next step (used by Momentum and Adam).
• Loss landscape — switch between a steep, elongated ravine (classic zig-zag demo) and a bumpy valley with local dips.
• View mode — compare all four optimizers at once, or isolate one to inspect its trail closely.
• ↺ Restart run — resets every ball to the starting point and clears trails.

Did you know?

Adam's name is an abbreviation of "Adaptive Moment Estimation" — it tracks both the first moment (mean) and second moment (uncentered variance) of the gradients, which is exactly the momentum term and the RMSProp term combined.