Optimizer Race: SGD vs Momentum vs Adam

Loss Surface

Hyperparameters

Learning rate η 0.15

Momentum β 0.85

Optimizers

SGD (plain gradient descent)

SGD + Momentum

Adam (adaptive moments)

Live Readouts

Optimizer	Loss	Step
SGD	—	0
Momentum	—	0
Adam	—	0

How it works

All three optimizers minimize the same scalar loss L(x, z) by following its gradient ∇L, but each one turns the gradient into a step differently:

SGD:       x ← x − η ∇L(x)

Momentum:  v ← β v + ∇L(x)
           x ← x − η v

Adam:      m ← β1 m + (1−β1) ∇L(x)
           s ← β2 s + (1−β2) ∇L(x)²
           x ← x − η · m̂ / (√ŝ + ε)

Plain SGD zig-zags across narrow ravines because the gradient points mostly across the valley, not along it. Momentum accumulates a running velocity v, so consistent downhill directions build speed while oscillating components cancel out — it swings through the ravine instead of crawling. Adam goes further: it keeps a per-dimension estimate of the gradient's second moment (s) and divides the step by its square root, effectively giving each coordinate its own adaptive learning rate — steep dimensions get smaller steps, shallow ones get larger steps, so it also handles saddle points where the gradient briefly vanishes.

Learning rate η — the base step size for all three optimizers, so the comparison stays fair.
Momentum β — the momentum decay used by the Momentum optimizer (Adam uses fixed β1=0.9, β2=0.999 internally, the standard defaults).
Surface — switch the loss landscape: a convex bowl (all three converge similarly), an elongated ravine (Momentum and Adam pull ahead), a saddle point (plain SGD can stall near zero gradient), or multiple local minima (different optimizers can land in different basins).

This is exactly the mechanism used to train real neural networks — Adam (or variants like AdamW) is the default optimizer in most deep learning because it converges reliably across the wildly different gradient scales found in a network's millions of parameters.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.