Nesterov Momentum: Look-Ahead Gradient on an Ill-Conditioned Valley

Optimizers Racing

Plain Gradient Descent Classical (Heavy-Ball) Momentum Nesterov Accelerated Gradient

Valley Shape & Update Rule

Condition number κ = a/b 15

Learning rate η 0.020

Momentum β 0.85

Live Readouts

Steps taken

Nesterov loss

—

Plain GD loss

—

How it works

The bowl is a quadratic loss f(x,y) = ½(κ·x² + y²) — steep along x, shallow along y once κ > 1. Its gradient is exact and linear: ∇f = (κ·x, y). Three walkers start at the same corner and take a step every tick using three different rules:

Plain GD:    θ ← θ − η∇f(θ)

Momentum:    v ← β·v − η∇f(θ)
             θ ← θ + v

Nesterov:    v ← β·v − η∇f(θ + β·v)   ← gradient evaluated at the
             θ ← θ + v                 look-ahead point, not θ itself

Plain GD is stable only while η < 2/κ, so a large κ forces a tiny step in the steep x-direction — and that same tiny step crawls painfully slowly down the shallow y-direction. Classical momentum accumulates velocity and reaches the shallow floor faster, but it overshoots and oscillates across the steep walls because it computes the gradient at the point it is currently standing on. Nesterov's trick is to evaluate the gradient at θ + β·v — where momentum is about to carry it — which acts as a correction that damps the overshoot before it happens, giving the accelerated O(1/k²) convergence rate that makes NAG the standard building block inside Adam and most modern deep-learning optimizers.

κ slider — elongates the valley; higher κ makes plain GD's zig-zag dramatically worse while barely slowing Nesterov.
η slider — step size shared by all three optimizers, for a fair race.
β slider — momentum coefficient; push it toward 0.98 to see classical momentum start ringing around the minimum while Nesterov settles.

⚠ diverged — lower η or κ

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.