Learning Rate Schedules: Gradient Descent on a Loss Surface

Schedule

Initial LR 0.22

Training length (epochs) 120

Epoch

0 / 120

–

Loss

–

How it works

The surface is a synthetic loss landscape: a broad convex bowl with sinusoidal ripples layered on top, the way a real neural-network loss surface has a dominant descent direction plus many small local wrinkles. Each epoch the marker takes one gradient-descent step, w ← w − η·∇L(w), where η is the learning rate produced by the selected schedule for that epoch. The instanced trail behind the marker is the actual optimization trajectory, not a decoration.

L(x,z) = 0.11(x²+z²) + 0.85·sin(1.4x)·cos(1.3z)
step decay:   η(t) = η₀ · 0.5^⌊t/(T/5)⌋
cosine:       η(t) = ηmin + ½(η₀−ηmin)(1+cos(πt/T))
warmup+cos:   η(t) = η₀·t/Twarm  for t
              
                Constant — never slows down, so it keeps hopping between neighbouring ripples instead of settling; loss plateaus above the true minimum.
                Step decay — large, sudden loss drops right after each cut (visible as a kink in the loss readout) because the smaller step finally lets the marker settle into whichever ripple it is in.
                Cosine annealing — the smooth η(t) curve gives smooth loss convergence: fast descent early, fine settling late, no kinks.
                Warmup + cosine — starting η at ~0 and ramping up avoids a violent first jump on a steep patch of the surface, then anneals down exactly like cosine.
              
              Real-world relevance: this is the same trade-off that governs training a real network — too high an LR for too long oscillates and never reaches a good minimum; decaying too early freezes the weights in the first ripple the optimizer happens to fall into, which is why schedules that start high and anneal smoothly (cosine, with or without warmup) are the default choice for modern training runs.

Current weights Trajectory (past steps) Loss surface

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.