📉 Learning Rate Scheduling Strategies

About this simulation

A neural network's loss landscape is rarely a smooth bowl — it's rugged, with ridges and shallow valleys. The learning rate controls how big a step the optimizer takes at each iteration across that landscape. Too high, and the ball bounces wildly across ridges without settling; too low, and it crawls so slowly it barely moves. This simulation rolls a ball down a 3D loss surface using real gradient descent, where the step size at each moment is set by the learning-rate schedule you choose — visualized live on the floating curve above the surface.

How it works

• Constant — the learning rate never changes; fast early but often oscillates near the minimum.
• Step decay — the rate is cut in half every quarter of training, in sudden drops.
• Cosine annealing — the rate follows a smooth cosine curve down to zero, popular for its gentle, predictable landing.
• Warmup + cosine — the rate ramps up linearly first (avoiding early instability) then anneals with a cosine curve.
• Exponential decay — the rate shrinks by a constant multiplicative factor every step.

Controls

• Schedule — pick which learning-rate strategy drives the descent.
• Base LR — the peak/starting learning rate for the schedule.
• Total steps — how many optimizer steps make up one training run.
• Warmup steps — length of the linear ramp-up (used by the warmup schedule).
• ⏸ Pause / ↺ Reset run — pause the descent or restart the ball at a fresh position. Drag to rotate, scroll to zoom.

Did you know?

Cosine annealing with warm restarts (SGDR) was introduced by Loshchilov & Hutter in 2016 and remains one of the most widely used schedules in modern deep learning, including in training large transformer models.