Learning Rate Schedules Comparison
Schedule Type
Parameters
Actions
| Schedule | Formula | Best For | Pros/Cons |
|---|---|---|---|
| Step Decay | LR Ć 0.1 every N epochs | Image classification | ā Simple ā Manual tuning |
| Exponential | LR Ć e^(-kt) | Smooth decay needed | ā Smooth ā Too aggressive |
| Cosine | LR Ć cos(Ļt/T) | Modern CNNs, transformers | ā Smooth ā No tuning |
| 1cycle | Triangle: up then down | Fast training, SGD | ā Fast convergence |
| Warmup | Linear increase first | Large batch, transformers | ā Stabilizes training |
Understanding Learning Rate Schedules
Learning rate scheduling adjusts the learning rate during training. Starting with a higher LR enables fast initial progress, while decreasing it later allows fine-tuning for better convergence.
Why Schedule Learning Rate?
- Early Training: High LR for fast progress through easy terrain
- Late Training: Low LR for precise convergence to minimum
- Escape Local Minima: LR changes can help escape poor solutions
- Better Generalization: Lower final LR finds flatter minima
Common Schedules Explained
- Step Decay:
- Reduce LR by factor (0.1) every N epochs
- Simple, widely used
- Example: Start 0.1, drop to 0.01 at epoch 30, 0.001 at epoch 60
- Exponential Decay:
- LR = LR_0 Ć e^(-kt)
- Smooth continuous decay
- Can be too aggressive
- Cosine Annealing:
- LR = LR_min + 0.5(LR_max - LR_min)(1 + cos(Ļt/T))
- Smooth curve from max to min
- No hyperparameters to tune
- Popular in modern training
- Cosine with Warm Restarts:
- Periodic restarts to high LR
- Can escape local minima
- Ensemble via checkpoints
- 1cycle Policy:
- Increase LR first half, decrease second half
- Proposed by Leslie Smith
- Trains faster (fewer epochs needed)
- Great with SGD + momentum
Learning Rate Warmup
- Gradually increase LR from 0 to initial value
- Typically first 5-10% of training
- Essential for: Large batch sizes, transformers, Adam
- Prevents instability early in training
Adaptive Learning Rates (Per-Parameter)
- Adam, RMSprop adapt LR per parameter automatically
- Still benefit from scheduling the global LR
- Cosine schedule commonly used with Adam
Choosing a Schedule
- Default Recommendation: Cosine annealing
- For SGD: Step decay or 1cycle
- For Adam: Constant with warmup, or cosine
- For Transformers: Warmup + inverse sqrt decay
- For RL: Often constant
Implementation Tips
- Most frameworks have built-in schedulers
- PyTorch: torch.optim.lr_scheduler
- TensorFlow: tf.keras.optimizers.schedules
- Use learning rate finder first to set max LR
- Monitor validation loss to detect overfitting
- Save checkpoints at different LRs
Common Mistakes
- Decaying too quickly (underfitting)
- Decaying too slowly (wasted computation)
- Not using warmup with large batches
- Same schedule for all optimizers (customize!)
Advanced Techniques
- Cyclical Learning Rates: Periodic oscillations
- Stochastic Gradient Descent with Warm Restarts (SGDR)
- Layer-wise LR: Different rates for different layers
- Discriminative Fine-tuning: Different rates when fine-tuning
Experiment with the Demo
Use the interactive tool above to:
- Compare different LR schedules visually
- Adjust initial learning rate and see effects
- Understand when LR drops occur
- See trade-offs between strategies
Learning rate scheduling is crucial for training! The right schedule can reduce training time and improve final model quality significantly.