Convergence Paths on Loss Surface
SGD
SGD + Momentum
RMSprop
Adam
Loss Convergence Over Time
Optimizer Selection
Loss Surface
Parameters
| Optimizer | Update Rule | Best For | Learning Rate |
|---|---|---|---|
| SGD | θ = θ - η∇θ | Convex problems, simplicity | 0.01 - 0.1 |
| SGD + Momentum | v = βv + ∇θ; θ = θ - ηv | Accelerating through ravines | 0.001 - 0.01 |
| RMSprop | E[g²] = βE[g²] + (1-β)g² | RNNs, non-stationary objectives | 0.001 |
| Adam | Combines momentum + RMSprop | Default choice, most problems | 0.001 (default) |
| AdaGrad | Adapts per-parameter rates | Sparse data, NLP | 0.01 |
| AdamW | Adam with weight decay fix | Transformers, modern architectures | 0.001 |
Understanding Optimizers
Optimizers are algorithms that adjust neural network weights to minimize loss. Choosing the right optimizer and tuning its parameters can dramatically affect training speed and final model quality.
Stochastic Gradient Descent (SGD)
The foundational optimizer:
- Update Rule: θ_new = θ_old - η · ∇L
- Pros: Simple, well-understood, memory efficient
- Cons: Slow convergence, gets stuck in local minima, sensitive to LR
- When to use: Convex problems, when you want fine control
SGD with Momentum
Accelerates SGD by accumulating velocity:
- Update Rule: v = β·v + ∇L; θ = θ - η·v
- Key Idea: Build up velocity in consistent directions
- Benefits: Faster convergence, dampens oscillations, escapes local minima better
- Hyperparameter: β (momentum coefficient, typically 0.9)
- Nesterov Momentum: Look-ahead variant, often better
AdaGrad
Adapts learning rate per parameter:
- Idea: Larger updates for infrequent parameters
- Benefit: Good for sparse data
- Problem: Learning rate decays too aggressively
- Used in: NLP with large vocabularies
RMSprop
Fixes AdaGrad's aggressive decay:
- Idea: Use exponential moving average of squared gradients
- Prevents: Learning rate from vanishing
- Good for: RNNs, non-stationary problems
- Hyperparameter: β (decay rate, typically 0.9)
Adam (Adaptive Moment Estimation)
Combines best of momentum and RMSprop:
- Maintains: First moment (mean) and second moment (variance) of gradients
- Bias Correction: Adjusts for initialization bias
- Why Popular: Works well out-of-the-box, requires little tuning
- Default Choice: Start here for most problems
- Hyperparameters: β₁=0.9, β₂=0.999, ε=1e-8
AdamW
Adam with correct weight decay:
- Fixes weight decay implementation in Adam
- Better generalization
- Preferred for transformers
- Used in BERT, GPT training
When to Use Each Optimizer
- Adam/AdamW: Default choice, works for most problems
- SGD + Momentum: When you need best final accuracy, have time to tune
- RMSprop: RNNs, reinforcement learning
- AdaGrad: Sparse data, NLP
Learning Rate Scheduling
- Step Decay: Reduce LR every N epochs
- Exponential Decay: Continuous reduction
- Cosine Annealing: Follows cosine curve
- Warm Restarts: Periodic LR resets
- 1cycle: Increase then decrease
Common Issues
- Oscillation: Learning rate too high - reduce LR
- Slow Convergence: LR too low - increase LR or use adaptive optimizer
- Getting Stuck: Try momentum or different optimizer
- NaN/Inf: Exploding gradients - lower LR, use gradient clipping
Best Practices
- Start with Adam (lr=0.001) as baseline
- Try SGD + momentum if have time to tune
- Use learning rate finder to set initial LR
- Apply learning rate scheduling
- Monitor gradient norms during training
- Use warmup for large batch sizes
- Different LR for different layer groups
Experiment with the Comparison
Use the interactive tool above to:
- Watch optimizers navigate loss surfaces
- Compare convergence speeds
- See how each handles different surface types
- Understand momentum effects
- Observe adaptive learning rate benefits
Choosing and tuning the optimizer is crucial for training success. Understanding their differences helps you train models faster and better!