Optimization Methods Explained
Optimization finds parameter values that minimize a loss function. Learn foundations, algorithms, and practical tips for convergence.
📚 Fundamentals
- Convexity: global vs local minima
- Gradients: steepest descent direction
- Hessian: curvature informs Newton steps
🧪 Algorithms
- Gradient Descent: x←x−η∇f
- Momentum/Adam: accelerate and adapt step sizes
- Newton/Quasi-Newton: second-order updates
- Line search/Trust region: robust step selection
❓ Frequently Asked Questions
1) Vanishing gradients?
Rescale inputs, use adaptive methods, or second-order info.
Rescale inputs, use adaptive methods, or second-order info.
2) Saddle points?
Noise and momentum help escape flat regions.
Noise and momentum help escape flat regions.
3) Step size choice?
Use schedules or line search; monitor loss decrease.
Use schedules or line search; monitor loss decrease.
4) Nonconvex problems?
Multiple runs and good initialization help.
Multiple runs and good initialization help.
5) Constraints?
Projected gradients, penalty or barrier methods.
Projected gradients, penalty or barrier methods.