Optimizer Hyperparameters: Complete Guide

Learn about optimizer hyperparameters in machine learning. Understanding SGD, Adam, and other optimizer parameters.

Introduction

Optimizer hyperparameters control how the optimization algorithm updates model parameters during training. Different optimizers have different parameters that significantly impact convergence speed, stability, and final performance. Understanding these parameters is essential for effective model training.

SGD Parameters

Learning Rate

Controls step size in parameter updates:

θ = θ - lr * ∇J(θ)

Typical range: 0.001 to 0.1
Start with 0.01
Use learning rate scheduling

Momentum

Accumulates gradient information from previous steps:

v = momentum * v + lr * ∇J(θ)
θ = θ - v
            

Typical range: 0.9 to 0.99
Helps escape local minima
Speeds up convergence

Weight Decay

L2 regularization term:

θ = θ - lr * (∇J(θ) + weight_decay * θ)

Typical range: 1e-4 to 1e-2
Prevents overfitting
Promotes smaller weights

Adam Parameters

Learning Rate

Step size for parameter updates:

Typical range: 1e-4 to 1e-2
Default: 0.001
Less sensitive than SGD

Beta1 (First Moment Decay)

Exponential decay rate for first moment estimates:

Typical range: 0.9 to 0.99
Default: 0.9
Controls momentum-like behavior

Beta2 (Second Moment Decay)

Exponential decay rate for second moment estimates:

Typical range: 0.99 to 0.999
Default: 0.999
Controls adaptive learning rates

Epsilon

Small constant for numerical stability:

Typical range: 1e-8 to 1e-6
Default: 1e-8
Prevents division by zero

RMSprop Parameters

Learning Rate

Step size for updates:

Typical range: 1e-4 to 1e-2
Default: 0.001

Alpha (Decay Rate)

Smoothing constant for moving average:

Typical range: 0.9 to 0.99
Default: 0.99
Controls gradient smoothing

Epsilon

Numerical stability constant:

Typical range: 1e-8 to 1e-6
Default: 1e-8

AdaGrad Parameters

Learning Rate

Initial learning rate:

Typical range: 0.01 to 0.1
Default: 0.01
Adapts automatically over time

Epsilon

Numerical stability constant:

Typical range: 1e-8 to 1e-6
Default: 1e-8

AdaDelta Parameters

Rho

Decay rate for moving averages:

Typical range: 0.9 to 0.99
Default: 0.95
Controls gradient smoothing

Epsilon

Numerical stability constant:

Typical range: 1e-6 to 1e-4
Default: 1e-6

AdamW Parameters

Learning Rate

Step size for parameter updates:

Typical range: 1e-4 to 1e-2
Default: 0.001

Weight Decay

L2 regularization strength:

Typical range: 1e-4 to 1e-2
Default: 0.01
Applied differently than SGD

Beta1 and Beta2

Same as Adam:

Beta1: 0.9 (default)
Beta2: 0.999 (default)

Optimizer Selection Guidelines

When to Use SGD

Well-tuned learning rate schedule
Large batch sizes
Convex optimization problems
When interpretability is important

When to Use Adam

Default choice for most cases
Robust to hyperparameter choices
Good for sparse gradients
Works well with default parameters

When to Use RMSprop

Recurrent neural networks
Non-stationary objectives
When Adam doesn't work well

Key Insight

Adam is often the best default choice due to its robustness and adaptive learning rates. For specific cases, SGD with momentum or RMSprop may work better. Always validate optimizer choice with your specific problem.

Parameter Tuning Strategies

Grid Search

learning_rates = [1e-4, 1e-3, 1e-2]
momentums = [0.9, 0.95, 0.99]
weight_decays = [1e-4, 1e-3, 1e-2]
            

Log Scale Search

learning_rates = [10**x for x in range(-4, -1)] # 0.0001 to 0.1

Frequently Asked Questions

What optimizer should I use?

Adam is often the best default choice due to its robustness and adaptive learning rates. For specific cases, SGD with momentum or RMSprop may work better. Test multiple optimizers.

How do I tune Adam parameters?

Start with default values (lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8). Tune learning rate first, then beta values if needed. Adam is robust to parameter choices.

What's the difference between Adam and AdamW?

AdamW applies weight decay differently than Adam, often leading to better generalization. Use AdamW when you need L2 regularization, as it's more principled than Adam's weight decay.

How do I choose learning rate for different optimizers?

SGD: 0.01-0.1, Adam: 0.001-0.01, RMSprop: 0.001-0.01. Start with literature values, use learning rate finder, or try log scale search around default values.

Should I use momentum with SGD?

Yes, momentum (0.9-0.99) helps SGD escape local minima and converge faster. It's especially important for non-convex optimization problems like neural networks.