🛡️ Regularization Techniques

Understanding Regularization

Regularization techniques prevent overfitting by constraining model complexity. They help models generalize to unseen data rather than memorizing training examples.

L2 Regularization (Weight Decay)

• Adds penalty proportional to sum of squared weights
• Loss = Data Loss + λ Σw²
• Encourages small weights
• Smooths decision boundaries
• Most common: λ = 0.0001-0.01

L1 Regularization (Lasso)

• Penalty proportional to sum of absolute weights
• Loss = Data Loss + λ Σ|w|
• Promotes sparsity (many weights → 0)
• Feature selection built-in
• Less common in deep learning

Dropout

• Randomly drop neurons during training
• Prevents co-adaptation of neurons
• Ensemble effect
• Rate 0.2-0.5 typical
• Apply after dense layers

Data Augmentation

• Artificially expand training set
• Images: flips, rotations, crops, color jitter
• Text: back-translation, synonym replacement
• Highly effective, no downside

Early Stopping

• Stop when validation loss stops improving
• Patience parameter (epochs to wait)
• Simple and effective

Batch Normalization

• Normalizes layer inputs
• Reduces internal covariate shift
• Has regularizing effect
• Can reduce need for dropout

When to Use Each

• Always: Data augmentation, early stopping
• CNNs: BatchNorm + data augmentation usually sufficient
• Dense layers: Dropout + L2
• Small datasets: Strong regularization needed

Experiment

See how different techniques prevent overfitting!