📦 Batch Size Impact

Understanding Batch Size

Batch size is the number of training examples processed together in one forward/backward pass. It's one of the most important hyperparameters, affecting training speed, memory usage, and model quality.

Batch Size Trade-offs

• Small Batches (1-32):
  • ✅ Better generalization
  • ✅ Regularization effect (noise in gradients)
  • ✅ Lower memory usage
  • ✅ Can escape sharp minima
  • ❌ Slower training (more iterations needed)
  • ❌ Noisy gradient updates
  • ❌ Underutilizes GPU
• Large Batches (256-2048):
  • ✅ Faster training (fewer iterations)
  • ✅ Better GPU utilization
  • ✅ Stabler gradients
  • ✅ Can use larger learning rates
  • ❌ Higher memory requirements
  • ❌ May converge to sharp minima (poor generalization)
  • ❌ Harder to fit in GPU memory

The Generalization Gap

Large batches often achieve lower training loss but worse test performance:

• Small batch noise acts as regularization
• Large batches converge to sharper minima (less robust)
• Solutions: Learning rate warmup, longer training, gradient noise

Computational Considerations

• Training Time:
  • Time per epoch ∝ 1 / batch_size (up to hardware limit)
  • But may need more epochs with larger batches
  • Sweet spot: maximize GPU utilization
• GPU Memory:
  • Memory usage ∝ batch_size
  • Larger batches = fewer fit in memory
  • OOM (Out of Memory) errors common
• Gradient Accumulation:
  • Simulate large batches with small memory
  • Accumulate gradients over multiple mini-batches
  • Update weights after N accumulation steps

Learning Rate Scaling

When increasing batch size, scale learning rate:

• Linear Scaling Rule: LR × (new_batch / old_batch)
• √ Scaling: LR × √(new_batch / old_batch)
• Warmup: Gradually increase LR for first few epochs
• Prevents instability with large batches

Optimal Batch Sizes

• CNNs: 32-256 typical
• Transformers: Can handle very large (1000+) with proper scaling
• RNNs: Often smaller (16-64)
• GANs: Typically smaller (32-128)
• Rule of Thumb: Largest that fits in GPU memory without hurting generalization

Gradient Noise

• Small batches: High gradient noise (variance)
• Large batches: Low gradient noise
• Noise can help escape local minima
• But also slows convergence

Distributed Training

For very large batch sizes:

• Data Parallelism: Split batch across GPUs
• Model Parallelism: Split model across GPUs
• Mixed Precision: Use FP16 to fit larger batches
• Gradient Checkpointing: Trade computation for memory

Batch Normalization Interaction

• BatchNorm statistics depend on batch size
• Very small batches: BatchNorm unstable
• Solution: GroupNorm, LayerNorm for small batches

Best Practices

• Start with 32 as default
• Increase to maximize GPU usage (watch nvidia-smi)
• Monitor validation performance
• Use learning rate warmup for large batches
• Consider gradient accumulation if memory-limited
• Test multiple batch sizes
• Balance speed vs generalization

Experiment with the Demo

Use the interactive tool above to:

• Compare training curves for different batch sizes
• See speed/memory/generalization trade-offs
• Understand gradient noise effects
• Find optimal batch size for your scenario

Batch size significantly impacts training! Understanding these effects helps you train models faster and achieve better results.