🔄 Backpropagation Step-by-Step

Interactive Visualization of Gradient Flow in Neural Networks

Training Control

Training Data

Auto Training

0
Epoch
-
Current Loss
-
Accuracy
0
Training Steps

Current Step: Initialization

Select a training example to begin. This network learns XOR function using backpropagation.

Understanding Backpropagation

Backpropagation (backward propagation of errors) is the fundamental algorithm for training neural networks. It efficiently computes gradients of the loss function with respect to all weights in the network, enabling gradient descent optimization.

The Four Steps of Backpropagation

Training a neural network with backpropagation involves four key steps:

  • Forward Pass: Input data flows through the network, computing activations layer by layer
  • Loss Calculation: Compare network output with target to compute error (loss)
  • Backward Pass: Gradients flow backward through network using chain rule
  • Weight Update: Adjust weights in direction that reduces loss

Forward Pass Mathematics

For each layer, we compute:

z = W · x + b (linear transformation)
a = σ(z) (activation function)

Where:

  • x = input from previous layer
  • W = weight matrix
  • b = bias vector
  • z = pre-activation (weighted sum)
  • σ = activation function (sigmoid, ReLU, etc.)
  • a = activation (output of layer)

Loss Calculation

For binary classification, we typically use Mean Squared Error or Cross-Entropy:

MSE Loss: L = 1/2 · (y_pred - y_true)²
Cross-Entropy: L = -[y·log(ŷ) + (1-y)·log(1-ŷ)]

Backward Pass - The Chain Rule

The backward pass uses the chain rule from calculus to compute how much each weight contributed to the error. Starting from the output layer, we compute gradients layer by layer going backwards.

For output layer:

δ_output = (a_output - y_true) · σ'(z_output)

For hidden layers:

δ_hidden = (W_next^T · δ_next) · σ'(z_hidden)

Where δ (delta) represents the error signal for each neuron.

Weight Updates - Gradient Descent

Once we have gradients, we update weights:

W_new = W_old - η · ∂L/∂W
b_new = b_old - η · ∂L/∂b

Where η (eta) is the learning rate controlling step size.

Why Backpropagation is Revolutionary

Before backpropagation, training neural networks was extremely slow. The key insight:

  • Instead of trying different weights randomly, we compute exact gradients
  • Chain rule lets us reuse computations, making it efficient
  • Complexity is linear in network size, not exponential
  • Works for any differentiable architecture

The XOR Problem

This visualization uses the classic XOR problem - a function that cannot be learned by single-layer networks but requires hidden layers. XOR returns 1 when inputs differ, 0 when they match:

  • 0 XOR 0 = 0
  • 0 XOR 1 = 1
  • 1 XOR 0 = 1
  • 1 XOR 1 = 0

This problem historically demonstrated the necessity of multi-layer networks and backpropagation.

Vanishing and Exploding Gradients

Two major challenges in backpropagation:

  • Vanishing Gradients: Gradients become tiny in early layers (especially with sigmoid), preventing learning. Solutions: ReLU, batch normalization, residual connections
  • Exploding Gradients: Gradients become huge, causing instability. Solutions: gradient clipping, proper initialization, batch normalization

Computational Efficiency

Backpropagation is remarkably efficient:

  • Forward pass computes outputs and saves intermediate values
  • Backward pass reuses saved values to compute gradients
  • Time complexity: O(W) where W is number of weights
  • Space complexity: O(W) to store gradients
  • Much faster than numerical gradient estimation

Modern Variations

Backpropagation has evolved with several improvements:

  • Stochastic Gradient Descent (SGD): Update after each example
  • Mini-batch Gradient Descent: Update after small batches
  • Momentum: Accelerate training in relevant directions
  • Adam: Adaptive learning rates per parameter
  • Batch Normalization: Stabilize gradient flow
  • Gradient Checkpointing: Trade computation for memory

Automatic Differentiation

Modern frameworks (TensorFlow, PyTorch) use automatic differentiation to implement backpropagation automatically. They build a computational graph during forward pass, then traverse it backwards to compute gradients - you just define the forward pass!

Tips for Effective Training

  • Use appropriate learning rate - too high causes oscillation, too low is slow
  • Initialize weights carefully (Xavier, He initialization)
  • Choose activation functions wisely (ReLU for hidden, softmax/sigmoid for output)
  • Monitor gradient magnitudes - watch for vanishing/exploding
  • Use batch normalization for deep networks
  • Apply gradient clipping if gradients explode
  • Consider learning rate scheduling

Interactive Learning

Use this visualization to:

  • Step through each phase of backpropagation manually
  • See how gradients flow backward through the network
  • Watch weights update in real-time
  • Observe the network learning the XOR function
  • Understand the relationship between loss and weight updates

Experiment with the controls above to build deep intuition about how neural networks learn!