Question 1

What is backpropagation?

Accepted Answer

Backpropagation is the algorithm for efficiently computing gradients in a neural network. Using the chain rule of calculus, it propagates the error signal backwards from the output layer through each hidden layer. For each weight w, it computes ∂L/∂w (the gradient of the loss with respect to that weight). Gradient descent then updates each weight: w ← w − α·(∂L/∂w), where α is the learning rate. A single forward pass + backward pass constitutes one training step.

Question 2

Why do deep networks need non-linear activation functions?

Accepted Answer

Without non-linear activations, any stack of linear layers collapses to a single linear transformation: f(W₂(W₁x)) = (W₂W₁)x. No matter how many layers you add, the network can only learn linear decision boundaries. Non-linear activations (ReLU, sigmoid, tanh) break this: they allow the network to compose non-linear transformations and learn arbitrarily complex functions — a property proven by the Universal Approximation Theorem.

Question 3

What causes a neural network to overfit?

Accepted Answer

Overfitting occurs when a neural network memorises the training data rather than learning general patterns. It happens when the model has too many parameters relative to the amount of training data, or when training runs too long. Overfitting manifests as low training loss but high validation loss. Solutions include: dropout (randomly zeroing activations during training), L1/L2 weight regularisation, data augmentation, early stopping, and reducing model capacity.

Question 4

What is the Universal Approximation Theorem?

Accepted Answer

The Universal Approximation Theorem (Cybenko 1989, Hornik 1991) states that a single hidden layer feedforward neural network with a sufficient number of neurons and non-linear activations can approximate any continuous function on a compact subset of ℝⁿ to arbitrary precision. This proves neural networks are theoretically as expressive as needed — but it says nothing about how many neurons are required or how easy the function is to learn via gradient descent.

🧪 Neural Network Playground — Visualise Training, Backprop & Decision Boundaries

What did you find?