Perceptron Learning
Click to add data points. The line represents the Neural Network's decision boundary training in real-time.
How Neural Networks Learn
The fundamentals of neural networks begin with neurons, weights, biases, and activation functions. During the forward pass, inputs are multiplied by weights, summed with a bias, and passed through an activation function to compute an output. Backpropagation then calculates the gradient of the loss with respect to each weight using the chain rule. Finally, gradient descent updates the weights to minimise the loss function. Each training iteration brings the decision boundary closer to correctly classifying the data.
Backpropagation and Gradient Descent
Backpropagation applies the chain rule backwards through the network, computing how much each weight contributed to the overall error. The learning rate controls step size during weight updates: too large and the network oscillates unstably around the minimum; too small and convergence becomes painfully slow.
This simulation visualises the gradient descent process in real time as the loss curve decreases and the decision boundary shifts to separate the two classes.
Activation Functions
Sigmoid � s(x) = 1/(1+e⁻ˣ) � squashes output between 0 and 1 but suffers from the vanishing gradient problem at extreme values. ReLU � max(0, x) � is computationally efficient and the modern default for deep networks, but neurons can �die� (permanently output zero gradients). Tanh outputs between -1 and 1 and is zero-centred, often making it better for hidden layers.
Each activation function produces different decision boundary shapes. The choice of activation profoundly affects how quickly and effectively a network learns.
Neural Networks in the Real World
Neural networks power an extraordinary range of modern technologies: image recognition (Convolutional Neural Networks / CNNs), language models (Transformers, GPT), self-driving cars, medical diagnosis, game AI (AlphaGo), and recommendation systems (Netflix, Spotify, YouTube). All of these are built on the same fundamental principle demonstrated in this simulation: adjusting weights through backpropagation to minimise a loss function.
Experiments to Try
- Create XOR-like data (linearly non-separable) and observe how a single-layer perceptron fails
- Increase hidden neurons from 2 to 16 and watch for overfitting
- Test different learning rates � compare fast vs slow convergence
- Watch the decision boundary converge as training progresses
Backpropagation
Backpropagation is the algorithm for computing gradients in a neural network. Using the chain rule of calculus, errors are propagated backwards from the output layer to earlier layers. Each weight receives a gradient ∂L/∂w, indicating how much it contributes to the loss. Gradient descent then adjusts weights in the direction that reduces loss: w ← w − η ⋅ ∂L/∂w.
Activation Functions
Activation functions introduce non-linearity, allowing networks to model complex relationships. Sigmoid σ(x) = 1/(1+e−x) squashes output to (0,1) but suffers from vanishing gradients. ReLU = max(0,x) is the default in deep learning computationally cheap and avoids vanishing gradients. Softmax converts logits to probabilities in the output layer.
Gradient Descent
Gradient descent minimises the loss function L (e.g., MSE or cross-entropy) by iteratively updating weights along the negative gradient. Batch gradient descent uses all training data; stochastic (SGD) uses one sample; mini-batch uses small batches (e.g., 32). Learning rate η controls step size too large causes divergence; too small makes training slow.
Universal Approximation
The Universal Approximation Theorem (Cybenko 1989; Hornik 1991) states that a feed-forward neural network with a single hidden layer and enough neurons can approximate any continuous function on a compact domain to arbitrary accuracy. This theoretical result underpins why neural networks are so powerful, though in practice depth (more layers) is more efficient than width (more neurons per layer).
Key Equations
| Concept | Formula | Notes |
|---|---|---|
| Forward pass | a(l) = f(W(l)a(l-1) + b(l)) | f: activation; W: weights; b: biases |
| Sigmoid | σ(x) = 1/(1 + e−x) | Output ∈ (0,1); used in binary classification |
| ReLU | f(x) = max(0, x) | Default in hidden layers; computationally efficient |
| MSE loss | L = (1/n)Σ(ŷ−y)² | Mean squared error for regression |
| Cross-entropy loss | L = −Σ y log(ŷ) | For classification; matches softmax output |
| Weight update (SGD) | w ← w − η ⋅ ∂L/∂w | η: learning rate; gradient via backprop |
| Softmax | σ(z)i = ezi / Σj ezj | Converts logits to probability distribution |
Curriculum Relevance
| Level | Topic | Relevance |
|---|---|---|
| A-Level / IB CS | Machine learning basics | Supervised learning, training vs testing, overfitting |
| AP CS / Data Science | Neural nets & AI | Perceptrons, layers, backpropagation concept |
| Undergraduate Year 1-2 | Machine learning | Linear regression, logistic regression, gradient descent |
| Undergraduate Year 3+ | Deep learning | CNNs, RNNs, transformers, batch normalisation |
| Postgraduate | Research methods | Optimisation theory, generalization bounds, architecture search |
🔒 Unlock All 32 Simulations
Get unlimited access to all 32 simulations with MySimulator Premium.
Upgrade to PremiumDiscussion and comments (Premium)
Premium members can discuss use-cases and ask questions inside this page.
Video tutorial (Premium)
A short guided walkthrough is available for premium users.
Unlock Tutorial ->Downloadable worksheet (Premium)
Printable worksheet with tasks and reflection questions.
Unlock Worksheet ->Share this simulation
Send this page to students or colleagues.
Related simulations
Continue with similar experiments.
📖 Recommended Reading
Deepen your understanding with our in-depth articles.