A deep neural network is a stack of layers of simple units called neurons. Each connection between two neurons carries a weight; every neuron sums its weighted inputs, adds a bias, and passes the result through an activation function that decides how strongly it "fires." Stacking several such layers lets the network build up increasingly abstract representations of the input.
Modern deep networks can have hundreds of billions of parameters, but every one of them is still trained with the same basic loop shown here: forward pass, compute loss, backpropagate the error, and nudge the weights — repeated millions of times.
An interactive 3D neural network: watch cyan signals fire forward through input, hidden and output layers, pink error signals flow backward, and connection weights redraw themselves purple or orange as the loss falls during training.
Every layer sums weighted inputs and passes them through an activation function; forward passes produce a prediction, backpropagation sends the error backward, and gradient descent nudges each weight to shrink the loss over many epochs.
Change the hidden layer width to resize the network, pick ReLU, Sigmoid or Tanh to see how activations behave differently, and adjust the learning rate to see faster but noisier or slower but smoother convergence. Pause training or reset the weights at any time.
ReLU became the default activation for deep networks largely because, unlike sigmoid or tanh, it does not saturate for large positive inputs — this keeps gradients from vanishing as they backpropagate through many layers.