Inside a Neural Network: How a Forward Pass Turns Input Into a Prediction

The single neuron: a weighted vote

Every prediction a neural network makes is the end result of thousands, millions, or billions of a single tiny operation repeated and layered together: the artificial neuron. A neuron receives several numeric inputs, multiplies each one by a learned weight, sums the results together, adds one more learned number called a bias, and finally passes that sum through a non-linear activation function to produce its single output value:

output = activation( (weight₁ × input₁) + (weight₂ × input₂) + ... + bias )

The weights represent how strongly each input influences this particular neuron — a large positive weight means that input strongly pushes the neuron's output up, a negative weight pushes it down, and a weight near zero means that input is effectively ignored. Every weight and bias in the network starts as a small random number and is gradually adjusted during training (via backpropagation and gradient descent) until the network's collective predictions become accurate. The forward pass, by contrast, is what happens when a trained (or partially trained) network is actually used to produce a prediction: input data flows forward, layer by layer, being transformed at each step, until it emerges as an output.

From one neuron to a layer, and a layer to a network

A single neuron on its own can only draw a straight decision boundary, which is far too limited for most real problems. Networks get their power by arranging many neurons into layers, and stacking several layers on top of each other. A typical feedforward network has three kinds of layers: an input layer, whose "neurons" simply hold the raw feature values (pixel brightnesses, word counts, sensor readings); one or more hidden layers, where the actual computation and feature-combining happens; and an output layer, which produces the final prediction in whatever form the task requires — a single number for regression, or a set of class probabilities for classification.

Within a fully-connected layer, every neuron receives the output of every neuron in the previous layer, computes its own weighted sum and bias, and applies its activation function. In compact matrix notation, if a[l-1] is the vector of outputs from the previous layer, W[l] is the matrix of weights for the current layer, and b[l] is its vector of biases, the current layer's pre-activation values are z[l] = W[l]·a[l-1] + b[l], and its output is a[l] = activation(z[l]). This same formula is applied layer after layer, with each layer's output becoming the next layer's input, until the final layer produces the network's prediction, a[L].

Why activation functions are not optional

If every neuron simply summed its weighted inputs and passed the result along unchanged, with no non-linear activation function anywhere in the network, then no matter how many layers were stacked on top of each other, the entire network would mathematically collapse into being equivalent to a single linear layer — because a chain of purely linear operations is itself just another linear operation. That would make deep networks pointless: a hundred-layer network with no activation functions could learn nothing more than a single layer could.

Non-linear activation functions are what let networks bend, curve, and combine inputs in complex ways, giving them the ability to approximate essentially any function given enough neurons and layers. A handful of activation functions dominate modern practice, each suited to a particular role:

• ReLU (Rectified Linear Unit), defined simply as max(0, x), is the default choice for hidden layers in most modern networks because it is cheap to compute and avoids much of the "vanishing gradient" problem that plagued earlier networks — though a neuron whose weighted sum is consistently negative can get permanently stuck outputting zero, a failure mode called the "dying ReLU" problem.
• Sigmoid squashes any input into a value between 0 and 1, making it a natural choice for the output layer of a binary classifier, where that value can be read directly as a probability.
• Softmax is sigmoid's generalisation to multiple classes: it converts a layer's raw outputs into a full probability distribution over several categories that sums to exactly 1, and is the standard choice for the output layer of a multi-class classifier.
• Tanh, which squashes inputs into the range -1 to 1, is centred on zero (unlike sigmoid) and is still commonly seen inside recurrent networks.

A worked example: three numbers becoming a prediction

Concretely tracing a forward pass helps make the abstraction solid. Suppose a tiny network takes two input numbers, has one hidden layer with two ReLU neurons, and a single output neuron with a sigmoid activation (for a yes/no prediction). Given input [1.0, 0.5]: the first hidden neuron might compute a weighted sum of, say, 0.9, and since ReLU passes positive numbers through unchanged, its output is 0.9. The second hidden neuron might compute a weighted sum of -0.3; ReLU clamps this to 0, meaning this neuron contributes nothing further downstream for this particular input — it has effectively "switched off". The output neuron then takes a weighted combination of those two hidden outputs (0.9 and 0), adds its own bias, and passes the result through sigmoid, producing something like 0.82 — which the network reports as an 82% probability of the positive class.

Every one of those weights and biases (six numbers, in this toy example: four connecting inputs to the hidden layer, two connecting the hidden layer to the output) was learned during training. The forward pass is simply the mechanical process of applying those learned numbers to a new input; the learning itself happens separately, through backpropagation and gradient descent, comparing this predicted 0.82 against the true label and nudging every weight slightly to make future predictions more accurate.

Depth, width, and what they buy you

Two structural choices define a network's capacity: its width (how many neurons sit in each layer) and its depth (how many layers are stacked). A wider layer can represent more distinct patterns at that particular stage of processing, while additional depth lets the network build increasingly abstract representations by composing earlier, simpler patterns into more complex ones — this is especially visible in convolutional networks trained on images, where early layers learn to detect simple edges and colour blobs, middle layers combine those into textures and shapes, and later layers assemble those shapes into recognisable objects. Both width and depth come at a cost: more parameters mean more computation per forward pass, a greater risk of overfitting without enough training data, and, in very deep networks, a greater risk of the vanishing or exploding gradient problems that make training unstable without countermeasures like careful weight initialisation, batch normalisation, or residual (skip) connections.