Convolutional Neural Networks: Filters, ReLU and Max-Pooling Explained

A filter is a question asked everywhere

A convolutional layer slides a small grid of numbers, the kernel or filter (often 3x3 or 5x5), across every position of the input image, and at each position computes the dot product between the kernel and the patch of pixels underneath it. The result is a single number that says how strongly that patch matches the pattern the kernel has learned to detect — an edge, a corner, a patch of colour. Repeating this at every position produces a whole new grid called a feature map.

output[y][x] = sum over (i,j) of  kernel[i][j] * input[y+i][x+j]   (+ bias)
                                                        # one filter → one feature map

The trick that makes convolution cheap and powerful is weight sharing: the same 9 or 25 numbers are reused at every position, so the network does not need a separate set of weights for "edge in the top-left corner" and "edge in the bottom-right corner" — it learns one edge detector and applies it everywhere. This also makes the layer roughly translation-equivariant: shift the input a few pixels and the feature map shifts with it, instead of the network having to relearn the pattern from scratch at every location.

ReLU: throwing away the sign

After convolution, each value passes through a non-linear activation, almost always the Rectified Linear Unit: ReLU(x) = max(0, x). It zeroes out negative responses and passes positive ones through unchanged. Without a non-linearity, stacking convolutional layers would collapse mathematically into one big linear filter no matter how many layers were stacked; ReLU is what lets depth actually add expressive power, and it is cheap — one comparison, no exponentials — which is why it replaced sigmoid and tanh in most modern networks.

Max-pooling: throwing away position

Max-pooling then shrinks each feature map by sliding a small window (typically 2x2) across it and keeping only the largest value in each window, discarding the rest. This does two things: it reduces the amount of computation for every layer that follows, and it makes the representation a little more tolerant to small shifts and distortions in exactly where a feature appeared — the network increasingly cares whether an edge or a texture is present in a neighbourhood, not its exact pixel coordinate.

2x2 max-pool, stride 2:
[ 3  1  7  2 ]        [ 3  7 ]
[ 0  2  4  9 ]   →   [ 5  9 ]
[ 5  1  0  3 ]
[ 1  0  2  1 ]

Stacking layers: from edges to objects

A single convolution layer only sees a small receptive field — one kernel's worth of pixels. Stack several conv/ReLU/pool blocks and each successive layer's receptive field, measured back in original-image pixels, grows, because it is built from features that were themselves built from smaller regions. In practice this produces a rough hierarchy: early layers respond to edges and simple gradients, middle layers combine those into corners, textures and simple shapes, and deeper layers respond to parts and whole objects. Nobody hand-designs this hierarchy — it falls out of training the kernels with backpropagation to minimise a classification or detection loss.

Why convolution beats a fully-connected layer for images

A fully-connected layer on an 8x8x3 image already needs 192 weights per output neuron; on a realistic 224x224x3 photo that is over 150,000 weights per neuron, and none of that structure is reused between neurons. A convolutional layer with a 3x3 kernel needs 27 weights total for the whole image, shared at every position. Fewer parameters means fewer things to learn from a fixed amount of data, faster training, and — because the same detector is tested at every location — a built-in assumption that a useful pattern (an edge, an eye, a wheel) is a local pattern that can appear anywhere in the frame, which is almost always true for natural images.

Frequently asked questions

Try it live

Everything above runs in your browser — open Convolutional Neural Network and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.