Radial Basis Function Networks: Learning With Bumps, Not Planes

A network built from bumps, not planes

A standard feedforward network builds its decision surface out of hyperplanes — each hidden unit's pre-activation is a dot product w·x, and a sigmoid or ReLU turns that into a soft or hard half-space boundary. A radial basis function network does something different: each hidden unit computes a distance from the input to a stored centre, and turns that distance into a bump that is largest at the centre and decays outward. The most common choice is the Gaussian:

φ_i(x) = exp( -‖x - c_i‖² / (2σ_i²) )

c_i = centre of basis function i (a point in input space)
σ_i = width — how fast the bump decays away from c_i
y(x) = Σ_i w_i · φ_i(x) + b     (output is a weighted sum of bumps)

Local versus global approximation

This locality is the defining trait. A hidden unit in a multilayer perceptron affects the output everywhere on one side of its hyperplane, all the way to infinity; a Gaussian basis function affects the output only in a neighbourhood of its own centre, and its influence fades to essentially zero a few widths away. That makes RBF networks local approximators: adding training data far from existing centres does not disturb the fit near those centres, which is a useful property when a model must be updated incrementally without retraining everything from scratch. The price is the curse of dimensionality — the number of bumps needed to tile a region grows very fast with input dimension, because a fixed-width Gaussian only covers a small hyper-volume in high dimensions.

Two-stage training

Because the model is linear in the weights once the bumps are fixed, RBF networks are almost always trained in two decoupled stages rather than by full backpropagation. First, choose the centres c_i — commonly by running k-means clustering on the training inputs, so centres land where the data actually is, or in engineering contexts by simply picking a subset of training points. Second, with centres and widths fixed, computing the output weights is linear least squares: build the matrix Φ of φ_i(x_j) for every training point and basis function, and solve w = (ΦᵀΦ)⁻¹Φᵀy (or its ridge-regularised form) in one shot — no gradient descent, no local minima, a closed-form solution. Widths σ_i are usually set from the spacing between neighbouring centres, often as some multiple of the average distance to the nearest few centres, to avoid gaps or excessive overlap between bumps.

Why they are universal approximators

Given enough centres, an RBF network with Gaussian (or any of several other admissible) basis functions can approximate any continuous function on a compact domain to arbitrary accuracy — the same universal approximation guarantee that holds for sigmoidal MLPs, proved for RBF networks by Park and Sandberg in 1991. Intuitively this is not surprising: a sum of narrow enough Gaussians centred densely enough can reproduce any smooth bump shape, in the same way a Riemann sum of narrow rectangles reproduces any integrable function as the rectangles narrow.

Where RBF networks still win

Exact interpolation problems, where you want the surface to pass through every training point precisely (one centre per data point, solved analytically), function approximation for control systems, and the ubiquitous RBF kernel in support vector machines — which is exactly this same Gaussian similarity measure used as a kernel rather than a hidden layer. The interpolation network and the SVM kernel trick are, mathematically, the same idea in two different frameworks.

Frequently asked questions

Try it live

Everything above runs in your browser — open Radial Basis Function Network and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.