Restricted Boltzmann Machines: Energy-Based Generative Learning

A Bipartite Network with No Peer Pressure

A Restricted Boltzmann Machine (RBM) consists of two layers of binary units: a visible layer v, which represents observed data such as pixel activations, and a hidden layer h, which represents learned latent features. Every visible unit connects to every hidden unit through a weight matrix W, but crucially, there are no connections within a layer — no visible-to-visible links, no hidden-to-hidden links. This is the 'restriction' that gives the model its name, and it turns out to be enormously useful: because units within a layer never talk to each other directly, all the hidden units become conditionally independent given the visible units, and vice versa. That independence means you can update an entire layer in one parallel sweep instead of updating each unit one at a time, which is what makes an otherwise cumbersome undirected graphical model tractable enough to train at scale.

The Energy Function: Grading Configurations

At the heart of an RBM is an energy function that assigns a single scalar number to every possible joint configuration of visible and hidden units. For binary units it is written as E(v,h) = -sum_i(a_i * v_i) - sum_j(b_j * h_j) - sum_ij(v_i * W_ij * h_j), where a_i and b_j are bias terms for the visible and hidden units and W_ij is the weight connecting visible unit i to hidden unit j. Low energy should correspond to configurations the network considers plausible, and high energy to configurations it considers unlikely. This energy is converted into a probability through the Boltzmann distribution: P(v,h) = exp(-E(v,h)) / Z, so that low-energy states get exponentially higher probability mass and high-energy states get exponentially suppressed. Training an RBM is, in a very literal sense, sculpting this energy surface so the valleys line up with the training data and the peaks line up with everything else.

The Intractable Z and Why Contrastive Divergence Exists

The normalizing constant Z in that formula, called the partition function, is defined as Z = sum over all v,h of exp(-E(v,h)) — it sums the unnormalized probability over every conceivable combination of visible and hidden states. For any network with more than a handful of units, this sum has an astronomically large number of terms, making Z, and therefore the exact probability P(v,h), computationally intractable. This matters because the true maximum-likelihood gradient for the weights takes the elegant form dlogP(v)/dW_ij = <v_i h_j>_data - <v_i h_j>_model, where the first term is an expectation easily estimated from real training examples but the second term requires sampling from the full model distribution defined by that same unreachable Z. Contrastive divergence (CD), introduced by Geoffrey Hinton in 2002, sidesteps this by replacing the intractable model expectation with a cheap approximation: start a Markov chain at a real data point, run just a handful of alternating Gibbs sampling steps (often only one, called CD-1), and use the resulting 'reconstruction' in place of a true equilibrium sample. The resulting update rule is DeltaW_ij ~= <v_i h_j>_data - <v_i h_j>_reconstruction, nudging weights up wherever a visible-hidden pair co-activates on real data more than it does after the short reconstruction chain, and down where the reverse holds.

Gibbs Sampling Between the Layers

The reconstruction step in contrastive divergence leans directly on the layer independence built into the RBM. Because hidden units are conditionally independent given the visible layer, you can sample all of h at once from P(h|v) = sigmoid(b + W^T v), and because visible units are conditionally independent given the hidden layer, you can sample all of v at once from P(v|h) = sigmoid(a + W h). Alternating these two sampling steps — visible to hidden, then hidden back to visible, then hidden again — is exactly a block Gibbs sampling chain, and it is what lets the network mock up a rough draw from its own model distribution without ever computing Z. Running this chain longer produces a more faithful model sample and a more accurate gradient estimate, but even the crude one-step version used in CD-1 turns out to reliably push the energy surface in the right direction, which is why contrastive divergence became the practical workhorse for training RBMs rather than exact but hopeless maximum-likelihood learning.

Stacking RBMs and Their Place in History

In the mid-2000s, RBMs found their most famous application as building blocks for Deep Belief Networks. Hinton and colleagues showed that you could train one RBM on raw data, freeze it, treat the activations of its hidden layer as 'data' for a second RBM stacked on top, and repeat this greedy layer-by-layer pretraining all the way up. This unsupervised pretraining gave deep networks a sensible starting point for their weights before a final round of supervised fine-tuning, which was a genuinely important trick in an era, roughly 2006 to 2012, when training deep networks from random initialization was notoriously unstable. Better weight initialization schemes, ReLU activations, batch normalization, and ultimately far larger labeled datasets eventually made layer-wise RBM pretraining unnecessary, and RBMs receded from mainstream use. But their conceptual DNA runs straight through modern generative AI: the idea of learning an energy or score function over data, and using iterative stochastic sampling to draw from it, resurfaces directly in today's score-based and diffusion generative models, which makes the RBM less a museum piece and more an early ancestor of ideas still very much alive.

Frequently asked questions

Try it live

Everything above runs in your browser — open Restricted Boltzmann Machines: Energy-Based Generative Learning and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.