Normalizing Flows: Exact Density Modeling via Invertible Transforms

The Missing Piece: Why GANs and VAEs Can't Tell You the Likelihood

A generative model's job is to learn a probability distribution p(x) over data, whether that's images, molecules, or audio waveforms. GANs learn this implicitly: a generator network maps random noise to samples, and a discriminator pushes those samples to look real, but nowhere in this process does the model ever compute a number for p(x). VAEs get closer by defining a probabilistic decoder, but because the mapping from the latent variable z to the data x is typically many-to-one and non-invertible, the true likelihood requires integrating over every possible z that could have produced x — an intractable integral. VAEs sidestep this with a lower bound, the ELBO, which approximates log p(x) rather than computing it exactly. Normalizing flows take a different route entirely: they insist that the mapping between latent space and data space be a bijection, a one-to-one, invertible function. That single design constraint is what makes an exact, tractable likelihood possible, no bounds or approximations required.

Invertible and Differentiable: The Two Non-Negotiables

A flow is built from a transformation f that maps a latent variable z, drawn from a simple base distribution like a standard Gaussian p_Z(z), to a data point x = f(z). For this to work as a generative model with exact likelihoods, f must satisfy two properties. First, it must be invertible: given any x, there must exist a unique z = f^-1(x) that produced it, with no two different z values ever mapping to the same x. Second, both f and its inverse must be differentiable, so that the transformation's local stretching or squashing of space can be measured smoothly at every point. Together these properties mean a flow never destroys information — you can always run the transformation backward and recover exactly the latent code that generated a given sample, which is precisely what lets us reason rigorously about how probability mass moves between the two spaces.

The Change-of-Variables Formula and the Jacobian Correction

Suppose you know the density p_Z(z) of the latent variable, and you push z through the invertible map x = f(z). The resulting density over x is not simply p_Z evaluated at the corresponding z — you also need to account for how much the transformation locally stretches or compresses volume around that point. This is the change-of-variables formula: p_X(x) = p_Z(f^-1(x)) · |det(∂f^-1/∂x)|. The term |det(∂f^-1/∂x)| is the absolute value of the determinant of the Jacobian matrix of the inverse transform, and it acts as a volume-correction factor. If f expands a small neighborhood of z, probability mass gets spread over a larger region in x-space, so the density there must drop, and the Jacobian determinant captures exactly this expansion or contraction. In practice, training works in log-space for numerical stability, giving the equivalent form log p_X(x) = log p_Z(z) - log|det(∂f/∂z)|, where z = f^-1(x). This log-likelihood is exact — nothing is bounded or approximated — and it is precisely the quantity we maximize during training by gradient ascent.

Making the Jacobian Cheap: Coupling Layers and Triangular Structure

Computing a determinant of a general n×n Jacobian costs O(n^3) operations, which is hopelessly expensive for high-dimensional data like images with thousands of pixels. Flow architectures get around this with a clever trick: design each layer so its Jacobian matrix is triangular, since the determinant of a triangular matrix is just the product of its diagonal entries, an O(n) computation. The classic mechanism for this is the coupling layer, used in models like RealNVP and Glow. It splits the input into two halves, x1 and x2, leaves x1 unchanged, and transforms x2 using a function whose parameters (like a scale and shift) are computed from x1 alone: y2 = x2 · exp(s(x1)) + t(x1), y1 = x1. Because y1 depends only on x1 and y2's dependence on x1 is entirely inside s and t while its dependence on x2 is a simple elementwise affine map, the resulting Jacobian is triangular, with the diagonal made of 1s (from y1) and the exp(s(x1)) terms (from y2). This keeps every layer both easy to invert and cheap to differentiate, even though s and t themselves can be arbitrarily complex neural networks.

Stacking Layers, and Where Flows Get Used

A single coupling layer is a fairly weak transformation — it leaves half of its input completely untouched. The real expressive power of normalizing flows comes from composition: chaining many such layers together, typically alternating which half of the variables gets transformed so that every dimension eventually gets updated. Because the composition of invertible, differentiable functions is itself invertible and differentiable, and because the log-determinant of a composed Jacobian is just the sum of the log-determinants of each layer (log|det(∂f/∂z)| = Σ_k log|det(∂f_k/∂z_k)|), you can stack dozens of simple, cheap layers and still get exact likelihoods and full invertibility, while the overall transformation becomes flexible enough to model highly complex, multimodal data distributions. This combination of exact density evaluation and exact sampling makes flows valuable for tasks where you genuinely need a likelihood, such as anomaly and out-of-distribution detection, scientific density estimation, and as building blocks in variational inference, alongside their role as standalone generative models for images and audio.

Frequently asked questions

Try it live

Everything above runs in your browser — open Normalizing Flows: Exact Density Modeling via Invertible Transforms and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.