GANs, VAEs and Diffusion Models: How Each One Actually Generates a Sample

A practical, mechanism-level comparison of the three major generative model families and why diffusion models came to dominate image generation.

▶ Open the simulation

GANs: two networks locked in an adversarial game

A Generative Adversarial Network is built from two neural networks with opposing objectives, trained simultaneously. The generator takes a vector of random noise as input and transforms it, through a stack of learned upsampling layers, into a full image in a single forward pass. The discriminator is a separate classifier network that looks at an image — sometimes a real one from the training set, sometimes one the generator just produced — and tries to predict which it is. Training alternates: the discriminator is updated to get better at telling real from fake, and the generator is updated to produce images that fool the discriminator's current judgement, and because both networks improve together, the generator is chasing a moving target the entire time, which is what makes GAN training notoriously unstable — if the discriminator gets too good too fast, its gradient signal to the generator collapses to almost nothing useful (it can already tell every fake apart perfectly, so there's no informative direction to push the generator in), and if it lags behind, the generator can exploit a specific weakness and collapse to producing a narrow set of images that fool that particular weakness, a failure mode called mode collapse where diversity collapses even as individual samples look plausible.

The structural payoff for surviving that instability is speed: once trained, generating a sample from a GAN is a single forward pass through the generator network, taking milliseconds, which made GANs the practical choice throughout the mid-to-late 2010s for anything needing fast, high-volume generation, including real-time applications.

VAEs: compressing into a smooth latent space and decoding back out

A Variational Autoencoder is built from an encoder network that compresses an input image down into a low-dimensional latent representation, and a decoder network that reconstructs an image back out of that representation, trained together to make the reconstruction match the original input as closely as possible. The variational part is the key structural difference from an ordinary autoencoder: rather than encoding an image to a single fixed point in latent space, the encoder outputs the parameters (a mean and a variance) of a probability distribution, and a point is sampled from that distribution before being passed to the decoder — and the training loss explicitly penalises those distributions for straying too far from a standard normal distribution, using a term called the KL divergence.

That regularisation term is what makes VAEs actually useful for generation rather than just compression: it forces the latent space to be smooth and continuous, without gaps or isolated clusters, so that any point you sample from a simple standard normal distribution at generation time — not just points corresponding to real training images — decodes into something that looks like a plausible image, because nearby points in that space have been trained to decode into visually similar outputs. The trade-off is that the KL regularisation term, balanced against the reconstruction quality term, tends to produce outputs that are noticeably blurrier and less sharp than GAN or diffusion samples, because the model is explicitly incentivised to hedge across a range of plausible reconstructions rather than commit confidently to one high-frequency-detail answer, which softens fine texture and edges in the average case.

Diffusion models: denoising your way from static to structure

A diffusion model is trained on a completely different idea: take a real image and progressively destroy it by adding a small amount of Gaussian noise over many steps — typically hundreds to a thousand — until, at the final step, the image is statistically indistinguishable from pure random noise. That forward noising process needs no learning at all; it's a fixed, known mathematical procedure. What the network learns is the reverse: given a noisy image at some step t, predict either the noise that was added or the slightly-less-noisy version at step t-1. Once trained on millions of examples of this noise-prediction task across every step of the process, generation works by starting from pure random noise and repeatedly applying the trained denoising network, step by step, each pass removing a little more noise and revealing a little more coherent structure, until after all the steps a full, sharp image has emerged from what began as static — visually, it looks like an image slowly resolving out of television snow, guided at every step by the model's learned prediction of what noise to subtract next.

The immediate cost of this approach is obvious from the description: generating one sample requires running the network many times sequentially (though modern samplers like DDIM have cut the practically required steps from around a thousand down to tens, and distillation techniques push it lower still), making diffusion models substantially slower to sample from than a single-pass GAN. The payoff that made this trade-off worth it is training stability and sample quality: there's no adversarial game to destabilise, just a well-behaved supervised regression problem (predict the noise) at every step, which converges far more reliably than adversarial training, and empirically produces sharper, more diverse, higher-fidelity images than either GANs or VAEs achieve at comparable model scale, particularly on the fine-grained texture and coherent global structure that made GAN mode collapse and VAE blurriness so visible by comparison.

Why diffusion won for images, and what the other two still do better

Diffusion models became the dominant approach for large-scale text-to-image generation (DALL-E 2 onward, Stable Diffusion, Midjourney's underlying approach) primarily because training stability at scale matters enormously once you're spending the compute budget of a major lab: a GAN that mode-collapses two-thirds of the way through a multi-million-dollar training run is a catastrophic, hard-to-diagnose failure, while a diffusion model's simple denoising objective degrades gracefully and predictably as you scale it up, making it a far safer bet for the kind of massive, expensive training runs that produce state-of-the-art image quality. The conditioning mechanism that lets diffusion models follow a text prompt — injecting a text embedding into the denoising network at every step so each denoising decision is guided by what the prompt describes — also composes naturally with the iterative, many-step generation process in a way that's proven easier to control precisely than steering a GAN's single-pass generation.

None of this makes GANs or VAEs obsolete for their own use cases. GANs remain attractive wherever real-time, single-pass generation speed genuinely matters and adversarial instability can be managed with a smaller, well-understood problem domain (certain style-transfer and super-resolution applications, for instance). VAEs remain valuable specifically because of their smooth, interpretable latent space — being able to interpolate meaningfully between two points in latent space, or do arithmetic on latent vectors to combine attributes, is a genuinely useful property that VAEs give more directly than diffusion models do, and VAE-style encoders are in fact used as a compression component inside modern latent diffusion models (Stable Diffusion runs its slow diffusion process not on raw pixels but on a much smaller VAE-compressed latent representation, then decodes back to pixels at the end), which is a good illustration of how these three families increasingly get combined rather than treated as mutually exclusive choices.

Frequently Asked Questions

Why is GAN training unstable while diffusion model training is not?

GANs train two networks with opposing objectives simultaneously, so progress by one can destabilise or stall the other, causing failures like mode collapse; diffusion models instead solve a straightforward supervised regression problem (predicting noise) with no adversarial component, which converges far more predictably.

Why do VAE-generated images look blurrier than GAN or diffusion images?

The VAE training objective includes a regularisation term that keeps the latent space smooth and forces the model to hedge across plausible reconstructions rather than commit to one sharp, high-frequency-detail answer, which softens fine texture and edges in the resulting images.

Why are diffusion models slower to generate a sample than a GAN?

A GAN generates an image in a single forward pass through its generator network, while a diffusion model must run its denoising network repeatedly across many sequential steps to gradually turn random noise into a coherent image, though modern samplers have reduced the required steps considerably.

Does Stable Diffusion actually run entirely on pixels?

No. It uses a VAE-style encoder to compress the image into a much smaller latent representation first, runs the slow diffusion denoising process on that compact latent space rather than full-resolution pixels, and then uses a VAE decoder to reconstruct the final pixel image, combining both techniques for speed and quality.