Autoregressive Models: Generating Sequences One Token at a…

Q: Is the chain rule factorization an approximation?

No. p(x_1, ..., x_n) = product_{t=1}^{n} p(x_t | x_1, ..., x_{t-1}) is an exact mathematical identity that follows from the definition of conditional probability, true for any joint distribution over any sequence. The approximation in practice comes entirely from how well the neural network p_theta approximates each true conditional p(x_t | x_<t), not from the factorization itself.

Q: Why not just predict all the tokens at once instead of one at a time?

Predicting every token simultaneously (as some non-autoregressive models attempt) sidesteps the sequential bottleneck at generation time but is a much harder learning problem, since the tokens depend on each other in complex ways that a single parallel prediction struggles to capture. The chain rule sidesteps that difficulty by breaking the joint distribution into a sequence of much simpler, well-defined conditional predictions, at the cost of needing multiple sequential forward passes to generate.

Q: What exactly does causal masking prevent?

It prevents a position's attention computation from incorporating information from any token that comes after it in the sequence. Without this mask, a Transformer trained on next-token prediction could trivially attend to the very token it's supposed to be predicting, since that token is already present in the input during training, making the training loss meaningless and the model useless for actual generation.

Q: Do RNNs and Transformers use a different mathematical objective?

No, they share the exact same training objective, maximizing sum_t log p_theta(x_t | x_<t). They differ only in how they compute that conditional distribution internally: an RNN processes tokens sequentially through a recurrent hidden state, while a causally-masked Transformer processes the whole sequence in parallel but restricts attention to earlier positions to respect the same left-to-right dependency.

Q: Why does generation feel slower than training?

During training, the entire target sequence is already known, so a causally-masked Transformer can compute all positions' losses in a single parallel forward pass. During generation, each token depends on the ones generated just before it, so the model must run a fresh forward pass for every new token, an inherently sequential process that techniques like key-value caching speed up but cannot fully eliminate.

Factorizing a Joint Distribution, Exactly

Suppose you want to model the probability of an entire sequence of tokens x_1, x_2, ..., x_n — a sentence, a line of code, a musical phrase. Modeling this joint distribution p(x_1, ..., x_n) directly seems hopeless: the number of possible sequences explodes exponentially with length, so there's no way to just tabulate every outcome. The probability chain rule rescues us, and it costs nothing in accuracy. It states that any joint distribution can be rewritten exactly as a product of conditional distributions: p(x_1, x_2, ..., x_n) = product_{t=1}^{n} p(x_t | x_1, ..., x_{t-1}). In words, the probability of the whole sequence equals the probability of the first token, times the probability of the second token given the first, times the probability of the third given the first two, and so on. This is not an approximation or a simplifying assumption — it follows directly from the definition of conditional probability, applied repeatedly. What it buys us is enormous: instead of one impossibly complex object (the joint distribution over all sequences), we only ever need to model one much simpler object, p(x_t | x_1, ..., x_{t-1}), the distribution over the next token given everything that came before. A model that is good at solving this one repeatable sub-problem is, by the chain rule, automatically a model of the entire joint distribution.

Training a Network to Predict the Next Token

An autoregressive model is simply a neural network trained to approximate p(x_t | x_1, ..., x_{t-1}) with parameters theta, written p_theta(x_t | x_<t). Early versions used recurrent neural networks (RNNs), which read tokens one at a time and compress everything seen so far into a hidden state vector, updating that summary at every step. Modern systems overwhelmingly use causal Transformers instead, which process the whole sequence in parallel using self-attention but restrict each position to only attend to itself and earlier positions. Either architecture is trained the same way: given a large corpus of real sequences, the training objective is to maximize the log-probability the model assigns to the actual next token at every position, summed over the sequence: L(theta) = sum_{t=1}^{n} log p_theta(x_t | x_1, ..., x_{t-1}). Equivalently, training minimizes the negative log-likelihood, which is exactly the cross-entropy loss familiar from classification, applied here at every single position in every sequence simultaneously. Because the chain rule guarantees this per-token objective is mathematically equivalent to modeling the full joint distribution, a model that reliably nails next-token prediction across huge amounts of diverse text is, formally, learning a compressed model of language itself.

Causal Masking: No Peeking at the Future

Transformers process an entire sequence at once rather than token by token, which is fantastic for training speed but creates a subtle danger: self-attention naturally lets every position look at every other position, including ones that come later. If left unchecked, the network could "solve" next-token prediction by simply copying the answer from a future position it can already see — a shortcut that produces near-perfect training loss but a completely useless model at generation time, since the future tokens obviously don't exist yet when you're actually writing text. Causal masking fixes this by forcing the attention mechanism at position t to only consider positions 1 through t, typically implemented by adding negative infinity to the attention scores for all future positions before the softmax, so their attention weight collapses to zero. This single constraint is what turns an otherwise bidirectional Transformer into a genuinely autoregressive one, guaranteeing that p_theta(x_t | x_1, ..., x_{t-1}) is computed using only information that would actually be available at generation time. It's a small architectural rule with an outsized consequence: it is precisely what keeps training-time next-token prediction and inference-time generation mathematically consistent with each other.

Generation: Sampling One Token, Feeding It Back In

Once trained, the model no longer sees ground-truth continuations — generation has to build the sequence from scratch, one token at a time, which is where the name "autoregressive" earns its keep: the model regresses on its own previous outputs. Starting from some prompt x_1, ..., x_k, the process repeats a simple loop: compute p_theta(x_{t} | x_1, ..., x_{t-1}) for the next position, sample (or greedily pick) a token x_t from that distribution, append it to the sequence, and feed the extended sequence back into the model to compute the distribution for x_{t+1}. This is why generation is inherently sequential and comparatively slow compared to training: each new token requires a fresh forward pass conditioned on everything generated so far. Sampling strategies vary — greedy decoding always takes the single most likely token, while temperature sampling, top-k, and nucleus (top-p) sampling introduce controlled randomness so the model doesn't produce the same bland, repetitive continuation every time. Whatever the sampling rule, the underlying probability being sampled from at every step is exactly the conditional p(x_t | x_<t) that the chain rule identified as the one thing the model ever needed to learn.

Why This Simple Idea Scales to GPT-Sized Models

What makes the chain-rule factorization so powerful in practice is that it turns an intractable modeling problem into a uniform, endlessly repeatable one: no matter how long the sequence gets, the model only ever has to answer the same question — "given everything so far, what comes next?" That uniformity means the same architecture and training recipe scale almost effortlessly from short sentences to documents spanning thousands of tokens, and from small networks to models with hundreds of billions of parameters, simply by adding more layers, more attention heads, and more training data. It also explains why next-token prediction, an objective that sounds almost too simple to matter, turns out to force a model to implicitly learn grammar, factual associations, reasoning patterns, and style: getting p(x_t | x_<t) right in general, across a vast and varied corpus, requires building rich internal representations of meaning, not just surface statistics. GPT and its successors are, at their mathematical core, exactly this: a causally-masked Transformer trained to maximize sum_t log p_theta(x_t | x_<t), which is the chain rule turned into an engineering blueprint.

Frequently asked questions

Is the chain rule factorization an approximation?

No. p(x_1, ..., x_n) = product_{t=1}^{n} p(x_t | x_1, ..., x_{t-1}) is an exact mathematical identity that follows from the definition of conditional probability, true for any joint distribution over any sequence. The approximation in practice comes entirely from how well the neural network p_theta approximates each true conditional p(x_t | x_<t), not from the factorization itself.

Why not just predict all the tokens at once instead of one at a time?

Predicting every token simultaneously (as some non-autoregressive models attempt) sidesteps the sequential bottleneck at generation time but is a much harder learning problem, since the tokens depend on each other in complex ways that a single parallel prediction struggles to capture. The chain rule sidesteps that difficulty by breaking the joint distribution into a sequence of much simpler, well-defined conditional predictions, at the cost of needing multiple sequential forward passes to generate.

What exactly does causal masking prevent?

It prevents a position's attention computation from incorporating information from any token that comes after it in the sequence. Without this mask, a Transformer trained on next-token prediction could trivially attend to the very token it's supposed to be predicting, since that token is already present in the input during training, making the training loss meaningless and the model useless for actual generation.

Do RNNs and Transformers use a different mathematical objective?

No, they share the exact same training objective, maximizing sum_t log p_theta(x_t | x_<t). They differ only in how they compute that conditional distribution internally: an RNN processes tokens sequentially through a recurrent hidden state, while a causally-masked Transformer processes the whole sequence in parallel but restricts attention to earlier positions to respect the same left-to-right dependency.

Why does generation feel slower than training?

During training, the entire target sequence is already known, so a causally-masked Transformer can compute all positions' losses in a single parallel forward pass. During generation, each token depends on the ones generated just before it, so the model must run a fresh forward pass for every new token, an inherently sequential process that techniques like key-value caching speed up but cannot fully eliminate.

Try it live

Everything above runs in your browser — open Autoregressive Models: Generating Sequences One Token at a Time via the Chain Rule and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open Autoregressive Models: Generating Sequences One Token at a Time via the Chain Rule simulation