Home▸Machine Learning & Neural Networks▸Autoregressive Models: Generating Sequences One Token at a Time

📜 Autoregressive Models: Generating Sequences One Token at a Time via the Chain Rule

Discover how autoregressive models like GPT factorize the joint probability of a whole sequence into a product of simple next-token conditional probabilities using the probability chain rule, and how that idea powers modern language generation.

Machine Learning & Neural Networks3DModerate60 FPS

The simulation visualizes a causal Transformer generating a sequence step by step in 3D, showing the causal attention mask blocking connections to future positions and the model's output distribution over the next token being sampled and fed back in as input.

🔬 What It Demonstrates

🎮 How to Use

Type or select a starting prompt, choose a sampling strategy (greedy, temperature, or top-k), and press play to watch the model compute p(x_t | x_<t) at each step, sample a token, and append it before repeating for the next position.

💡 Did You Know?

The probability chain rule behind autoregressive models predates deep learning by centuries — it's the same basic rule of conditional probability used in classical statistics, and GPT-style language models are essentially this rule scaled up with a causally-masked neural network and trillions of tokens of training text.

⚙ Under the hood

Watch a causal-masked model generate tokens left to right, conditioning only on past positions as the chain-rule factorization comes to life.

autoregressive modelchain rulecausal maskinglanguage modelsequence generationmachine-learning

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install