The simulation visualizes a causal Transformer generating a sequence step by step in 3D, showing the causal attention mask blocking connections to future positions and the model's output distribution over the next token being sampled and fed back in as input.
Type or select a starting prompt, choose a sampling strategy (greedy, temperature, or top-k), and press play to watch the model compute p(x_t | x_<t) at each step, sample a token, and append it before repeating for the next position.
Prompt input, sampling strategy select (greedy/temperature/top-k), generation speed slider, play/pause, reset sequence
The probability chain rule behind autoregressive models predates deep learning by centuries — it's the same basic rule of conditional probability used in classical statistics, and GPT-style language models are essentially this rule scaled up with a causally-masked neural network and trillions of tokens of training text.
The simulation visualizes a causal Transformer generating a sequence step by step in 3D, showing the causal attention mask blocking connections to future positions and the model's output distribution over the next token being sampled and fed back in as input.
The simulation visualizes a causal Transformer generating a sequence step by step in 3D, showing the causal attention mask blocking connections to future positions and the model's output distribution over the next token being sampled and fed back in as input.
Type or select a starting prompt, choose a sampling strategy (greedy, temperature, or top-k), and press play to watch the model compute p(x_t | x_<t) at each step, sample a token, and append it before repeating for the next position.
The probability chain rule behind autoregressive models predates deep learning by centuries — it's the same basic rule of conditional probability used in classical statistics, and GPT-style language models are essentially this rule scaled up with a causally-masked neural network and trillions of tokens of training text.