Tools like GitHub Copilot, Cursor and Code Llama generate code one token at a time. At every step a transformer decoder computes self-attention: each new token forms a query and compares it against the keys of every earlier token in its context window, turning those similarities into a weighted mix of their values.
Attention(Q,K,V) = softmax( Q·Kᵀ / √d_k ) · V
Next-token probability: P(tokenᵢ) = softmax( logitsᵢ / T )
T → 0 sharp, greedy, near-deterministic completions
T → 2 flat distribution, more exploratory / creative output
- Temperature — rescales the logits before the softmax that turns them into probabilities; the bars over the next slot show exactly this distribution across candidate token types.
- Generation speed — tokens emitted per second, same knob as a "streaming" completion in an editor.
- Heads — a real transformer runs several attention computations in parallel ("multi-head attention"), each free to focus on a different pattern (syntax, recent identifiers, matching brackets…); every colour here is one head.
- Context window — how many previous tokens the model is allowed to attend to; a wider window costs more compute (attention is O(n²) in sequence length) but keeps more of the file "in mind".
This is a simplified, illustrative model — not a running neural network — but the softmax-over-scaled-similarity mechanism and the temperature-controlled sampling step are exactly the ones inside real code-completion and debugging-agent models.