Each token starts as a small feature vector h⁰. A self-attention layer refines every token's vector by mixing in every other token's vector, weighted by how relevant they are:
Attention(Q,K,V) = softmax( QKᵀ / √dₖ · (1/τ) ) V
Q = h·Wq K = h·Wk V = h·Wv
h(l+1) = h(l) + Attention(Q,K,V)
Stacking this L times (the layers slider) lets information travel further: layer 1 mixes only direct neighbours' signal, layer 2 already carries second-hand context, and so on — exactly how a real transformer encoder deepens its view of the sentence.
- Bidirectional (BERT-style) — the masked token's query can attend to every other position, before and after it. This is literally BERT's Masked-Language-Model pretraining objective: 15% of tokens are replaced with [MASK] and the encoder must reconstruct them using the full sentence.
- Causal only (GPT-style) — attention scores to any position after the query are forced to −∞ before the softmax, so the mask can only see words to its left. This is the causal mask that makes GPT-style decoders autoregressive: at generation time the model genuinely cannot see the future, so this toggle reproduces that constraint exactly, not just visually.
- Temperature τ — divides the attention logits before softmax; low τ sharpens attention onto the single best-matching token, high τ spreads it almost uniformly across the sentence.
- Prediction bars — after L layers the mask token's final vector is compared (cosine similarity, softmaxed) against a handful of candidate words' embeddings; the closest match wins. Watch how removing right-side context (Causal) starves the model of exactly the clue that made the bidirectional guess easy — the same reason BERT beats a causal encoder at fill-in-the-blank tasks and why GPT is instead built to generate left-to-right.
Real-world relevance: this simplified single-head attention is the same core operation (scaled dot-product attention, residual stacking, causal vs bidirectional masking) inside every production transformer — BERT, GPT, T5 and their descendants — just with one attention head and a 6-dimensional toy embedding instead of thousands.