Masked Language Modeling: How BERT Predicts Missing Words (2D)

Sentence

Model

Softmax temperature T 1.00

Readouts

Top prediction

–

Top-1 probability

–

Entropy H(p)

–

Match?

–

Top 5 candidates

Click "Predict masked word" to run inference.

How it works

Each vocabulary word w has a fixed embedding vector e_w (here, a unit vector in 2D so the whole space fits on a flat canvas — real BERT uses 768+ dimensions, the 3D version of this sim used 3). A masked position is filled in from its bidirectional context: both the word(s) before and after [MASK] are averaged into one context vector, unlike an autoregressive model that only ever looks left.

c = normalize( Σ e_context_word )
score(w) = cos(c, e_w) = c · e_w      (both unit vectors)
P(w) = softmax( score(w) / T )
     = exp(score(w)/T) / Σ_v exp(score(v)/T)

Context vector (white arrow) — the bidirectional average of the two words surrounding [MASK], drawn from the origin of the embedding circle.
Vocabulary points — every candidate word's embedding, colored by semantic cluster (animals, furniture, weather, food, verbs, adjectives), arranged around a unit circle at 60° apart — the flat, 2D analogue of the 3D version's six octahedron axes. Words trained together end up nearby, exactly like real word/subword embeddings.
Temperature T — divides the similarity scores before the softmax. Low T (sharp) makes the model commit strongly to its top guess; high T (flat) spreads probability across more candidates — the same knob real language models expose as "sampling temperature".
Re-embed vocabulary — resets the random jitter that places each word around its cluster center, showing that the exact geometry (and hence exact prediction) depends on how the embeddings were initialized/trained, not on some fixed "true" answer.

This is a deliberately simplified surrogate: real BERT builds a contextual embedding for [MASK] with many layers of self-attention over the whole sequence, not a plain average of two static vectors. What survives here is the essential mechanism — aggregate context from both directions, then score every vocabulary word against it — which is exactly what "bidirectional" means in "Bidirectional Encoder Representations from Transformers". The context-averaging, cosine-scoring and temperature-softmax math is identical to the 3D original; only the embedding space's dimensionality (3D → 2D) and the view (WebGL scene → pannable/zoomable 2D canvas) changed, so the model behaves exactly the same way.

Drag to pan · scroll or +/− to zoom