Masked Language Modeling: How BERT Predicts Missing Words (2D)

BERT's headline trick is the masked language modeling objective: hide a token, then predict it from everything around it — left and right — rather than only what came before, as an autoregressive model like GPT must. This 2D rebuild makes that mechanism tangible with the same fully computed toy model as the 3D original: a sentence with one word masked sits above a pannable, zoomable embedding circle where every vocabulary word occupies a fixed 2D unit vector, and pressing Predict averages the embeddings of the words on both sides of the mask into a single context vector, scores every candidate word by cosine similarity to it, and turns those scores into a probability distribution with a temperature-controlled softmax. Live readouts track the top prediction, its probability, the distribution's entropy, and whether the model actually recovered the true masked word, while the temperature slider and a vocabulary re-embedding button let you see how both the sharpness of the guess and the underlying geometry shape the outcome. Drag or scroll on the embedding view to inspect any cluster up close.