Masked Language Modeling: How BERT Predicts Missing Words

BERT's headline trick is the masked language modeling objective: hide a token, then predict it from everything around it — left and right — rather than only what came before, as an autoregressive model like GPT must. This simulator makes that mechanism tangible with a small, fully computed toy model: a row of 3D token blocks renders a sentence with one word masked, a point cloud below it is a real embedding space where every vocabulary word sits at a fixed 3D unit vector, and pressing Predict averages the embeddings of the words on both sides of the mask into a single context vector, scores every candidate word by cosine similarity to it, and turns those scores into a probability distribution with a temperature-controlled softmax. Live readouts track the top prediction, its probability, the distribution's entropy, and whether the model actually recovered the true masked word, while the temperature slider and a vocabulary re-embedding button let you see how both the sharpness of the guess and the underlying geometry shape the outcome.