BERT's Blindfold: Bidirectional vs Causal Masked-Word Prediction

This simulator runs a small, real multi-layer self-attention engine over a sentence with one word hidden behind a [MASK] token, then asks it to guess the missing word — exactly the masked-language-model objective BERT is pretrained on. A toggle swaps the encoder between full bidirectional attention (every token can see every other token, left and right) and a causal mask that blocks the mask position from seeing anything to its right, the same constraint that forces GPT-style decoders to generate strictly left-to-right. Stacked attention layers, softmax temperature and live entropy/coverage readouts show, token by token, exactly which words the model is allowed to lean on before it commits to a guess — and why hiding the right-hand context can turn an easy fill-in-the-blank into a coin flip.