🙈 3D Masked-Word Prediction: Bidirectional versus Causal Attention Masks against the Exact Bayes Limit on a Hidden-Theme Cloze Task
Two Blender-built decks train the same single-head attention model with a bidirectional (BERT-style) and a causal (GPT-style) mask to guess a hidden word in template sentences, with attention pillars, candidate-word bars, and the exact Bayes predictor as the limit that no model can beat.
⚙ Under the hood
Two Blender-built decks train the same single-head attention model with a bidirectional (BERT-style) and a causal (GPT-style) mask to guess a hidden word in template sentences, with attention pillars, candidate-word bars, and the exact Bayes predictor as the limit that no model can beat.
Masked Language ModelSelf-AttentionCausal MaskBidirectional AttentionBERTGPTBayes OptimalNatural Language Processing
3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install