📚 2D Lost in the Middle: Softmax Attention with Primacy and Recency Bias, Needle Recall and Context Length
Attention heights over a long context with a needle at a chosen depth: primacy and recency bias, match logit and distractor noise give the U-shaped recall curve, the dependence on context length and sampled retrieval trials, with six independent numerical checks.
⚙ Under the hood
Attention heights over a long context with a needle at a chosen depth: primacy and recency bias, match logit and distractor noise give the U-shaped recall curve, the dependence on context length and sampled retrieval trials, with six independent numerical checks.
Lost in the MiddleLong ContextSoftmax AttentionPositional BiasNeedle in a HaystackRetrievalContext WindowLarge Language Models
2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install