🧠 2D Attention Cost: Sliding-Window, Strided Sparse and Linear-Kernel Attention Against Full Softmax in Error, Flops and Memory
Heat-map of one causal attention head on a seeded synthetic sequence: full softmax versus sliding window with sink tokens, strided sparse and linear-kernel attention, with counted multiply-adds, error against full attention, cost curves up to 130,000 tokens and six independent numerical checks.
⚙ Under the hood
Heat-map of one causal attention head on a seeded synthetic sequence: full softmax versus sliding window with sink tokens, strided sparse and linear-kernel attention, with counted multiply-adds, error against full attention, cost curves up to 130,000 tokens and six independent numerical checks.
Sparse AttentionLinear AttentionSliding WindowTransformerAttention SinkComputational CostLong ContextMachine Learning
2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install