Home▸AI & Machine Learning▸2D Attention Cost: Sliding-Window, Strided Sparse and Linear-Kernel Attention Against Full Softmax in Error, Flops and Memory

🧠 2D Attention Cost: Sliding-Window, Strided Sparse and Linear-Kernel Attention Against Full Softmax in Error, Flops and Memory

Heat-map of one causal attention head on a seeded synthetic sequence: full softmax versus sliding window with sink tokens, strided sparse and linear-kernel attention, with counted multiply-adds, error against full attention, cost curves up to 130,000 tokens and six independent numerical checks.

AI & Machine Learning2DAdvanced60 FPS📱 Mobile-adapted🟠 Blender⇄ 3D version
2D Attention Cost: Sliding-Window, Strided Sparse and Linear-Kernel Attention Against Full Softmax in Error, Flops and Memory ↗ Open standalone
⚙ Under the hood

Heat-map of one causal attention head on a seeded synthetic sequence: full softmax versus sliding window with sink tokens, strided sparse and linear-kernel attention, with counted multiply-adds, error against full attention, cost curves up to 130,000 tokens and six independent numerical checks.

Sparse AttentionLinear AttentionSliding WindowTransformerAttention SinkComputational CostLong ContextMachine Learning

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install

What did you find?

Add reproduction steps (optional)