Sliding-Window Attention & Segment Recurrence (2D)

Full transformer self-attention costs O(N²), which is why models built to read long documents or long chat histories don't attend across the whole sequence at once. This simulator lays a token sequence out as a 2D tape and processes it in fixed-size windows, caching a small recurrent memory from each segment before advancing to the next — the same segment-recurrence idea behind Transformer-XL and the sliding-window/chunking strategies long-context retrieval pipelines rely on. Tune the sequence length, window size and memory size, then play through the sequence and watch the current window (blue) and its cached memory (orange) light up with attention lines drawn between them, while a live cost chart tracks windowed attention pairs segment-by-segment against the flat O(N²) cost of full attention. Drag the tape to pan and scroll to zoom when the sequence is wider than the view.