Sora and other video Diffusion Transformers (DiT) don't feed raw pixels to the transformer. A video is first compressed by a VAE into a smaller latent tensor of shape (T, H, W, C), then cut into 3D "spacetime patches" — small cuboids spanning both a patch of space and a span of time. Each cuboid becomes one token, exactly like an image patch becomes one token in a Vision Transformer:
tokens along height = H / p_s
tokens along width = W / p_s
tokens along time = T / p_t
total tokens N = (H/p_s) · (W/p_s) · (T/p_t)
Self-attention compares every token to every other token, so its compute and memory cost scales quadratically with the sequence length:
attention cost ∝ N² · d (d = embedding width)
Shrinking the patch size (more, smaller cuboids) gives the model finer spatial/temporal detail, but N grows and the N² attention bill grows even faster. Algebraically, N²/(per-frame windowed cost) always reduces to exactly the number of time-groups t — this simulator verifies that identity live in the chart panel every time you move a slider.
- Grid / Frames — set the size of the compressed latent video before patchifying.
- ps / pt — the spatial and temporal patch size; watch total tokens N and the N² attention-pair count react live in the cost chart.
- Noise σ — patch color blends toward random static, standing in for the Gaussian noise a diffusion model adds at an earlier timestep.
- Full vs. Windowed attention — click a patch in the top-down frame view to draw its attention edges. "Full" connects it to every patch in the whole clip; "Windowed" only connects it to patches in its own time-group.
- Isometric view — drag the top canvas to rotate the projected patch volume by hand (an independently-computed 3D→2D projection, no WebGL), scroll/zoom-slider to scale, and use the time-group slider to pick which frame the bottom-left panel dissects in full 2D.
Real-world relevance: this patch-count/attention-cost trade-off is the main reason video generators like Sora, Veo and Movie Gen use compressed latents and moderate patch sizes rather than patchifying raw pixels — full-resolution spacetime attention on raw video would be computationally impossible at today's scale.