Sora and other video Diffusion Transformers (DiT) don't feed raw pixels to the transformer. A video is first compressed by a VAE into a smaller latent tensor of shape (T, H, W, C), then cut into 3D "spacetime patches" — small cuboids spanning both a patch of space and a span of time. Each cuboid becomes one token, exactly like an image patch becomes one token in a Vision Transformer:
tokens along height = H / p_s
tokens along width = W / p_s
tokens along time = T / p_t
total tokens N = (H/p_s) · (W/p_s) · (T/p_t)
Self-attention compares every token to every other token, so its compute and memory cost scales quadratically with the sequence length:
attention cost ∝ N² · d (d = embedding width)
Shrinking the patch size (more, smaller cuboids) gives the model finer spatial/temporal detail to work with, but N grows and the N² attention bill grows even faster — this trade-off is exactly why patch size is one of the first knobs a video-DiT team tunes for a given compute budget.
- Grid / Frames — set the size of the compressed latent video before patchifying.
- ps / pt — the spatial and temporal patch size; watch total tokens N and the N² attention-pair count react live.
- Noise σ — patch color blends toward random static, standing in for the Gaussian noise a diffusion model adds at an earlier timestep; the transformer's job is to predict and remove it, patch by patch.
- Full vs. Windowed attention — click a patch to draw its attention edges. "Full" connects it to every patch in the whole clip (what Sora's global spacetime attention does, giving true temporal coherence). "Windowed" only connects it to patches in its own frame — cheaper, but frames drift out of sync because no token ever looks across time.
Real-world relevance: this patch-count/attention-cost trade-off is the main reason video generators like Sora, Veo and Movie Gen use compressed latents and moderate patch sizes rather than patchifying raw pixels — full-resolution spacetime attention on raw video would be computationally impossible at today's scale.