Spacetime Patch Tokenizer: How Sora Turns Video Into Transformer Tokens
Interactive 3D visualizer of Sora-style spacetime-patch tokenization: slice a compressed video latent into space x space x time patch cuboids, watch the token count and quadratic self-attention cost explode as patch size shrinks, and compare full spacetime attention against per-frame windowed attention.
Before a video Diffusion Transformer can denoise a clip, the clip has to become a sequence of tokens — and Sora's key architectural idea is to patchify a compressed video latent in space and time at once, producing "spacetime patches" instead of per-frame image patches. This simulator renders that latent volume as a real 3D grid of cuboids: change the latent resolution, clip length, and spatial/temporal patch size to watch the token count N and the O(N²) self-attention cost react live, blend patches toward Gaussian noise to see the tensor a DiT actually denoises at each diffusion timestep, and click any patch to compare full spacetime self-attention (every token sees every other token, the source of temporal coherence) against a cheaper per-frame windowed variant that never looks across time.
Interactive 3D visualizer of video-diffusion-transformer-style spacetime-patch tokenization: slice a compressed video latent into space×space×time patch cuboids, watch total tokens and the quadratic self-attention cost react live, and compare full spacetime attention against cheaper per-frame windowed attention.
3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install