Parameters
Camera
Quality
Model checks
The science
What is modelled
A neural network is cut into p consecutive stages, one per GPU. A batch is split into m micro-batches. Every micro-batch must pass forward through stages 1..p (time tf per stage) and then backward through stages p..1 (time tb). A stage can only start an operation when its input has arrived and the GPU is free. Two orderings are compared: GPipe (all forwards, then all backwards) and 1F1B (after a short warm-up each stage alternates one forward, one backward).
Equations
- The simulator solves the dependency graph: forward
(s, j)waits for forward(s-1, j); backward(s, j)waits for backward(s+1, j)(or for the last stage’s own forward) and for the previous operation on its GPU. - Ideal work per GPU is
m (tf + tb). The fill and drain adds(p-1)(tf + tb), so the makespan is(m + p - 1)(tf + tb)for both schedules. - Bubble fraction (idle share of all GPU-time)
= (p-1)/(m+p-1). It falls asmgrows, which is why many micro-batches are used; efficiency ism/(m+p-1). - Memory: a micro-batch’s activations are stored from its forward until its backward ends. GPipe holds
mof them on a stage; 1F1B holds at mostmin(p, m)(stage 1 has the most), so it trades nothing in speed for much less memory.
Checks run in this page
- Makespan: both simulated schedules finish at exactly
(m+p-1)(tf+tb). - Bubble: idle share counted from the simulated busy times equals
(p-1)/(m+p-1). - Memory: the peak stored activations found by scanning the simulated events equals
m(GPipe) andmin(p, m)(1F1B).
Honest simplifications
- All stages take identical time and communication between GPUs is free. Real stages are unbalanced and activation transfers, optimiser steps and gradient all-reduce add time; unbalanced stages make the bubble larger than shown.
- No interleaved or zero-bubble schedules, no recomputation (activation checkpointing), and no data or tensor parallelism are modelled.
- Memory is counted in whole micro-batch activations, not bytes; weights and optimiser state are ignored.
The GPU cards, bench, micro-batch cubes and baked stone texture are Blender-built; the schedule is computed live in your browser.