Pipeline parallelism Gantt chart (2D)

The schedule as a timeline: one row per GPU, forward and backward blocks per micro-batch, idle bubbles hatched. Click a block to follow one micro-batch through the pipeline.

Switch to 3D

Presets

Parameters

Model checks

The science

What is modelled

A neural network is cut into p consecutive stages, one per GPU. A batch is split into m micro-batches. Every micro-batch must pass forward through stages 1..p (time tf per stage) and then backward through stages p..1 (time tb). A stage can only start an operation when its input has arrived and the GPU is free. Two orderings are compared: GPipe (all forwards, then all backwards) and 1F1B (after a short warm-up each stage alternates one forward, one backward).

Equations

  • The simulator solves the dependency graph: forward (s, j) waits for forward (s-1, j); backward (s, j) waits for backward (s+1, j) (or for the last stage’s own forward) and for the previous operation on its GPU.
  • Ideal work per GPU is m (tf + tb). The fill and drain adds (p-1)(tf + tb), so the makespan is (m + p - 1)(tf + tb) for both schedules.
  • Bubble fraction (idle share of all GPU-time) = (p-1)/(m+p-1). It falls as m grows, which is why many micro-batches are used; efficiency is m/(m+p-1).
  • Memory: a micro-batch’s activations are stored from its forward until its backward ends. GPipe holds m of them on a stage; 1F1B holds at most min(p, m) (stage 1 has the most), so it trades nothing in speed for much less memory.

Checks run in this page

  • Makespan: both simulated schedules finish at exactly (m+p-1)(tf+tb).
  • Bubble: idle share counted from the simulated busy times equals (p-1)/(m+p-1).
  • Memory: the peak stored activations found by scanning the simulated events equals m (GPipe) and min(p, m) (1F1B).

Honest simplifications

  • All stages take identical time and communication between GPUs is free. Real stages are unbalanced and activation transfers, optimiser steps and gradient all-reduce add time; unbalanced stages make the bubble larger than shown.
  • No interleaved or zero-bubble schedules, no recomputation (activation checkpointing), and no data or tensor parallelism are modelled.
  • Memory is counted in whole micro-batch activations, not bytes; weights and optimiser state are ignored.

What the 2D view adds

  • The Gantt chart is the schedule solved as a dependency graph: every block starts at the later of “GPU free” and “input ready”. Hatched areas are the bubble.
  • Click a block to highlight one micro-batch on every stage; the diagonal wavefront is the fill and drain that creates the (p-1) term.

The GPU card and micro-batch sprites and the bench background are Blender 5.2 renders of the same models used in the 3D version; the schedule is computed live in your browser on one shared maths module (pp_core.js).

Loading sprites...
Stored activations on GPU 1 over time
Bubble fraction against micro-batches m
Peak stored activations against m
Made with 🟠 Blender