Before PyTorch popularised define-by-run, most deep learning
frameworks used a define-and-run (static) model: you first
declared the entire computation graph symbolically, compiled it once, and then
fed data through the frozen structure. PyTorch instead builds the graph fresh
every single forward pass, using ordinary Python control flow — an if
statement or a for loop can change which operations run and how many
times, and autograd simply records whatever actually executed.
torch.jit.trace or a compiled TensorFlow 1.x graph when live inputs no longer match the traced shape or branch.PyTorch's define-by-run design (introduced 2016) let researchers set breakpoints and inspect tensors mid-model like ordinary Python, which is widely credited with accelerating research iteration speed enough that TensorFlow itself adopted eager execution by default in TF2 (2019).
A 3D computation graph that tears itself down and rebuilds each forward pass — following live input values and sequence length through Python-style control flow — set against a frozen static graph that silently goes stale when its inputs change underneath it.
In dynamic mode, changing the input value or sequence length rebuilds the graph instantly, mirroring PyTorch's define-by-run execution. In static mode, the graph is traced once and frozen, so later parameter changes no longer affect which path the data actually takes — until you recompile.
Move the input slider past zero to flip the branch, change the sequence length to unroll more recurrent cells, then switch between Dynamic and Static mode to see the graph respond immediately or fall stale. Try autoplay to watch dynamic mode reshape itself batch after batch.
PyTorch's eager, define-by-run design let researchers debug models with ordinary Python tools — print statements, breakpoints, pdb — instead of a symbolic graph compiler, a productivity gain TensorFlow eventually matched by making eager execution the default in TF2.