PyTorch's Dynamic Computation Graph: Why Define-by-Run Changed Research Speed

How PyTorch's define-by-run autograd works under the hood, and why it mattered so much more for research iteration than static graph frameworks.

▶ Open the simulation

Define-and-run versus define-by-run

Early deep learning frameworks like the original TensorFlow (pre-2.0) and Theano used a define-and-run model: you first wrote a full description of the computation — every layer, every operation, every branch — as a static graph, a kind of blueprint with no data flowing through it yet. Only after that blueprint was completely assembled and compiled did you feed real data in through a session, at which point the framework executed the graph efficiently, often after optimising it. PyTorch, by contrast, uses define-by-run: there is no separate blueprint phase. Every time you run your Python model code, PyTorch builds the computation graph as the operations actually execute, tensor by tensor, using real data, and throws that graph away once backpropagation is done, ready to build a fresh one on the very next call.

This sounds like a purely technical implementation detail, but it changes what kind of model you can write with ordinary Python control flow. In a static graph framework, an if-statement or a for-loop whose length depends on the input data has to be encoded as a special graph operation (TensorFlow's tf.cond and tf.while_loop, for instance) because the graph has to be fully determined before any data arrives. In PyTorch, you just write a normal Python if-statement or for-loop, and because the graph is built fresh on every execution, the graph itself can legitimately be different on different inputs — a variable-length sentence, a tree-structured input, a recurrent network that runs for a different number of steps depending on when it decides to stop. The computation graph literally follows the shape of the current example through the code.

What the graph actually looks like and how autograd walks it

Picture the graph as a directed acyclic structure where each node is a tensor and each edge represents an operation that produced it from its inputs. When you write y = w @ x + b, PyTorch's autograd engine doesn't just compute y numerically — every tensor that requires gradients (has requires_grad=True) silently accumulates a grad_fn attribute recording which operation created it and what its inputs were. So y's grad_fn points back to the addition, which points back to the matrix multiplication, which points back to w and x, forming a chain all the way back to your original inputs and parameters. This is the dynamic graph: an audit trail of exactly what happened, built as a side effect of ordinary computation.

Calling .backward() on a scalar loss at the end triggers a reverse traversal of that exact chain. Starting from the loss, the derivative of the loss with respect to itself is trivially 1, and autograd walks backwards through each grad_fn, applying the chain rule at every node: the local derivative of that specific operation (a matrix multiplication has a well-known derivative rule, as does addition, as does a sigmoid) is multiplied by the gradient flowing in from downstream, and the result is passed further upstream. Picture it as a wave of gradient signal spreading backwards through the exact same wiring the forward pass used, depositing a numeric gradient value into the .grad attribute of every leaf tensor (typically your model's weights) it reaches. Once .backward() completes, the temporary graph structure — all those grad_fn links — is discarded, which is why by default you can't call .backward() twice on the same forward pass without explicitly asking PyTorch to retain it.

Why this mattered so much for research velocity specifically

Debugging a static graph model in the 2015-2017 era meant debugging a compiled artefact: if a tensor had the wrong shape three layers deep, the error surfaced as an opaque graph-construction failure, often disconnected from the specific line of Python that logically caused it, and you had no way to simply insert a print statement or drop into a debugger mid-computation because the computation hadn't happened yet — you were describing a computation, not performing one. PyTorch's define-by-run model means the forward pass is just regular, eager Python execution: you can set a breakpoint anywhere inside your model's forward method, inspect the actual numeric values and shapes of any intermediate tensor at that exact point, and step through line by line with a standard debugger, because every operation runs immediately when the interpreter reaches it, exactly like any other Python code.

For research specifically, where the actual bottleneck is trying an architectural idea, discovering it's subtly wrong, and iterating to the next idea, this collapsed the debug cycle from potentially hours (reasoning about a graph compilation error) to minutes (reading a normal Python stack trace pointing at the exact failing line, with real tensor values visible). It also made architectures with genuinely data-dependent control flow — recursive tree networks that branch differently for every parse tree, or transformers with variable-length sequences and dynamic attention masking, or reinforcement learning loops where the number of steps depends on when an episode terminates — dramatically easier to express, because you write the branching logic once in plain Python rather than encoding it as special-purpose static graph operators. This is a large part of why PyTorch became the dominant framework in academic research within a few years of its 2017 release, even though TensorFlow had a significant head start and, at the time, better production deployment tooling.

The trade-off, and why it narrowed over time

The historical cost of define-by-run was runtime performance and deployment: because the graph is rebuilt from scratch on every forward pass using the Python interpreter, there was no opportunity for the framework to do the kind of whole-graph optimisation a static, compiled graph allows — operator fusion (combining several small operations into one larger, faster kernel), dead-code elimination, or memory planning across the entire computation ahead of time. Static graphs, once compiled, could in principle run with the Python interpreter almost entirely out of the loop, which mattered enormously for production inference speed and for exporting models to run on mobile devices or specialised hardware without shipping a Python runtime alongside them.

Both major frameworks converged on a middle path in response. TensorFlow 2.0 adopted eager execution as its default in 2019, essentially adopting PyTorch's debugging-friendly model, while offering @tf.function decorators to trace and compile eager code into an optimised static graph when you're ready to deploy. PyTorch, in turn, introduced TorchScript and later torch.compile, which trace or otherwise capture a dynamic PyTorch model into a graph representation that can be optimised and compiled ahead of time for production. The practical result today is that the framework you write your research code in and the representation your model is actually deployed as are no longer forced to be the same thing — you get eager-mode iteration speed while developing and something closer to static-graph efficiency once you compile for production, which is roughly the best of both worlds that the field spent the better part of a decade converging towards.

Frequently Asked Questions

What does 'define-by-run' literally mean?

It means the computation graph is constructed as a side effect of actually executing your model's forward pass with real tensors, rather than being fully specified in advance before any data flows through it, as in older static-graph frameworks.

Why does PyTorch's autograd need to store a grad_fn on each tensor?

grad_fn records which operation produced that tensor and from what inputs, forming the chain that backward() walks in reverse to apply the chain rule and compute gradients for every parameter that fed into the final loss.

Does the dynamic graph get reused across training steps?

No, by default a fresh graph is built on every forward pass and discarded after the corresponding backward pass, which is why calling backward() twice on the same forward pass raises an error unless you explicitly set retain_graph=True.

Is define-by-run slower than a static graph?

Historically yes for raw execution speed, because a static graph can be globally optimised ahead of time; modern PyTorch narrows this gap significantly with tools like torch.compile, which trace a dynamic model and compile it into an optimised graph for deployment while keeping eager execution for development.