TensorFlow and Keras: From Static Graphs to Production-Grade Deep Learning

The graph-and-session era, and why it existed

TensorFlow 1.x, released in 2015, forced you to build a static computation graph — every layer, loss and optimiser step described symbolically — and then open a tf.Session to actually push data through it and get numbers out. This two-phase design (build the graph, then run it) was not an accident of engineering taste; it existed because Google needed to train and serve models across large distributed clusters of CPUs, GPUs and its own custom TPU chips, and a fully specified graph is exactly the artefact a distributed scheduler needs: it can be partitioned across devices, operations can be fused together for speed, memory can be planned ahead of time rather than allocated reactively, and the whole thing can be serialised to a protocol-buffer file and shipped to a serving environment that has no Python interpreter at all. The cost, as covered in detail elsewhere, was a genuinely painful debugging experience, because you were debugging a graph-construction bug rather than stepping through a running computation.

Keras began as a separate, higher-level library sitting on top of TensorFlow (and, for a period, also Theano and CNTK), offering a much friendlier layer-stacking API — model.add(Dense(64)) rather than manually wiring tensors together — that hid most of the raw graph-and-session mechanics from the user. Its popularity was itself a signal: the ergonomics of the underlying framework mattered enormously to adoption, and by TensorFlow 2.0 in 2019, Google folded Keras in as the framework's official high-level API rather than continuing to compete with it.

Eager execution and the tf.function compromise

TensorFlow 2.0's biggest architectural change was making eager execution the default: operations now run immediately, tensor by tensor, exactly like NumPy or PyTorch, with real values you can print and inspect at any point. This closed the debugging gap that had driven so many researchers to PyTorch in the intervening years. But Google didn't want to give up the deployment and performance advantages of a static graph, so TensorFlow 2.x offers the tf.function decorator as a deliberate bridge: wrap a Python function in @tf.function and, the first time it's called, TensorFlow traces the operations it performs and compiles them into a static graph behind the scenes (using a technique called AutoGraph to convert Python control flow like if and while into graph-native operations), which then runs with the same distributed scheduling and optimisation benefits the old graph-and-session model had. Subsequent calls with the same input signature reuse the compiled graph rather than re-tracing, giving most of the speed of a static graph with the development experience of eager mode for the parts of the workflow where debugging matters most.

What a production model needs that a notebook doesn't

A model that scores well on a held-out test set in a Jupyter notebook is a research artefact, not a production system, and the gap between the two is where most of the actual engineering effort in an ML team goes. The first requirement is a stable, versioned serialisation format: TensorFlow's SavedModel format bundles the computation graph, the trained weights, and metadata about input and output signatures into a single directory that can be loaded by a completely different process — a serving container written in C++ for latency, a mobile app via TensorFlow Lite, a browser via TensorFlow.js — without needing the original Python training code at all. This decoupling between how a model is trained and how it's served is the single most important production requirement that a notebook workflow doesn't naturally provide.

The second requirement is a serving layer that can handle real traffic characteristics that training never has to deal with: batching requests that arrive at different times into efficient GPU batches without making any individual request wait too long, running multiple model versions side by side so a new model can be A/B tested or gradually rolled out against the currently live one, and exposing metrics (request latency percentiles, error rates, prediction distribution drift) that an on-call engineer can actually monitor. TensorFlow Serving is purpose-built for exactly this, and it deliberately does none of the training-time conveniences (no Python callback system, no easy plotting) because production serving optimises for a completely different set of constraints: predictable low latency, high throughput, and operational observability, not iteration speed.

Quantisation, pruning and the hardware-shaped model

Production deployment frequently requires shrinking a model rather than just packaging it as-is, and TensorFlow's tooling for this illustrates a broader truth about production deep learning: the model that trains best is rarely the model you actually want to run at inference time on constrained hardware. Post-training quantisation converts a model's 32-bit floating point weights down to 8-bit integers, which shrinks the model file by roughly 4x and can meaningfully speed up inference on hardware with efficient integer arithmetic, at the cost of a small, usually acceptable accuracy drop caused by the coarser numeric representation losing some precision in the weight values. Pruning goes further, identifying and zeroing out connections whose weights are close enough to zero that removing them barely changes the model's output, then using sparse matrix representations to skip the resulting empty computation entirely.

Both techniques exist because the deployment target is often not a data-centre GPU with abundant memory and power but a phone, a car's onboard computer, or an embedded sensor with a power and memory budget measured in milliwatts and megabytes, and a model trained without any regard for that target will simply not fit. This is a genuinely different optimisation problem from the one solved during training — during training you're searching for weights that minimise a loss function; during deployment optimisation you're searching for a version of those weights, or a restructured version of the architecture itself, that preserves most of the accuracy while fitting a hardware budget that has nothing to do with the loss function at all.