Operator Fusion & the Roofline Model

Edge-inference runtimes like TensorFlow Lite, Core ML and ONNX Runtime spend as much effort fusing operators as they do quantizing weights — because on battery-powered hardware, DRAM traffic often costs more time and energy than the arithmetic itself. This lab renders a five-stage Conv-BatchNorm-ReLU-Conv-Pool pipeline as a 3D scene: data packets travel up from a memory plane into each operator and back down again, unless that operator has been fused into its neighbor, in which case the packet stays in flight and never touches memory. A live roofline plot — the standard performance-modeling tool for exactly this problem — tracks arithmetic intensity and achieved throughput as you toggle fusion, switch precision (FP32/FP16/INT8) and swap between a mobile CPU, mobile GPU and edge NPU, showing numerically why fusion is worth more on memory-bound stages than compute-bound ones.