Operator Fusion & the Roofline Model (2D)

Kernel Fusion

Precision

Edge Hardware

Tensor size 1.0M elements

Live Readouts

Memory traffic

–

Arithmetic intensity

–

Achieved throughput

–

Latency

–

Roofline verdict

–

Speedup from fusion (this config)

–

How it works

An inference pipeline is a chain of operators — here Conv → BatchNorm → ReLU → Conv → Pool. Every operator that runs on its own must read its input tensor from DRAM and write its output back before the next operator starts. Operator fusion — what TensorFlow Lite, Core ML and ONNX Runtime's graph optimizers all do — merges a chain like Conv-BN-ReLU into one kernel so the intermediate tensors never leave on-chip registers/cache.

Arithmetic Intensity  AI = FLOPs / Bytes moved
Roofline:  Achievable GFLOP/s = min( PeakCompute, AI × PeakBandwidth )
Unfused bytes  ≈ 2·N·precisionBytes  per op   (read + write, every op)
Fused bytes    ≈ 2·N·precisionBytes  per fused GROUP (read once, write once)

Fusion toggle — merges [Conv1,BN,ReLU] and [Conv2,Pool] into two kernels instead of five; the top panel shows packets skipping the DRAM dip between fused ops.
Precision — INT8/FP16 shrink bytes-per-element (and, on the NPU, raise peak compute via native low-precision math), shifting the operating point along both axes of the roofline.
Hardware — CPU, GPU and NPU differ in peak FLOP/s and peak DRAM bandwidth; the same pipeline can be memory-bound on one and compute-bound on another.
The bright dot on the roofline plot is the current config; the faint dot is the same config with fusion flipped. Drag the plot to pan, scroll/pinch to zoom, and turn on Trail to leave a fading history as you sweep sliders.

This is the exact reasoning edge-inference engines use to decide which ops to fuse: a memory-bound elementwise op (BatchNorm, ReLU) gains almost nothing from a faster ALU — its bottleneck is DRAM bandwidth — so folding it into its neighbor's kernel is the only way to speed it up.

Pipeline & memory traffic

Roofline (drag to pan, scroll to zoom)

Compute-heavy op (Conv) Memory-heavy op (BN/ReLU/Pool) Data packet in flight

scroll = zoom · drag = pan