Operator Fusion & the Roofline Model

Kernel Fusion

Precision

Edge Hardware

Tensor size 1.0M elements

Live Readouts

Memory traffic

–

Arithmetic intensity

–

Achieved throughput

–

Latency

–

Roofline verdict

–

Speedup from fusion (this config)

–

How it works

An inference pipeline is a chain of operators — here Conv → BatchNorm → ReLU → Conv → Pool. Every operator that runs on its own must read its input tensor from DRAM and write its output back before the next operator starts. Operator fusion — what TensorFlow Lite, Core ML and ONNX Runtime's graph optimizers all do — merges a chain like Conv-BN-ReLU into one kernel so the intermediate tensors never leave on-chip registers/cache.

Arithmetic Intensity  AI = FLOPs / Bytes moved
Roofline:  Achievable GFLOP/s = min( PeakCompute, AI × PeakBandwidth )
Unfused bytes  ≈ 2·N·precisionBytes  per op   (read + write, every op)
Fused bytes    ≈ 2·N·precisionBytes  per fused GROUP (read once, write once)

Fusion toggle — merges [Conv1,BN,ReLU] and [Conv2,Pool] into two kernels instead of five; particles skip the DRAM round-trip between fused ops (no dip to the memory plane).
Precision — INT8/FP16 shrink bytes-per-element (and, on the NPU, raise peak compute via native low-precision math), shifting the operating point along both axes of the roofline.
Hardware — CPU, GPU and NPU differ in peak FLOP/s and peak DRAM bandwidth; the same pipeline can be memory-bound on one and compute-bound on another.
The dot on the roofline plot is the current config; the fainter dot is the same config with fusion flipped, so you can see the operating point slide right (higher AI) and often up (higher achieved throughput) when kernels fuse.

This is the exact reasoning edge-inference engines use to decide which ops to fuse: a memory-bound elementwise op (BatchNorm, ReLU) gains almost nothing from a faster ALU — its bottleneck is DRAM bandwidth — so folding it into its neighbor's kernel is the only way to speed it up.

Compute-heavy op (Conv) Memory-heavy op (BN/ReLU/Pool) Data packet in flight

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.