A ~7B-parameter transformer is shown as a stack of layers. Quantization stores each weight in fewer bits (FP32 → FP16 → INT8), shrinking memory and speeding up matrix multiplies at the cost of numerical noise (visible as node jitter). Pruning removes the weakest connections (by |weight|), so fewer multiply-adds run per token — but unstructured sparsity rarely gives a fully linear speedup on real hardware, which TensorRT-style kernels try to close by fusing ops and using structured/sparse kernels.
size = params · bits/8 · (1 − sparsity)
latency = L0 · k(precision) · (1 − 0.55·sparsity)
E[tok/round] = (1 − α^(γ+1)) / (1 − α)
throughput = E[tok/round] / (latency_target + γ·latency_draft)
- Quantization — sets bit-width and the per-op speed factor
k; INT8 is ~3.4× faster than FP32 here but also lowers the small draft model's accuracy α.
- Sparsity — fraction of weights zeroed out; shrinks size linearly but latency sub-linearly (sparse-kernel overhead), and above ~50% starts eroding draft accuracy too.
- Draft+verify — toggles speculative decoding: a cheap draft model proposes γ tokens per round, the full target model verifies them in one parallel pass, accepting a prefix and rejecting at the first mismatch.
- Draft length γ — how many tokens the draft proposes before verification; more tokens raise best-case throughput but waste more compute when rejected early.
Real engines like TensorRT (or TensorRT-LLM) apply exactly this stack — INT8/FP8 quantization, structured pruning, kernel fusion and speculative/lookahead decoding — to cut LLM serving cost per token.