Neural Network Quantization Explained

Fundamentals

PTQ vs QAT

PTQ calibrates int8 scales using representative data; QAT simulates quantization during training for higher fidelity.

Scales

Per-tensor vs per-channel; symmetric vs asymmetric; observers and calibration strategies.

Deployment

Export to TFLite/ONNX/TensorRT; verify opset support; benchmark on target hardware.

How the Algorithm Works

PTQ Calibration

1. Collect representative dataset
2. Run observers to estimate scales/zero-points
3. Quantize weights/activations; evaluate KPIs
4. Adjust observers (percentile/entropy) if needed

QAT

1. Insert fake-quant modules
2. Train with schedule; freeze observers late
3. Convert to int8 and validate on target

Real-World Applications

Edge/Mobile

Int8 reduces memory/latency significantly; verify kernels and operators on device.

Cloud

Higher throughput and lower costs; mixed precision for accuracy-sensitive components.

Best Practices

Checklist

• Representative calibration dataset
• Per-channel for conv weights, QAT when needed
• Validate on target hardware with KPIs
• Track observer stats and seeds
• Govern with approvals and audits

Pitfalls

• Relying on minmax with outliers
• Unsupported ops without fallbacks
• Comparisons on proxy hardware only
• No reproducibility or logs
• Skipping fairness/calibration checks

Evaluation

Metrics and Protocols

• Accuracy, ECE; latency (P50/P95), memory footprint
• Benchmark on target device; discard warmup
• Repeated runs with CIs

Worked Examples

PTQ

# Calibration with representative dataset; convert and validate

QAT

# Prepare QAT, train, freeze observers, convert

Mixed Precision

# Selective fp16/int8 assignment for sensitive layers

Implementation

PyTorch

# torch.ao.quantization: observers, prepare, convert (PTQ/QAT)

TFLite

# Converter with dynamic range and full int8; representative dataset

TensorRT

# INT8 calibrators; per-layer precision overrides

Validation

• Golden tests; KPI checks on target hardware
• Numerical tolerances; outlier handling

The Math Behind It

Error Models

Uniform quantization incurs bounded error; error propagates through layers with amplification factors.

Scale Selection

Minmax, percentile, and entropy criteria; per-channel reduces variance in conv layers.

Optimization

QAT minimizes task loss with simulated quantization; straight-through estimators approximate gradients.

Training Strategy

Schedules

• Warmup, observer freeze, and final fine-tune
• Cosine decay of LR; patience for stability

Stability

• Grad clipping; careful with BN folding
• Selective higher precision for fragile layers

Monitoring

• Accuracy, calibration (ECE), latency, memory
• Activation histograms and saturation checks