Model Compression

Model compression reduces model size and computational requirements while preserving accuracy. This enables deployment on edge devices, reduces serving costs, and improves latency. Techniques include quantization, pruning, knowledge distillation, and efficient architectures.

Where It Is Used

Model compression enables ML everywhere:

• Mobile apps: On-device inference for privacy, offline use
• IoT devices: Sensors, cameras, embedded systems
• Edge computing: Retail, manufacturing, vehicles
• Cloud cost reduction: Faster inference, lower costs
• Real-time systems: Low-latency requirements
• Browser ML: TensorFlow.js, ONNX.js applications

Products: Siri (on-device), Google Lens, smart home devices, autonomous vehicles.

Compression Techniques

Typical Targets

What You Do in Practice

1. Profile baseline model - Size, latency, accuracy, memory
2. Set deployment constraints - Target device specifications
3. Choose compression approach - Based on constraints and model type
4. Apply compression - Quantization, pruning, distillation
5. Fine-tune if needed - Recover accuracy loss
6. Validate on target hardware - Real device testing
7. Optimize runtime - TensorRT, ONNX Runtime, etc.
8. Benchmark thoroughly - Latency, accuracy, memory
9. Deploy and monitor - Track real-world performance
10. Iterate - Further optimization if needed

Quantization

Reduce numerical precision of weights and activations.

Post-training quantization (simple):

import torch

# Load FP32 model
model = torch.load('model.pth')

# Dynamic quantization (weights only)
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.LSTM},
    dtype=torch.qint8
)

# Static quantization (weights + activations)
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
torch.quantization.prepare(model, inplace=True)

# Calibrate with representative data
with torch.no_grad():
    for batch in calibration_dataloader:
        model(batch)

torch.quantization.convert(model, inplace=True)

Quantization-aware training (higher accuracy):

import torch.quantization as quant

# Prepare model for QAT
model.qconfig = quant.get_default_qat_qconfig('fbgemm')
model_prepared = quant.prepare_qat(model, inplace=False)

# Train with fake quantization
optimizer = torch.optim.Adam(model_prepared.parameters())
for epoch in range(num_epochs):
    for batch in train_loader:
        loss = train_step(model_prepared, batch)
        loss.backward()
        optimizer.step()

# Convert to quantized model
model_quantized = quant.convert(model_prepared)

Pruning

Remove weights or structures that contribute little to accuracy.

Magnitude pruning:

import torch.nn.utils.prune as prune

# Prune 30% of weights with lowest magnitude
for name, module in model.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        prune.l1_unstructured(module, name='weight', amount=0.3)

# Make pruning permanent
for name, module in model.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        prune.remove(module, 'weight')

Structured pruning (remove entire filters/channels):

# Prune entire filters based on L1 norm
def prune_filters(model, amount=0.3):
    for name, module in model.named_modules():
        if isinstance(module, torch.nn.Conv2d):
            prune.ln_structured(
                module,
                name='weight',
                amount=amount,
                n=1,  # L1 norm
                dim=0  # Prune filters
            )

Iterative pruning (prune-retrain cycle):

def iterative_pruning(model, target_sparsity, steps=5):
    current_sparsity = 0
    step_sparsity = target_sparsity / steps

    for step in range(steps):
        # Prune
        prune_model(model, amount=step_sparsity)

        # Retrain
        train_model(model, epochs=5)

        current_sparsity += step_sparsity
        print(f"Step {step}: Sparsity = {current_sparsity:.1%}")

Knowledge Distillation

Train a smaller "student" model to mimic a larger "teacher".

import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels,
                      temperature=4.0, alpha=0.5):
    """
    Combine soft targets (from teacher) with hard targets (labels).
    """
    # Soft targets: KL divergence with temperature
    soft_loss = F.kl_div(
        F.log_softmax(student_logits / temperature, dim=1),
        F.softmax(teacher_logits / temperature, dim=1),
        reduction='batchmean'
    ) * (temperature ** 2)

    # Hard targets: cross-entropy with true labels
    hard_loss = F.cross_entropy(student_logits, labels)

    return alpha * soft_loss + (1 - alpha) * hard_loss

# Training loop
teacher.eval()
student.train()

for batch in train_loader:
    inputs, labels = batch

    with torch.no_grad():
        teacher_logits = teacher(inputs)

    student_logits = student(inputs)
    loss = distillation_loss(student_logits, teacher_logits, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Efficient Architectures

Use architectures designed for efficiency:

# Example: MobileNetV3
from torchvision.models import mobilenet_v3_small

model = mobilenet_v3_small(pretrained=True)
# Already ~2.5M parameters vs ResNet50's 25M

Evaluation and Metrics

Compression ratio:

def compression_stats(original_model, compressed_model):
    orig_size = sum(p.numel() for p in original_model.parameters())
    comp_size = sum(p.numel() for p in compressed_model.parameters())

    orig_bytes = sum(p.numel() * p.element_size()
                     for p in original_model.parameters())
    comp_bytes = sum(p.numel() * p.element_size()
                     for p in compressed_model.parameters())

    return {
        'param_reduction': 1 - comp_size / orig_size,
        'size_reduction': 1 - comp_bytes / orig_bytes,
    }

Failure Modes and Debugging Checklist

• [ ] Accuracy drop too large: Need fine-tuning or less aggressive compression
• [ ] Latency not improved: Runtime not optimized for compression
• [ ] Memory unchanged: Need to use optimized kernels
• [ ] Quantization artifacts: Outlier values causing issues
• [ ] Pruning instability: Pruning too aggressively
• [ ] Hardware incompatibility: Target device doesn't support format
• [ ] Calibration issues: Unrepresentative calibration data

Production Checklist

Before compression: - [ ] Baseline metrics documented - [ ] Target constraints defined - [ ] Compression approach selected - [ ] Calibration data prepared - [ ] Target device available for testing

After compression: - [ ] Accuracy validated on test set - [ ] Latency benchmarked on target device - [ ] Memory usage verified - [ ] Edge cases tested - [ ] Model format compatible - [ ] Deployment pipeline tested

Deployment: - [ ] Runtime optimized (TensorRT, ONNX Runtime) - [ ] Model versioning in place - [ ] Monitoring for quality degradation - [ ] Rollback plan ready

Tools and Stack

Example Mini-Case

Scenario: Deploy image classification model on mobile app.

Baseline: ResNet50, 98MB, 150ms latency on mobile.

Target: < 10MB, < 50ms latency, < 1% accuracy drop.

Approach: 1. Architecture: Switch to MobileNetV3-Small (10MB, 85ms) 2. Quantization: INT8 post-training (2.5MB, 45ms) 3. Fine-tuning: 5 epochs to recover accuracy

Results:

Key learnings: - Architecture choice > post-hoc compression - Calibration data quality matters for quantization - Test on actual device, not emulator - INT8 quantization gave best size/accuracy trade-off

Related Pages

• Index: ML Field Guide
• Image Classification
• Object Detection
• MLOps Foundations

Further Reading

• PyTorch Quantization
• TensorFlow Model Optimization
• ONNX Runtime
• Efficient Deep Learning Book

Try it live

Everything above runs in your browser — open Model Compression Lab — Shrink a Network Without Wrecking It and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.