Neural Network Pruning Explained

Fundamentals

Types of Pruning

• Unstructured weight pruning (magnitude, gradient)
• Structured channel/filter/pruning
• Lottery ticket and dynamic sparsity

Schedules and Recovery

Gradual pruning with fine-tuning yields better accuracy retention than one-shot pruning.

Deployment

Structured sparsity maps better to kernels; evaluate end-to-end on target hardware.

How the Algorithm Works

Algorithm

1. Train baseline; compute saliency (magnitude/gradient/Hessian)
2. Prune weights/channels by threshold or target sparsity
3. Fine-tune; reassess saliency
4. Repeat to target KPI; export masks

Scheduling

• Gradual pruning (polynomial/cubic)
• One-shot pruning for quick baselines
• Layer-wise budgets by sensitivity

Real-World Applications

Edge/Mobile

Latency/memory improvements with structured channel pruning; validate on ARM/embedded targets.

Cloud

Throughput and cost reductions; balanced with accuracy via careful recovery.

Best Practices

Checklist

• Establish KPIs and baselines
• Select structured pruning for real speedups
• Use gradual schedules and recovery
• Benchmark on target hardware
• Track masks, seeds, and KPIs
• Govern with approvals and audit logs

Anti-Patterns

• One-shot extreme pruning without recovery
• Ignoring hardware kernel support
• No tracking or reproducibility
• Overfitting to proxy latencies
• Skipping fairness and calibration checks

Evaluation

Metrics and Protocols

• Accuracy/calibration, latency (P50/P95), memory, energy
• Benchmark on target hardware
• Use repeated runs and report CIs

Robustness

Test under distribution shifts and noisy inputs; verify stability of predictions.

Worked Examples

Iterative Magnitude Pruning

# Schedule sparsity increase + fine-tuning cycles

Structured Channel Pruning

# Channel selection by saliency and FLOPs

Implementation

PyTorch

# torch.nn.utils.prune.l1_unstructured / ln_structured

Masks and Persistence

Persist masks with model; export pruned weights after making pruning permanent.

Recovery

• Lower LR and longer patience
• Rewinding to early checkpoints

The Math Behind It

Saliency

Magnitude and gradient-based criteria approximate contribution to loss; Hessian-informed methods refine importance.

Lottery Ticket

Subnetworks with proper initialization can match full model performance; rewinding improves reproducibility.

Bounds

Sparsification affects Lipschitz properties and generalization; structured pruning retains tensor topology.

Training Strategy

Recovery

• Lower LR and cosine decay
• Increase patience and use EMA

Stability

• Clip gradients; maintain BN statistics
• Freeze fragile layers early

Monitoring

• Track accuracy, latency, memory
• Log sparsity per layer