Knowledge Distillation: Training a Small Model to Think…

Why compress a model that already works?

State-of-the-art neural networks often owe their accuracy to massive size: hundreds of millions or billions of parameters spread across deep, wide layers. That scale makes them slow to run, expensive to host, and impractical for phones, browsers, or latency-sensitive applications. Knowledge distillation tackles this by training a much smaller 'student' network to reproduce the behaviour of the large 'teacher' network, aiming to keep most of the teacher's accuracy at a fraction of the computational cost.

The key insight: learn from soft labels, not just right answers

A model trained normally only sees hard labels — a photo is either 'cat' or 'dog', full stop. But a well-trained teacher's raw output before the final decision, its probability distribution over all classes, contains far more information: it might say 90% cat, 9% dog, 1% fox, revealing that cats and dogs look more alike to the model than cats and foxes. These are called soft labels, and Hinton, Vinyals and Dean showed in 2015 that training a student to match this whole distribution transfers 'dark knowledge' about class similarity that hard labels alone never capture.

Temperature and the distillation loss

Raw softmax outputs are often extremely confident, with the correct class near 1 and everything else near 0, which hides the useful similarity information. Distillation introduces a temperature parameter that divides the logits before the softmax, producing a softer, more spread-out distribution where secondary probabilities become visible. The student is then trained on a combined loss: one term matches the student's softened outputs to the teacher's softened outputs, and another term matches the student's predictions to the true hard labels, letting the student benefit from both the teacher's nuance and ground truth.

Where distillation is used, and its limits

Distillation underpins many production systems, from compressing large language models into smaller, cheaper variants to shrinking vision models for on-device or edge deployment, cutting inference cost and latency dramatically. It is not magic, however: a student's own architecture and parameter count impose a ceiling on how much it can absorb, and a mediocre teacher cannot produce a great student. In practice, distillation works best as one tool among several, often combined with pruning, quantisation, or additional fine-tuning.

Frequently asked questions

Who invented knowledge distillation?

The technique was formalised in the 2015 paper 'Distilling the Knowledge in a Neural Network' by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean at Google. It built on earlier model compression ideas but popularised the specific soft-label, temperature-based approach now widely used.

Does the student model ever outperform the teacher?

Occasionally, yes. Because soft labels act as a form of regularisation and reveal relationships between classes that hard labels obscure, a student can sometimes generalise better on certain tasks or datasets, even though it typically trails the teacher on raw accuracy.

Is knowledge distillation the same as pruning or quantisation?

No, they are complementary compression techniques. Pruning removes weights or neurons from an existing network, and quantisation reduces the numerical precision of weights, while distillation trains an entirely separate, smaller network from scratch to imitate the teacher's outputs. All three are often combined in real deployment pipelines.

Try it live

Everything above runs in your browser — open Knowledge Distillation: Training a Small Model to Think Like a Big One and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open Knowledge Distillation: Training a Small Model to Think Like a Big One simulation