Home▸Machine Learning & Neural Networks▸Knowledge Distillation: Training a Small Model to Think Like a Big One

🎓 Knowledge Distillation: Training a Small Model to Think Like a Big One

Explore how knowledge distillation compresses a large 'teacher' neural network into a fast, lightweight 'student' by learning from soft probability distributions rather than hard labels.

Machine Learning & Neural Networks3DModerate60 FPS

The simulation visualises a large teacher network and a compact student network side by side, showing how the teacher's softened probability distribution over classes is passed to the student as a training signal.

🔬 What It Demonstrates

🎮 How to Use

Drag the temperature slider to soften or sharpen the teacher's output distribution and watch how the student's learned predictions change, then use the speed and play/pause controls to step through training and rebuild the networks with different sizes.

💡 Did You Know?

Hinton's original paper found that distilling a huge ensemble of models into a single small network preserved nearly all of the ensemble's accuracy, while requiring far less compute at inference time.

⚙ Under the hood

Interactive 3D lab visualizing a teacher network's softened output distribution training a smaller student network via knowledge distillation.

knowledge-distillationmodel-compressionteacher-studentmachine-learningdeep-learningneural-networks

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install