The simulation visualises a large teacher network and a compact student network side by side, showing how the teacher's softened probability distribution over classes is passed to the student as a training signal.
Drag the temperature slider to soften or sharpen the teacher's output distribution and watch how the student's learned predictions change, then use the speed and play/pause controls to step through training and rebuild the networks with different sizes.
Temperature slider, training speed slider, play/pause, rebuild network
Hinton's original paper found that distilling a huge ensemble of models into a single small network preserved nearly all of the ensemble's accuracy, while requiring far less compute at inference time.
The simulation visualises a large teacher network and a compact student network side by side, showing how the teacher's softened probability distribution over classes is passed to the student as a training signal.
The simulation visualises a large teacher network and a compact student network side by side, showing how the teacher's softened probability distribution over classes is passed to the student as a training signal.
Drag the temperature slider to soften or sharpen the teacher's output distribution and watch how the student's learned predictions change, then use the speed and play/pause controls to step through training and rebuild the networks with different sizes.
Hinton's original paper found that distilling a huge ensemble of models into a single small network preserved nearly all of the ensemble's accuracy, while requiring far less compute at inference time.