Distillation

Knowledge distillation is a model compression technique where a smaller student model is trained to mimic the output distribution of a larger teacher model. The student learns from soft probability distributions (teacher logits) rather than hard labels, transferring knowledge more efficiently.

Distillation is used to produce smaller, faster models that retain much of the teacher's accuracy.