Knowledge Distillation Isn't Model Shrinking — It's Teaching a Student to Think Like the Teacher
Running large models in production is expensive and slow. Distillation produces smaller models that retain much of the teacher's judgment at a fraction of the inference cost, making real-time, on-device, and high-throughput deployments practical without sacrificing as much accuracy as training a small model from scratch.
Knowledge distillation transfers capability from a large teacher model to a smaller student by having the student imitate the teacher's softened probability distribution, not just memorize correct answers. Hard labels discard valuable information about class similarity — a cat photo labeled only as "cat" loses the signal that dogs and tigers are visually closer to cats than cars are. The teacher's output distribution captures these relationships as "dark knowledge."
A temperature parameter in the softmax function controls how much of that secondary information the student sees. Higher temperatures smooth the distribution, exposing subtle class affinities that a hard label would flatten to zero. The student optimizes a combined loss: cross-entropy against ground-truth labels plus a divergence term that pulls its softened output toward the teacher's softened output.
Distillation is not lossless compression. Student capacity sets a hard ceiling on what can transfer, teacher errors propagate, and the training data's coverage bounds what the student learns. The technique sits alongside quantization and pruning in the model-compression toolkit, and the methods are often stacked — distill first, then quantize.
The term "dark knowledge" captures something counterintuitive: the most valuable information a teacher model provides is not the correct answer but the relative probabilities it assigns to wrong answers. Those probabilities encode structural knowledge about the data that a one-hot label destroys.
Temperature is a dial for how much secondary information the student receives, not a hyperparameter to maximize. Too high, and class differences wash out; too low, and distillation collapses into ordinary hard-label training.
The distinction between logit-level distillation and response-based distillation matters practically. Many teams that think they are doing knowledge distillation are actually just generating synthetic training data from an API, which provides far less supervisory signal.
Distillation's cost is easy to underestimate. Running the teacher to generate training data, storing outputs, and training the student all consume resources. It is a deployment-time trade-off, not a free lunch.