Knowledge Distillation Isn't Model Shrinking — It's Teaching a Student to Think Like the Teacher

Large models typically have stronger understanding and generation capabilities, but they also come with larger parameter scales, higher inference latency, and greater deployment costs.

If a model performs well but cannot fit into a phone, browser, or resource-limited server, those capabilities are hard to put into practice.

A natural idea arises:

Can we let a large model act as a teacher and pass the knowledge it has learned to a smaller model?

This is the problem that Knowledge Distillation aims to solve.

It does not mean directly copying the large model's parameters to the small model, nor is it just about collecting a batch of answers for the small model to memorize by rote. Instead, it involves making the student model imitate the teacher model's way of understanding data as closely as possible.

Why is it called "Distillation"?

In chemistry, distillation uses differences in the volatility of components to separate a target substance from a mixture.

Knowledge Distillation borrows this name:

A model with stronger capabilities and a larger structure
        ↓ extracts knowledge
A model with smaller capabilities and lighter deployment

Where:

• The stronger model is called the Teacher Model
• The small model that needs to learn is called the Student Model
• The training process where the student imitates the teacher is called Knowledge Distillation

The teacher model is usually already trained. During distillation, the student model learns not only the correct answers in the training data but also the predictive information provided by the teacher model.

Why is learning only the standard answer not enough?

Suppose we want to train an image classification model, and the input is a photo of a cat.

An ordinary label might only tell the model:

Cat: 1
Dog: 0
Tiger: 0
Car: 0

This kind of explicit standard answer is called a Hard Label.

A hard label can tell the student that "the correct answer is cat," but it does not explain:

• How much does this image resemble a cat?
• Why is it easily confused with a dog?
• Do cats and tigers share similar features?
• Why is a car almost certainly not the answer?

For a hard label, all incorrect options are 0; there is no difference between dog, tiger, and car. Much valuable information about class relationships is thus discarded.

What information does a soft label provide?

When the teacher model processes the same image, its internal output might be closer to the following probability distribution:

Cat:   80%
Dog:   10%
Tiger:  9%
Car:    1%

This kind of information with relative probabilities is usually called a Soft Label.

It not only gives the conclusion "cat" but also exposes the teacher model's judgment about the relationships between different classes:

Cats and dogs share some similarities
Cats and tigers also have related visual features
Cats and cars have almost no relationship

This kind of class correlation learned by the teacher model from vast amounts of data is often called "Dark Knowledge."

Therefore, what the student model in knowledge distillation needs to imitate is not an isolated answer, but the shape of the teacher model's output distribution.

Why does the temperature parameter appear?

Before obtaining the final probabilities, a model typically outputs a set of unnormalized scores, known as logits.

Softmax converts these scores into probabilities:

pᵢ = exp(zᵢ / T) / Σ exp(zⱼ / T)

Where:

• zᵢ is the logit for the i-th class
• T is the Temperature

Ordinary classification often uses T = 1. Distillation training usually uses a temperature greater than 1 to make the probability distribution smoother.

For example, an originally very concentrated distribution:

Cat: 96%　Dog: 3%　Tiger: 1%

After being softened by a higher temperature, it might become easier to see the differences between secondary classes:

Cat: 60%　Dog: 23%　Tiger: 17%

The numbers here are only to illustrate the distribution change and are not fixed calculation results.

When the temperature is too low, the teacher's output will be very close to a hard label; when the temperature is raised, the student can see more information about non-target classes. But a higher temperature is not always better; excessive smoothing can make class differences too weak.

What exactly is the student model optimizing?

Classic knowledge distillation usually retains two learning signals simultaneously.

1. Learning from true labels

The student model must still be accountable for the standard answers in the training data, for example, that this image is indeed a cat.

This part typically uses ordinary cross-entropy loss.

2. Imitating the teacher's distribution

The student model must also make its softened output close to the teacher model's softened output.

This part often uses KL divergence or cross-entropy to measure the difference between the two distributions.

The overall objective can be simplified as:

Total Loss
= α × Learning from true answers
+ β × Imitating the teacher's distribution

In common implementations, the distillation loss is also multiplied by T² to compensate for the gradient scaling caused by the temperature change.

Thus, the student neither completely abandons the real data to blindly imitate the teacher, nor does it only see black-and-white standard answers.

The complete process can be summarized as:

The same batch of training data
   ├──→ Teacher Model ──→ Softened teacher distribution ──┐
   │                                                      ├─→ Calculate loss → Update student model
   └──→ Student Model ──→ Softened student distribution ──┘
                     +
                   True labels

The teacher model generally only participates in forward computation and no longer updates its parameters; it is the student model that is truly being trained.

Understanding Knowledge Distillation through "Learning to Cook"

Training a student model can be imagined as training a new chef.

The first method is to only give him a recipe:

Salt 3 grams
Oil 20 milliliters
Stir-fry for 2 minutes

This is very much like a hard label. The newcomer knows what to do in the end, but does not understand why the heat must be controlled that way, nor how to adjust when the state of the ingredients changes.

The second method is to have an experienced chef give a live demonstration.

The newcomer not only sees the finished product but can also observe when the master chef adjusts the heat, how to judge doneness, and how to change operations when encountering different ingredients.

This information, which is not fully written in the recipe but reflects experience, is like the dark knowledge contained in the teacher model's probability distribution.

Knowledge distillation does not guarantee that the student will end up exactly the same as the teacher; rather, it uses richer supervisory signals to allow a student with smaller capacity to learn the teacher's way of judgment as much as possible.

Does "collecting large model answers" all count as classic knowledge distillation?

Not necessarily.

In classic classification distillation, the trainer can usually access the teacher model's logits or complete probability distribution and then have the student directly fit this information.

But what is often obtained through a large model API is just a final text answer:

Question → Teacher Model → Text Answer

If the interface does not provide logits or token probabilities, the student cannot see the teacher's complete output distribution. This method of training a student using teacher-generated content is usually closer to Response-based Distillation or distillation based on synthetic data.

In large language models, common practices also include:

• Having the teacher generate high-quality Q&A data
• Having the student imitate the teacher's answering style and task format
• Learning the teacher's token probabilities when accessible
• Aligning intermediate features, hidden states, or attention information

They all embody the idea of "the teacher transferring capability to the student," but the information available differs, and the training methods are not exactly the same.

Therefore, knowledge distillation cannot be simply understood as "sending requests to a large model and saving the answers." Whether probabilities can be accessed, how the training data is constructed, and how the loss function is designed all affect which type of distillation it belongs to.

What is the difference between knowledge distillation, fine-tuning, and model compression?

These concepts often appear together, but they solve problems in different ways.

Method	Core Operation	Requires a Teacher Model?	Primary Purpose
Fine-tuning	Continue training the model with task-specific or domain-specific data	Not necessarily	Adapt the model to a new task or domain
Knowledge Distillation	Have a student model imitate a teacher model	Yes	Transfer the teacher's capability to a smaller model
Quantization	Represent weights or computations with lower precision	No	Reduce memory, storage, and computation overhead
Pruning	Remove unimportant connections, channels, or parameters	No	Reduce model size and computation

They are not mutually exclusive.

A model can first be distilled to get a smaller student model, then be quantized; it can also be further fine-tuned for a specific business domain. Multiple methods are often combined in actual deployment.

What can knowledge distillation bring?

Lower inference cost

Student models have fewer parameters and typically require less memory and computation resources, making them suitable for high-concurrency services.

Faster response speed

Smaller models make it easier to reduce inference latency, usable for mobile devices, edge devices, and real-time applications.

Better results than training a small model alone

For a student model of the same scale, if it only learns from hard labels, the supervisory information received is relatively limited; after adding the teacher's signal, it can often learn more nuanced decision boundaries.

Utilizing unlabeled data

The teacher model can generate soft labels or answers for a large number of unlabeled samples, and this data can then be used to train the student model.

Distillation also has clear limitations

Knowledge distillation is not a lossless copy.

Student capacity determines the upper limit of capability

If the student model is too small, it does not have enough parameter capacity to carry the teacher's knowledge. Even with a lot of training data, it cannot fully replicate the teacher's capabilities.

The teacher's errors can also be passed on

When the teacher model has biases, hallucinations, or errors, the student may learn these problems along with it. The teacher's output is not naturally equivalent to the correct answer.

The coverage of training data is very important

The student will only imitate the teacher on the tasks and distributions covered by the distillation data. If the dataset only contains math problems, the student will not automatically gain the teacher's full capabilities in coding, writing, and visual understanding.

Distillation still requires cost

Generating training data, running the teacher model, saving outputs, and training the student model all require resources. It is a trade-off between model capability and deployment cost, not a way to gain large model capabilities at zero cost.

Data and licensing boundaries cannot be ignored

When using a third-party model to generate training data, you must also comply with the corresponding terms of service, data licenses, and intellectual property requirements. Being technically able to collect outputs does not mean they can be directly used for training in any scenario.

Summary

The core of knowledge distillation is not "shrinking a large model," but having a student model learn the additional supervisory information provided by a teacher model.

Hard labels only tell the student the final answer; soft labels can also express the teacher's judgment on class similarities and decision boundaries. By learning from both true labels and the teacher's distribution simultaneously, the student has the opportunity to achieve better results at a smaller parameter scale than training independently.

The entire article can be condensed into the following chain:

Teacher model outputs knowledge
        ↓
Soft labels, logits, answers, or intermediate features
        ↓
Student model imitates the teacher
        ↓
Trade a smaller model for lower deployment costs

But distillation is not replication. The student model's capacity, the coverage of the distillation data, the quality of the teacher's output, and the design of the training objective together determine how much capability can ultimately be transferred.