Artificial Intelligence · LLM · Agent

Distillation Is the Real Reason Big Models Ship

By 吴佳浩Alben · Jul 24, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Distillation is what makes large models economically viable. Without it, a 671B-parameter model would never leave the research cluster; with it, the same capability gets packaged into a 7B model that runs on a single GPU. For teams without eight-figure training budgets, distilling from a commercial API into a private, controllable small model is the only realistic path to owning their AI stack.

Summary

Distillation is the engineering backbone that gets large models out of the lab and into production. A 671B-parameter model can cost tens of millions to train and requires dozens of GPUs just to run; a distilled 7B student runs on a single consumer card while retaining most of the teacher's capability. The technique works by having the student learn not just the final answer but the teacher's full probability distribution—Hinton's "dark knowledge"—along with intermediate hidden states, attention maps, and reasoning chains.

The process spans data generation, cleaning, scoring, and a multi-loss training regime that combines supervised fine-tuning with KL-divergence soft-label losses. Temperature scaling exposes the teacher's internal class relationships, while feature and attention distillation transfer how the teacher understands input. The hottest frontier, reasoning distillation, copies the teacher's entire chain-of-thought, which is why DeepSeek-R1's 1.5B student can still reason through math problems.

A complete enterprise walkthrough shows the full pipeline: historical tickets seed a teacher model that generates structured question-reasoning-answer triples; rules and an AI judge filter the output; a ChatML dataset feeds LoRA training via LLaMA Factory; and vLLM serves the final student. The result is a domain-specific model that costs two orders of magnitude less to run than the teacher.

Takeaways

— Distillation transfers a teacher model's behavior, not its weights; student and teacher can have completely different architectures and parameter counts.

— Temperature scaling (T > 1) smooths the teacher's softmax output, exposing "dark knowledge"—the relative similarity between incorrect classes—that a one-hot label discards.

— The standard KD loss combines cross-entropy on ground-truth labels with KL divergence between teacher and student softmax distributions at temperature T, weighted by α.

— Feature distillation aligns intermediate hidden states via a projector layer and MSE loss, letting the student copy how the teacher represents input internally.

— Attention distillation directly matches attention weight matrices between teacher and student layers, transferring what the model focuses on when reading text.

— Reasoning distillation (CoT distillation) copies the teacher's full chain-of-thought—analysis, hypotheses, verification steps—not just the final answer, which is why small distilled models still reason well.

— Preference distillation uses DPO/IPO/ORPO to align the student with the teacher's value judgments by learning from chosen-vs-rejected answer pairs.

— Data quality dominates: garbage teacher outputs get learned verbatim and amplified; a typical training mix is 70% teacher-generated data, 20% human-annotated data, and 10% other synthetic data.

— Multi-teacher distillation lets a student learn from GPT, Claude, DeepSeek, and Gemini simultaneously, cross-validating outputs and combining each model's strengths.

— A student rarely surpasses its teacher in general capability because its capacity ceiling is bounded by the teacher's supervision signal, though iterative self-distillation can yield local improvements.

— DeepSeek-R1 distilled a full family of models (32B down to 1.5B) from a single reasoning teacher, proving that reasoning chains, not just factual knowledge, are transferable.

— The complete enterprise pipeline—seed questions → teacher generation → rule/AI-judge filtering → ChatML dataset → LoRA training → vLLM deployment—is reproducible with LLaMA Factory and a commercial API.

Conclusions

Distillation is the unglamorous infrastructure that separates companies shipping models from those publishing papers; the model that runs in production is almost never the largest one the company has built.

Temperature is a deceptively simple knob that controls how much of the teacher's internal structure gets exposed to the student—too low and the student sees only the answer, too high and noise drowns the signal.

The industry's shift toward reasoning distillation reflects a deeper insight: factual knowledge compresses poorly, but reasoning procedures compress well, which is why a 1.5B model can solve math problems its size shouldn't permit.

Multi-teacher distillation is effectively an arbitrage play—exploiting the fact that different vendors' models have uncorrelated strengths and weaknesses, then blending them into a single student that outperforms any individual teacher on aggregate.

The 70/20/10 data mix rule of thumb encodes a real constraint: purely synthetic data causes distribution collapse over successive training generations, so human-annotated anchors are necessary to keep the data manifold from drifting toward the teacher's biases.

Distillation's real competitive moat is not the algorithm—KD loss is public and trivial to implement—but the proprietary data-generation pipeline, cleaning heuristics, and teacher-student layer-mapping strategies that each lab guards as trade secrets.

Concepts & terms

Dark Knowledge

The relative probability relationships among incorrect classes in a teacher model's softmax output. A teacher might assign 0.15 probability to 'dog' and 0.03 to 'fox' for a cat image, revealing that cats are more similar to dogs than foxes—information completely absent from a one-hot label.

Temperature (T) in Softmax

A scaling parameter applied inside softmax: p_i = exp(z_i / T) / Σ exp(z_j / T). T=1 gives standard softmax; T>1 flattens the distribution, exposing subtle class relationships. The gradient is scaled by T² during backpropagation to compensate.

KL Divergence Loss

Measures how one probability distribution diverges from another. In distillation, KL(teacher_softmax || student_softmax) quantifies how much the student's output distribution differs from the teacher's, computed at temperature T and multiplied by T² for gradient scaling.

Feature Distillation

Instead of matching only final output probabilities, the student learns to reproduce the teacher's intermediate hidden states. A projector layer (linear transform) aligns the student's smaller hidden dimension to the teacher's, then MSE loss minimizes the distance.

Attention Distillation

Directly aligns the attention weight matrices (Q·K^T) between corresponding teacher and student layers. Transfers what tokens the model attends to during processing, which captures more structural information than output probabilities alone.

Reasoning Distillation (CoT Distillation)

The teacher generates full reasoning chains—thinking steps, intermediate calculations, verification—not just final answers. The student learns to reproduce the entire chain, which is why small distilled models (e.g., DeepSeek-R1-1.5B) retain strong reasoning on math and code tasks.

Preference Distillation

The teacher provides preference pairs (chosen vs. rejected answers) for the same question. The student optimizes directly on these pairs using DPO, IPO, or ORPO, aligning with the teacher's value judgments without needing a separate reward model or PPO loop.

Multi-Teacher Distillation

A student learns simultaneously from multiple teacher models (e.g., GPT, Claude, DeepSeek, Gemini). Combines each teacher's strengths, cross-validates outputs to reduce hallucinations, and produces a student that can outperform any single teacher on aggregate metrics.

Model Collapse

When models are trained exclusively on synthetic data generated by other models, the data distribution gradually narrows toward the generator's biases over successive generations. Retaining a fraction of human-annotated data anchors the distribution to real-world diversity.

LoRA (Low-Rank Adaptation)

A parameter-efficient fine-tuning method that inserts trainable low-rank matrices into attention layers while freezing the original weights. Dramatically reduces memory requirements, making it practical to distill a 7B student on consumer hardware.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗