OPD: How Large Models Now Distill Judgment, Not Just Answers

AI Core Technology Analysis | OPD: Large Models Are No Longer Replicating Knowledge, but Judgment

Insert image description here

Author: 吴佳浩Alben

Writing Time: 2026.7.23

Update Time: 2026.7.29

In the previous article, we explained traditional Knowledge Distillation (KD): the Teacher teaches the "answer" to the Student. This article addresses a further question — Why are companies like OpenAI, DeepSeek, and Google no longer satisfied with just distilling knowledge, but are now distilling "preferences"? What problem does OPD solve, where does it stand in the evolution of Alignment technology, and how is it implemented in engineering?

1. Why is Traditional Distillation No Longer Enough?

Let's review the core logic of traditional distillation:

flowchart LR
    A[Teacher] --> B[Student learns Logits]
    B --> C[Replicates Knowledge]

This logic holds true in the dimension of "knowledge": the Teacher knows more facts and can perform more complex reasoning, and the Student can inherit this capability simply by fitting the Teacher's output distribution.

But today, what truly differentiates large models is no longer just "how much they know," but:

Whether the answers align with human preferences;
Whether they are safe and not easily misused;
Whether the reasoning process is correct and free of hallucinations;
Whether the expression is natural and truly "helpful."

The underlying knowledge bases used by models like GPT-5, DeepSeek, Gemini, and Claude are becoming increasingly similar. The real differentiator lies in how well Alignment is done. And Alignment is precisely what traditional KD cannot learn.

A single sentence summarizes the core point of this chapter:

Large model distillation is upgrading from "replicating answers" to "replicating the ability to judge the quality of answers."

2. The Evolution of Alignment: From Pretrain to OPD

Before explaining how OPD is implemented, it's necessary to place it on a longer timeline—OPD is not a concept that appeared out of thin air, but a natural evolution of large model Alignment technology.

flowchart TD
    A[Pretrain] --> B[SFT]
    B --> C[RLHF: Train Reward Model + PPO]
    C --> D[DPO: Directly Learn Preference]
    D --> E[RFT: Reinforcement Fine-Tuning]
    E --> F[OPD: Online Preference Distillation]

Each stage solves a different problem:

Pretrain: The model learns language patterns and world knowledge from massive unlabeled text. At this stage, the model cannot yet "converse."
SFT (Supervised Fine-Tuning): Uses human-annotated Q&A pairs to teach the model to respond in an instruction-following format. The model begins to "speak."
RLHF (Reinforcement Learning from Human Feedback): SFT data alone cannot cover all scenarios. Human annotators are introduced to rank multiple responses, training a Reward Model. A reinforcement learning algorithm like PPO then optimizes the policy model to learn "what humans prefer."
DPO (Direct Preference Optimization): The RLHF pipeline of training a Reward Model + PPO is complex and unstable. DPO proposes skipping the explicit Reward Model and directly using an equivalent supervised learning objective on preference data pairs.
RFT (Reinforcement Fine-Tuning): Further simplifies data requirements based on DPO, using a small number of samples with scoring criteria combined with reinforcement learning signals to allow the model to self-improve on specific tasks, reducing reliance on large-scale human preference annotation.
OPD (Online Preference Distillation): When the Teacher model's own judgment is strong and stable enough, instead of having every Student go through the full process of "annotating preference data → training a Reward Model → reinforcement learning," the Teacher can directly teach this "judgment" to the Student online—this is the protagonist of this article.

Looking at this timeline reveals a clear trend: With each step forward, the reliance on "human-annotated preference data" decreases, and the utilization of the "model's own judgment" increases. OPD is the current product of this trend. The following chapters will break down how it works.

3. Review: What Does Traditional Distillation Actually Learn?

Let's first go through the complete traditional KD process as a baseline for comparison.

Take a concrete example. For the question "What is the capital of France?", the probability distribution output by the Teacher might be:

    Paris   96%
    London   2%
    Tokyo    2%

The Student needs to fit not just the single word "Paris," but the entire probability distribution—including the subtle relative relationship that "London is slightly closer to the correct answer than Tokyo."

Described mathematically: The Teacher's output distribution is denoted as $P_{teacher}(y|x)$, and the Student's output distribution as $P_{student}(y|x)$. The training objective is to minimize the KL divergence between the two:

$\mathcal{L}{KD} = KL\big(P{teacher}(y|x) ,|, P_{student}(y|x)\big)$

Here, the Student learns the Knowledge Distribution—it knows "which answer is closer to the truth," but it has no idea "which answer is more popular with humans." This is precisely the problem the next chapter will address.

4. Why is KD Unsuitable for Alignment?

Consider a real-world scenario. A user asks: "How do I learn Python?"

The Teacher provides two answers, both factually correct:

Answer A: Step-by-step, with code examples, easy to understand, suitable for beginners;
Answer B: Involves more engineering practice advice (virtual environments, type annotations, testing frameworks), more professional, but less friendly for newcomers.

Both answers are perfectly tied in terms of "factual correctness"—KD's Loss will only tell the Student that "the tokens in both answers are reasonable," but it will not tell the Student which answer better suits the current user's needs and human preferences.

This is the fundamental limitation of traditional distillation in Alignment scenarios: It can only fit "correctness," and cannot learn "preference." Preference, precisely, cannot be inferred from the token probabilities of a single answer; it must come from "comparison"—which leads us to OPD.

5. What Exactly is OPD?

The core shift of OPD (Online Preference Distillation) can be summarized in one sentence:

No longer learning Tokens, but learning Preference (ranking).

In traditional KD, the Teacher provides answers; in OPD, the Teacher provides rankings. The Student no longer asks "What is the correct answer?" but rather "Given two candidate answers, which one is better, and why?" This shift seems minor, but it transforms the supervisory signal for Student training from "approximating a point" to "learning a relative relationship," which is the core requirement of Alignment training.

6. The Overall OPD Process (Key Section)

The complete OPD training process is as follows:

flowchart TD
    A[Question] --> B[Student generates multiple candidate answers]
    B --> C[Teacher scores/ranks candidate answers]
    C --> D[Obtain Ranking]
    D --> E[Calculate Preference Loss]
    E --> F[Update Student parameters]
    F --> G[Student uses new parameters to generate next round of candidates]
    G --> B

The most fundamental change here is: The Teacher no longer outputs Logits, but a Preference Signal. What the Teacher sees is not "what the next token should be," but "which of these complete candidate answers is better." The granularity of this signal is coarser (sentence-level/answer-level judgment), but its information density is higher (directly aligned with the goal of "good or bad").

7. Why is Online Called Online?

The data production method in traditional distillation is offline:

flowchart LR
    A[Teacher fixed] --> B[Batch generate data] --> C[Student offline learning]

The Teacher first generates data in one go, saves it to disk as a static dataset, and the Student then trains repeatedly on this fixed dataset. Data generation and model training are two completely separate phases.

OPD is different:

flowchart LR
    A[Teacher] --> B[Student]
    B --> C[Student parameter update]
    C --> D[Re-generate candidates with new Student]
    D --> A

Every time the Student updates a round of parameters, the candidate answers generated in the next round will be different. The Teacher needs to re-score and rank this new batch of freshly generated candidates, rather than working from a static set of historical data. The Teacher and Student co-evolve during the training process—this is why it is called Online: the preference signal is generated dynamically in real-time, not prepared in advance as static corpus.

8. KD vs. OPD Comparison (Core Chapter)

Comparison Item	KD	OPD
Learning Objective	Logits	Preference
Learning Content	Token Probabilities	Answer Ranking
Loss	KL Loss	Preference Loss (Bradley-Terry)
Teacher	Fixed, offline data generation	Online participation, real-time scoring
Alignment-Oriented	No	Yes
Learns Human Preference	No	Yes
Requires Multiple Candidates	No	Yes
Can Combine with RL	Not directly supported	Naturally supported

Summary in one sentence:

KD replicates knowledge; OPD replicates judgment.

9. The Training Principle Behind OPD

Let's look specifically at how the Teacher provides the supervisory signal. For the same Question, the Student generates multiple candidates:

    Answer A
    Answer B
    Answer C

What the Teacher provides is not a score, but a ranking:

$B > A > C$

The Student's training objective is to maximize the probability of the event "the generated ranking matches $B>A>C$"—rather than maximizing the probability of a fixed token sequence:

    Maximize: P(B > A > C)     ← OPD's objective
    Not: P(Token)              ← Traditional KD's objective

This means the Student's update direction is to "make the winning answer more likely to appear in its own generation distribution, and the losing answer less likely to appear." This is a relative objective, not an absolute one.

10. How is OPD Loss Calculated?

The most common engineering practice is to simplify the ranking problem into Pairwise comparisons, i.e., comparing only one Chosen (better) and one Rejected (worse) at a time. This is precisely the idea behind the Bradley-Terry model: given two candidates, the probability that one is preferred is the sigmoid function of the difference in their "latent quality scores."

$P(\text{chosen} \succ \text{rejected}) = \sigma\big(r_{chosen} - r_{rejected}\big)$

Taking the negative log-likelihood of this probability during training yields the Preference Loss:

$\mathcal{L}{pref} = -\log \sigma\big(r{chosen} - r_{rejected}\big)$

This formula looks simple, but it is the common mathematical foundation for understanding RLHF's Reward Model training, DPO's core objective function, and OPD's Preference Loss—the Loss for all three essentially comes from the Bradley-Terry modeling approach. The only difference lies in how $r$ is specifically defined and whether an independent Reward Model needs to be explicitly trained.

11. Actual Code Implementation (PyTorch, All Real and Runnable)

This chapter provides five code segments, progressively demonstrating the implementation from "traditional KD" to a "complete OPD training loop," and finally to a "real production environment version." The first four code segments have all been run and tested locally, with the results attached after each segment; the fifth segment is a reference implementation that can be directly used in a production environment, requiring a GPU and model weights to run, which will be noted at the corresponding location.

Example 1: Traditional KD (as a comparison baseline)

First, run a traditional KD for direct comparison with the OPD later. Here, a small MLP simulates the Teacher/Student, and the logic is completely consistent with real LLM distillation:

"""
Example 1: Traditional Knowledge Distillation
Dependency: pip install torch --break-system-packages
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(42)
NUM_CLASSES = 5
NUM_SAMPLES = 1000
FEATURE_DIM = 16

X = torch.randn(NUM_SAMPLES, FEATURE_DIM)
true_w = torch.randn(FEATURE_DIM, NUM_CLASSES)
y = (X @ true_w).argmax(dim=1)
loader = DataLoader(TensorDataset(X, y), batch_size=64, shuffle=True)

class MLP(nn.Module):
    def __init__(self, in_dim, hidden_dim, out_dim, num_layers=2):
        super().__init__()
        layers = [nn.Linear(in_dim, hidden_dim), nn.ReLU()]
        for _ in range(num_layers - 1):
            layers += [nn.Linear(hidden_dim, hidden_dim), nn.ReLU()]
        layers.append(nn.Linear(hidden_dim, out_dim))
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)

teacher = MLP(FEATURE_DIM, 128, NUM_CLASSES, num_layers=3)
student = MLP(FEATURE_DIM, 16, NUM_CLASSES, num_layers=1)

# First train Teacher to a good level
teacher_optim = torch.optim.Adam(teacher.parameters(), lr=1e-3)
for epoch in range(15):
    for xb, yb in loader:
        loss = F.cross_entropy(teacher(xb), yb)
        teacher_optim.zero_grad(); loss.backward(); teacher_optim.step()
teacher.eval()

# Student learns Teacher's output distribution (Logits) via KL divergence
T = 2.0
student_optim = torch.optim.Adam(student.parameters(), lr=1e-3)
for epoch in range(10):
    for xb, yb in loader:
        with torch.no_grad():
            teacher_logits = teacher(xb)
        student_logits = student(xb)
        soft_teacher = F.softmax(teacher_logits / T, dim=1)
        soft_student = F.log_softmax(student_logits / T, dim=1)
        # KD Loss: Make Student's distribution approximate Teacher's distribution
        loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (T ** 2)
        student_optim.zero_grad(); loss.backward(); student_optim.step()

print(f"KD Loss (KL) for the last batch: {loss.item():.4f}")

Actual run output:

    KD Loss (KL) for the last batch: 4.0525

Key Point: Here, the Student learns the "relative probability of each class." The Loss uses KL divergence because we want to match the two probability distributions themselves. But note, this Loss has no concept of "comparing two candidates to see who is better"—it is just fitting a static target distribution. Next, let's see how OPD breaks this limitation.

Example 2: Constructing Preference Pairs (Chosen / Rejected)

The first step of OPD training is to construct (prompt, chosen, rejected) triplet data. In real scenarios, chosen/rejected come from human annotation or from pairwise comparisons made by a large Teacher model on multiple candidate answers. Here, synthetic data simulates this data structure to conveniently demonstrate the complete process of subsequent Loss calculation:

"""
Example 2: Constructing Preference Data Pairs
Note: In real scenarios, chosen/rejected come from human annotation or pairwise scoring results from a Teacher model.
      Here, we assume the existence of a "human preference direction" (true_preference_direction).
      Answer vectors closer to this direction represent more preferred answers, used to construct a batch of verifiable Chosen/Rejected samples.
"""
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

PROMPT_DIM = 8
ANSWER_DIM = 4
NUM_PAIRS = 200

true_preference_direction = torch.randn(ANSWER_DIM)
true_preference_direction = true_preference_direction / true_preference_direction.norm()

prompts = torch.randn(NUM_PAIRS, PROMPT_DIM)
base_answers = torch.randn(NUM_PAIRS, ANSWER_DIM)
chosen = base_answers + 0.8 * true_preference_direction    # Closer to human preference direction
rejected = base_answers - 0.8 * true_preference_direction  # Further from human preference direction

print("Preference dataset construction complete:")
print(f"  prompts.shape  = {tuple(prompts.shape)}")
print(f"  chosen.shape   = {tuple(chosen.shape)}")
print(f"  rejected.shape = {tuple(rejected.shape)}")
print(f"  Sample chosen[0]   = {[round(v, 4) for v in chosen[0].tolist()]}")
print(f"  Sample rejected[0] = {[round(v, 4) for v in rejected[0].tolist()]}")

Actual run output:

    Preference dataset construction complete:
      prompts.shape  = (200, 8)
      chosen.shape   = (200, 4)
      rejected.shape = (200, 4)
      Sample chosen[0]   = [0.5363, -1.3464, 0.2815, -1.0671]
      Sample rejected[0] = [-0.3622, -1.1753, 1.5518, -1.3985]

In real scenarios, the construction method for this triplet is usually: feed the same prompt to the Student (or multiple candidate models) to generate several candidate answers, then hand them to a large Teacher model for pairwise comparison. The Teacher outputs "which is better," thereby determining the chosen and rejected.

Example 3: Calculating Preference Loss (Bradley-Terry)

With the (prompt, chosen, rejected) data, the next step is to train a scoring model (which can be understood as a simplified Reward Model) to learn to give higher scores to the chosen:

"""
Example 3: Train a scoring model using Bradley-Terry Loss,
so it learns to distinguish between chosen (higher preference) and rejected (lower preference)
"""
class RewardModel(nn.Module):
    """Input (prompt, answer) concatenated vector, output a scalar score representing the quality/preference level of this answer"""
    def __init__(self, prompt_dim, answer_dim, hidden_dim=32):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(prompt_dim + answer_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1),
        )

    def forward(self, prompt, answer):
        x = torch.cat([prompt, answer], dim=-1)
        return self.net(x).squeeze(-1)


reward_model = RewardModel(PROMPT_DIM, ANSWER_DIM)
optimizer = torch.optim.Adam(reward_model.parameters(), lr=1e-2)

for step in range(200):
    chosen_reward = reward_model(prompts, chosen)
    rejected_reward = reward_model(prompts, rejected)

    # Bradley-Terry preference loss: -log(sigmoid(r_chosen - r_rejected))
    # Intuition: The higher the chosen's score is compared to rejected's, the smaller the loss
    loss = -torch.log(torch.sigmoid(chosen_reward - rejected_reward)).mean()

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (step + 1) % 50 == 0:
        acc = (chosen_reward > rejected_reward).float().mean().item()
        print(f"Step {step+1}: preference_loss={loss.item():.4f}  chosen>rejected accuracy={acc:.2%}")

Actual run output:

    Step 50: preference_loss=0.0021  chosen>rejected accuracy=100.00%
    Step 100: preference_loss=0.0008  chosen>rejected accuracy=100.00%
    Step 150: preference_loss=0.0005  chosen>rejected accuracy=100.00%
    Step 200: preference_loss=0.0003  chosen>rejected accuracy=100.00%

The Loss converges quickly to near 0, indicating the scoring model has fully learned the preference relationship that "chosen should have a higher score than rejected"—this is the exact same Loss used for Reward Model training in DPO and RLHF.

Example 4: Complete Training Loop (Minimal OPD)

The previous three examples demonstrated the "KD baseline," "preference data construction," and "preference Loss calculation" respectively. Now, let's string them together into a truly Online closed loop: Student generates candidates → Teacher scores and ranks → Update Student → Use the updated Student to generate the next round of candidates, and so on.

Here, the Student is modeled as a "policy network": given a prompt, it outputs a Gaussian distribution, and candidate answers are obtained by sampling (using continuous vectors to simulate text generation in real scenarios, avoiding the need for a real LLM to run). The Teacher uses a fixed scoring function to simulate "human preference":

"""
Example 4: Complete Online Preference Distillation Training Loop (Minimal Runnable Version)
Core Structure:
  1. Student samples multiple candidate answers for each prompt (corresponds to "Student generates multiple answers")
  2. Teacher scores the candidate answers (corresponds to "Teacher scoring")
  3. Take the highest score as chosen and the lowest as rejected for each group (corresponds to "Obtain Ranking")
  4. Update Student using Bradley-Terry style Preference Loss (corresponds to "Update Student")
  5. In the next round, use the updated Student to re-generate candidates, and Teacher re-scores them — this is the meaning of "Online"
"""
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

PROMPT_DIM = 8
ANSWER_DIM = 4

# Simulate the "answer direction truly recognized by human/Teacher". In real scenarios, this is the implicit value judgment within the large Teacher model.
true_preference_direction = torch.randn(ANSWER_DIM)
true_preference_direction = true_preference_direction / true_preference_direction.norm()

def teacher_score(prompt, answer):
    """
    Simulate Teacher's scoring of (prompt, answer).
    In real scenarios, this is calling large models like GPT-5/Claude to score or pairwise compare candidate answers.
    Here, the cosine similarity between the answer vector and the "preference direction" simulates the score; higher score means more preferred.
    """
    return F.cosine_similarity(answer, true_preference_direction.unsqueeze(0).expand_as(answer), dim=-1)


class StudentPolicy(nn.Module):
    """
    Student Policy Network: Given a prompt, output the mean and variance of a Gaussian distribution.
    Multiple candidate answers are obtained by sampling, corresponding to "sampling multiple different answers from the same prompt" in a real LLM.
    """
    def __init__(self, prompt_dim, answer_dim):
        super().__init__()
        self.mean_head = nn.Linear(prompt_dim, answer_dim)
        self.log_std = nn.Parameter(torch.tensor(-0.5))

    def forward(self, prompt):
        mean = self.mean_head(prompt)
        std = self.log_std.exp()
        return mean, std

    def sample_and_logprob(self, prompt, num_candidates):
        mean, std = self.forward(prompt)
        mean_exp = mean.unsqueeze(1).expand(-1, num_candidates, -1)
        eps = torch.randn_like(mean_exp)
        # The sampled candidate answers are treated as "established facts already generated," detached to avoid incorrect gradient paths
        candidates = (mean_exp + std * eps).detach()
        # Recalculate the log probability of this batch of candidate answers using the current policy parameters, gradients can flow back normally to mean/std
        log_prob = (
            -0.5 * ((candidates - mean_exp) / std) ** 2
            - torch.log(std)
            - 0.5 * torch.log(torch.tensor(2 * torch.pi))
        ).sum(-1)
        return candidates, log_prob


student = StudentPolicy(PROMPT_DIM, ANSWER_DIM)
optimizer = torch.optim.Adam(student.parameters(), lr=3e-2)

NUM_ROUNDS = 150
NUM_CANDIDATES = 8
BATCH_SIZE = 32
STD_MAX = 1.5  # Reference policy constraint: upper bound for exploration variance,
               # corresponds to the KL penalty term in real RLHF/DPO that "constrains the Student not to deviate too far from the reference policy."
               # Without this constraint, the Student would "cheat" by infinitely expanding the variance, causing training to diverge.

score_history = []
for round_idx in range(NUM_ROUNDS):
    prompts_batch = torch.randn(BATCH_SIZE, PROMPT_DIM)

    # Step1: Student generates multiple candidate answers for each prompt
    candidates, log_probs = student.sample_and_logprob(prompts_batch, NUM_CANDIDATES)

    # Step2: Teacher scores each candidate answer
    flat_candidates = candidates.view(-1, ANSWER_DIM)
    flat_prompts = prompts_batch.unsqueeze(1).expand(-1, NUM_CANDIDATES, -1).reshape(-1, PROMPT_DIM)
    scores = teacher_score(flat_prompts, flat_candidates).view(BATCH_SIZE, NUM_CANDIDATES)

    # Step3: In each group of candidates, the highest score is chosen, the lowest is rejected, obtaining the Ranking
    chosen_idx = scores.argmax(dim=1)
    rejected_idx = scores.argmin(dim=1)
    batch_idx = torch.arange(BATCH_SIZE)
    chosen_log_prob = log_probs[batch_idx, chosen_idx]
    rejected_log_prob = log_probs[batch_idx, rejected_idx]

    # Step4: Calculate Preference Loss and update Student
    loss = -F.logsigmoid(chosen_log_prob - rejected_log_prob).mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    with torch.no_grad():
        student.log_std.clamp_(max=torch.log(torch.tensor(STD_MAX)))

    score_history.append(scores.mean().item())

    if (round_idx + 1) % 15 == 0:
        recent_avg = sum(score_history[-15:]) / 15
        print(f"Round {round_idx+1}: preference_loss={loss.item():.4f}  "
              f"avg_teacher_score (mean of last 15 rounds)={recent_avg:.4f}")

print("OPD training complete: The average Teacher score of Student-generated candidate answers continuously increases with training rounds.")

Actual run output:

    Round 15: preference_loss=0.7223  avg_teacher_score (mean of last 15 rounds)=0.1288
    Round 30: preference_loss=0.8042  avg_teacher_score (mean of last 15 rounds)=0.3134
    Round 45: preference_loss=0.6510  avg_teacher_score (mean of last 15 rounds)=0.4013
    Round 60: preference_loss=0.4437  avg_teacher_score (mean of last 15 rounds)=0.4865
    Round 75: preference_loss=0.3708  avg_teacher_score (mean of last 15 rounds)=0.5650
    Round 90: preference_loss=0.3688  avg_teacher_score (mean of last 15 rounds)=0.6388
    Round 105: preference_loss=0.1544  avg_teacher_score (mean of last 15 rounds)=0.6866
    Round 120: preference_loss=0.3311  avg_teacher_score (mean of last 15 rounds)=0.7620
    Round 135: preference_loss=0.3000  avg_teacher_score (mean of last 15 rounds)=0.8240
    Round 150: preference_loss=0.1034  avg_teacher_score (mean of last 15 rounds)=0.8820
    OPD training complete: The average Teacher score of Student-generated candidate answers continuously increases with training rounds.

This result very intuitively demonstrates the effect of OPD: The Student was never told what the "correct answer" is, only repeatedly informed "who is better than whom in this batch of candidates." After 150 rounds of online iteration, the Teacher score rose from near 0 to 0.88 (the upper limit of cosine similarity is 1). The Student's generation distribution gradually shifted towards the Teacher's preferred direction, which is a direct manifestation of "replicating judgment ability" rather than "replicating answers."

The line STD_MAX in the code is particularly noteworthy: if this constraint is removed, actual testing shows that training diverges after a few dozen rounds—the Student learns to "game" the Preference Loss by infinitely expanding its exploration variance (the larger the variance, the harder it is to accurately assess the relative gap between chosen and rejected, making the Loss easier to decrease), but at the cost of overall generation quality collapsing. This is precisely why a KL constraint of "do not deviate too far from the reference policy" must be added to the Student in real RLHF/DPO training—the STD_MAX in this example is a simplified implementation of this constraint.

Example 5: Real Production Environment Version (Qwen + GPT Ranking + TRL DPOTrainer)

The first four examples used MLPs and continuous vectors to simulate the core logic of OPD, so that the code could run completely without a GPU or API Key, allowing you to witness the convergence process firsthand. But in a real production environment, the Student is an LLM that generates text (like Qwen2.5-7B-Instruct), and the Teacher is a large model that truly makes judgments (like GPT-5). The following code is a reference implementation that can be directly put into a production pipeline:

"""
Production Environment Reference Implementation: Qwen generates candidates -> GPT ranks -> Construct preference data -> TRL DPOTrainer updates Qwen
Dependencies: pip install torch transformers trl vllm openai datasets --break-system-packages
Runtime Requirements: Requires GPU (recommended 24GB+ VRAM), Qwen2.5-7B-Instruct model weights, and a usable OpenAI API Key.
Note: The writing environment for this article had no GPU and no network access to huggingface.co/api.openai.com,
      so this code was not actually executed in the sandbox, but every line is real, usable production code.
      The logic completely corresponds to Example 4 above, just replacing "MLP simulated vectors" with "real text generation and ranking."
"""
import json
from openai import OpenAI
from vllm import LLM, SamplingParams
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig
from datasets import Dataset

STUDENT_MODEL_NAME = "Qwen/Qwen2.5-7B-Instruct"
TEACHER_MODEL_NAME = "gpt-5"

# ---------- 1. Teacher: Call GPT-5 to rank multiple candidate answers generated by Student ----------
teacher_client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

def teacher_rank(prompt: str, candidates: list) -> dict:
    """Let Teacher rank candidate answers, only requiring it to return an array of indices sorted by quality from high to low"""
    numbered = "\n".join(f"[{i}] {c}" for i, c in enumerate(candidates))
    ranking_prompt = (
        f"Please rank the following {len(candidates)} answers by quality from high to low, "
        f"only return a JSON array with elements being the original indices (starting from 0), do not output any other content.\n"
        f"Question: {prompt}\n{numbered}"
    )
    resp = teacher_client.chat.completions.create(
        model=TEACHER_MODEL_NAME,
        messages=[{"role": "user", "content": ranking_prompt}],
        temperature=0,
    )
    order = json.loads(resp.choices[0].message.content)
    return {"best_idx": order[0], "worst_idx": order[-1]}


# ---------- 2. Student: Use vLLM to batch generate multiple candidate answers ----------
student_llm = LLM(model=STUDENT_MODEL_NAME)
sampling_params = SamplingParams(n=4, temperature=0.8, max_tokens=512)

def student_generate(prompts: list) -> list:
    """For a batch of prompts, sample 4 candidate answers each"""
    outputs = student_llm.generate(prompts, sampling_params)
    return [[o.text for o in out.outputs] for out in outputs]


# ---------- 3. Construct preference dataset: each prompt corresponds to a (chosen, rejected) pair ----------
def build_preference_dataset(prompts: list) -> Dataset:
    all_candidates = student_generate(prompts)
    records = []
    for prompt, candidates in zip(prompts, all_candidates):
        rank = teacher_rank(prompt, candidates)
        records.append({
            "prompt": prompt,
            "chosen": candidates[rank["best_idx"]],
            "rejected": candidates[rank["worst_idx"]],
        })
    return Dataset.from_list(records)


# ---------- 4. Use TRL's DPOTrainer to update Student on the preference dataset ----------
# Note: Specific parameter names for DPOConfig/DPOTrainer may change with trl version iterations. Please check the official documentation for your installed version before use.
tokenizer = AutoTokenizer.from_pretrained(STUDENT_MODEL_NAME)
policy_model = AutoModelForCausalLM.from_pretrained(STUDENT_MODEL_NAME)
ref_model = AutoModelForCausalLM.from_pretrained(STUDENT_MODEL_NAME)  # Reference policy, used for KL constraint

dpo_config = DPOConfig(
    output_dir="./opd_output",
    per_device_train_batch_size=2,
    learning_rate=5e-6,
    beta=0.1,               # Corresponds to the role played by STD_MAX in Example 4: constrains Student not to deviate too far from the reference policy
    num_train_epochs=1,
)

seed_prompts = [
    "How to learn Python?",
    "Please explain what overfitting is.",
    "Write a short poem about autumn.",
]
preference_dataset = build_preference_dataset(seed_prompts)

trainer = DPOTrainer(
    model=policy_model,
    ref_model=ref_model,
    args=dpo_config,
    train_dataset=preference_dataset,
    tokenizer=tokenizer,
)
trainer.train()

This code corresponds completely to Example 4: student_generate corresponds to "Student generates multiple answers," teacher_rank corresponds to "Teacher scores and ranks," build_preference_dataset corresponds to "Obtain Ranking and construct Chosen/Rejected," and the DPOTrainer internally implements the Bradley-Terry Preference Loss discussed in Chapter 10. The beta parameter corresponds to the role played by STD_MAX in Example 4, which is "preventing deviation from the reference policy." Wrapping this pipeline in a while loop, and after each round of training, re-generating candidates with the latest Student, creates a truly Online version.

12. Why is the Industry Starting to Embrace OPD?

The previous chapters discussed the technical principles of OPD. This chapter answers a more practical question: Why are companies like OpenAI and DeepSeek shifting from "continuing to invest in RLHF" to "starting to do OPD"? Based on the direction of public industry discussions, the reasons can be roughly summarized into five points.

First, the cost of RLHF is getting higher. The complete RLHF process requires training a Reward Model and then running PPO reinforcement learning. The entire pipeline has high engineering complexity and unstable training, with significant debugging costs and computational overhead. This is especially true as the model matrix grows larger (one Teacher needs to produce several Students of different sizes), making it unrealistic to re-run the full RLHF for each size.

Second, high-quality human preference data is becoming increasingly scarce. Early RLHF relied heavily on human annotators making pairwise comparisons of answers. But as model capabilities improve, ordinary annotators struggle to reliably distinguish "which answer is better" (especially in specialized scenarios like coding and math). The marginal cost of high-quality preference annotation is continuously rising, while supply is declining.

Third, the Teacher model itself is already strong enough. When the judgment of models like GPT-5 and Claude approaches or even surpasses that of average human annotators, instead of spending heavily to re-collect human preference data, it's better to directly let this already strong Teacher do the ranking judgment—it is itself a ready-made, scalable "preference source."

Fourth, top-tier large models are themselves an implicit Reward Model. The step of specifically training a Reward Model in RLHF is essentially about wanting a "model that can score." Today's strongest large models, after their own RLHF/DPO training, have internal preference judgments highly aligned with humans. Using them directly as judges eliminates the need to train a separate Reward Model.

Fifth, the number of Students is increasing, and their sizes are getting smaller. One Teacher often needs to distill a whole set of Students like 70B, 32B, 14B, 7B, 3B. If each size had to individually go through the complete process of human annotation plus RLHF, the cost would grow linearly or even super-linearly with the number of Students. OPD only requires the Teacher to continuously provide ranking signals, making the cost of large-scale Student replication much lower.

Combining these five points yields the core engineering logic behind the industry's embrace of OPD:

The Teacher directly teaches judgment to the Student, eliminating the need to redo a round of human preference annotation for every single Student. This is not a purely technical choice, but an engineering trade-off made between cost, scalability, and effectiveness.

13. What if the Teacher is GPT-5?

In real engineering, the roles of Teacher and Student are very clearly divided:

The sole responsibility of the Teacher in OPD is Ranking—it does not need to be deployed in a low-latency online environment, nor does it need to participate in the Student's backpropagation. It only needs to provide a judgment of "who is better" for the candidates generated by the Student during the training phase. This means:

The Teacher can be very large and very slow, because it is only called during the offline/semi-online training phase;
The Student can be very small, focused on "internalizing" the Teacher's judgment into its own generation preferences;
Even if the Teacher is a closed-source API (like GPT-5), as long as its ranking/scoring results can be obtained, the entire OPD process can be driven. This is why teacher_rank in Example 5 directly calls the OpenAI API.

14. Real Industrial Case: GPT-5 Teaches Qwen3-14B

The previous chapters discussed principles and code. This chapter pieces them together into a real industrial case specific enough to "directly plan a project schedule against."

Scenario Setup:

Teacher: GPT-5—powerful, but high inference cost, high latency, impossible to deploy at scale in one's own product;
Student: Qwen3-14B—slightly weaker than GPT-5, but inference cost is a fraction of GPT-5's, fully deployable, with controllable latency and cost;
Goal: Enable Qwen3-14B to approximate GPT-5's "judgment" and response style as closely as possible while maintaining low cost.

The complete engineering process is as follows:

Why is Qwen deployed in the end, and not GPT-5? The reasons are direct and straightforward:

GPT-5 is too expensive: Charged per token, in high-concurrency business scenarios, the call cost grows linearly with request volume, becoming a huge expense in the long run;
GPT-5 has high latency and is uncontrollable: As a closed-source API, response time is affected by the other party's service load. Enterprises cannot perform fine-grained latency optimization or private deployment;
Qwen3-14B is cheap enough and controllable enough: It can be deployed on one's own GPU cluster, using inference frameworks like vLLM to push latency very low, with the cost structure completely in one's own hands;
GPT-5's value here is not "being deployed," but "acting as a referee once": It only needs to rank the candidates generated by Qwen during the training phase. After training, it can completely exit the system—its judgment has been "taught" to Qwen.

This is the simplest and most compelling value of OPD in real engineering: Use a one-time, offline/semi-online Teacher call cost to exchange for a Student that can run at low cost long-term. GPT-5 never appears in the final product chain; what appears in the product chain is always the cheaper, faster Qwen3-14B—except this Qwen3-14B, after OPD training, carries the shadow of GPT-5 in its response style and judgment.

And this chain does not stop at Qwen3-14B.

In the past, humans taught large models what a "good answer" is—human annotators wrote preference labels one by one, training the Reward Model in RLHF. Now, large models have begun teaching other large models what a "good answer" is—GPT-5 teaches its judgment to Qwen, and Qwen can continue to pass this judgment down to even smaller 3B, 1.5B models:

flowchart TD
    H[Human Preference Annotation] --> G[GPT-5]
    G -->|OPD| Q[Qwen3-14B]
    Q -->|OPD| M[3B]
    M -->|OPD| S[1.5B]

With each step down this chain, the model's size and cost decrease exponentially, but the judgment of "what is a good answer" is preserved through the step-by-step transfer. This is the most intuitive form of Preference Transfer—it is not a one-time teacher-student relationship, but a transfer chain that can continuously extend downwards.

15. The Relationship Between OPD, RLHF, and DPO (Key Section)

Combined with the evolution history in Chapter 2, let's clarify the relationship between RLHF, DPO, and OPD at the loss function level:

flowchart TD
    A[Pretrain] --> B[SFT]
    B --> C[RLHF: Train Reward Model + PPO]
    C --> D[DPO: Directly Learn Preference]
    D --> E[OPD: Distill Learned Preference]

RLHF: First train an independent Reward Model to fit human preferences, then use a reinforcement learning algorithm like PPO to make the policy model maximize this Reward—essentially "training a teacher who can score."
DPO: Skip the explicit Reward Model and PPO, directly optimize an equivalent objective function on preference data pairs, achieving effects similar to RLHF with a simpler supervised learning method.
OPD: Assume the "scoring/ranking ability" already exists in a stronger Teacher model (whether this Teacher was trained via RLHF or DPO), and directly let the Student learn this "ranking ability" from the Teacher online.

Summarize the relationship between the three in one sentence:

RLHF trains the teacher; DPO is a more efficient way for the teacher to learn to score; OPD is the teacher teaching this "scoring ability" to the student.

16. Advantages and Limitations of OPD

Advantages:

Better Alignment effect—directly optimizes the goal of "human preference," rather than just "correctness."
Higher safety—preference signals can naturally encode "whether it is safe and compliant" into the Ranking.
More aligned with Human Preference—the Student learns the relative judgment of "what is a good answer," rather than memorizing a specific answer.
The Student is more likely to surpass models trained with traditional KD on subjective dimensions like style and tone.

Limitations:

High Teacher cost—each round requires the Teacher to participate in scoring/ranking. Especially when the Teacher is a large model API, the call cost grows linearly with the number of training rounds.
High Ranking cost—high-quality ranking judgment is itself a non-trivial task. Especially when subjective preferences are involved, judgments from different Teachers (or even the same Teacher at different times) may be inconsistent.
High online training engineering complexity—the loop of "generate candidates—score—update parameters" needs to be engineered, which is much more complex than an offline distillation pipeline.
Preference drift is easy—without constraint mechanisms like STD_MAX in Example 4 or beta in Example 5 of this article, the Student's generation distribution might deviate from a reasonable range to "game" the Preference Loss. This is why real systems introduce some form of KL constraint or reference policy regularization.

17. Summary

Traditional distillation answers:

How to transfer knowledge to a small model?

OPD answers:

How to transfer the ability to "judge what is a good answer" to a small model?

Summarize the entire article in one sentence:

Knowledge distillation solves "making the small model know more," while preference distillation solves "making the small model judge better." As large models enter the Alignment era, the real competitive advantage between models is shifting from knowledge reserves to value judgment capabilities, and OPD is the engineering implementation of this capability transfer.

Folks, from the runnable code in Chapter 11 of this article, you can see an intuitive conclusion: a Student that was never told the "standard answer," merely by continuously receiving relative signals like "who is better in this batch of candidates," was able to significantly shift its generation preferences towards the Teacher after 150 rounds of online iteration (Teacher score rose from near 0 to 0.88). Combined with the technology evolution history in Chapter 2 and the industry motivations in Chapter 12, it's clear that this is not an isolated technical choice, but an inevitable step in the current progression of the Alignment main thread—the competition among large models is shifting from "who knows more" to "whose judgment can be replicated more efficiently."

OK, that's all for today. If it was helpful, feel free to like, follow, and leave a comment. See ya!!!