OPD: How Large Models Now Distill Judgment, Not Just Answers
RLHF pipelines are expensive and brittle, and high-quality human preference data is drying up. OPD offers a cheaper, scalable path to alignment by turning an existing strong model into a reusable judge, making it practical to align a whole fleet of smaller, deployable models.
Traditional knowledge distillation copies a Teacher model's output probabilities, transferring facts but not taste. OPD changes the signal: the Teacher ranks multiple candidate answers from the Student, and the Student learns to maximize the probability of that ranking. This shifts the training objective from fitting a static token distribution to learning a relative preference relationship.
The process is online because the Student generates new candidates after each parameter update, and the Teacher re-scores them in a co-evolving loop. A full PyTorch implementation shows a simulated Student's Teacher score rising from near zero to 0.88 over 150 rounds, without ever seeing a correct answer. A production-ready example uses Qwen as the Student and GPT-5 as the ranking Teacher, with TRL's DPOTrainer handling the Bradley-Terry loss.
Industry adoption is driven by cost: running full RLHF for every Student size is impractical. OPD reuses a single strong Teacher's judgment to train an entire family of smaller models, cutting the per-model alignment cost. The technique also enables preference transfer chains, where a 14B model trained via OPD can itself become the Teacher for a 3B model.
OPD reframes alignment as a ranking problem, which is a better fit for subjective quality than the point-estimate approach of traditional distillation.
The online loop creates a co-evolution dynamic that can amplify small initial preference signals into strong behavioral shifts.
Preference transfer chains suggest a future where alignment is a cascading, automated process rather than a one-off human annotation effort per model size.
The Bradley-Terry loss is the common mathematical thread linking RLHF, DPO, and OPD; the innovation is in who provides the ranking and when.