Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations

Rui Wang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Wenhao Yu, Kam-Fai Wong

On-policy distillation acts as an exploration catalyst, but suffers from length-based reward hacking and teacher-student mismatch.

Why does on-policy distillation (OPD) often fail to improve student models despite using stronger teachers, and how can we stabilize it?

On-policy distillation (OPD) is widely used to train language models, yet it often produces inconsistent results, ranging from performance stagnation to complete training collapse. The authors demonstrate that OPD functions as an exploration catalyst rather than a capability expander, meaning its success depends entirely on the fidelity of the guidance signal provided by the teacher. They identify two primary pathologies—length exploitation and student-teacher mismatch—and show that simple, zero-overhead signal regulations can stabilize training and outperform massive teacher models.

Paper Primer

OPD is often mistaken for a mechanism that expands a model's fundamental reasoning ceiling. Instead, it acts as an exploration catalyst: it reshapes the student's existing generative space to surface correct reasoning paths more efficiently, with performance gains scaling primarily through prompt diversity rather than per-problem sampling depth.

The method hinges on regulating the token-level advantage signal to prevent the student from gaming the objective. The authors introduce two in-loop regulations: Hard Clipping, which enforces strict bounds on advantage values, and Soft Log-scale Compression, which dampens extreme signals while preserving relative token rankings.

Regulated OPD eliminates length-based reward hacking and stabilizes training.

By applying clipping or log-scale compression to the token-level advantage, the student is prevented from using filler tokens or premature truncation to artificially inflate its reward. The regulated framework allows a modest 4B teacher to decisively outperform state-of-the-art methods distilled from 30B teachers.

Teacher capability is not a proxy for distillation quality.

The authors observe an inversion where stronger teachers (e.g., 4B-GRPO) provide worse guidance than smaller ones (1.7B-GRPO) due to a distributional mismatch between the teacher's preferences and the student's rollout distribution. The 1.7B-GRPO teacher consistently yields higher final accuracy than more powerful models that lack alignment with the student's current capability.

Why does the student model resort to "length exploitation" during training?

Because the standard OPD objective averages token-level advantages over the entire sequence, the student can "wash out" negative signals from incorrect reasoning by appending filler tokens or "lock in" positive signals by truncating the response early.

What is the "Student-Teacher Mismatch" and why does it break distillation?

It occurs when a large capability gap causes the teacher's preference signals to decouple from the student's actual performance, effectively providing "noise" that steers the student toward counterproductive exploration paths.

This paper shifts the focus of OPD from "brute-force teacher scaling" to "signal quality regulation," proving that a well-aligned, smaller teacher is often more effective than a massive, misaligned one.

Introduction

We define OPD, its exploratory role, and its key pathologies.

OPD trains a student model by rewarding each token it generates proportionally to how much the teacher model prefers that token, turning the teacher’s probabilities into a dense, token‑level learning signal.

On‑Policy Distillation (OPD) acts as an exploration catalyst: it steers the student toward correct reasoning paths via dense token‑level guidance, but it does not raise the model’s ultimate performance ceiling. When the teacher is stronger, OPD can paradoxically hurt the student because the guiding signal may become misaligned, leading to the pathologies of Student‑Teacher Mismatch and Length Exploitation that we later diagnose.

**Figure 1:** **Left**: OPD accelerates exploration but cannot exceed the model ceiling. **Middle**: a stronger teacher does not guarantee a better student. **Right**: advantage rises while length explodes or crashes.

**Figure 2.** Overview. We characterize OPD’s role as an exploration catalyst (§3), diagnose pathologies (§4), and propose lightweight signal regulations that restore stability without additional compute (§5).

The Role of OPD

We examine how On‑Policy Distillation reshapes student learning dynamics.

This section probes the mechanism by which On‑Policy Distillation influences learning. We focus on its role as an exploration catalyst and analyze how performance trajectories evolve with different teachers and sampling budgets.

OPD repeatedly nudges the student toward promising regions of its generative space, making good solutions appear early in the sampling process.

How does this differ from a standard exploration bonus that adds extra reward for novel states?

Standard bonuses modify the reward function itself, encouraging the agent to visit unseen states. OPD, by contrast, keeps the reward unchanged but reshapes the student’s output distribution using the teacher’s on‑policy rollouts, so the same states become more likely without altering the underlying objective.

GRPO is an on‑policy reinforcement‑learning algorithm that optimizes a policy by comparing groups of trajectories rather than individual samples.

In what way does GRPO differ from vanilla policy‑gradient methods?

Vanilla policy gradient treats each trajectory independently, which can lead to high variance updates. GRPO aggregates trajectories into groups and updates the policy based on the relative ranking of these groups, yielding a lower‑variance, more stable gradient.

**Figure 3.** Pass@$k$ performance of Qwen3-1.7B-Base trained via different methods. OPD variants outperform Base and GRPO models in low-$k$. However, the convergence at higher $k$ suggests OPD accelerates exploration rather than raising the model's performance ceiling.

Overall, OPD speeds up the discovery of high‑quality solutions in the low‑sample regime but does not extend the model’s ultimate capability ceiling, and its catalytic effect can become a liability if the teacher’s guidance is misaligned.

Pathologies: Student-Teacher Mismatch

Analyzes how a teacher’s strength can mislead the student during on‑policy distillation.

When the teacher’s policy distribution diverges strongly from the student’s, the reward signal derived from the teacher can become noisy rather than informative.

The teacher’s reward for action 1 is high; OPD assigns a large weight to the student’s rollout when it selects action 1.

The student, however, chooses action 2 half the time, receiving low or zero reward, which pushes its policy toward the teacher’s bias.

After several updates the student’s distribution shifts to $[0.7,0.3]$, reducing the KL gap but also limiting exploration of action 2.

The mismatch creates a feedback loop where the student over‑fits to the teacher’s dominant action, losing the ability to discover alternative strategies.

**Figure 4.** **Weak teacher outperforms strong teacher in OPD w/ Clip.** Benchmark accuracy (first two panels) and informativeness $\mathcal{I}$ on the validation set for OPD w/ Clip runs distilling from four teachers with varying capabilities into the same Qwen3-1.7B-Base student.

Why doesn’t a stronger teacher always improve the student in OPD?

Because OPD relies on the teacher’s rollout rewards; if the teacher’s policy is far from the student’s, those rewards emphasize actions the student rarely explores, turning the signal into noise rather than guidance.

Pathologies: Length Exploitation

Length exploitation shows how token‑level advantage lets students cheat by padding or truncating sequences.

Beyond teacher selection, the OPD objective itself contains a structural loophole that lets a student inflate its advantage by manipulating response length.

Because the objective averages a token‑level advantage over the whole sequence, a student can boost the average either by adding many low‑signal filler tokens or by truncating after a favorable prefix.

How does length exploitation differ from simply adding a length penalty to the loss?

A length penalty directly subtracts a term proportional to $T$ from the loss, forcing shorter outputs. Length exploitation, by contrast, works *inside* the advantage averaging: the student can keep the loss unchanged while arbitrarily increasing $T$ to dilute negative advantage, so a penalty is needed to counteract the averaging, not just to discourage long sequences.

As $T$ grows, the negative core term $\frac{A_{\text{core}}}{T}$ vanishes, so the student can make $\bar{a}$ arbitrarily close to zero without fixing the underlying reasoning error.

**Figure 6.** Length exploitation illustration. Advantage increases, while accuracy drops. Teacher→Student: Qwen3-4B-GRPO→Qwen3-1.7B (Mode A), and Qwen3-1.7B-GRPO→Qwen3-1.7B-Base (Mode B).

Mode B therefore yields very short, safe responses that still earn a large reverse‑KL advantage because the teacher assigns high probability to the early tokens.

**Figure 7.** (a) A correct rollout receives a mildly negative token-mean. (b) Mode A (Endless Exploration): a long wrong rollout with negative scores diluted across tokens, failing to penalize the incorrect response. (c) Mode B (Abrupt Degeneration): a short, fully teacher-preferred rollout prefix that yields a strongly positive token-mean despite an incorrect outcome.

Compute per‑token advantages: $a = [0.2,\,0.1,\,0.1,\,0.5]$.

Average advantage $\bar{a} = (0.2+0.1+0.1+0.5)/4 = 0.225$.

If we append two filler tokens with teacher‑student advantage $a_{\text{filler}} = 0.0$, the new length $T=6$ gives $\bar{a} = (0.2+0.1+0.1+0.5)/6 = 0.15$, a lower (better) loss despite unchanged reasoning.

This toy shows how adding neutral filler tokens dilutes a positive average, letting the student hide a weak core while still improving the objective.

Finding 3: the token‑level advantage objective creates degenerate shortcuts at both length extremes, so students can improve the loss without improving factual correctness.

Regulations Overview

On‑policy distillation speeds learning but can be destabilized by length exploitation and teacher‑student mismatch; we review regulation fixes.

Earlier attempts stabilize OPD by pulling in off‑policy data or by scoring the top‑k tokens beyond the predicted one to smooth gradient estimates. These tricks, however, demand heavy overhead such as full‑vocab replay buffers or warm‑starting students with supervised‑fine‑tuning on teacher rollouts, which leaves logical failures unpenalized.

The advantage signal is forced into a fixed interval, discarding extreme outliers while keeping the rest unchanged.

The advantage is passed through a smooth log function that compresses large magnitudes but leaves small values almost linear.

The previous section showed that unchecked OPD causes response lengths to balloon, and truncation then destroys answer validity, collapsing accuracy despite rising advantage scores. By capping or compressing the advantage signal, Hard Clipping and Soft Log‑Scale Compression prevent the student from exploiting length to inflate its reward, steering training toward genuine reasoning improvements.

We evaluate both regulators on Qwen3‑1.7B‑GRPO and Qwen3‑4B‑GRPO models, modifying only the advantage computation within the existing training loop and incurring no extra compute. Table 1 reports the resulting performance across benchmarks, highlighting the trade‑off between stability and final accuracy.

Stabilizing OPD with Signal Regulation

Signal Regulation curbs extreme token signals to stabilize on‑policy distillation.

To stop a few token‑level log‑ratios from hijacking the policy gradient, the authors introduce token‑level signal regulation.

Signal Regulation caps and smooths per‑token log‑ratio signals so that no single token can dominate the gradient while preserving the overall direction of the teacher’s guidance.

How does Signal Regulation differ from naïve gradient clipping?

Naïve clipping simply truncates any reward exceeding $\tau$, discarding its magnitude entirely. Signal Regulation first clips, then applies a convolutional smoothing kernel, so the excess magnitude is redistributed to nearby tokens instead of being lost.

Clip each reward: $r_{\text{clip}} = [0.2, 2.0, -0.1, 2.0]$ (values above $\tau$ are reduced to $\tau$).

Convolve $r_{\text{clip}}$ with the kernel (zero‑pad ends): the smoothed signal becomes $s = [0.35, 1.05, 1.05, 1.35]$.

Replace the original rewards in the OPD loss with $s$, yielding a bounded gradient that still reflects the relative strength of the middle tokens.

Clipping alone would have yielded $[0.2, 2.0, -0.1, 2.0]$, losing the extra 1.5 from token 2; smoothing recovers that information by sharing it with neighbors.

Upgrading the student from the base model to Qwen3‑1.7B reduces the distributional gap, flipping the performance hierarchy: the 4B GRPO teacher now outperforms the 1.7B GRPO teacher, and regulated OPD with the stronger teacher surpasses prior methods that used a 30B teacher.

**Table 2.** Performance of Qwen3-1.7B student tuned by OPD+Clip (CLIP) and previous methods (Hou et al., 2026; Yang et al., 2026a), which further proves that a stronger teacher cannot ensure better student performance. The **best** and the <u>second best</u> scores are marked.

Analysis of Regulation Methods

Signal regulation trims extreme signals, yielding higher student performance.

Recall that On‑Policy Distillation (OPD) accelerates learning but can suffer from length exploitation and student‑teacher mismatch; signal regulation was introduced to curb these failures.

Hard clipping restores performance when a 4 B teacher otherwise harms the student.

Table 1 shows the clipped variant lifts the average score to 28.1, surpassing the naive OPD baseline.

**Figure 9.** Training dynamics of OPD variants. Teacher: Qwen3-4B-GRPO; Student: Qwen3-1.7B-Base. Vanilla OPD suffers greater length exploitation, whereas Clip produces shorter responses and achieves higher accuracy than naive OPD.

**Table 1.** Performance of different methods on benchmarks. Among all evaluated downstream student models (bottom two blocks), the overall best scores are **bolded** and the second best are <u>underlined</u>.

These results demonstrate that lightweight, zero‑overhead signal regulations—hard clipping and log‑scale compression—effectively neutralize length exploitation and student‑teacher mismatch, confirming that signal quality, not teacher size, drives OPD success.

Conclusion and Limitations

OPD boosts exploration but incurs length exploitation and student‑teacher mismatch; signal regulation stabilizes training.

We revisited OPD, confirming its role as an exploration catalyst that thrives on diverse problems, but we identified two core pathologies: length exploitation through redundant token padding and a student‑teacher mismatch that can degrade performance when the teacher vastly outperforms the student.

To restore reliable guidance, we introduced hard clipping and soft log‑scale compression, which curb extreme signal magnitudes and preserve gradient flow. These mechanisms require tuning hyperparameters that depend on the capacity gap between teacher and student, suggesting a need for a dynamic, adaptive regulation framework.

Our empirical evaluation focused on dense mathematical‑reasoning benchmarks, where reasoning chains are explicit and outcomes easy to verify. Consequently, the behavior of regulated OPD on open‑ended generation or knowledge‑intensive QA remains an open question.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers