DOPD: Dual On-Policy Distillation
Xinlei Yu, Gen Li, Qingyi Si, Guibin Zhang, Yuqi Xu, Congcong Wang, Shuai Dong, Kaiwen Tuo, Xiangyu Zeng, Kaituo Feng, Qunzhong Wang, Yang Shi, Xiaobin Hu, Xiangyu Yue, Jiaqi Wang, Shuicheng Yan
DOPD dynamically routes token-level supervision between teacher and student policies to prevent privilege illusion.
How can we improve student model performance in on-policy distillation by mitigating the information asymmetry between privileged teachers and students?
On-policy distillation (OPD) often uses privileged information to boost student performance, but this frequently triggers "privilege illusion," where students learn to mimic information-asymmetry shortcuts rather than acquiring genuine transferable capabilities. DOPD (Dual On-Policy Distillation) solves this by calculating a "privilege advantage gap" for every token, then dynamically routing supervision between the teacher and the student itself based on their relative confidence and performance discrepancy. This adaptive approach consistently outperforms vanilla OPD, recovering nearly 90% of the teacher-student performance gap on LLM benchmarks and delivering significant gains in multimodal settings.
Paper Primer
The core mechanism hinges on partitioning the token space into four regimes based on the privilege advantage gap ($A$) and the relative probabilities of the teacher ($q_T$) and student ($q_S$). By comparing these metrics against batch averages, the system selects the optimal distillation objective—ranging from full-vocabulary JS divergence for high-value teacher signals to weak self-regularization for uncertain tokens—to ensure the student internalizes core competencies rather than superficial cues.
DOPD significantly narrows the teacher-student performance gap across diverse tasks.
Across eight LLM benchmarks, DOPD achieved a 12.3-point average gain, recovering 89.8% of the original teacher-student performance gap. Outperforms vanilla OPD by 7.5 points on LLM tasks and 6.0 points on VLM tasks.
The method maintains robustness and scalability across varying model size ratios.
In the largest scale-mismatch setting (Qwen3-8B to Qwen3-0.6B), DOPD achieved a 14.1-point gain compared to vanilla OPD's 3.5-point gain. Consistent improvements ranging from 6.2–10.6 points across five different model pairs.
Why does adding privileged information to both teacher and student fail to solve the "privilege illusion" on its own?
Uniformly distilling all tokens from a monolithic source causes the student to passively adapt to privileged inputs rather than internalizing transferable abilities, leading to entropy collapse and shortcut learning.
How does the "privilege advantage gap" distinguish between genuine capability and information asymmetry?
A large gap indicates a genuine capability discrepancy under identical privileged conditions, whereas a small gap suggests the teacher's advantage is primarily attributable to the privileged information itself.
The effectiveness of privileged information in this paradigm relies on providing capability-oriented guidance (e.g., step-wise hints) rather than final answers, which the authors found can actually degrade performance by encouraging rigid overfitting.
Researchers should move away from uniform, monolithic distillation objectives when using privileged data; dynamic, advantage-aware routing is essential to ensure students learn transferable skills rather than fitting privileged shortcuts.
Motivation and Problem Framing
Identifies why standard on‑policy distillation falters when teachers use privileged data.
Standard on‑policy distillation (OPD) promises efficient capacity transfer by supervising a student on its own sampled trajectories, yet this promise collapses when the teacher relies on privileged information inaccessible to the student.
OPD lets a student generate its own experience, then asks a stronger teacher to label each token of that trajectory, providing dense, on‑policy supervision.
Privileged information is data available to the teacher (e.g., external reasoning hints or annotated visuals) that the student cannot observe during its own roll‑outs.
Because supervision signals vary wildly across tokens—only a handful encode decisive reasoning—optimizing every token with the same privileged teacher signal amplifies the privilege illusion.
We therefore introduce Dual On‑Policy Distillation (DOPD), which dynamically selects, for each token, whether to follow the privileged teacher or a self‑supervised student signal based on the advantage gap and relative probabilities.
The core failure of standard OPD is that privileged teachers inject an information‑asymmetry gap, causing students to chase unattainable signals rather than genuine capability improvements.
Distillation Paradigms and Baselines
Background on distillation paradigms and the impact of privileged information.
Teacher‑student distillation has become a standard way to transfer capability from a stronger model to a weaker one, especially when the student generates its own rollouts for supervision.
In vanilla on‑policy distillation the student samples trajectories, the teacher evaluates the same states, and the student is penalized for deviating from the teacher’s action distribution.
Introducing privileged information—extra cues available only to the teacher or the student—creates an asymmetry that can inflate performance without improving true capability.
**Figure 2.** Comparison of existing (a) standard distillation, (b) self distillation, and (c) adaptive distillation paradigms with our proposed (d) dual distillation paradigm.
**Figure 3.** Comparison of (a) performance and (b) entropy on OPD variants with privileged information. Here, T., S., and Priv. denote teacher policy, student policy and with privileged information, respectively.
Beyond the dual objective, OPD research explores three divergence‑based losses: forward KL (covers the teacher’s full support), reverse KL (focuses on the teacher’s modes), and Jensen‑Shannon (balances both sides).
The DOPD Mechanism
DOPD dynamically routes distillation objectives based on token-level privilege advantage to isolate transferable capability from privileged shortcuts.
Standard distillation often fails because it treats all teacher signals as equally valid, inadvertently transferring privileged shortcuts alongside genuine capability. Dual On-Policy Distillation (DOPD) resolves this by dynamically routing supervision based on the token-level privilege advantage gap, ensuring the student only imitates the teacher when a credible capability advantage exists.
DOPD is a selective distillation framework that uses a privileged student and teacher to measure the "advantage" of the teacher's signal at every token, then routes the student to either imitate the teacher, anchor to its own privileged counterpart, or explore independently.
**Figure 5.** Overview of our proposed DOPD.
Advantage-aware filtering prevents the student from overfitting to teacher signals that arise solely from privileged information asymmetry, rather than superior reasoning capability.
The condition $(A_n \ge \bar{A}) \land (q_T \ge q_S)$ evaluates to True, triggering the IHT regime.
The system selects the full-vocabulary JS divergence objective $L_{\text{HT}}$.
The student policy is updated to align its full probability distribution with the teacher's, capturing both the teacher's primary decision and its secondary preferences.
By identifying this specific regime, DOPD isolates tokens where the teacher's privileged access provides a clear capability signal, allowing for dense supervision without the risk of transferring shortcuts from other regimes.
Experimental Setup
Details of models, data, baselines, and training settings for DOPD experiments.
All experiments use the Qwen3 family (non‑thinking variants) as both teacher and student policies. The primary LLM pair is Qwen3‑8B (teacher) to Qwen3‑1.7B (student), while the VLM pair is Qwen3‑VL‑8B to Qwen3‑VL‑2B. Additional smaller pairs test scalability.
Training data for LLM‑based OPD combine high‑quality mixtures from RaR‑Science‑20K, DAPO‑Math‑17K, and Skywork‑OR1‑Coding‑14K, covering general, reasoning, and coding tasks. VLM‑based OPD uses the ViRL39K dataset, which spans general, visual‑reasoning, and visual‑understanding tasks.
Privileged inputs are generated by GPT‑5.4 (2026‑03‑05). For LLM tasks the model produces step‑wise decomposition hints (verified rationales) without exposing execution traces or final answers. For VLM tasks it creates structured visual annotations: object labels with bounding‑box coordinates.
We evaluate on eight LLM benchmarks (C‑Eval, LiveBench, MATH500, AIME25, ZebraLogic, AutoLogi, BFCLv3, LCBv5) and eight VLM benchmarks (RealWorldQA, MMStar, MathVision, DynaMath, LogicVista, MMMU, MMMU‑Pro, VSI‑Bench), each using its official metric.
Baselines span three OPD paradigms. Standard OPD includes Vanilla OPD, OPCD, ExOPD, and Uni‑OPD. Self‑distillation covers SDFT, OPSD, and SDPO. Adaptive distillation comprises EOPD and TIP. For VLM we also compare against Vision‑OPD and VA‑OPD∗.
All runs use 8 NVIDIA H200 141 GB GPUs. The teacher policy is frozen during distillation; the student is optimized with AdamW and a cosine learning‑rate schedule (5 × 10⁻⁶). Batch sizes are 128 (LLM) and 64 (VLM) with four rollout samples per instance, a Top‑K of 128, and distillation weights $\beta_{w}$=0.3, $\beta_{l}$=0.6.
**Figure 12.** Demonstrations of VLM-based privileged input.
**Table 1.** Performance comparison of different methods across various benchmarks, including General, Visual Reasoning, and Visual Understanding categories, with the average performance shown in the final column.
**Table 2.** Performance comparison of our proposed DOPD with counterparts on general, visual reasoning, and visual understanding tasks. *The codes of VA-OPD are not officially released, so we use the results of our reproduced version.
Main Performance Results
DOPD’s gap‑closing gains and benchmark superiority are quantified.
DOPD narrows the student‑teacher performance gap by 12.3 points, recovering 89.8 % of the original gap.
Table 1 reports the average gap reduction across eight benchmarks.
**Figure 1.** Performance comparison of our DOPD with competing approaches across eight benchmarks in terms of average across all benchmarks (upper bigger bars) and individual values of each benchmark (lower small bars).
Ablations and VLM Analysis
Ablation studies dissect each component’s impact, confirming DOPD’s robustness and scalability.
We now probe DOPD’s components through targeted ablations, measuring how each removal affects student performance.
**Figure 4.** Token ablations on random tokens, and tokens with high or low advantage gap.
DOPD improves the student policy by a 10.1‑point absolute gain over the baseline.
Table 2 reports a 10.1‑point improvement relative to the student‑only policy.
DOPD recovers 69.2 % of the teacher‑student advantage gap.
Table 2 lists a 69.2 % gap‑recovery figure for the proposed method.
DOPD outperforms Vanilla OPD by 6.0 points on average across VLM‑based tasks.
Comparisons in Table 2 show a 6.0‑point lead over Vanilla OPD.
On the largest teacher‑student mismatch (8B → 0.6B) DOPD gains 14.1 points, versus Vanilla OPD’s 3.5‑point gain.
Scalability results in Figure 6 (panel a) report these gains.
In the same mismatch DOPD reduces the teacher‑student gap by 53.0 %.
Figure 6 (panel b) quantifies the gap‑reduction ratio.
**Figure 6.** Scalability comparison of proposed DOPD and Vanilla OPD on (a) performance gain and (b) teacher-student gap reduction ratio. Here, the solid and dashed lines represent the 0.6B and 1.7B student policy, respectively.
**Figure 7.** Comparison of proposed DOPD and Vanilla OPD on (a) continual learning, where we conduct a three-stage continual learning with general, reasoning, and coding training sub-datasets sequentially. The solid and dashed lines denote the results on general benchmark (LiveBench) and corresponding specific benchmarks (MATH500 and BFCLv3); and (b) out-of-distribution tasks, where we optimize the student policy on coding or reasoning dataset, but evaluated on another out-of-domain benchmarks (MATH500 and BFCLv3).
**Figure 8.** Training stability comparison of proposed DOPD and representative baselines, reporting the (a) performance and (b) entropy trends over training steps on LiveBench.
**Figure 9.** Token-level visualization of the four token types, where each token is colored based on their privilege advantage gap $\mathcal{A}$ and predicted probabilities of teacher $q_T$ and student $q_S$ policies.
Privileged‑information experiments (Table 4, Table 5) reveal that providing step‑wise hints without execution yields the largest LLM gains (10.4 pts) and that bounding‑box + object‑label inputs give the biggest VLM boost (7.2 pts).
Step‑wise hints without execution improve LLM distillation by 10.4 points.
Table 4 lists a 10.4‑point increase for this variant.
Bounding‑box with object label raises VLM performance by 7.2 points.
Table 5 shows a 7.2‑point uplift for this input type.
Step‑wise hints with execution still add a solid 8.3‑point LLM gain.
Table 4 records an 8.3‑point improvement for the execution‑aware variant.
Token‑level analysis (Section 4.3.3) groups tokens by advantage gap and prediction confidence, distinguishing stable knowledge, low‑value connectives, key privileged knowledge, and self‑consistent exploration.
Divergence experiments (Section 4.3.4) compare forward KL, reverse KL, and JS divergence across sampling strategies. JS with the full vocabulary attains the highest score (42.5 at step 160).
JS divergence with full‑vocabulary distillation reaches 42.5, the best among all objectives.
Table 7 reports 42.5 for JS + Full Vocabulary at step 160.
Sensitivity studies (Section 4.3.5) find $\beta_{w}$ = 0.3 and $\beta_{l}$ = 0.6 optimal, delivering the peak performance of 49.8 at step 160.
Setting $\beta_{w}$ = 0.3 and $\beta_{l}$ = 0.6 yields a final score of 49.8 at step 160.
Table 8 (final row) lists 49.8 for the full DOPD configuration.
The additional question illustrates a geometric intuition: cutting an equilateral triangle with two parallel interior lines leaves at least one triangular piece.
Limitations and Conclusion
We discuss DOPD’s remaining limits, future work, and detail privileged input examples.
Our analysis confirms that privileged teachers can appear superior without delivering transferable gains, because supervision signals concentrate on a subset of tokens. This asymmetry stems from an information gap between teacher and student.
DOPD mitigates the gap by routing token‑level supervision: tokens exhibiting a measurable advantage gap receive full‑vocabulary teacher distillation, while the remaining tokens are optimized primarily by the student itself. The routing decision leverages relative token probabilities to balance capacity transfer.
Extensive LLM and VLM experiments show consistent improvements over vanilla OPD across robustness, continual‑learning behavior, out‑of‑distribution generalization, and training stability. These gains hold across diverse model scales and data regimes.
First, DOPD’s effectiveness hinges on the availability and quality of privileged information, whose collection incurs additional annotation, generation, and filtering costs. Producing high‑fidelity privileged inputs is non‑trivial for many domains.
Second, the method adds computational overhead relative to vanilla OPD, requiring an extra forward pass of the student model for each batch. This overhead modestly increases training time and memory consumption.
Third, the current routing strategy, while empirically stable, relies on heuristic thresholds rather than a learned policy. Future work could replace these hand‑crafted rules with a principled, learnable routing mechanism.
Promising directions include cheaper, more reliable acquisition of privileged information and alternative advantage‑gap detectors that are grounded in theory. More broadly, dynamic distillation that jointly leverages teacher and student signals may inspire interpretable, efficient, and trustworthy distillation paradigms beyond LLMs and VLMs.
This image displays an object detection annotation example. A person surfing is enclosed in a dashed bounding box labeled "surfing" with coordinates [181, 142, 310, 275]. A surfboard is enclosed in a smaller dashed bounding box labeled "surfboard" with coordinates [318, 311, 392, 349].
DOPD still depends on costly privileged data and heuristic routing, leaving a capacity gap when teacher signals are unavailable or noisy.