Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
Shi-Yu Tian, Yong Liu, Ximan Liu, Tianrui Zhu, Sule Bai, Xinwei Yang, Ruiqi Liu, Kun-Yang Yu, Ming Yang, Zhi Zhou, Yu-Feng Li, Yipeng Sun, Gu Simiu
ROAD MOPD coordinates multi-teacher distillation by using auxiliary teachers to gate token-level rewards and constrain gradient updates.
How can we prevent conflicting gradient updates when training a single student model on multiple specialized teacher domains simultaneously?
Multi-teacher on-policy distillation (MOPD) routes domain-specific data to specialized teachers, but updates from different domains often conflict when applied to shared model parameters. The authors introduce Cross-Teacher Guarding (ROAD MOPD), which uses non-routed teachers as auxiliary critics to assess the same student trajectories. This mechanism gates token-level rewards based on teacher agreement and applies preventive gradient projection to suppress conflicting updates while retaining target-domain signals. In high-conflict settings, this approach improves accuracy by up to 8.8 percentage points over standard MOPD and consistently outperforms existing gradient-conflict baselines.
Paper Primer
Domain-routed MOPD assumes that assigning samples to specialized teachers is sufficient for capability integration. However, because these teachers operate on shared parameters, their independent updates often interfere, leading to performance degradation in the disadvantaged domain.
ROAD MOPD treats non-routed teachers as cross-domain critics: it uses their feedback on the same student-generated trajectories to identify and mitigate interference. The core move is a dual-level coordination: agreement-aware token weighting modulates rewards based on teacher consensus, while preventive gradient projection constrains target-domain updates to avoid conflict with auxiliary teacher preferences.
ROAD MOPD mitigates performance degradation in high-conflict multi-domain settings.
Comparison against Vanilla MOPD in Math × Medical and Code × Medical domain pairings. 5.6–8.8 percentage point improvement in Medical accuracy.
Preventive gradient projection outperforms post-hoc gradient surgery (e.g., PCGrad).
Direct constraint of target-domain updates using auxiliary teacher feedback on the same rollout. 0.18–2.12 percentage point improvement across domain-average accuracies.
Why does domain routing alone fail to prevent performance degradation?
Domain routing only separates the source of supervision; it does not coordinate the resulting gradient updates. When teachers have different response styles or expertise, their independent updates to shared parameters can induce persistent conflicts that degrade performance in one or more domains.
What is the role of the auxiliary teacher in this framework?
The auxiliary teacher acts as a cross-domain critic that evaluates the same student-generated trajectories as the routed teacher. Its feedback provides a local compatibility signal used to gate token-level rewards and define a reference direction for gradient projection.
For researchers building multi-teacher distillation systems, this paper demonstrates that update compatibility—not just data routing—is the primary bottleneck for joint capability integration. ROAD MOPD provides a template for using auxiliary teacher feedback to actively constrain shared-parameter updates.
The Challenge of Multi-Teacher Distillation
We expose why naïve domain‑routed MOPD can sabotage learning across domains.
Domain‑routed MOPD routes each rollout to a teacher specialized for that domain, but all updates share the same student parameters. When teachers propose divergent corrections, the shared parameters receive conflicting gradients, leading to persistent update interference. Figure 1 illustrates how this incompatibility harms medical domains while leaving math‑code pairs largely unaffected.
MOPD sends each domain’s rollouts to its own teacher, then uses the teacher’s token‑level corrections to update a single student model.
Update incompatibility occurs when gradient directions from different domain teachers point oppositely on the same student parameters, causing interference.
**Figure 1.** Update conflicts reveal a failure mode of domain-routed MOPD. (a) Fraction of conflicting layers during training; labels show training averages. (b–d) Per-benchmark score changes of Vanilla MOPD relative to single-domain OPD. Math–Code remains compatible, whereas both Medical pairings consistently degrade Medical performance.
The core failure of domain‑routed MOPD is update incompatibility: shared parameters receive opposing gradients from different domains, causing performance loss in poorly compatible pairs.
ROAD MOPD: Cross-Teacher Guarding
ROAD MOPD introduces token‑wise credit and gradient projection to resolve update conflicts.
Domain‑routed supervision in vanilla MOPD often sends conflicting gradient signals to the shared student, stalling learning. ROAD MOPD tackles this by (A) adjusting token‑level rewards using cross‑teacher agreement and (B) projecting gradients to preserve a target‑gradient norm while eliminating conflict.
ROAD MOPD first checks whether two teachers agree on each token; if they do, it amplifies that token’s reward, otherwise it dampens it. Then it reshapes the resulting gradient so that the update stays close to the original direction while removing the component that would increase the auxiliary loss.
How does the token‑wise $\lambda_{t}$ differ from using a single scalar $\lambda$ for the whole sequence?
A global $\lambda$ scales every token equally, so it cannot suppress a harmful teacher signal on a single token while still amplifying useful signals elsewhere. $\lambda_{t}$ varies per token based on the agreement $s_t$, allowing the model to keep beneficial corrections and mute detrimental ones.
For each token we ask two teachers: does the routed teacher and an auxiliary teacher both prefer the same correction relative to a frozen reference? If yes, we boost that token’s reward; if not, we attenuate it.
Compute the tanh terms: tanh(0.8/1)=0.664, tanh(‑0.4/1)=‑0.379.
Form the agreement score: $s_t$ = 0.664 × (‑0.379)=‑0.252.
Apply the modulation: $\lambda_{t}$ = clip(1 + 0.5 × (‑0.252), 0.5, 1.5)=clip(0.874, 0.5, 1.5)=0.874.
Thus the routed‑teacher advantage for this token is multiplied by 0.874, slightly attenuating it because the teachers disagree.
Even a modest disagreement reduces the reward, preventing a harmful teacher signal from dominating the update.
After computing the target‑domain gradient $g_d$ and an auxiliary gradient g′_d, we restrict the update to a line segment that removes the conflicting component while guaranteeing that the resulting gradient retains at least a fraction m of its original norm.
Why not simply add the auxiliary gradient to the target gradient and hope the optimizer balances them?
Adding the auxiliary gradient can increase the loss on the target domain because the two gradients may point in opposite directions. The projection explicitly removes the conflicting component while guaranteeing that the resulting update still moves substantially in the target direction (norm‑retention), which a naïve sum cannot ensure.
**Figure 2.** **Vanilla MOPD and ROAD MOPD.** Left: Vanilla MOPD’s domain-routed supervision can produce conflicting updates to shared parameters. Right: ROAD MOPD uses same-trajectory auxiliary-teacher feedback for (A) agreement-guided token credit assignment and (B) preventive gradient projection under a target-gradient norm-retention constraint.
**Figure 3.** Teacher disagreement and local supervision. Left: raw gradient conflict by disagreement quintile (Math: blue; Medical: orange). Right: Math (top) and Medical (bottom) response windows; blue attenuates ($\lambda_t < 1$) and orange amplifies ($\lambda_t > 1$) token supervision.
One optimizer step of ROAD MOPD (Algorithm 1)
Empirical Evaluation and Ablations
ROAD MOPD beats baselines across domains, with top All‑Avg scores and robust gradient handling.
In the low‑conflict Math × Code setting, ROAD MOPD attains the highest All‑Avg (55.40 %) and beats the strongest baseline ExOPD by 0.35 points.
Table 2 shows ROAD MOPD’s All‑Avg of 55.40 % versus ExOPD’s 55.05 %.
In the three‑domain Math × Code × Medical scenario, ROAD MOPD leads all methods with an All‑Avg of 52.89 %, 1.18 points above the next best CaMOPD.
Table 3 reports ROAD MOPD’s All‑Avg of 52.89 % versus CaMOPD’s 51.71 %.
PCGrad resolves gradient conflicts by projecting each domain’s gradient onto the normal plane of the others, preserving components that do not interfere.
How does ROAD MOPD’s preventive projection differ from PCGrad’s gradient‑conflict handling?
PCGrad projects gradients after they are computed on independent domain rollouts, whereas ROAD MOPD projects the target‑domain gradient using auxiliary‑teacher gradients that share the exact same rollout. This same‑trajectory constraint can resolve conflicts that PCGrad cannot see, leading to the consistent gains reported above.
**Table 2.** Low-conflict generalization: Math $\times$ Code. Qwen3-4B domain averages (%) and their mean (All).
**Table 3.** **Three-domain generalization: Math $\times$ Code $\times$ Medical.** Qwen3-4B domain averages (%) and their mean (All).
**Figure 4.** Preventive projection vs. PCGrad. (a) Mean layerwise cosine; labels average shaded steps 39–58. (b) Domain accuracy (%).
**Table 4.** Ablations of ROAD MOPD. Domain-average accuracy (%). Avg.: mean across the two domains.
The table lists data sources categorized by domain (Math, Code, and Medical) and the number of sampled prompts for each source.
Extended Qualitative Analysis
Derivations and extra visualizations support the ablation analysis of ROAD MOPD.
This appendix supplies the mathematical foundations of ROAD MOPD and visual evidence for the ablation studies. We first derive the token‑level OPD loss, then the norm‑constrained projection, and finally present the full training pipeline. The figures illustrate how agreement‑aware attenuation and the projection affect token dynamics and overall training progress.
**Figure 5.** Extended Math and Medical examples from the right panels of Figure 3. Outlines identify the main-text windows within the longer contexts. Blue and orange denote attenuation and amplification on the unchanged scale $\lambda_t \in [0.5, 1.5]$. Token indices are zero-based, with half-open intervals. Omitted context and the generation limit are marked.
**Figure 6.** Three additional Math response excerpts. Panels show the beginning, middle, and end of three distinct sampled responses to `AIME2024_0`. Original token coefficients and the [0.5, 1.5] color scale are preserved. A dot represents a whitespace-only token; omitted context and the generation cap are marked.
**Figure 7.** Three additional Medical response excerpts. Panels show the beginning, middle, and end of three distinct sampled responses to `MedQA_test_0`, using the same fixed-window rule and unchanged color scale as Figure 6. These are generated reasoning excerpts, not verified answer explanations.
Ablation experiments confirm that each component of ROAD MOPD is essential. Removing the agreement‑aware token weighting leads to noticeably higher conflict loss and lower average performance across Math, Medical, and Code domains. Disabling the preventive gradient projection similarly degrades the norm‑retention ratio, showing that unchecked conflicts harm the student update.
Experimental Setup and Reproducibility
Reproducibility details: models, data, hyperparameters, baselines, and evaluation.
Section B.1 introduces the student and three frozen domain teachers. The Math teacher checkpoint is Qwen3‑4B‑Non‑Thinking‑RL‑Math‑Step500, the Code teacher is Qwen3‑4B‑Non‑Thinking‑RL‑Code‑Step300, and the Medical teacher is II‑Medical‑8B (a RL‑fine‑tuned Qwen3‑8B). All teachers share the same Qwen3 tokenizer, enabling token‑wise scoring of student outputs.
Section B.2 describes the paired training sets: each domain contributes 10 000 prompts, for a total of 20 000 shuffled prompts. Math prompts are drawn from DeepMath (difficulty ≥ 6) after filtering 57 K candidates; Code prompts come from Eurus‑RL‑Code; Medical prompts are assembled from four sources (4 000 MedMCQA, 3 000 MedReason, 2 000 Medical‑R1‑Distill‑Data, 1 000 m23k).
Section B.3 lists the shared hyperparameters (Table 5) and infrastructure. PPO minibatch size is 1 008 prompts, rollout temperature and top‑p are both 1.0, and the optimizer is AdamW with learning rate $10^{-5}$ (constant). Global gradient norm clipping is set to 1.0 after merging domain gradients; the modulation parameters are $τ = 1$, $ρ_g = 0.5$, and $[λ_{\min}, λ_{\max}] = [0.5, 1.5]$.
Section B.4 details baseline implementations within the same MyOPD/verl stack. Vanilla MOPD routes each prompt to its domain teacher and clips token‑advantage logits to $[-10, 10]$ without cross‑teacher modulation. ExOPD applies a fixed extrapolation coefficient of 1.25 and disables gradient projection. CaMOPD alternates homogeneous domain batches, using a gap‑based selection criterion with an 80 % score‑mass threshold. Open MOPD rescales domain loss weights by the product of target share and observed token share, clipped to $[0.05, 20]$. PCGrad projects opposing gradients when their dot product is negative, then sums the corrected gradients.
Section B.5 explains the evaluation protocol across 11 benchmarks (Code, Math, Medical). Generation uses vLLM with temperature 1.0, top‑p 1.0, and a maximum of 16 384 tokens. Math benchmarks report Avg@32 (mean correctness over 32 completions); Code benchmarks report Avg@4 (mean correctness over four completions); Medical benchmarks report Avg@1 (single‑completion accuracy). Scores are percentages and domain averages are arithmetic means over their constituent benchmarks.
**Table.** Hyperparameters used for training.
**Table C.2.** Full results for low-conflict and three-domain settings
**Table 7.** Per-benchmark breakdown of the ablations and PCGrad comparison. Avg. is the benchmark mean; rounding may cause small differences from the mean of displayed scores. Scores are percentages (higher is better). (a) Math × Medical (b) Code × Medical
Detailed Benchmark Results
Appendix C provides full benchmark tables, visualizations, and training details for ROAD MOPD.
Table 7 reports the ablation study for the Math × Medical and Code × Medical settings. Each column removes one component of ROAD MOPD (credit assignment, gradient protection, or adaptive retention) and compares against the PCGrad baseline. The numbers are percentages, higher is better, and the average rows summarize overall performance.
Table 8 lists single‑domain checkpoints trained on 10 k in‑domain prompts for three epochs. The scores across the 11 benchmarks illustrate the baseline performance of each domain‑specific OPD model before any multi‑domain distillation.
Tables 9 and 10 give the full per‑benchmark results underlying the summary tables in the main paper. Table 9 covers the low‑conflict Math × Code scenario, while Table 10 presents the three‑domain evaluation (Math, Code, Medical). Bold entries indicate the best multi‑domain method, underlined the second‑best.
Figure 5 extends the token‑level trajectories from Figure 3 to 256‑token contexts, showing both attenuation and amplification around answer mentions. The visualizations use the token‑modulation‑only checkpoint at update 58 with temperature/top‑p = 1 and a 2 048‑token generation cap.
This figure displays a color scale representing the parameter $\lambda_t$. The scale ranges from 0.5 to 1.5, with a blue color gradient on the left side labeled "attenuation" and an orange color gradient on the right side labeled "amplification". The center point at 1.0 is represented by a neutral white color.
The training dynamics section presents full traces from three archived ROAD MOPD runs (Math × Medical, Code × Medical, Math × Code). Each run contains 58 updates with 1 008 prompts per update, one PPO epoch, and a 16 384‑token response cap. Gradient‑norm retention curves stay at or above the floor $m = 0.6$, preventing excessive norm loss while allowing full projection when the norm already satisfies the floor.
Limitations: ROAD MOPD adds computational overhead due to auxiliary‑teacher scoring and gradient processing, although reusing student trajectories mitigates rollout costs. The current evaluation spans only three domains; future work should test scalability to larger domain collections and bigger teacher pools.