Latent-MOPD: Latent Multi-Teacher On-Policy Distillation
Zhengyu Fang, Seoyeon Hong, Jie Yang, Muyang Li, Koyoshi Shindo, Brandon Joseph Lwowski, Jing Li
Latent-MOPD integrates multiple LLM specialists by distilling both their output distributions and internal hidden states.
How can we distill multiple specialist teachers into a single student model by supervising both their output distributions and their internal hidden-state representations?
Multi-teacher on-policy distillation (MOPD) trains a single student to combine the strengths of several domain specialists, but current methods rely solely on output-level token predictions. This ignores the intermediate reasoning steps encoded in hidden representations, which can provide richer guidance for the student's internal computation. Latent-MOPD routes each student response to its domain-specific teacher and aligns both the teacher's token predictions and selected late-layer hidden states. It uses domain-pure updates and a per-teacher crossfade schedule to shift supervision from latent states to token predictions as training progresses. In same-family settings, this approach outperforms token-only and representation-only baselines across all nine benchmarks, surpassing the best individual teacher on a majority of them.
Paper Primer
The method hinges on a dual-channel supervision strategy: a routed token channel provides dense output guidance, while a selective late-layer alignment channel forces the student to mimic the teacher's internal reasoning states. To bridge architectural differences, the system uses a shared linear projection map that is initialized via ridge regression and optimized jointly with the student.
Latent-MOPD consistently outperforms single-channel distillation baselines.
In same-family 1.5B teacher experiments, the student surpasses the per-benchmark best teacher on five of nine benchmarks (BBH, MuSR, MBPP, MBPP+, and Minerva). The aggregate performance (Norm score) reaches 1.05, exceeding the best-teacher envelope of 1.0.
The method remains robust when scaling to cross-family teachers (separately developed 7B models). In this setting, Latent-MOPD outperforms both token-only and representation-only baselines on all six tested benchmarks, demonstrating that the latent alignment effectively transfers capabilities across different model lineages.
Why is domain-pure batching critical for this method?
Interleaving samples from different teachers within a single optimizer update causes representation-only training to collapse. Domain-pure updates stabilize the latent supervision by ensuring each parameter update is guided by a consistent specialist.
How does the crossfade schedule manage the two supervision channels?
The method uses a per-teacher clock to gradually shift supervision from latent states to token predictions. This transient latent signal prevents the student from over-relying on intermediate states while ensuring it learns the teacher's reasoning process early in training.
Latent-MOPD proves that a single student can effectively integrate multiple specialists by supervising both the "what" (token predictions) and the "how" (hidden representations) of teacher reasoning.
Introduction
We expose why token‑only distillation fails in multi‑teacher LLMs and introduce Latent‑MOPD.
Token‑only on‑policy distillation works well for a single specialist but falters when multiple teachers are combined. The student receives only the teachers’ output distributions, ignoring the internal reasoning that produced those outputs. This omission limits the ability to fuse diverse capabilities into a single model.
OPD trains a student model on the responses it generates, using a teacher’s token‑level predictions as supervision.
Hidden‑state supervision—aligning the student’s internal representations with those of the teacher—addresses the missing intermediate reasoning. Prior work (LastOPD, OPRD) shows that matching late‑layer activations can boost transfer, but those methods consider a single teacher. Extending this to multiple specialists raises new questions about target selection, width mismatches, and update scheduling.
The limitation of token‑only distillation in multi‑teacher settings is that it discards the rich internal reasoning captured in hidden states.
The Latent-MOPD Mechanism
Latent-MOPD adds hidden‑state alignment to token distillation, letting the student learn a teacher’s internal reasoning.
Token‑only distillation forces the student to mimic only the teachers’ output distributions, ignoring the internal representations that give those teachers their expertise.
Like a student copying not just the final answer but also the teacher’s scratch work, we require the student to match the teacher’s hidden states in addition to token probabilities.
Normalize both vectors: $\nu(z^{T}) = (0.2,0.5,-0.1,0.3)/\|z^{T}\|_2$, $\nu(z^{S}) = (0.1,0.4,0.0,0.2)/\|z^{S}\|_2$.
Apply the identity projector $g_{\psi}$ (same‑family) to $\nu(z^{S})$, yielding the same normalized vector.
Compute the squared L2 distance: $\|\nu(g_{\psi}(z^{S})) - \text{sg}(\nu(z^{T}))\|_2^{2}=0.018$.
Scale by $\lambda d_T = 0.5 \times 4 = 2$, giving a latent loss contribution of $0.036$.
Combine with the token loss (e.g., $0.12$) to obtain a total loss of $0.156$ for this token.
Matching hidden states forces the student to internalize the teacher’s intermediate computation, not just the final probability distribution.
How does aligning hidden states differ from just averaging token predictions?
Token averaging only aligns the final probability distribution, while hidden‑state alignment forces the student to reproduce the intermediate feature activations that the teacher uses to compute those probabilities, capturing richer knowledge.
Each prompt carries a domain label that selects a single specialist teacher, so the student receives token and latent targets exclusively from that teacher.
Why not route tokens to multiple teachers and average their losses?
Averaging dilutes a specialist’s confident prediction and introduces conflicting gradients from unrelated domains; routing preserves the strong signal from the appropriate teacher and yields cleaner updates.
We project the student’s hidden vector through $g_{\psi}$, normalize both sides, and penalize their L2 distance, optionally using a trainable linear map for cross‑family teachers.
What prevents the student from simply copying the teacher’s hidden state without learning?
The stop‑gradient operator blocks gradients from reaching the teacher side, so the student must adjust its own parameters to reduce the distance; the teacher remains a fixed target.
Sample a student response $\hat{y}\sim\pi_{\theta}(\cdot\mid x)$ for a domain‑labeled prompt $x$.
Select the specialist teacher $\pi_{T_d}$ based on the domain label $d$.
Compute token‑level advantage $A_{t,v}$ for the top‑$k$ candidates and evaluate $L_{\text{route}}^{\text{OPD}}$.
Extract hidden states $z^{S}_t$ and $z^{T_d}_t$, apply $g_{\psi}$ and normalization, and evaluate $\ell^{\text{rep}}_{\text{route},t}$.
Combine the two losses using the crossfade schedule $L(s)$.
Back‑propagate through the student (teacher parameters remain frozen) and update $\theta$ (and $g_{\psi}$ in the cross‑family case).
Increment the per‑teacher counter $u$; when $u$ reaches $T_w$, switch to the next teacher.
Crossfade schedule for a single teacher.
**Figure 2.** Three supervision designs on shared student-generated prefixes. (a) Uniform averaging gives each teacher's token probabilities equal weight. (b) MOPD-style routes token supervision to one frozen teacher. (c) Latent-MOPD adds a representation target from the same teacher, matching $g_{\psi}(z^S)$ to its hidden state. PG denotes the policy-gradient token update from outputs after the LM heads; bars are schematic distributions. Routed columns illustrate a math prompt. One layer pair is drawn: same-family aligns the last three using the identity, cross-family the last using a shared trainable linear map. Both use crossfade (Figure 4).
Experimental Results
Latent‑MOPD beats token‑only baselines with a strong normalized gain.
Latent‑MOPD attains a normalized performance of 1.05, exceeding the best‑teacher envelope.
Table 1 reports Norm = 1.05 for the same‑family configuration, while the token‑only baseline scores 0.90.
Same‑family uses teachers that share the student’s scale and training history; cross‑family mixes larger teachers with a smaller student.
How does the cross‑family regime differ from same‑family despite using the same student architecture?
Cross‑family teachers are larger (7 B) and have a different hidden width, so Latent‑MOPD inserts a linear projection that maps the teacher’s hidden states into the student’s space before applying hidden‑state alignment.
**Figure 1.** Same-family gains beyond individual teachers, with a cross-family extension. (a) Same-family; (b) cross-family (last 1, Linear). Both panels use shared per-axis scales, with each outer vertex fixed at the same-family Latent-MOPD score. Solid polygons show Latent-MOPD; the dotted ring in (b) repeats the same-family reference. Representation-only denotes OPRD-style (all layers) in (a) and Rep-only (last 1) in (b). Labels give Latent-MOPD scores; bold with (>Teacher) marks those above the strongest teacher. Table 1 reports all nine same-family benchmarks.
**Figure 5.** **Training dynamics, general retention and representation alignment.** (a) Default same-family (last-three-layer) and cross-family (last-layer) Latent-MOPD on MATH-500, with 8 samples per problem. (b) Same-family general retention under 0-shot likelihood scoring. (c) Same-family raw representation loss, normalized to each domain's first logged value and excluding $\beta$. Crossfade lasts 10 updates per domain (shaded), followed by token-only supervision. Appendix A specifies scoring and loss measurement.
Latent‑MOPD consistently outperforms token‑only baselines across domains.
Ablations and Analysis
We quantify how each component of Latent‑MOPD contributes to performance.
We isolate each design choice of Latent‑MOPD by removing it and measuring the resulting drop in performance.
Dropping three‑layer supervision (using last‑layer only) reduces the CKA‑based Norm from $1.05$ to $0.99$.
Table 1 shows the Norm drop when the layer selection is limited to the last layer.
Replacing three‑layer representation‑only supervision with last‑layer supervision lowers Norm from $0.73$ to $0.64$.
Table 1 reports the Norm values for representation‑only controls across layer choices.
Removing the linear projector in the cross‑family regime (using three layers instead of last layer) changes Norm from $0.26$ to $0.25$.
Table 1 includes the cross‑family comparison with and without the linear projection.
Eliminating the cross‑fade schedule (using constant weights) cuts the same‑family student’s score at step 20 from $90.3$ to the base $84.1$, a loss of $6.2$ points.
Figure 5a tracks the same‑family run with cross‑fade versus the constant‑weight baseline.
Replacing domain‑pure updates with interleaved updates collapses MATH‑500 validation from $91.7$ to $22.8$ at step 20, a drop of $68.9$ points.
Same‑family all‑layer representation‑only control (Appendix F) reports the two update regimes.
Omitting latent‑state alignment (using MOPD‑style) reduces the math‑teacher matching margin by $0.0648$ on selected positions.
Figure 6 measures the margin increase achieved by Latent‑MOPD over MOPD‑style.
**Figure 4.** Crossfade across rotating teachers. Domain-pure updates rotate through math, code and logic. Two selected cycles illustrate token weights rising and latent weights fading on each teacher’s own clock (Equation 3); bar lengths are schematic. Both regimes use this schedule, with different windows.
**Figure 8.** Layer selection, crossfade and logic-domain training dynamics. (a) Last-layer and last-three-layer comparisons: representation-only and Latent-MOPD in the same-family (SF) setting, and Latent-MOPD with linear projection in the cross-family (CF) setting. Norm is computed separately within Tables 1 and 2; the two regimes have separate axes. (b) Same-family score gains from crossfade over constant weights, in percentage points, computed from the reported Table 1 scores. Both configurations use last-three-layer alignment; the comparison changes the coupled token and representation schedule. (c) Latent-MOPD on GYM-200, with 8 samples per problem: same-family last-three-layer and cross-family last-layer linear configurations; the dashed line marks the base. The validation set is separate from the 925-problem GYM evaluation in the main tables.
**Figure 10.** **Interleaved representation-only training under an extended budget.** The same-family all-layer control in Figure 3c, continued to 120 steps. (a) MATH-500; (b) GYM-200. Scores average 8 samples per problem. Solid lines connect all measured checkpoints. Horizontal dashed lines mark the base; vertical dotted lines mark the original 62-step budget, and diamonds the step-62 values. Vertical scales differ.
Training Objectives
We define a joint token‑latent loss and a baseline MOPD‑style objective for multi‑teacher distillation.
The central premise is that token‑only distillation ignores the internal reasoning of specialist teachers; adding hidden‑state alignment captures those latent patterns.
Our token channel follows the on‑policy distillation (OPD) recipe: for each generated prefix we score the student’s top‑$k$ tokens, normalize their weights, and back‑propagate through the student’s distribution only.
A token‑only routed baseline that supervises the student solely through the token‑level loss, without any hidden‑state target.
Compute the average token loss: $(0.8 + 0.6)/2 = 0.7$.
Compute the average latent loss: $(0.4 + 0.2)/2 = 0.3$.
With $\lambda = 2$, the joint loss is $0.7 + 2 \times 0.3 = 1.3$.
The scalar $\lambda$ directly controls how much the hidden‑state signal influences the total gradient; increasing $\lambda$ amplifies the latent term proportionally.
We extend the token channel to a multi‑teacher setting: $N$ frozen teachers $\{\pi_{T_i}\}$ each own a domain, and prompts are routed to the teacher matching their domain label $d$.
**Computational structure: constraints before counting.** Skywork and Latent-MOPD use 8 right and 8 up moves, giving $2\binom{7}{2}\binom{7}{1} = 294$. MOPD-style's $7 + 7$ formulation gives 180 for a length-14 path.
**Figure 7.** **Three-teacher supervision: an illustrative schematic.** All panels use the same math prompt. (a) The math, code and logic teachers contribute equally to the averaged next-token distribution, with weight 1/3 each. (b) MOPD-style selects the math teacher's distribution; the code and logic teachers remain inactive. (c) Latent-MOPD retains this token target and adds a hidden-state target from the same math teacher. Hidden-feature tiles are separate from the token distributions; their positions do not denote token or layer correspondence. Values and features are schematic.
Representation Similarity Analysis
Analyzes how layer representations differ across teachers and how masking and batch design affect distillation performance.
We probe how closely the student’s internal states match each teacher’s by measuring linear Centered Kernel Alignment (CKA) layer‑wise, then examine how masking high‑activation dimensions and different batch‑organization strategies influence these similarities and downstream performance.
Domain‑pure batch organization keeps training stable, reaching 91.7 % MATH and 56.7 % GYM after 62 steps, while interleaved updates collapse to 43.2 % / 12.8 %.
Interleaved runs fall below the untrained student by step 10 and end at 43.2 % MATH, 12.8 % GYM at step 62.
Extending interleaved training to 120 steps recovers only modest performance, ending at 84.5 % MATH and 41.6 % GYM, still 7.2 % and 15.1 % below the domain‑pure reference.
Domain‑pure reference after 62 steps is 91.7 % MATH and 56.7 % GYM.
When using same‑family teachers, targeting the last three layers outperforms the single last layer on eight of nine benchmarks.
Table 1–2 results show eight wins for three‑layer targets versus one win for the last‑layer target.
**Figure 3.** Representation targets and update organization. (a,b) Native linear CKA between matched block outputs on fixed base-student responses before training. Same-family differences increase near the output (magnified scale); cross-family similarities show a middle-layer trough. Hatching marks the selected depths; bands give one prompt-bootstrap standard deviation (Appendix E). (c) Same-family all-layer representation-only supervision remains stable with domain-pure updates; interleaved updates collapse and finish far below the base (dotted line). Both use MATH-500 with 8 samples per problem (Appendix F).
**Figure 9.** Depth-matched CKA after masking high-activation dimensions. (a) Same-family, with a magnified vertical scale; (b) cross-family, on the full 0–1 scale. Each teacher is compared with the base student on the same 192-prompt diagnostic pool as Figure 3a,b. Before centering, the top 1% of dimensions by mean absolute activation are zeroed separately in each model and block. Curves show linear CKA at matched decoder-block outputs, before final normalization; shading is one standard deviation over 200 prompt-bootstrap replicates. The masked cross-family curves retain a middle-layer trough, while the same-family minima occur before the final block. Hatching marks the selected training depths: the final three blocks in (a) and the final block in (b).
Teacher Output Alignment
Latent‑MOPD improves teacher‑matching margins over MOPD‑style across domains and readout depths.
We evaluate how Latent‑MOPD aligns student outputs with domain‑specialist teachers compared to the MOPD‑style baseline.
Latent‑MOPD raises the math‑teacher matching margin on selected positions by 0.0648.
Figure 6 shows the margin shifting from –0.0282 (MOPD‑style) to 0.0366 (Latent‑MOPD), a $\Delta$ of 0.0648 (95 % CI [0.0184, 0.1324]).
**Figure 6.** Domain-conditioned teacher matching after same-family distillation. (a,b) Mean next-token probability cosine on teacher-disagreement positions: input domains are rows and teachers are columns, on one shared color scale; outlines mark row maxima. (c) Change in matching-teacher margin (Latent-MOPD minus MOPD-style) on selected and all positions, with paired 95% instance-bootstrap intervals. Appendix G defines the selection rule, margin and measurement protocol.
**Figure 11.** **Teacher matching across readout depth in the same-family setting.** (a) Latent-MOPD minus MOPD-style matching-teacher margin, by input domain. Shading and endpoint bars show paired 95% pointwise instance-bootstrap intervals. (b) Per-teacher cosine changes on selected math positions at the last four fitted indices and native head; positive values indicate greater similarity under Latent-MOPD. Both panels use the same teacher-disagreement selection rule. Native-head measurements are separate from the fitted readouts.
Case Studies
Latent‑MOPD outperforms MOPD‑style across three case studies, fixing token and latent errors.
Latent‑MOPD solves all three case studies correctly, while MOPD‑style fails on each.
In the Eulerian number task Latent‑MOPD produces 4 versus 0, in the logic rewrite task it yields the reachable final state, and in the counting problem it returns 294 versus 180.
**Figure a.** Code: initializing the computation correctly. Same-family, MBPP. Compute the Eulerian number $a(n, m)$; the supplied test is `eulerian_num(3, 1) == 4`. Inner-loop excerpts:
Across the three domains—code generation, logical rewriting, and combinatorial counting—Latent‑MOPD consistently captures the hidden‑state constraints that MOPD‑style overlooks, leading to correct solutions and adherence to task‑specific invariants.
Related Work
We compare prior distillation approaches and highlight how Latent‑MOPD integrates multiple teachers, hidden‑state supervision, and on‑policy training.
On‑policy distillation trains the student on its own sampled responses, letting the teacher score each token the student generates. This reduces the train‑inference mismatch that plagues fixed‑teacher imitation and works alongside supervised fine‑tuning and reinforcement learning.
All token‑level OPD variants draw their targets from the teacher’s output distribution, but they leave the hidden states that produced those targets unsupervised. Representation‑level distillation adds a second supervision source by aligning student hidden states to those of the teacher, as pioneered by FitNets and later extended to attention statistics.
Multi‑teacher distillation aggregates several specialists into a single student. Off‑policy approaches combine teacher predictions by averaging or adaptive weighting, while feature‑level methods align the student to multiple teachers’ intermediate representations on a static corpus.
For image generators, Poly‑OPD fuses heterogeneous flow‑model teachers via a pixel bridge, and STEP‑OPD aligns diffusion teachers on‑policy by matching representation changes across blocks. Our representation channel mirrors these ideas by matching routed LLM hidden states on student‑generated token prefixes.
MOPD proceeds in three stages—general supervised fine‑tuning, domain‑specific reinforcement learning, and integration—freezing the teachers during distillation. We retain per‑prompt routing and on‑policy scoring, reusing specialist checkpoints without extra teacher training.
Weight‑space merging (parameter averaging, task arithmetic) provides a non‑distillation baseline; we initialize the student from a merged model for 62 steps before applying Latent‑MOPD to test whether distillation can improve on the merged weights.
Centered Kernel Alignment (CKA) measures representation similarity across networks of differing widths. Our diagnostics show same‑family teachers stay close to the student through most layers, while cross‑family teachers exhibit a pronounced middle‑layer dip, highlighting where latent supervision is most needed.
The evaluation setting fixes a set of specialist teachers and compares five integration strategies: output mixing, routed token supervision, representation‑only supervision, our joint objective, and parameter merging. All teachers share the student’s tokenizer, simplifying token‑level alignment.
Tables 1–2 isolate the contribution of each supervision channel. Token‑only and representation‑only baselines are outperformed by the joint Latent‑MOPD method, confirming that combining both signals yields a synergistic gain.
Beyond the reported experiments, measuring learned projectors and update directions at the supervised hidden states would clarify how representation geometry translates into downstream performance improvements.
Experimental Setup
Details of models, data, optimization, and training protocol for reproducing Latent-MOPD experiments.
The experiments use the DeepSeek‑R1‑Distill‑Qwen‑1.5B student together with two teacher panels—same‑family teachers (JustRL‑DeepSeek‑1.5B for math, Archer2‑Code‑1.5B for code, Nemotron‑ProRL‑1.5B for logic) and cross‑family teachers (Skywork‑OR1‑Math‑7B, AceReason‑Nemotron‑7B, DeepSeek‑R1‑Distill‑Qwen‑7B)—routed by domain labels to provide both token‑level and hidden‑state supervision; optimization follows a constant‑learning‑rate AdamW schedule over 62 steps, sampling four responses per prompt at temperature $1.0$ across a pool of 17,856 prompts (5,952 per domain) arranged in domain‑pure blocks of 32, yielding 7,936 generated responses.