Verify Before You Distill: Prompt-Level Teacher Gating for on-Policy Distillation

Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng, Bin Liang, Huayu Deng, Yao Hu, Kam-Fai Wong, Mu Chuan

TGOPD improves on-policy distillation by using verifier-scored probes to gate unreliable teacher signals.

How can we improve on-policy distillation by selectively ignoring teacher supervision when the teacher is unreliable for a specific prompt?

On-policy distillation (OPD) accelerates training by using a frozen teacher to provide dense token-level guidance, but it blindly trusts the teacher even when it is confidently wrong. This leads to negative transfer, where the student learns the teacher's errors rather than the correct task logic. Teacher-Gated On-Policy Distillation (TGOPD) solves this by auditing the teacher on each prompt before admitting its signal. It uses the teacher's otherwise-idle compute to generate a small set of probe rollouts, which are scored by a verifier to estimate reliability. If the teacher passes the audit, the system uses dense OPD; otherwise, it falls back to verifier-grounded reinforcement learning. This selective routing consistently outperforms vanilla OPD across mathematics, code, and instruction-following tasks.

Paper Primer

The core mechanism is a prompt-level gate that routes supervision based on empirical outcome evidence. By generating $K_T$ probe rollouts and calculating their verifier pass rate, the system makes a binary decision: admit the teacher's dense token-level signal or withhold it in favor of a trajectory-level reinforcement learning update.

TGOPD consistently outperforms vanilla OPD across all tested domains and scales.

In single-domain experiments, TGOPD achieved the highest performance in all six domain-scale settings, with the largest gains observed in code generation where teacher confidence is least informative. On 35B-scale code models, TGOPD achieved positive transfer (+3.0 on LiveCodeBench) while all other distillation baselines caused negative transfer.

Why is a verifier-based gate necessary when existing methods already use entropy or likelihood to measure teacher uncertainty?

Distributional signals like entropy or teacher-student agreement measure uncertainty, but they cannot distinguish between a confident correct answer and a confident error. TGOPD uses direct outcome evidence from a verifier to ensure the teacher is actually correct before admitting its signal.

Does the probe generation process significantly slow down the training loop?

No. Because the teacher's scoring pass is much faster than the student's autoregressive decoding, the teacher node is typically idle. Probes are generated during this idle window, resulting in only modest end-to-end overhead.

TGOPD demonstrates that "verify-before-distill" is a robust strategy for post-training, effectively turning idle compute into a reliability filter that prevents the propagation of teacher errors.

Introduction

The paper exposes wasted compute and degraded quality caused by indiscriminate teacher supervision in OPD.

On‑policy distillation (OPD) speeds up post‑training by giving a frozen teacher dense token‑level supervision on the student’s own rollouts. Vanilla OPD, however, forces this supervision on every prompt without checking whether the teacher’s answer is reliable for that specific input.

In asynchronous OPD the teacher’s GPU sits idle while the student generates rollouts, so the teacher node is dramatically under‑utilized.

**Figure 1.** **Teacher-side GPU underutilization in asynchronous OPD.** (a) Under Vanilla OPD, teacher-node utilization remains below 5% for 59% of the measured hour; TGOPD uses this otherwise-idle capacity for reliability probes. (b) Mean teacher-node utilization increases from 9.8% to 78.9%.

Forcing teacher supervision on every prompt wastes compute and can degrade student quality.

Setup and Notation

Preliminaries define the student, teacher, verifier, and the two baseline learning signals.

On‑policy distillation applies teacher supervision to every token of the student’s own rollouts, which becomes wasteful when the teacher’s output is unreliable for a given prompt.

OPD forces the student to imitate the teacher token by token, using a reverse‑KL advantage that treats the teacher’s probability as a target distribution.

Compute the teacher log‑likelihood for token $a$: $\log 0.8 \approx -0.22$.

Compute the student log‑likelihood for token $a$: $\log 0.5 \approx -0.69$.

Form the log‑likelihood ratio: $-0.22 - (-0.69) = 0.47$.

Apply stop‑gradient (no effect on the numeric value) and multiply by $\beta$: advantage $=0.47$ for token $a$.

Repeat the same steps for token $b$: teacher log $\log 0.2 \approx -1.61$, student log $-0.69$, ratio $-0.92$, advantage $=-0.92$.

Positive advantages push the student toward the teacher’s high‑probability token, while negative advantages pull it away from low‑probability tokens, yielding a dense, token‑wise learning signal.

When teacher supervision is unavailable, the student can instead rely on the verifier alone, which yields a coarser, trajectory‑level signal.

GRPO replaces token‑level teacher signals with a single scalar advantage derived from the verifier, applying the same value to every token in a rollout.

Teacher-Gated On-Policy Distillation

TGOPD gates teacher supervision per prompt using a verifier‑driven reliability estimate.

Vanilla on‑policy distillation (OPD) forces the teacher to supervise every prompt, even when its answer is unreliable, wasting compute and hurting student quality. TGOPD addresses this by auditing the teacher per prompt and only applying dense supervision when the audit passes.

Instead of always trusting the teacher, TGOPD first asks a verifier whether the teacher’s answer is likely correct for the current prompt; only if the answer passes does the student receive the dense token‑level teacher signal.

How does TGOPD differ from simply weighting OPD by the teacher’s confidence?

Confidence is a model‑internal scalar that cannot distinguish a confident wrong answer from a confident correct one. TGOPD’s verifier evaluates the actual outcome of several teacher rollouts, producing a binary gate that either fully trusts the teacher or completely discards its signal, avoiding any ambiguous interpolation.

The audit treats the teacher as a black‑box oracle and measures how often its sampled answers solve the prompt, using a tiny probe budget.

Why not use the teacher’s self‑confidence scores instead of probe rollouts?

Self‑confidence reflects the teacher’s internal uncertainty but cannot tell whether a confident answer is actually correct. Probe rollouts provide direct outcome evidence, which the verifier can evaluate, yielding a more trustworthy reliability signal.

Sum the verifier rewards: $1+0+1=2$.

Divide by the probe budget: $q_T(x)=2/3\approx0.67$.

Since $q_T(x)\ge\tau=2/3$, the gate will open for this prompt.

The estimator $q_T(x)$ directly reflects how often the teacher’s sampled answers succeed, unlike confidence scores that convey only internal uncertainty.

We fix $K_T=3$ and $\tau=2/3$ in all main experiments, i.e. a two‑of‑three majority decides to open the gate. A sweep with $K_T=5$ shows a broad optimum around simple majority, confirming that a coarse reliability decision suffices.

For each prompt $x$ in the batch, launch $K_T$ teacher probe rollouts concurrently with student generation.

Collect $G$ student rollouts $\{y_i\}$ from the frozen policy $\pi_{\text{old}}$ and record their log‑probabilities.

Score each teacher probe with the verifier and compute the reliability estimate $q_T(x)$.

Evaluate the binary gate $g(x)=\mathbf{1}[q_T(x)\ge\tau]$.

If $g(x)=1$, compute the OPD advantage $\hat A^{\text{OPD}}_{i,t}$; otherwise compute the GRPO advantage $\hat A^{\text{GRPO}}_{i,t}$.

Form the routed advantage $\hat A^{\text{TGOPD}}_{i,t}$ using the selector equation.

Assemble the PPO‑clipped surrogate loss $L_{\text{TGOPD}}(\theta)$ and take a gradient step.

**Figure 2.** Teacher confidence does not consistently indicate prompt-level reliability. This diagnostic uses ten offline teacher responses for each of 2,400 prompts per domain; $q_T^{(10)}$ is their verifier pass rate and is distinct from the $K_T = 3$ online estimator used for training. (a,b) Teacher self-confidence under higher ($q_T^{(10)} \ge 0.7$) and lower ($q_T^{(10)} < 0.7$) reliability. Confidence separates the groups at AUROC 0.51 on code and 0.73 on math. (c) In the low-reliability regime ($q_T^{(10)} < 0.5$), the teacher's highest-confidence sampled response is incorrect 84% of the time on code and 61% of math.

**Figure 3.** **TGOPD overview.** (A) Vanilla OPD: the student generates $G$ on-policy rollouts, and the frozen teacher scores every token. The resulting teacher–student log-probability gap provides dense token-level supervision (equation 1) on every prompt. (B) TGOPD: while the student decodes, the teacher uses its otherwise-idle capacity to generate a small set of probe rollouts. A verifier scores these probes, and their pass rate $q_T(x)$ (equation 4) estimates prompt-level teacher reliability. Teacher scoring and routine verifier evaluation of student rollouts make both candidate advantages available; the latter is omitted from the diagram for visual clarity. The gate selects dense OPD when $q_T(x) \geq \tau$ and verifier-grounded GRPO (equation 2) otherwise. The two supervision branches are mutually exclusive.

Experimental Setup

Key results and the experimental pipeline for TGOPD across model scales and domains.

TGOPD (ours) achieves the highest IFEval score on the 35B model, reaching 91.5% and surpassing the next best baseline by 0.7 points.

Table 1 shows TGOPD at 91.5% versus 90.8% for the best competing method (Vanilla OPD) on the 35B‑A3B model.

Both student models are trained on domain‑specific prompt pools—DAPO‑Math‑17K for mathematics, CodeI/O for code, and filtered Nemotron‑Cascade 2 for instruction following—using the same rollout pipeline and hyper‑parameters across all methods.

**Table 1.** Performance comparison of different distillation methods across MATH, CODE, and IF benchmarks for Qwen3.5-4B and Qwen3.6-35B-A3B models.

Main Results

TGOPD consistently beats vanilla OPD, delivering up to +3.0 points on code benchmarks.

TGOPD consistently outperforms Vanilla OPD across all domains and model scales, achieving the largest improvements on code benchmarks.

Table 1 reports gains of +3.0 (4B) and +2.9 (35B) on code, +1.6/+1.9 on IF, and +1.5/+1.2 on math; Table 2 confirms the same trend in the multi‑domain setting.

**Table 2.** TGOPD applied to multi-domain OPD (MOPD). Each run trains a single student on math, code, and IF simultaneously, routing prompts to domain-specialist teachers. Results are reported at 199 training steps with $K_T=3$, $\tau=2/3$. Bold: better of the two methods per column.

Compute Efficiency and Utilization

Analyzing how TGOPD reclaims idle teacher capacity and the impact of gating choices.

On‑policy distillation wastes compute by forcing a teacher to supervise every prompt, even when its output is unreliable; TGOPD audits the teacher first and only distills when the audit passes.

Without any probing, the teacher node runs at only 5.0 %–9.8 % average utilization, with 57 %–78 % of 15‑second samples below 5 %.

Table 3 reports these idle‑time statistics across all four configurations.

Activating probes raises SOPD teacher utilization to 78.9 % for the 4B model.

Figure 4 (left panel) shows the distribution shift for SOPD 4B.

Activating probes raises SOPD teacher utilization to 82.8 % for the 35B model.

Figure 4 (left panel) shows the distribution shift for SOPD 35B.

With probes, MOPD teacher utilization reaches 66.6 % for the 4B model.

Figure 4 (right panel) displays the bimodal MOPD distribution for 4B.

With probes, MOPD teacher utilization reaches 57.7 % for the 35B model.

Figure 4 (right panel) displays the bimodal MOPD distribution for 35B.

**Figure 4.** GPU utilization before and after TGOPD, across four configurations. (a) Split-violin view of teacher-node utilization. Under vanilla OPD the mass piles up near 0% in every configuration; TGOPD shifts the whole distribution upward, showing that probes use the idle window. The MOPD distributions under TGOPD are visibly bimodal because probing is domain-routed: the teacher node is partitioned into three domain-specialist engines and only the engine owning the current prompt's domain is active, so the node alternates between partial and full occupancy rather than saturating as a single undivided teacher does. (b) Cluster-average utilization. Gains track both the teacher's share of the cluster (1/5 at 4B vs. 1/7 at 35B) and the saturation the probe achieves (higher under SOPD than under MOPD).

**Figure 5.** **Gate-threshold sweep on math benchmarks** (Qwen3.5-4B, $K_T = 5$, 99 steps). Solid: TGOPD; dotted: Vanilla OPD. All three benchmarks peak near $\tau = 3/5$.

GRPO fallback improves average score over the Mask‑only policy by +0.43 points for the 4B model.

Table 4 shows 54.54 vs. 54.97 average scores.

GRPO fallback improves average score over the Mask‑only policy by +0.16 points for the 35B model.

Table 4 shows 61.94 vs. 61.78 average scores.

Mask‑only policy raises average score over vanilla OPD by +1.08 points on single‑domain runs.

Table 5 compares 62.20 (vanilla) to 63.52 (mask‑only) for Qwen3.5‑4B.

GRPO fallback raises average score over vanilla OPD by +1.20 points on single‑domain runs.

Table 5 compares 62.20 (vanilla) to 63.40 ≈ 63.67 (GRPO) for Qwen3.5‑4B.

Overall, GRPO fallback outperforms masking by +0.57 average points across all settings.

Aggregating the differences in Tables 4 and 5 yields a net +0.57 advantage.

Related Work

We survey on-policy distillation variants and multi-teacher approaches, highlighting reliability‑aware extensions.

This section positions our work among prior on‑policy distillation and multi‑teacher systems, emphasizing how reliability‑aware gating differentiates the proposed approach.

Multiple expert teachers each specialize in a domain; a student routes each prompt to the teacher whose expertise best matches the input.

Introduces a forward‑KL term that emphasizes high‑entropy teacher tokens, encouraging the student to focus on uncertain predictions.

Defines token‑level trust regions by measuring agreement between teacher and student decodings, limiting distillation to mutually agreed tokens.

Stabilizes implicit token rewards by clipping mixtures and sampling student entropy, mitigating instability in dense supervision.

Extrapolates token rewards from compatibility signals under a batch‑level budget, without requiring an external verifier.

Uses consensus among multiple sampled teacher traces as a trajectory‑level reliability proxy when no external grader is available.

Shows that high teacher entropy can indicate either harmful uncertainty or useful solution diversity, depending on token position.

Identifies a position‑dependent degradation of off‑policy teacher guidance, revealing limits of KL‑based or entropy‑based signals.

Conditions trajectory‑level teacher supervision on agreement between verifier feedback and the teacher‑student likelihood gap.

Determines update direction from environmental correctness while self‑distillation modulates the magnitude of the update.

Evaluates teacher‑proposed candidates using verifier‑scored student continuations and distills outcome‑calibrated local targets.

Routes prompts to a verifier when the teacher fails the reliability audit, withholding dense supervision.

Adapts domain scheduling dynamically to balance teacher expertise and compute, improving efficiency of multi‑teacher distillation.

Routes each prompt to a domain‑expert teacher, integrating multiple capabilities into a single student without verification.

Training Hyperparameters

Training configurations and evaluation benchmarks for both model scales.

Table 6 lists the full training configuration for the two model scales. Both Vanilla OPD and TGOPD share student initialization, data, optimizer settings, and system topology, differing only in model‑dependent system parameters.

**Table 6.** Training configuration by model scale. Both scales follow the same optimization and online-probing protocol; only model-dependent system settings differ.

The 4B experiments run on five 8‑GPU nodes (40 GPUs) split into trainer, student‑rollout, and teacher roles, while the 35B experiments use seven 8‑GPU nodes (56 GPUs) with a larger trainer pool.

For each prompt, the frozen teacher generates three fresh probe responses online, and the gate opens when at least two pass the verifier, applying the same reliability test across both scales.

**Table 7.** Evaluation benchmark details. *Runs*: number of independent generations per checkpoint; the score reported in all main-text tables is the mean over these runs.

The evaluation suite covers three domains—Math, Code, and IF—with metrics such as accuracy and Pass@1, and the reported scores in the main text are means over the listed number of runs.

GPU Utilization Traces

GPU utilization traces reveal how TGOPD reclaims idle teacher capacity.

This appendix presents the raw one‑hour GPU utilization traces for each configuration, sampled at 15‑second intervals.

**Table 3.** Compute Efficiency: Reclaiming Idle Teacher Capacity

**Figure 6.** Teacher-node GPU utilization over a one-hour window for each configuration. Light blue: vanilla OPD. Dark blue: TGOPD. The shaded region indicates the teacher capacity reclaimed through probe generation. Dashed lines indicate the mean utilization of each run. (a,b) Single-domain OPD at the 4B and 35B scales. (c,d) Multi-domain OPD at the corresponding scales.

To avoid inferring wall‑clock neutrality from utilization alone, we compare the completed 35B CodeIO TGOPD and OPD runs under identical settings, finding TGOPD increases mean step time by 5.9 % while decode throughput changes by less than 0.1 %.

The baseline teacher node is bursty, not uniformly slow: each light‑blue spike marks a scoring pass, while the flat intervals correspond to the student’s rollout phase.

Under TGOPD, the dark‑blue curve occupies a high band and rarely returns to zero, but its peaks never exceed those of the baseline, indicating probes fill idle gaps without adding load.

Multi‑domain probing oscillates because the teacher node is partitioned into domain‑specialist engines; only the engine matching the current prompt’s domain is active, producing variable occupancy while keeping idle fraction low (0–2 %).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers