Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian
CriticOPD improves on-policy distillation by using teacher critiques to rewrite erroneous student rollouts, restoring reliable supervision.
How can we improve student model training when the teacher model fails to provide reliable supervision on incorrect student rollouts?
On-policy distillation (OPD) trains students on their own rollouts, but teacher supervision fails when the student errs: the teacher often follows the student's mistakes rather than correcting them, rendering the target unreliable. CriticOPD restores this reliability by having the teacher critique incorrect rollouts, allowing the student to rewrite the solution from the point of error. The student then learns from this revised, corrected trajectory without needing the critique at inference. On mathematical reasoning benchmarks, this method consistently outperforms standard OPD, achieving gains of up to 7.36 points on challenging competition problems.
Paper Primer
The core move is to replace unreliable supervision on erroneous student paths with supervision on corrected, student-written rewrites. The teacher acts as a critic that identifies specific errors, and the student uses this feedback to generate a revised suffix, ensuring the training data remains student-generated while the teacher's scoring target becomes accurate.
CriticOPD significantly improves student accuracy on mathematical reasoning tasks compared to standard OPD.
Across six benchmarks, CriticOPD consistently outperformed OPD, with the largest gains observed on AIME 2024, AIME 2025, and AMC 2023. Average accuracy gains of 4.57 points for Qwen3-1.7B and 4.06 points for Qwen3-0.6B.
The method hinges on the observation that while a teacher's ability to continue a sequence correctly (recoverability) drops sharply after a student error, its ability to identify those errors remains stable. By using the critique to guide a rewrite, the teacher's recoverability on the revised suffix increases from 0.14 to 0.67.
Why does this approach use a critique-guided rewrite instead of simply having the teacher take over the generation?
Teacher takeover introduces significant occupancy mismatch, training the student on sequences it would rarely generate itself. CriticOPD keeps all supervised tokens student-generated, minimizing this mismatch while still providing a reliable corrective target.
What is the scope of this method — does it apply to any task?
The current implementation requires reference answers to ground the critic, making it directly applicable to verifiable tasks like mathematics or code. Tasks without verifiable answers would require a different reward model or critic design.
By decoupling error recognition from continuation, CriticOPD demonstrates that teachers can provide high-quality supervision even on paths they cannot initially solve, provided the student is guided to a correctable state.
The Reliability Gap in Distillation
OPD struggles when teachers cannot correct student mistakes, prompting a critique‑driven fix.
OPD assumes the teacher can reliably continue from any student prefix, but when the student deviates into erroneous continuations the teacher often propagates those errors instead of correcting them. This failure stems from a mismatch between the distribution of prefixes the student encounters at inference and the distribution seen during training, a phenomenon we call occupancy mismatch. The resulting unrecoverable errors limit OPD’s ability to improve student performance.
OPD trains a student model on the sequences it generates, using a teacher’s token‑level predictions as supervision at each step.
The teacher's inability to recover from student errors is the primary bottleneck in OPD.
Quantifying Teacher Failure
We quantify teacher supervision reliability via recoverability and expose the occupancy mismatch that hampers OPD.
We assess how reliably a teacher can continue a student‑generated prefix by measuring recoverability and by exposing the occupancy mismatch that makes OPD supervision brittle.
It is the expected correctness of the teacher’s continuation when starting from a given student prefix.
It occurs when the distribution of student prefixes drifts into regions where the teacher’s recoverability is low, so supervision becomes unreliable.
**Figure 1.** Where the supervision of each method falls after the student errs, and who writes it.
We evaluate on a Qwen3‑1.7B student after 80 OPD steps, a Qwen3‑4B‑Instruct‑2507 teacher, and DAPO‑Math‑17k problems. For each of 900 problems we generate one rollout, keep 150 correct and 150 incorrect rollouts, cut each at the empty prefix and at seven equally spaced paragraph boundaries (≈ 1/8 to 7/8 of its length), and sample 16 continuations per cut to estimate $\Lambda_T$ and $\Lambda_S$.
**Figure 3.** Teacher ($\Lambda^T$) and student ($\Lambda^S$) continuation accuracy along (a) correct and (b) incorrect rollouts.
Occupancy mismatch occurs when the student’s training distribution drifts away from the teacher’s reliable regime, causing a large fraction of supervision to be drawn from low‑recoverability prefixes.
The CriticOPD Mechanism
CriticOPD trains the student on rewrites that isolate errors, letting the teacher give reliable feedback.
The student keeps the prefix up to the last mistake, the teacher critiques that point, and the student rewrites the suffix under the critique, so supervision is reliable exactly where the teacher can continue.
How does CriticOPD differ from a teacher‑takeover approach?
In teacher‑takeover the teacher generates the entire continuation; in CriticOPD the student writes every token, only the suffix is guided by the critique, preserving the student’s agency while still receiving reliable feedback.
The teacher produces a textual critique that pinpoints each error; the student truncates at the last error and generates a new suffix conditioned on that critique, turning error signals into a constructive learning signal.
Why not let the teacher generate the corrected continuation directly?
Doing so would remove the student’s responsibility for producing the tokens; CriticOPD forces the student to write the suffix, so it learns to correct itself and can operate without a critique at inference.
Retain prefix $h_{\tau} = (x, y_{<4}) = [\,\text{"The} , \text{"answer} , \text{"is} \,]$.
Form critique $c$ = “Error at token 4: ‘3’ → should be ‘4’.”.
Append $c$ to the prefix and sample a new suffix; the student generates token 4 = “4” and token 5 = “.”, yielding $\tilde y = [\,\text{"The} , \text{"answer} , \text{"is} , \text{"4} , \text{"."}\,]$.
Teacher scores the rewrite with $c$ in context; the student’s log‑probability is computed without $c$.
The rewrite replaces the erroneous token while keeping the rest of the rollout unchanged, demonstrating how the critique isolates the exact failure point for targeted correction.
Sample a student rollout $y$ for problem $x$ using prompt $P_{\text{train}}$.
If $y$ is correct, compute standard OPD loss $L_{\text{OPD}}$ on all prefixes $h_t$.
If $y$ is incorrect, run the teacher critic to produce critique $c$ and cut point $\tau$.
Form retained prefix $h_{\tau} = (x, y_{<\tau})$ and append $c$.
Sample a new suffix conditioned on $c$, obtaining the rewritten rollout $\tilde y$.
Compute feedback‑distillation loss $L_{\text{fb}}$ on suffix tokens $t\ge \tau$.
Update student parameters using the total loss $L_{\text{ours}} = L_{\text{OPD}} + L_{\text{fb}}$.
**Figure 2.** Overview of CriticOPD. The student rewrites an incorrect rollout from the last error that the teacher critiques. The teacher scores the rewrite with the critique, and the student learns from it without the critique.
**Figure 5.** An example of critique-guided rewriting. Left: the original solution, with the critiqued paragraphs marked. Middle: the critique, which quotes each erroneous sentence, gives a fix, and ends with the reference answer. Right: the suffix that the student writes from the last error with the critique.
Experimental Setup and Baselines
CriticOPD’s accuracy dominates all baselines on six math benchmarks.
We compare four classes of baselines on six mathematical‑reasoning benchmarks. The first class contains standard post‑training methods (SFT, KD, GRPO, OPD). The second class uses privileged information like OPSD‑T and Critique‑GRPO. The third class reduces supervision via FastOPD, and the fourth class rewrites student trajectories (TRD, SKD, RelayOPD). All methods share the same training recipe and token budget of 32,768.
Supervised fine‑tuning trains the student directly on the teacher’s ground‑truth solutions.
FastOPD truncates long student rollouts, thereby reducing the amount of supervision the teacher must provide.
**Table 1.** Results on six mathematical reasoning benchmarks. Entries are means and standard deviations over three seeds, and the subscript after each method gives the selected step. Bold marks the best student at each scale.
CriticOPD is evaluated against standard post‑training methods across diverse reasoning benchmarks.
Performance and Ablation Results
CriticOPD’s gains over OPD and component ablations are quantified.
CriticOPD raises OPD student accuracy from 50.08 % to 54.65 % on Qwen3‑1.7B.
Table 1 reports the main results.
**Figure 6.** Recoverability of the teacher and the student on original rollouts (dashed) and critique-guided rewrites (solid), plotted against the position relative to the cut point. On the rewrite, the teacher sees the critique, and the student does not.
Overall, CriticOPD consistently outperforms OPD and privileged‑information baselines, with the largest gains on AIME and AMC benchmarks.
Analyzing Teacher Recoverability
Ablation results linking reliability, occupancy mismatch, and student accuracy.
Recall that OPD fails when the teacher cannot reliably continue from the student’s incorrect prefixes; CriticOPD mitigates this by forcing the teacher to rewrite and correct errors before the student learns. In this ablation we examine how each component influences teacher reliability, occupancy mismatch, and the resulting student accuracy.
**Figure 7.** Occupancy mismatch and teacher reliability on the states that each method supervises, and the average accuracy of the Qwen3-1.7B student.
**Figure 12.** Token positions of the prefixes that each method supervises (colored) and of the inference prefixes (gray), with the total variation distance in each title. The dotted line marks the training limit of 16,384 tokens.
CriticOPD raises the student’s average accuracy by +4.57 percentage points compared with standard OPD.
Table 1 reports an average accuracy of 56.94 % for CriticOPD versus 52.37 % for OPD on the Qwen3‑1.7B student, a +4.57 point gain.
Extended Analysis and Dynamics
We quantify how each component of CriticOPD contributes to student performance.
This section isolates the effect of each design choice in CriticOPD by removing it and measuring the resulting change in student accuracy.
Removing the answer from the feedback reduces the average AIME score by 0.21 points.
CriticOPD AIME Avg = 34.79 vs w/o answer = 34.58.
Removing the critique reduces the average AIME score by 6.57 points.
CriticOPD AIME Avg = 34.79 vs w/o critique = 28.22.
Replacing the targeted critique with a generic one lowers the average AIME score by 0.94 points.
CriticOPD AIME Avg = 34.79 vs critique + answer → generic = 33.85.
Switching the suffix writer from student to teacher drops the average AIME score by 1.61 points.
CriticOPD AIME Avg = 34.79 vs student → teacher = 33.18.
Scoring the suffix without any feedback reduces the average AIME score by 0.66 points.
CriticOPD AIME Avg = 34.79 vs w/o feedback = 33.18 (−0.66).
When the suffix is scored without feedback, the second metric falls by an additional 1.19 points.
Second metric = 53.06 vs 52.53 (−1.19) under the w/o‑feedback condition.
Without feedback, the third metric drops by 4.39 points.
Third metric = 50.18 vs 45.79 (−4.39) when feedback is omitted.
Overall, the full CriticOPD configuration improves average accuracy by +3.80 points compared to the baseline.
Table 2 reports a $\Delta$ of +3.80 for the full system.
When the answer component is omitted, the gain shrinks to +1.79 points.
$\Delta$ = +1.79 for the “w/o answer” ablation.
Removing the critique reduces the gain to +0.66 points.
$\Delta$ = +0.66 for the “w/o critique” ablation.
Using a generic critique + answer yields a gain of +4.06 points.
$\Delta$ = +4.06 for the “critique + answer → generic” ablation.
When the suffix writer is changed to the teacher, the gain rises to +4.57 points.
$\Delta$ = +4.57 for the “student → teacher” ablation.
**Figure 8.** (a) Advantage per token of two continuations of 64 tokens at the cut point, which either continue or correct the error, under the scoring of OPD and of CriticOPD. Numbers give correct minus continue. (b) Success rate of the student of CriticOPD minus that of OPD from the same prefixes on OlympiadBench, with 95% confidence bands.
Training dynamics show consistent improvement in student accuracy over time.
Implementation Details
Appendix A‑C lists all prompts, ablations, and diagnostic details for CriticOPD.
This appendix records the exact prompts used throughout the experiments, with typewriter‑style placeholders for variable parts. In training the student sees each DAPO‑Math‑17k problem as the user message $P_{\text{train}}$, while evaluation uses $P_{\text{eval}}$.
A.1 Prompts of CriticOPD – The critic receives a system prompt $P_{\text{critic}}$ that defines the task and output format, and a user message $P_{\text{critic-user}}$ containing the problem, reference answer, and numbered paragraphs of the student solution. If a rollout hits the length limit, the first paragraph of $P_{\text{critic}}$ is swapped for $P_{\text{len\_critic}}$; the critic’s reply is parsed into error listings, after which $P_{\text{fb}}$ (the feedback) is appended to the kept prefix.
A.2 Prompts of the Ablations – All ablation variants reuse the CriticOPD critic. The “no‑answer” variant drops the final answer line from $P_{\text{fb}}$, while the “no‑critique” variant keeps only that line. Variants (b) and (c) modify these behaviours, and a generic feedback variant replaces $P_{\text{fb}}$ with $P_{\text{generic}}$: “Your solution contains errors. Check the previous reasoning and redo it.”
**Table.** Ablation study of CriticOPD variants.
A.3 Prompts of the Diagnosis – The diagnosis (Section 2.2) uses the CriticOPD critic without the reference answer. For a complete solution the system prompt $P_{\text{diag\_critic}}$ replaces the first paragraph of $P_{\text{critic}}$; for a truncated solution $P_{\text{part\_critic}}$ is used. The line with the reference answer is omitted from $P_{\text{critic-user}}$.
B Details of the Diagnosis – This section expands on the diagnostic procedure, describing rollouts, cut points, continuations, and the evaluation pipeline.
B.1 Rollouts and Cut Points – SFT, KD, GRPO, OPD, FastOPD, SKD, and RelayOPD all use $P_{\text{train}}$. TRD rewrites a student solution with $P_{\text{sys\_TRD}}$ and $P_{\text{TRD}}$. OPSD‑T scores a rollout with a teacher that receives $P_{\text{OPSD}}$. Critique‑GRPO follows the same critique pattern as CriticOPD, appending the reference answer to the feedback.
B.2 Continuations and Success Rates – From each cut point the teacher and student each generate 16 continuations of the prefix $h$. Fractions of correct continuations are denoted $Λ_T(h)$ for the teacher and $Λ_S(h)$ for the student. A problem is deemed solvable by the teacher if $Λ_T(\emptyset) \geq 8/16$.
B.3 Critique and Judgment – For Figure 4(b) the teacher critiques the truncated solution using $P_{\text{part\_critic}}$, decoding greedily up to 2,048 tokens. The Qwen3‑8B judge labels each listed error as `NEW_ERROR`, PROPAGATED, RESTATEMENT, or `NOT_AN_ERROR`, and any label other than `NOT_AN_ERROR` counts as a real error.
**Figure 4.** (a) Outcomes of teacher continuations. (b) Valid error-flag rate across teacher-recoverability bins.
B.4 Rewrite Analysis – Figure 6 isolates rollouts where the teacher’s success rate drops below 0.5 after a cut point. After applying CriticOPD critiques, 161 of 163 rollouts retain at least one error. Continuations are sampled at 25 %, 50 %, and 75 % of the rewrite, comparing teacher and student behavior with and without critique.
B.5 Diagnosis with the Qwen3‑0.6B Student – Repeating the diagnosis with the Qwen3‑0.6B student after 40 OPD steps reproduces the patterns of Figures 9, 10, and 11. Teacher success $Λ_T$ falls from 0.89 to 0.38, while the student’s success remains low, and the fraction of real errors stays between 77 % and 84 %.
**Figure 9.** Expected accuracy of the teacher and the Qwen3-0.6B student from prefixes of the student on (a) correct and (b) incorrect rollouts.
B.6 Occupancy Mismatch – Position distributions are measured over 48 logarithmic intervals (merged to 45). Total variation distance grows as the length limit shortens: OPD (0.10) → 8 k tokens (0.21) → 4 k (0.46) → 2 k (0.68) → 1 k (0.84). RelayOPD and TRD show intermediate distances, while SFT, KD, and SKD remain close to inference.
**Figure 11:** Success rates of the teacher and the Qwen3-0.6B student around the cut point.
Hyperparameters and Configuration
Appendix D‑E details computational costs, extra results, and sensitivity analyses for CriticOPD.
We report the wall‑clock and GPU‑hour costs of each distillation variant on the Qwen3‑1.7B student, then present the extra experimental tables and figures that support the paper’s claims.
**Figure 15:** Sensitivity of CriticOPD on the Qwen3-1.7B student with seed 42. The star marks the default, and the numbers in (b) give the share of truncated critiques.
**Figure 16.** Training dynamics, averaged over seeds and smoothed over five steps.
**Figure 14.** Training cost of the Qwen3-1.7B students. (a) A100 GPU-hours until the selected step, including the generation of offline data. (b) Seconds per training step, averaged over five steps.
**Figure 10.** (a) Outcomes of the continuations of the teacher and (b) fraction of the listed errors that are real, for the Qwen3-0.6B student.
**Figure 13.** Hidden states of the student on the prefixes that each method supervises, projected onto the first two principal components of the inference states. The contour encloses 95% of the inference states, and each title gives the classifier bound on the total variation distance.
**Table 5.** Results outside mathematics for Qwen3-1.7B. Code uses EvalPlus pass@1 on the extended tests.
The table compares different methods (OPD, FastOPD, CriticOPD, and two variants of CriticOPD) across three metrics: AIME length, Avg@16k, and Avg@32k.
**Table 1.** Performance comparison of different methods across various benchmarks (MMLU-Pro, MMLU, ARC-C, OpenBookQA, HumanEval+, MBPP+) and their average scores.
**Algorithm 1** CriticOPD Training **Require:** Student $\pi_{\theta}$, teacher $\pi_{T}$, batch $\{(x, a)\}$ 1: $\mathcal{L}_{\text{ours}} \leftarrow 0$ 2: **for** each $(x, a)$ in the batch **do** 3: $\quadSampley \sim \pi_{\theta}(\cdot \mid x)$ 4: $\quad$ **if** $y$ is incorrect or truncated **then** 5: $\quad \quad$ Generate critique $c$ using $\pi_{T}$ given $(x, y, a)$ 6: $\quad \quad$ Discard incomplete error items from $c$ 7: $\quad \quad$ **if** a valid critique is available **then** 8: $\quad \quad \quad \tau \leftarrow$ first position after the last critiqued paragraph 9: $\quad \quad \quad \tilde{y}_{<\tau} \leftarrow y_{<\tau}$ 10: $\quad \quad \quadSample\tilde{y}_{\geq\tau} \sim \pi_{\theta}(\cdot \mid h_{\tau}, c)$ 11: $\quad \quad \quad \ell \leftarrow \mathcal{L}_{\text{fb}}(\tilde{y}; c)$ 12: $\quad \quad$ **else** 13: $\quad \quad \quad \ell \leftarrow \mathcal{L}_{\text{OPD}}(y)$ 14: $\quad \mathcal{L}_{\text{ours}} \leftarrow \mathcal{L}_{\text{ours}} + \ell$ 15: Update $\theta$ using $\mathcal{L}_{\text{ours}}$
**Table 3.** Shared hyperparameters.