Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun
On-Policy Reverse Distillation (OPRD) accelerates student training by amplifying verifier-driven gradients along the teacher's learned policy shift.
Can we improve a stronger student model by using a weak teacher's policy shift as a directional gradient constraint, rather than relying on off-policy generations?
Standard distillation forces a student to match a weak teacher's entire policy, which often imposes a capacity ceiling and suppresses the student's own reward-supported discoveries. OPRD instead extracts only the teacher's learned policy shift—the change from its reference policy—and uses it to rescale the student's own verifier-driven policy gradient. This amplifies updates where the student and teacher align while allowing the student to diverge when the verifier rewards it. In successive model transfer, OPRD reaches teacher-level performance with up to 67% fewer updates than standard reinforcement learning and continues to improve beyond the teacher's plateau.
Paper Primer
OPRD treats the teacher's post-training policy change as a directional guide rather than an optimization target. By projecting the student's verifier-driven gradient onto this teacher-shift direction and amplifying the projected component, the method accelerates convergence without moving the stationary points of the student's objective.
OPRD significantly improves sample efficiency in weak-to-strong generalization.
In successive model transfer, OPRD reaches weak-teacher performance with 33–67% fewer student updates than standard GRPO. Up to 22.7 percentage points higher performance at early training checkpoints.
OPRD avoids the performance saturation typical of standard distillation.
Unlike On-Policy Distillation (OPD), which plateaus near the teacher's performance, OPRD continues to improve, ultimately surpassing both the teacher and reinforcement learning baselines. 11.09 points higher average performance than Mix-RL in multi-teacher consolidation.
Why does standard distillation fail to fully exploit weak-to-strong generalization?
Standard distillation treats the weak teacher's final policy as a target, forcing the student to inherit the teacher's capacity limitations and biases even when the student could discover better solutions via verifier rewards.
Does OPRD require the teacher to be stronger than the student?
No; OPRD is effective in both weak-to-strong and strong-to-weak settings because it uses the teacher's policy shift only to rescale the student's own verifier-driven gradient, rather than requiring the student to match the teacher's absolute performance.
Introduction and Motivation
We expose why off‑policy weak supervision stalls strong models and set up the paper’s core premise.
Conventional distillation forces a student to match a weak teacher’s policy, which caps the student’s ultimate performance. This off‑policy reliance on weak supervision creates a bottleneck when we try to transfer post‑training gains to stronger models in successive‑model transfer or multi‑domain consolidation.
Stronger models can learn from weaker supervisors and still surpass them, provided the supervision does not constrain the student’s capacity.
**Figure 1.** On-policy reverse distillation (OPRD) enables faster and stronger weak-to-strong generalization across two key settings. (Left) For successive model transfer, a checkpoint from a post-trained 4B-scale model serves as the teacher for an 8B-scale student. We average evaluations conducted every 30 training steps: Mean@16 over AIME'24, AIME'25, HMMT'25, and OlympiadBench for math, and Pass@1 over Knights & Knaves, Quantum Lock, String Manipulation, and Countdown for reasoning tasks. (Right) In multi-domain consolidation, four domain-specialized 4B-scale teachers are distilled into a single 8B-scale student. Training examples are randomly mixed within each batch, with the corresponding domain teacher activated for each example. We report performance every 60 steps for Logic (averaged over Knights & Knaves and Quantum Lock), Algorithms (String Manipulation), and Games (Countdown). The gray dashed lines denote the performance of the corresponding weak teachers.
The bottleneck of off‑policy weak supervision limits strong models; OPRD sidesteps this by leveraging the teacher’s policy shift as a directional gradient cue.
On-Policy Reverse Distillation
OPRD rescales the student’s gradient along the teacher’s policy‑shift direction to accelerate verifier‑driven learning.
On‑Policy Reverse Distillation (OPRD) tackles the core pain of weak‑to‑strong generalization: the student’s on‑policy updates are limited by a weak teacher’s static policy. By reshaping the student’s own gradient using the teacher’s policy‑shift direction, OPRD injects useful signal without turning the teacher into a hard target.
RLVR lets a language‑model policy improve by receiving rewards from programmatic verifiers—e.g., exact‑answer checks or code execution—so the learning signal is grounded in concrete, testable outcomes.
OPRD extracts the direction in which the teacher’s policy changed during its own RL fine‑tuning, then amplifies the student’s gradient only along that direction, leaving all orthogonal components untouched.
Projection coefficient $u_t = d_t^\top g_t = 0.6\cdot0.2 + 0.8\cdot(-0.1) = 0.12 - 0.08 = 0.04$.
Projected component $u_t d_t = 0.04 \times (0.6,\;0.8,\;0) = (0.024,\;0.032,\;0)$.
Orthogonal remainder $g_t^\perp = g_t - (0.024,\;0.032,\;0) = (0.176,\; -0.132,\; 0.05)$.
Scaled gradient $\tilde{g}_t = (1+\lambda) (0.024,\;0.032,\;0) + (0.176,\; -0.132,\; 0.05) = 3 \times (0.024,\;0.032,\;0) + (0.176,\; -0.132,\; 0.05) = (0.072,\;0.096,\;0) + (0.176,\; -0.132,\; 0.05) = (0.248,\; -0.036,\; 0.05)$.
Amplifying the aligned component boosts the update in the teacher’s preferred direction while the orthogonal part remains exactly as before, preserving any verifier‑driven signal that lies off the teacher’s shift.
How does OPRD differ from standard On‑Policy Distillation (OPD), which also samples the student’s own trajectories?
OPD treats the teacher’s full policy as a target distribution and minimizes a reverse‑KL loss, forcing the student to match every token probability. OPRD, by contrast, never forces the student to adopt the teacher’s policy; it only rescales the student’s own gradient along the teacher’s *direction of change* $d_t$, leaving the orthogonal component untouched. This means OPRD can accelerate learning when the teacher’s shift aligns with the verifier’s objective, yet still permits the student to move beyond the weak teacher.
**Figure 2.** Conceptual overview of On-Policy Reverse Distillation (OPD). The figure illustrates OPRD’s gradient correction procedure for a single query. Here, $z_T$ denotes the logits of the weak teacher after post-training and $z_T^{ref}$ those of its reference policy, and $C$ denotes mean-centering. Their centered difference $\Delta_t$ is the teacher’s policy shift at that student-visited prefix, and OPRD keeps only its unit direction $d_t$. In practice, we use a simple top-10 truncation under the student policy to focus the correction on its high-probability vocabulary region. The rightmost panel provides a conceptual view of the resulting student trajectory in the optimization landscape, where the student follows the verifier-driven policy gradient $g_t$ with its component along $d_t$ amplified by $1 + \lambda_t$ at each token and its orthogonal component left unchanged.
Experimental Results
OPRD delivers sizable gains over off‑policy baselines across reasoning benchmarks.
OPRD outperforms all off‑policy baselines, achieving the highest average scores on both Math Reasoning and Reasoning Gym.
Table 1 shows OPRD leads with average 73.30 on Math Reasoning and 58.77 on Reasoning Gym, surpassing the next best baseline by over 7 points.
**Table 1.** Performance comparison of different distillation methods on Math Reasoning and Reasoning Gym benchmarks.
**Table 2.** (Left) Experimental results for multi-teacher distillation on Reasoning Gym. We consolidate four task-specific Qwen3-4B-Base teachers into a single Qwen3-8B-Base student. The teacher and initial-student rows report fixed-checkpoint performance, while each trained-policy row averages Pass@1 over checkpoints at steps 60, 120, 180, 240, and 300. Detailed learning curves for each task are provided in Appendix E. (Right) Experimental results for strong-to-weak distillation. We evaluate Qwen3-8B → Qwen3-1.7B on AIME'24 and Qwen3-8B-Base → Qwen3-0.6B on Knights & Knaves. Each trained-policy row averages Mean@16 and Pass@1, respectively, over five checkpoints. Detailed learning curves are provided in Appendix F. The best result in each column is shown in bold.
**Figure 3.** (a) Ablations of scaling-direction construction. All OPRD variants use the step-60 GRPO checkpoint as the weak teacher. For OPSD, a verified draft generated by this teacher is provided as privileged context. (b) Ablation of directional amplification strength. We vary $\lambda$, the coefficient applied to OPRD's directional correction term. $\lambda = 0$ corresponds to GRPO. The gray dashed line marks the performance of the teacher checkpoint. All other settings follow the default configurations in Section 3.1. (c) Alignment dynamics between $d_t$ and $g_t$. On Knights & Knaves, we track $\theta_t$ between $d_t$ and $g_t$ during OPRD with a Qwen3-8B-Base student and Qwen3-4B-Base weak teacher. Excluding rollout groups with $g_t = 0$ (identical rewards within the group), we report token-averaged angles for aligned ($u_t \geq 0$) and opposed ($u_t < 0$) tokens.
OPRD consistently outperforms off‑policy baselines.
Ablations and Comparative Analysis
We evaluate how each component of OPRD contributes to its performance gains.
Recall that OPRD steers on‑policy learning by rescaling the verifier gradient with the teacher’s policy shift, preserving directions not captured by the weak policy delta.
**Table 3.** 4B (-Base) $arrow$ 8B (-Base) performance comparison across AIME'24, Knights, and String tasks.
Scaling the directional correction factor $\lambda$ to 0.5 yields strong early gains and matches the best‑performing baseline.
Performance improves monotonically up to $\lambda$ = 0.5 and changes little beyond that value.
A step‑60 teacher (29.0 % Pass@1) still enables the OPRD student to reach 88 % Pass@1, far surpassing the teacher.
Despite the teacher’s modest performance, the student rapidly climbs to ~88 % Pass@1 under OPRD.
Using a step‑0 reference policy causes OPRD to plateau at 52.5 % Pass@1 due to length‑bias amplification.
When the reference is taken from the initial checkpoint, OPRD shortens responses early but stalls at 52.5 %.
Switching the reference to the step‑30 checkpoint lifts the plateau, achieving 89.5 % Pass@1.
With a later reference the length bias is reduced, and OPRD climbs to 89.5 % Pass@1.
**Figure 4.** (a) Results under limited policy-gradient signal. On Knights & Knaves, we transfer a step-105 Qwen3-8B-Base teacher to a Qwen3-1.7B-Base student, with OPRD’s negative-branch scale $\lambda_t$ warmed up over the first 75 steps. (b, c) Effect of reference-policy selection on length bias. On Color Cube, we transfer a step-105 Qwen3-4B-Base teacher $\pi_T$ to a Qwen3-8B-Base student, using either the step-0 or step-30 checkpoint from the same GRPO run as $\pi_T^{ref}$. OPRD’s negative-branch scale $\lambda_t$ is warmed up over the first 75 steps. The gray dashed line marks teacher performance, while the colored stars denote the mean response lengths of the two choices of $\pi_T^{ref}$, and the white star marks that of $\pi_T$. All other settings follow Section 3.1.
**Figure 5.** (a) Visualizing token alignment and reasoning continuations. An AIME'25 response from the OPRD student at update 150. Green and red indicate positive and negative cosine similarity between $d_t$ and the student policy gradient $g_t$ with $A_t = 1$, respectively (see Appendix J.1). The token outlined in black, 0, has the lowest cosine similarity among displayed tokens. The student's top-1 token 0 completes 2016 directly. Forcing 5, the top-1 token under $d_t$, leads the same student to this result through an intermediate sum. The plots show the student's top-10 token probabilities above and their teacher-shift values ($d_t$) below. (b) Measuring similarity to teacher and student response styles. On AIME'24, we compare response styles using 101 standardized features across five categories. Normalized distance differences indicate whether each method's average style is closer to the weak teacher (red) or the GRPO-trained student at update 150 (green).
Related Work
Survey of prior techniques for bridging weak and strong supervision in RL and language models.
Weak-to-Strong Generalization has been reported in language understanding, reward modeling, and reasoning, yet weak supervision typically recovers only part of the gap to strong supervision.
Analyses attribute gains to correcting weak pseudo‑labels, extending coverage beyond the weak teacher, and mismatches between teacher and student hypothesis classes; by contrast, naive fine‑tuning can overfit the weak errors.
Methods such as W2SR‑P, S2L‑PO, and weak‑critiques modify the student’s training data, exploration, or feedback, whereas OPRD leaves all three unchanged and only rescales the student’s own policy gradient.
On‑Policy Distillation (OPD) makes distillation fully on‑policy by querying the teacher along the student’s current rollouts, fixing the prefix mismatch between training and inference.
Subsequent work leverages the same interface to merge specialist teachers, exploit privileged training‑time information, or extrapolate the implicit reward, yet the student still matches the teacher’s token distribution, so the weak policy remains the optimum in the weak‑to‑strong regime.
Distillation combined with verifier‑based reinforcement learning (e.g., KDRL) adds a teacher‑matching term to the reward objective, while other variants modify teacher guidance via policy ratios, reward‑based selection, or token‑level interventions.
These additional objectives can compete with pure reward maximization; OPRD avoids any teacher‑matching loss and optimizes reward alone.
Several works transfer the policy shift between a post‑trained policy and its reference, using it to steer larger frozen models during decoding or as an alignment target during training (e.g., W2S‑OPD, Direct‑OPD).
OPRD instead uses the weak‑policy delta to rescale the student’s gradient, transferring direction without making the shift itself an optimization target.
Gradient‑manipulation approaches such as gradient surgery, moving‑average projections, and cosine‑gated auxiliary gradients modify gradient directions to resolve conflicts.
OPRD is closest to this family in form but merely amplifies the component of the student’s policy gradient that already aligns with the teacher direction, leaving stationary points unchanged.
Conclusion
We wrap up OPRD’s benefits and outline key directions for extending its impact.
OPRD transfers a weak teacher’s post‑training policy shift by amplifying the aligned component of the student’s policy gradient, rescaling rather than replacing it. This accelerates learning while preserving the stationary points of the policy objective. Empirically, OPRD reaches teacher‑level performance in far fewer updates and continues improving after on‑policy distillation plateaus.
The student’s response style stays closer to the reward‑only baseline than to the teacher, indicating that the shift is expressed through the student’s own policy rather than pure imitation. Thus OPRD enables efficient transfer from smaller specialists without fixing the student’s optimization target.
Future work should explore broader tasks and settings, scaling to larger models, system considerations at scale, and recursive self‑improvement.
Mathematical and logical reasoning provide controlled environments where verifier feedback and policy improvement can be measured directly. Evaluations on code generation and agentic environments would test whether OPRD transfers useful policy shifts under diverse feedback and interaction forms, where early actions affect later observations and rewards.
Our experiments span Qwen3 models from 0.6 B to 8 B parameters, covering both weak‑to‑strong and strong‑to‑weak orderings. At larger scales, learning a policy shift with a smaller model could be far cheaper than optimizing the large model directly, and we need to assess whether the same efficiency gains persist.
OPRD adds only a frozen‑teacher forward pass and a correction of the student’s logit gradient, incurring modest overhead: 11.9 % more wall‑clock time and 10.2 % more peak GPU memory compared to GRPO. Scaling to frontier models will require efficient placement, sharding, and communication‑efficient correction across model and vocabulary shards.
Connecting weak‑to‑strong distillation with recursive self‑improvement could let each generation both supervise and directly update its successor via post‑training policy shifts. Prior work such as GPT‑6 Astra and Gemini 3.8 Flash illustrate pipelines where earlier models feed into later ones, and incorporating reverse distillation may further improve sample efficiency.
We thank Kee‑Eung Kim for computational resources, Rishabh Agarwal for discussions, and Reza Bayat for manuscript feedback.
Optimization Properties
Supplementary technical details, proofs, and training configurations.
Section A formalizes how OPRD scales the token‑level policy gradient by the matrix $I + \lambda_t d_t d_t^{\top}$, amplifying the component along the teacher direction $d_t$ while leaving orthogonal components unchanged.
For logits $z^k$, let $g_t=\nabla_{z_t}J(z^k)$ and $u_t = d_t^{\top}g_t$ with $\|d_t\|_2=1$, $\lambda_t\ge0$, and step size $\eta>0$. Defining $\tilde g_t = (I+\lambda_t d_t d_t^{\top})g_t$ and $z^{k+1}_t = z^k_t + \eta\tilde g_t$, we have $\tilde g_t=0$ for all $t$ iff $\nabla_z J(z^k)=0$, and if $\eta L(1+\bar\lambda)\le1$ (where $\bar\lambda=\max_t\lambda_t$) then
Section B investigates the asymmetric effects that arise when scaling only the positively aligned component of the gradient.
In B.1 the authors show that with positive‑only scaling ($\lambda_-=0$) the correction coefficient $c_t$ contains a term proportional to $|u_t|$, which remains non‑negative regardless of the sign of $u_t$, creating a one‑sided amplification.
B.2 isolates the two branches by activating scaling exclusively for $u_t\ge0$ (positive‑only) or $u_t<0$ (negative‑only), revealing that positive‑only scaling yields rapid early gains but also inflates response length, whereas negative‑only scaling shortens responses and quickly collapses performance.
B.3 proposes three scheduling strategies—default OPRD schedule, $\lambda_+$ annealing, and fixed symmetric scaling—and reports that the default schedule (gradually ramping up $\lambda_-$) preserves early acceleration while later allowing the negative branch to contribute, outperforming the other two variants.
Section C lists the concrete training hyper‑parameters used for the experiments, separating shared defaults (Table 4) from method‑specific overrides (Table 5).
**Table.** Comparison of optimization settings across GRPO, OPD, KDRL, and OPRD methods.
**Table 5.** Method-specific training settings. All distillation-based methods use the same task-specific frozen teacher checkpoint specified in Table 4.
Section D provides the full learning curves and additional quantitative results for the weak‑to‑strong transfer experiments, complementing the summary plots in the main text.
Detailed Experimental Results
Appendix compiles detailed experimental curves, ablations, and resource costs for OPRD.
D.1 reports learning curves for successive model transfer within the Qwen3 family, comparing OPRD against GRPO, OPD, and KDRL on four math benchmarks and four Reasoning‑Gym tasks.
D.2 shows that OPRD’s speed advantage persists when response‑length confounds are reduced by using Qwen3‑Base models, and that OPRD improves faster even when responses shorten.
D.3 provides a response‑resampling robustness check (Table 8) and confirms that OPRD still outperforms the strongest baseline by roughly 8.0 points on Math and 9.9 points on Reasoning Gym.
E demonstrates that multi‑teacher OPRD transfers specialist capabilities across heterogeneous tasks, achieving the highest Pass@1 on all four Reasoning‑Gym domains.
F reports strong‑to‑weak distillation results: OPRD lifts the Qwen3‑1.7B student’s Mean@16 from 10.0 to >41 after 150 updates, whereas GRPO reaches only ~25 at the same point.
G.1 details baseline implementations (W2SR‑P, S2L‑PO, OPSD, Direct‑OPD, W2S‑OPD) ensuring fair comparison under the default training configuration.
G.2 (Figure 14) shows OPRD’s consistently strongest gains among methods that rely on off‑policy generations from the weak teacher.
Figure 15 (G.2) compares OPRD with Direct‑OPD and W2S‑OPD, highlighting OPRD’s ability to surpass teacher‑level performance.
H (Figure 16) evaluates three guidance‑direction constructions; the weak‑policy delta consistently outperforms OPD and OPSD.
I investigates length‑bias in the teacher‑policy shift; using a step‑45 reference eliminates the bias, raising Pass@1 to 96.0 % versus 81.5 % with a step‑0 reference.
J.1 visualizes token‑level alignment between OPRD‑trained student gradients and the guidance direction, confirming that OPRD preserves beneficial alignment.
J.2 (Figure 18) reports energy‑distance and style‑category metrics; OPRD maintains a positive energy‑distance advantage of 0.447, indicating closer similarity to the GRPO student than to the teacher.
K quantifies computational overhead: OPRD adds 11.9 % wall‑clock time and 13.98 GiB (10.2 %) peak GPU memory compared with GRPO.
**Figure 6.** Evaluation performance and response length over training for OPRD variants with only the positive- or negative-alignment branch active. For Math and Knights & Knaves, we use Qwen3-4B and Qwen3-4B-Base teacher checkpoints obtained after 75 and 105 RL training steps, respectively. The two single-branch OPRD variants are trained for 45 steps with $\lambda = 0.5$, while GRPO is shown through 150 steps for reference. The maximum generation lengths are 20K and 8K tokens for the two settings, respectively. The gray dashed lines denote the performance or response length of the corresponding weak teachers. All other training settings follow the dataset-specific default configurations described in Appendix C.
**Figure 7.** Comparison of three branch-scheduling strategies. We compare the default $\lambda_-$ ramp-up ($\lambda_+ = 0.5, \lambda_- : 0 \to 0.5$), $\lambda_+$ annealing ($\lambda_+ : 0.5 \to 0, \lambda_- = 0$), and fixed symmetric scaling ($\lambda_+ = \lambda_- = 1.0$) against GRPO. The two scheduled variants use horizons of 30 updates for Math and 75 updates for Knights & Knaves. We use Qwen3-4B and Qwen3-4B-Base teacher checkpoints obtained after 75 and 105 RL training steps, respectively. Gray dashed lines denote teacher performance. All other training configurations follow Appendix C.
**Figure 8.** Learning curves for successive model transfer on individual math benchmarks. We use Qwen3-4B as the teacher and Qwen3-8B as the student. The gray dashed line denotes the performance of the weak teacher. All training settings follow the default Math configuration described in Appendix C.
**Figure 9.** Learning curves for successive model transfer on individual reasoning benchmarks. We use Qwen3-4B-Base as the teacher and Qwen3-8B-Base as the student. The gray dashed line denotes the performance of the weak teacher. All training settings follow the default Reasoning Gym configuration described in Appendix C.
**Figure 10.** Learning curves on individual math benchmarks using Qwen3-Base models. We use Qwen3-4B-Base as the teacher and Qwen3-8B-Base as the student. The gray dashed line denotes the performance of the weak teacher. All other settings follow the default Math configuration in Appendix C, but we omit the system prompt and reduce the mini-batch size to 32, yielding two optimizer steps per training batch.
**Figure 11.** Learning curves on additional three Reasoning Gym tasks. We use Qwen3-4B-Base as the teacher and Qwen3-8B-Base as the student. The gray dashed line denotes the performance of the weak teacher. For Color Cube Rotation and Binary Matrix, we use the teacher checkpoints from steps 30 and 45, respectively, as the reference policies instead of the raw step-0 models to mitigate length bias (see Appendix I for details).
**Figure 12.** Learning curves on individual Reasoning Gym tasks under multi-teacher distillation. We use four task-specific Qwen3-4B-Base models as teachers and jointly train a Qwen3-8B-Base student. The gray dashed line denotes the performance of the corresponding specialist teacher. All other settings follow the configurations described in Section C.
**Figure 13.** Learning curves for strong-to-weak distillation on two tasks. We evaluate Qwen3-8B $arrow$ Qwen3-1.7B on AIME'24 and Qwen3-8B-Base $arrow$ Qwen3-0.6B on Knights & Knaves, using teachers from step 105 of task-specific GRPO. Gray dashed lines mark teacher performance. All other settings follow Appendix C, with method-specific distillation coefficients scheduled over the first 45 updates.