Weak-to-Strong Generalization via Direct On-Policy Distillation

Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

Direct-OPD transfers RL-induced policy shifts from small models to larger ones as dense implicit rewards.

Can we transfer the reasoning improvements gained by RL on a small model to a larger model without repeating the expensive RL process on the larger model?

Reinforcement learning with verifiable rewards (RLVR) is computationally expensive, and repeating it for every new, larger model creates a significant post-training bottleneck. Direct On-Policy Distillation (Direct-OPD) solves this by isolating the RL-induced improvement as a log-ratio between a small model's post-RL and pre-RL checkpoints. It treats this ratio as a dense implicit reward, applying it to a larger student model's own on-policy states without needing to run sparse-reward RL on the target. This approach consistently improves stronger models, lifting Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs.

Paper Primer

Direct-OPD treats the teacher's RL-induced policy shift as a transferable signal: it calculates the log-ratio between the post-RL teacher and its pre-RL reference, then uses this as a dense per-token reward to guide the student. This is like a coach who doesn't show the student the final answer, but instead provides a real-time "hot or cold" signal based on how the coach's own training improved their performance.

Direct-OPD enables weak-to-strong generalization that outperforms direct RL on the target model.

Comparing matched RL-step budgets, transferring the policy shift from a 1.5B model to a 7B model yields higher accuracy than running RL directly on the 7B model. Lifts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in 4 hours, matching the performance of a week-long direct RL run on 32 A100s.

The method is robust to teacher-student capability gaps: it successfully improves students (like R1-Distill-7B) that already outperform the teacher model, proving it transfers a direction of improvement rather than imitating the teacher's absolute policy.

Why is this approach better than simply distilling the teacher's final policy?

Distilling the final policy entangles useful RL gains with the weak model's intrinsic capacity limits. Direct-OPD isolates only the RL-induced change, preventing the student from inheriting the teacher's performance ceiling.

Does this method require the student to mimic the teacher's specific thinking patterns?

No. Direct-OPD does not require high token-overlap with the teacher; it transfers the RL-induced direction of improvement on the student's own visited states, allowing it to work across different model families and architectures.

Motivation and Core Claim

We expose why RLVR costs explode with model size and propose Direct‑OPD to transfer RL gains from a cheap small model to a larger target.

Reinforcement learning with verifiable rewards (RLVR) yields strong reasoning but its cost scales with the size of the model being trained, turning post‑training into a bottleneck as models grow. The paper therefore asks: how can we reap the benefits of RL without incurring the full expense on every new large model? The answer is Direct‑OPD, which transfers the RL‑induced policy shift from a cheap small‑model run to a stronger target model.

**Figure 1.** Direct-OPD transfers the effect of small-model RL rather than imitating the small model. (a) Starting from R1-Distill-7B, vanilla OPD toward the post-RL JustRL-1.5B teacher degrades performance, whereas Direct-OPD transfers the JustRL-1.5B – R1-Distill-1.5B policy shift and improves the student. (b) The same policy shift improves Qwen3-1.7B, Qwen3-4B, and R1-Distill-7B on AIME 2024, including students whose initial accuracy already exceeds the post-RL teacher.

The cost of RLVR scales poorly with model size.

Foundations of On-Policy Distillation

We formalize on‑policy distillation and introduce policy‑shift signals for direct transfer.

Standard on‑policy distillation forces the student to mimic the teacher’s full output distribution, which can be noisy and costly to query exhaustively. In the weak‑to‑strong regime the teacher’s improvement over its pre‑RL reference encodes the useful learning signal we actually want to transfer.

OPD lets a student policy improve by learning from the teacher only on the trajectories it actually generates, avoiding a mismatch between training data and deployment behavior.

Identify $S_t$ = {a,b} (the two highest probabilities in $p_t$).

Renormalize $p_t$ on $S_t$: \bar $p_{S_t}$ = [0.6/(0.6+0.3), 0.3/(0.6+0.3)] = [0.667, 0.333].

Renormalize $q_t$ on the same set: \bar $q_{S_t}$ = [0.4/(0.4+0.35), 0.35/(0.4+0.35)] = [0.533, 0.467].

Compute $D_{KL}$(\bar $p_{S_t}$‖\bar $q_{S_t}$) = 0.667·log(0.667/0.533) + 0.333·log(0.333/0.467) ≈ 0.028.

Repeat for each prefix; the total loss is the sum of these per‑step KLs.

Restricting to the top‑k tokens preserves the dominant learning signal while cutting teacher queries from five to two per step.

How does OPD differ from standard offline distillation that uses a fixed teacher dataset?

Offline distillation trains on teacher outputs for a static corpus, so the student may see states the teacher never visits during deployment. OPD samples its own trajectories, ensuring the supervision is always aligned with the student’s current behavior.

Instead of copying the teacher’s absolute probabilities, we transfer the change the teacher underwent after reinforcement learning – the direction in policy space that encodes the learned reward.

Why not simply distill the teacher’s full distribution instead of its shift?

Distilling the full distribution forces the student to reproduce the teacher’s absolute preferences, including any residual biases. The shift isolates only the reward‑driven component, allowing the student to inherit the improvement while retaining flexibility to adjust its baseline behavior.

The Direct-OPD Objective

Direct-OPD turns the teacher’s RL shift into a dense reward while anchoring the student with a KL penalty.

Scaling RL with verifiable rewards (RLVR) to large models is prohibitively expensive, so we need a way to inherit the teacher’s policy improvement without re‑running RL on the target.

Direct‑OPD trains a larger student to reproduce the exact shift that RL induced on a small teacher, treating that shift as a dense per‑token reward while keeping the student tethered to its own initialization through a KL penalty.

Top‑k set $S_t$ = {a, b} because they have the two highest probabilities.

Renormalized probabilities \bar $p_t$(a)=0.4/(0.4+0.25)=0.615, \bar $p_t$(b)=0.385.

Assume teacher‑shift rewards $r_t$(a)=+0.6, $r_t$(b)=‑0.2.

Weighted coefficients $t$(a)=\bar $p_t$(a)·$r_t$(a)=0.369, t(b)=\bar $p_t$(b)·$r_t$(b)=‑0.077.

The gradient contribution for this step is t(a)∇_$\theta$ log $\pi$_$\theta$(a) + t(b)∇_$\theta$ log $\pi$_$\theta$(b).

Renormalizing on the top‑k set discards low‑probability tokens, so the gradient only pushes on actions the student actually considers, dramatically reducing variance.

Because both teacher and reference factorize autoregressively, the sequence‑level shift splits into a sum of per‑token rewards, enabling a token‑wise surrogate.

To keep the signal focused, we restrict each prefix to its top‑k student actions and renormalize the probabilities.

Using the dense per‑token reward $r_t$(v) as a local advantage, we first write the naïve Monte‑Carlo gradient and then Rao‑Blackwellize it.

Because \bar $p_t$(v) depends on $\theta$, differentiating it would inject unwanted Jacobian terms; we therefore detach it before forming the final surrogate.

The dense reward’s magnitude is fixed by the teacher’s training, while $\alpha$ is a free scalar; a single $\alpha$ cannot be tuned a‑priori for every teacher–student pair, so we adapt $\alpha$ during training.

How does Direct‑OPD differ from standard on‑policy distillation?

Standard on‑policy distillation copies the teacher’s action distribution directly, assuming the teacher is already optimal. Direct‑OPD instead extracts the *shift* induced by RL on a small teacher, treats that shift as a dense per‑token reward, and regularizes the student toward its own initialization, thereby transferring the *improvement* rather than the raw policy.

Empirical Results

Direct‑OPD transfers policy shifts and outperforms direct RL across model families.

Reinforcement learning with verifiable rewards (RLVR) is costly at scale, so Direct‑OPD transfers the RL‑induced policy shift from a small model to a larger one, avoiding full‑scale RL on the target.

Direct‑OPD lifts the accuracy of students that already surpass their post‑RL teachers, and the effect holds for two distinct teacher pairs and three student families.

Figure 2 shows consistent gains for R1‑Distill‑7B, Qwen3‑4B and Qwen3‑1.7B when the shift from R1‑Distill‑1.5B → JustRL‑1.5B or Nemotron‑1.5B → QuestA‑Nemotron‑1.5B is transferred.

**Figure 2.** Direct-OPD transfers RL-induced policy shifts across teacher pairs and student families. Left: R1-Distill-1.5B $arrow$ JustRL-1.5B transfer into R1-Distill-7B, Qwen3-1.7B, and Qwen3-4B, evaluated on AIME 2024 and AIME 2025. Right: Nemotron-1.5B $arrow$ QuestA-Nemotron-1.5B transfer into R1-Distill-7B and Qwen3-1.7B on AIME 2024. The two teacher pairs come from different training data and pipelines, showing that Direct-OPD is not specific to a single post-RL teacher.

Running RL on a small model and then applying Direct‑OPD yields higher accuracy and substantially lower wall‑clock time than training the large target directly with RL.

Figure 3 plots accuracy versus elapsed time; every transfer point (T600–T1500) lies above the direct‑RL curve at comparable time.

**Figure 3. Running RL on a small model and transferring its policy shift with Direct-OPD beats running RL directly on the large target under a shorter wall-clock training path.** We compare two routes to improving R1-Distill-7B: direct RL on R1-Distill-7B, versus a weak-to-strong route that runs RL on the smaller R1-Distill-1.5B and transfers the resulting policy shift into R1-Distill-7B with Direct-OPD. $TN$ denotes the transfer that uses the R1-Distill-1.5B RL checkpoint at step $N$ as the post-RL teacher $\pi_T$, with the base model as $\pi_{ref}$. **Left:** AIME 2025 accuracy against total wall-clock time; the wiggly curve is direct R1-Distill-7B RL, and each $TN$ point sums the elapsed time for small-model RL and the short Direct-OPD transfer. Later transfers (T600–T1500) sit above the direct-RL curve at comparable elapsed time. **Middle:** Direct-OPD transfer trajectories into R1-Distill-7B from the five small-teacher checkpoints; the early T300 carries a weaker shift than T900–T1500. **Right:** the same recipe with Qwen3 nonthinking models—transferring a Qwen3-1.7B RL shift into Qwen3-4B reaches the 0.635 accuracy of direct Qwen3-4B RL (dashed) on AIME 2025.

Sequentially applying two independently learned policy shifts compounds their benefits, raising AIME 2024/2025 scores by up to 15.5 points.

Figure 4 reports the stepwise gains: +10.0 after the first stage (JustRL) and an additional +5.5 after the QuestA stage.

**Figure 4.** Sequential policy-shift transfer into Qwen3-1.7B on AIME 2024 and AIME 2025. Left: trajectories after aligning the second stage to global steps 300–600. Right: endpoint scores on AIME 2024/2025. The first stage uses the R1-Distill-1.5B → JustRL-1.5B signal; the second stage continues from that checkpoint with the Nemotron-1.5B → QuestA-Nemotron-1.5B signal.

Mechanism Analysis

We dissect why Direct‑OPD succeeds by ablating overlap, horizon, and KL control.

We now ask whether Direct‑OPD truly needs the student to imitate the teacher’s high‑probability tokens, whether training on short rollouts propagates to longer generations, and how the KL coefficient shapes the dense teacher‑shift reward.

Direct‑OPD improves performance even when student‑teacher top‑k overlap stays low.

Figure 5 shows pattern‑aligned transfers reaching ~0.70 overlap while cross‑pattern transfers remain around ~0.64, yet both gain similar validation scores. +0.07 overlap advantage for pattern‑aligned transfer

A 2 k‑token training horizon yields higher long‑rollout alignment than a 6 k horizon.

Figure 8’s cumulative gap $G_T$ for the 2 k model stays above the 6 k model across 16 k positions, while Figure 7 shows the 2 k setting achieving the most stable validation accuracy.

Adaptive KL keeps the dense teacher‑reference reward near zero, avoiding the over‑optimistic spikes of fixed KL.

Figure 9 shows adaptive KL’s reward curve collapsing to zero after an initial correction, whereas the best fixed KL (pair‑dependent) yields a larger positive reward but no validation gain.

**Figure 5.** Teacher–student top-$k$ overlap during Direct-OPD training. Left: R1-Distill-1.5B $arrow$ JustRL-1.5B teacher pair. Right: Nemotron-1.5B $arrow$ QuestA-Nemotron-1.5B teacher pair. Solid curves measure overlap with the post-RL teacher, and dashed curves measure overlap with the teacher reference. The pattern-aligned R1-Distill transfer enters a higher-overlap regime, while the cross-pattern transfers remain lower and do not become imitation of the post-RL teacher.

**Figure 6.** Entropy diagnostics for R1-Distill-1.5B $arrow$ JustRL-1.5B policy-shift transfer. **Top row:** transfer into Qwen3-1.7B. **Bottom row:** transfer into R1-Distill-7B. Each row shows student entropy, post-RL teacher entropy, teacher-reference entropy, and teacher entropy minus reference entropy. Actor entropy does not collapse, while the teacher/reference entropy gap narrows over training.

**Figure 7** Response-length sweep for R1-Distill-1.5B → JustRL-1.5B transfer with fixed KL coefficient 1. We report the average of AIME 2024 and AIME 2025 validation accuracy (ave@32) during training for Qwen3-1.7B and R1-Distill-7B students. The 2k setting gives stable validation behavior across the two students, while shorter or longer rollouts do not consistently improve the validation curves.

**Figure 8.** Short-horizon Direct-OPD training changes behavior beyond the supervised prefix. On a fixed set of 64 long rollouts we track the cumulative gap $G_T = \sum_{t \le T} g_t$, where $g_t$ weights, over the actor's top-16 tokens at position $t$, the log-probability the post-RL teacher (JustRL) assigns minus that of the reference (R1-Distill-1.5B); higher $G_T$ means the actor's likely tokens look more like the post-RL teacher, while lower means the actor remains closer to the reference. **Left:** the 64 per-rollout trajectories for the untrained Qwen3-1.7B actor all drift negative—its long rollouts are reference-like, and this is not driven by a single outlier. **Middle:** the mean $G_T$ for the base actor and for actors trained with 2k/4k/6k response length (40 steps, fixed KL=1); every trained actor sits above the base across the full ~16k positions, and the 2k actor shifts well past its 2k training horizon (dashed line). **Right:** AIME 2024/2025 (ave@32) at the same 40-step checkpoint—the 2k setting validates best.

**Figure 9.** The best KL coefficient is pair-dependent, and adaptive KL pulls the dense token reward toward a balanced regime. 2k-response runs across fixed KL coefficients, with adaptive KL as the black curve. **Top row:** AIME 2024/2025 validation accuracy (ave@32); **bottom row:** dense token reward. The best fixed coefficient differs across teacher-student pairs, and a larger reward does not imply better validation. Adaptive KL instead pulls the dense token reward toward zero after an initial correction, keeping the student from simply maximizing the dense teacher/reference reward.

**Figure 10.** Entropy diagnostics for QuestA-Nemotron into Qwen3-1.7B. The panels show student entropy, post-RL teacher entropy, teacher-reference entropy, and teacher entropy minus reference entropy. Together with Figure 6, this shows that the non-collapse pattern is not specific to the JustRL teacher pair.

**Figure 11** QuestA transfer curves on AIME 2025 for the cross-pattern transfer setting. The main text reports the corresponding AIME 2024 curves.

Related Work and Conclusion

Survey of prior distillation approaches and a concise wrap‑up of Direct‑OPD.

Reinforcement learning with verifiable rewards (RLVR) now yields strong reasoning models, and knowledge distillation is the standard way to propagate those gains to other models. On‑policy distillation (OPD), where the student learns from its own sampled states under teacher supervision, has become the dominant variant for reasoning systems and is the focus of many recent works. Analyses attribute OPD’s success to teacher–student top‑k overlap on student‑visited states and to its interpretation as dense KL‑constrained RL.

Weak‑to‑strong generalization asks whether a weak supervisor can elicit capabilities from a stronger model, extending a long line of learning from imperfect supervision such as semi‑supervised learning and noisy‑label learning. Recent work applies this idea to reasoning and alignment, typically supervising the student with the weak model’s labels, which caps performance near the supervisor. Direct‑OPD instead transfers the reward implied by the difference between a small RL teacher and its pre‑RL reference, allowing students that already surpass the teacher to improve further.

Direct‑OPD leverages the policy‑as‑reward identity: under KL‑regularized RL the log‑ratio of policy to reference recovers the reward up to a constant, a principle used by Direct Preference Optimization to fit policies from preferences. We invert this relationship, reading a dense implicit reward from a post‑RL checkpoint and its reference, which links to dense and process rewards for reasoning and avoids over‑optimizing an explicit reward model. The signal is a behavioral log‑ratio evaluated on the student’s own on‑policy states, not a weight delta.

Direct‑OPD demonstrates that the policy shift learned by a small RL teacher serves as a useful dense reward for weak‑to‑strong generalization, improving stronger students across teacher pairs and model families while using far less compute than step‑matched direct RL. The approach reframes RL outcomes as reusable improvement signals rather than final policies to imitate. Limitations include the conditional nature of the signal—if the teacher/reference improvement is irrelevant on student‑visited states the method fails—and the need to tune response length and KL strength per teacher–student pair.

Experimental Details

Appendix details prompts, evaluation protocol, and training hyperparameters for Direct‑OPD.

We train Direct‑OPD using the math subset of Skywork‑OR1‑RL‑Data and a DAPO‑style prompt that asks the model to solve a problem step‑by‑step.

Direct‑OPD prompt template used for training rollouts and evaluation.

``` Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem. {Question} Remember to put your answer on its own line after "Answer:". ```

Evaluation follows the protocol in Table 2: 32 samples per problem, temperature 0.7, top‑p 0.95, and a maximum generation length of 31 744 tokens.

**Table 2.** Evaluation protocol for AIME validation.

**Table 3.** Training hyperparameters for Direct-OPD unless otherwise specified.

Sections B and C contain additional entropy diagnostics and extra results; their figures and tables have been omitted from this appendix.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers