Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang
Waypoint-OPD improves policy transfer by gating endpoint RL shifts based on temporal consistency between intermediate teacher checkpoints.
How can we improve the transfer of reinforcement learning (RL) policy shifts from teacher to student models by using intermediate "waypoint" checkpoints to gate the distillation signal?
Standard distillation methods transfer the total policy shift between a teacher's initial and final reinforcement learning (RL) states. This endpoint signal is often noisy because it cannot distinguish between consistent improvements and early-to-late reversals in the teacher's reasoning preferences. Waypoint-OPD introduces an intermediate "Waypoint" checkpoint to measure directional agreement between early and late teacher shifts. It uses this agreement to gate the endpoint signal, suppressing updates where the teacher's preferences were inconsistent. This approach consistently outperforms endpoint-only transfer across math and code reasoning tasks, achieving the highest domain-average scores for three different student model scales.
Paper Primer
The method hinges on temporal consistency: it treats the teacher's RL trajectory as a sequence of two shifts—Initial-to-Waypoint and Waypoint-to-Final—and compares their alignment at student-visited prefixes. By calculating the cosine similarity between these two shift vectors, the system generates a gate that scales the final transfer signal, effectively filtering out inconsistent or contradictory policy changes.
Waypoint-OPD consistently improves domain-average performance over endpoint-only transfer (Direct-OPD).
Evaluations across six settings (Qwen3-8B, 14B, and 30B-A3B students on math and code benchmarks) show consistent gains. Average improvements range from 1.19 to 2.66 percentage points across all tested student scales and domains.
Temporal gating is the primary driver of performance gains.
Ablation studies on Qwen3-8B math show that removing the gate or shuffling gate assignments across prefixes significantly reduces reasoning performance. The method achieves these gains with only one additional frozen-teacher scoring pass and no verifier calls during transfer.
Why is the endpoint-only approach (Direct-OPD) insufficient for transferring RL improvements?
The endpoint signal captures only the net change between the start and end of training. It fails to reveal if the teacher's policy evolved consistently or if early and late shifts partially canceled each other out, leading to potentially misleading supervision for the student.
Does this method require training additional teacher models or verifiers?
No. It uses three frozen checkpoints (Initial, Waypoint, Final) from a single teacher RL run. The transfer process requires only one additional scoring pass compared to standard endpoint distillation, with no new teacher training or verifier queries.
Motivation and Contribution
We expose why endpoint‑only distillation signals are noisy and propose Waypoint‑OPD to fix it.
Reinforcement‑learning fine‑tuning improves language‑model instruction following, but the process is costly. Reusing a teacher’s RL improvements via on‑policy distillation offers a way to share that effort, yet the supervision remains tied to the teacher’s absolute policy.
The endpoint policy‑shift signal (teacher’s log‑probability change from before to after RL) can be misleading because the same net shift may arise from early and late changes that either reinforce or cancel each other.
We hypothesize that checking whether the early‑to‑Waypoint and Waypoint‑to‑Final shifts agree in direction offers evidence for how much weight to assign the endpoint signal at each student‑generated prefix.
Waypoint‑OPD keeps three teacher checkpoints (Initial, Waypoint, Final) and compares the teacher’s relative preferences over the student’s top‑k token candidates; if the early‑to‑Waypoint and Waypoint‑to‑Final changes agree, the endpoint shift is kept, otherwise it is suppressed.
The core limitation of endpoint‑only distillation signals is their inability to detect contradictory intermediate policy shifts, which can misguide student training.
Problem Setup and Notation
Defines OPD, Direct‑OPD, and Final‑Teacher Imitation baselines and sets notation.
We consider a prompt $x\sim D$ and a response $y=(y_1,\dots,y_T)$. The student policy $\pi_\theta$ generates on‑policy trajectories $y\sim\pi_\theta(\cdot\mid x)$, while a fixed student initialization $\pi_S$ serves as a reference during transfer.
OPD aligns the student’s next‑token distribution with the teacher’s distribution at every prefix the student actually visits, using the final teacher as the target.
Direct‑OPD treats the teacher’s RL‑induced log‑probability shift between the initial and final checkpoints as an immediate token‑level reward for the student.
Final‑Teacher Imitation trains the student to directly mimic the final teacher’s policy across all tokens, ignoring any intermediate policy changes.
**Table.** Teacher reference: Qwen3-4B (Initial: 37.64, Waypoint: 52.22, Final: 55.37)
These definitions establish the baseline landscape: OPD provides dense alignment, Direct‑OPD uses raw teacher shifts as rewards, and Final‑Teacher Imitation copies the final policy outright.
The Waypoint-OPD Mechanism
Method details the Waypoint‑OPD mechanism and its agreement gate for consistent distillation.
Endpoint signals in on‑policy distillation are noisy because they depend on a single final teacher snapshot. This instability harms student updates, especially when the student visits prefixes that the teacher never saw. Waypoint‑OPD injects an intermediate teacher checkpoint to provide a consistency check before the final signal is applied.
The mechanism adds a mid‑trajectory teacher checkpoint (the “waypoint”) and uses the shift between the initial, waypoint, and final teacher policies to gate the distillation signal, so only consistent policy changes are transferred to the student.
How does Waypoint‑OPD differ from standard OPD that only uses the final teacher?
Standard OPD applies the final teacher’s log‑probability shift directly to the student, which can be noisy if the teacher’s policy changed dramatically. Waypoint‑OPD inserts an intermediate checkpoint and measures the consistency between the two shifts; only when they agree (high cosine similarity) does the endpoint signal get passed on, reducing variance.
Teacher shift $\delta^{(1)}_t$ values: $[0.1, -0.05, 0.0]$; $\delta^{(2)}_t$ values: $[0.08, -0.04, 0.01]$.
Center $\delta^{(1)}_t$: subtract its $q_t$‑weighted mean $0.1\cdot0.5 + (-0.05)\cdot0.3 + 0\cdot0.2 = 0.035$, yielding $u_t=[0.065, -0.085, -0.035]$.
Center $\delta^{(2)}_t$ similarly to get $v_t=[0.058, -0.074, 0.016]$.
Compute cosine agreement $c_t = \frac{\langle u_t,v_t\rangle_{q_t}}{\|u_t\|_{q_t}\|v_t\|_{q_t}} \approx 0.97$, indicating strong agreement.
Gate $g_t = \max(0,c_t) = 0.97$ (norms exceed $\varepsilon$), so the final coefficient is scaled by $0.97$.
Centering removes the uniform component of the shifts, so the gate reacts only to relative changes among the top‑$k$ actions, which is what drives consistent learning.
The agreement gate $g_t$ evaluates the cosine similarity $c_t$ between the two centered teacher shifts; if the shifts align (positive cosine), the gate passes a scaled version of the endpoint signal, otherwise it suppresses it.
Why does the gate use the cosine of centered shifts instead of the raw shift values?
Raw shifts contain a uniform component that affects all candidates equally and does not convey any preference information. Centering removes that component, and the cosine measures the directional agreement between the two shifts, which is the signal that indicates consistent policy evolution.
Student model generates a prefix $s_t$ and selects top‑$k$ tokens $K_t$ based on its policy $\pi_\theta$.
Three frozen teacher policies ($\pi_0$, $\pi_m$, $\pi_1$) compute log‑probability shifts $\delta^{(1)}_t$ and $\delta^{(2)}_t$ for each token in $K_t$.
Center each shift with $C_{q_t}$ to obtain $u_t$, $v_t$, and combine them into $d_t$.
Compute cosine agreement $c_t$ and derive the gate $g_t$ (with fallback).
Scale gated rewards by the batch factor $\kappa_B$ and form the dense coefficient $A_t(a)$.
Perform a dual‑clipped PPO update on the student using the surrogate objective $J_{\text{WP}}(\theta)$.
**Figure 2.** Training pipeline of Waypoint-OPD. Initial, Waypoint, and Final score candidates at student-generated prefixes. Their centered segment shifts define an agreement gate on the endpoint shift. Batch scaling and detached candidate advantages drive the student update Waypoint measures consistency only.
Experimental Setup and Performance
Waypoint-OPD consistently beats Direct‑OPD on math and code benchmarks.
Waypoint‑OPD improves mean@32 accuracy over Direct‑OPD by 2.18 percentage points on the Qwen3‑8B student model.
Figure 1 shows Waypoint‑OPD at 57.18 % versus Direct‑OPD at 55.00 % mean@32 for Qwen3‑8B.
We ask three questions: (1) whether Waypoint‑OPD scales across math, code, and model sizes; (2) whether temporal consistency boosts endpoint transfer and final‑teacher imitation; and (3) how gate assignment, mean‑normalization, and Waypoint position influence performance.
**Figure 1.** Mathematical reasoning across three student models. mean@32 accuracy (%), averaged over seven benchmarks. Waypoint-OPD improves over Direct-OPD by 2.18, 1.59, and 1.89 percentage points, respectively. Teacher scores are from the Final Qwen3-4B checkpoint.
**Figure 3.** Evaluation accuracy during training. Qwen3-8B results on AIME 2025, AIME 2026, and HMMT November 2025 at steps 0, 20, 40, 60, 80, and 100. Scores are mean@32 percentages. Solid red curves denote Waypoint-OPD and dashed blue curves denote Direct-OPD. Gray dashed lines show the Final Qwen3-4B teacher scores.
**Table 7.** Full code reasoning results across three student sizes. Scores are mean@8 percentages. Avg. is the unweighted mean across the five evaluations. Bold marks the best score within each student block. Teacher checkpoints are reference rows.
**Table 2.** Average code reasoning performance. Average mean@8 scores (%) over five evaluations. Qwen3-4B teacher checkpoints are shared references. Bold marks the best score per student. Full results: Appendix D.
Waypoint‑OPD consistently outperforms Direct‑OPD across math and code tasks.
Analysis of Gating and Dynamics
We quantify how each component of Waypoint‑OPD contributes to final performance.
Recall that Waypoint‑OPD inserts intermediate teacher checkpoints and an Agreement Gate to filter noisy endpoint signals, aiming for more reliable student distillation.
Removing the Agreement Gate drops the average math score from 56.14 to 53.50.
Table 3 shows the “No gate” variant achieving 53.50 Avg. versus 56.14 Avg. for the full Waypoint‑OPD.
**Figure 4.** Gate diagnostics on Qwen3-8B mathematical reasoning. Statistics for Waypoint-OPD over 100 training steps. From left to right: mean gate $g_t$, zero-gate fraction, batch-scale normalization multiplier $\kappa_B$, and mean per-prefix reward-vector L2 norm. Raw uses the gated, student-weighted endpoint signal $q_t(a)g_t d_t(a)$; Normalized uses the final candidate coefficients $A_t(a)$ in Eq. 12 from the same run. All statistics aggregate valid response prefixes.
**Figure 5.** Response length and response truncation rate. The two panels compare Qwen3-8B trained with Waypoint-OPD (solid red) and Direct-OPD (dashed blue) over 100 training steps. (a) Mean number of tokens per rollout response at each training step. (b) Fraction of rollout responses reaching the maximum length of 16,384 tokens; 0.2 corresponds to 20%.
The table displays performance metrics across different "Waypoint" steps (500, 600, 700) for various benchmarks: AIME24, AIME25, AIME26, HMMT25 Feb, HMMT25 Nov, HMMT26 Feb, AMC23, and an average (Avg.).
**Table 9.** **Training-time comparison on Qwen3-8B math.** Total training time over 100 training steps on eight H20 GPUs, excluding evaluation.
**Table 3.** Removing the gate. On Qwen3-8B math, removing the gate lowers Avg. from 56.14 to 53.50, a decrease of 2.64 points, with lower scores on all seven benchmarks (Table 3). Retaining centering and normalization without consistency-dependent weighting therefore does not recover the full method's gain. This comparison does not isolate the effect of centering.
Related Work
We position Waypoint‑OPD within prior on‑policy distillation and checkpoint selection literature.
Knowledge distillation transfers teacher output distributions to a student, while on‑policy variants supervise student‑generated trajectories. Recent work refines divergence objectives and leverages both teacher‑ and student‑generated data (Wen 2023; Gu 2024; Ko 2024; 2025). Methods such as Direct‑OPD, W2S‑OPD, OPRD, and Waypoint‑OPD differ in how they construct and weight the policy‑shift signal transferred to the student.
Selective supervision studies how teacher–student compatibility influences on‑policy transfer, with approaches like Prune‑OPD, EOPD, PW‑OPSD, OPD2, and RG‑OPD introducing various gating criteria. Waypoint‑OPD uniquely uses agreement between consecutive teacher shifts as a prefix‑level gate, without requiring verifier feedback. Checkpoint selection research shows that early stopping and intermediate teacher checkpoints can improve distillation, as explored in Pro‑KD, progressive distillation, and Direct‑OPD checkpoint studies.
Mathematical Derivations
Derives gate properties and details the agreement‑gated OPD algorithm.
Agreement‑gated on‑policy distillation (Algorithm 1)
**Table 4.** GRPO training configuration shared by the math and code teachers.
This table lists the configuration details for teacher and student models, as well as the benchmarks used for math and code evaluation.
**Table 6.** Model and evaluation configuration.
Experimental Protocol and Diagnostics
Detailed training setups, ablations, and performance diagnostics for Waypoint‑OPD.
We train with vLLM (Sheng et al., 2025) and FSDP (Zhao et al., 2023) on a single node equipped with eight NVIDIA H100 GPUs. All rollouts also use vLLM (Kwon et al., 2023). Tables 4–6 list the hyperparameters, model sizes, and benchmarks used throughout the appendix.
Teacher training draws ~57 K math examples (difficulty ≥ 6) from DeepMath‑103K and ~25 K code examples from Eurus‑RL‑Code. Both Qwen3‑4B teachers are optimized with GRPO for 1,200 steps using the settings in Table 4.
Evaluation uses temperature 0.7, top‑p 0.95, and a maximum generation length of 16,384 tokens. Math results are reported as mean@32; code results as mean@8, averaging over the respective number of responses per problem.
Table 4 shows the shared GRPO configuration: batch size 128, rollout count 8, learning rate $10^{-6}$, and zero KL coefficient. These settings are identical for the math and code teachers.
Table 5 details the student transfer setup: batch size 256, mini‑batch 64, rollout count 1, learning rate $10^{-6}$, KL coefficient $β=2.5$, and a target gate mean $τ=0.8$ with scale clipping in $[0.75, 1.5]$.
Table 6 enumerates the model variants (Qwen3‑4B, Qwen3‑8B, Qwen3‑14B, Qwen3‑30B‑A3B) and the benchmarks (AIME 2024‑2026, HMMT, AMC, MBPP, LiveCodeBench v4‑v6) on which they are evaluated.
Table 7 provides the full code‑evaluation numbers that underlie the summary in the main text’s Table 2, covering all three teacher checkpoints and every student method.
E COMPONENT ABLATION DETAILS investigates three controls on Qwen3‑8B math: “No gate”, “Shuffled gate”, and “No normalization”. All controls keep qt‑centering and the student KL penalty, using the same training and evaluation settings as Appendix C.
“No gate” forces $gt=1$ for every valid response prefix while preserving the centered endpoint signal and the gate‑mean normalization rule (Eq. 11), which reduces to the constant multiplier $κ_B=0.8$ under default settings.
“Shuffled gate” computes gates normally (including the low‑norm fallback) but randomly permutes them across prefixes in a rollout batch, breaking the association between each prefix and its own temporal‑agreement weight while leaving the overall gate distribution unchanged.
“No normalization” fixes $κ_B=1$, retaining centering, the positive‑cosine gate, and the low‑norm fallback; the distillation signal therefore lacks the batch‑dependent rescaling present in the default rule.
F SENSITIVITY TO WAYPOINT CHECKPOINT SELECTION tests Waypoint checkpoints at RL steps 500, 600, and 700 on Qwen3‑8B math, keeping the initial and final teachers fixed. The average scores are 56.18, 56.14, and 56.13 respectively, a total range of only 0.05 percentage points.
Table 8 reports the seven benchmark scores for each Waypoint choice and their unweighted average, confirming that all three checkpoints surpass Direct‑OPD (average 53.96) by roughly 2.2 points.
H TRAINING TIME AND RESPONSE LENGTH compares the total training time for 100 steps of Qwen3‑8B math on eight H100 GPUs. Waypoint‑OPD finishes in 8 h 2 min 21 s versus 8 h 21 min 3 s for Direct‑OPD, a reduction of 18 min 42 s (3.73 %).
Table 9 details the per‑step seconds (289.41 s vs 300.63 s) and GPU‑hour totals (64.31 h vs 66.81 h), showing that the additional scoring pass in Waypoint‑OPD does not increase overall training cost.