On-Policy Delta Distillation

Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han

On-Policy Delta Distillation (OPD2) uses the teacher's reasoning-tuning "delta" as a distillation signal to improve student reasoning.

Standard on-policy distillation trains students to mimic a teacher's output, but this often conflates the teacher's hard-won reasoning capabilities with its pre-existing stylistic biases. The authors propose On-Policy Delta Distillation (OPD2), which replaces the standard distillation reward with a "delta signal"—the difference between the reasoning-tuned teacher and its own base model. This isolates the specific knowledge acquired during reasoning tuning. Across math, science, and code benchmarks, OPD2 consistently outperforms conventional distillation, enabling smaller models to achieve reasoning performance that rivals or exceeds larger models.

Paper Primer

Conventional distillation treats the teacher's entire output distribution as the target, which forces the student to learn both the teacher's reasoning logic and its incidental conversational habits. OPD2 isolates the reasoning signal by calculating the log-probability difference between the teacher's final reasoning-tuned state and its original base state.

OPD2 is a reward-design framework: it centers the delta signal to handle bias and applies a joint condition that reverts to standard distillation if the delta signal conflicts with the teacher's output, ensuring the student remains grounded in the teacher's actual performance.

OPD2 significantly improves reasoning performance across diverse domains and model architectures.

In non-thinking mode, Qwen3-4B trained with OPD2 outperforms the larger Qwen3-8B trained with standard distillation on math and science benchmarks. OPD2 increased the average math score for Qwen3-1.7B from 34.8 to 54.6, a 57% relative improvement over the baseline.

OPD2 effectively improves models that already possess strong reasoning capabilities.

While standard distillation often degrades performance in "thinking" modes, OPD2 consistently yields gains, such as improving the HMMT25 math score of Qwen3-8B from 44.3 to 52.3. OPD2 achieved the best average performance across all model sizes and domains in thinking mode, where competing methods often reduced baseline performance.

Why use the "delta" between the teacher and its base model instead of just the teacher's output?

The delta signal captures the specific learning trajectory of reasoning tuning, effectively filtering out the teacher's pre-existing stylistic tendencies and "natural" preferences that are irrelevant to reasoning tasks.

Does this method require significantly more compute than standard distillation?

It introduces a modest overhead of approximately 8–28% in wall-clock training time due to the additional forward pass of the teacher-base model, but this remains comparable to other advanced distillation methods like ExOPD.

The delta signal is the primary driver of performance; ablation studies show that replacing it with standard distillation signals causes the most significant performance drop, while centering and joint conditioning provide secondary stability benefits.

For researchers performing post-training, OPD2 provides a more precise signal for transferring reasoning capabilities, making it a robust choice for both non-thinking and thinking-mode models.

Abstract and Introduction

We expose the limited reward design of on‑policy distillation and propose a delta‑signal based improvement.

On‑Policy Distillation (OPD) is a post‑training technique that replaces handcrafted reward models with dense token‑level supervision from a high‑performing teacher. By sampling student outputs and scoring them with the teacher, OPD preserves the student’s existing knowledge while nudging it toward the teacher’s behavior. However, the reward formulation—simply the log‑probability gap between teacher and student—has received little design scrutiny.

We introduce the *delta signal*, defined as the difference between the reasoning‑tuned teacher and its base (pre‑tuning) model, and build OPD² around it. The delta captures the knowledge acquired during reasoning tuning, offering a more direct supervision signal than the raw teacher distribution. OPD² therefore replaces the original log‑probability reward with the teacher‑base difference.

**Figure 1.** **Comparison of On-Policy Distillation (OPD) and our OPD$^2$.** While conventional on-policy distillation trains students to follow the teacher’s outputs, our OPD$^2$ utilizes the *delta signal*, the difference between the teacher and its base model, to focus on the knowledge of the learning trajectory required for reasoning capability.

To understand $\Delta$’s behavior we perform three analyses: (1) word‑cloud visualizations of the signal distribution, (2) token‑level heatmaps comparing $\Delta$ to raw teacher probabilities, and (3) statistical summaries of word‑level changes. These reveal that $\Delta$ emphasizes reasoning‑specific tokens while attenuating stylistic preferences retained from pre‑tuning. Building on this insight, we design two reward refinements: centering (subtracting the expected reward to obtain zero‑mean signals) and joint conditioning (combining the centered $\Delta$ with the original OPD reward to stabilize convergence).

We evaluate OPD² on a mixed‑domain training set (equal numbers of Math, Science, and Code questions) and test on 7 Math, 3 Science, and 4 Code benchmarks across Qwen‑3 (1.7B–8B) and Gemma‑4 models, both in think and no‑think modes. Across all settings OPD² yields consistent gains over standard OPD, demonstrating that the delta‑signal reward more effectively transfers reasoning capability during the short post‑training phase.

Delta Signal Definition

We define a delta distillation signal that isolates the teacher’s reasoning improvements over its base model.

On‑policy distillation (OPD) treats the teacher‑student log‑probability gap as a reward, but this mixes basic next‑token prediction with reasoning ability. We therefore introduce a delta signal that compares the fine‑tuned teacher to its own pre‑fine‑tuned base, isolating the reasoning improvements.

The delta signal measures how the teacher’s reasoning‑focused fine‑tuning changes its token predictions relative to the same model before that fine‑tuning.

Word‑cloud visualizations (Figure 2) show that $\Delta$ strongly promotes logical‑connective words like “hence”, “however”, and “instead”, while it suppresses exploratory verbs such as “see”, “try”, and “verify”. This shift reflects a focus on reasoning rather than surface‑level verification.

**Figure 2.** Word clouds for OPD, Base and $\Delta$. The figure illustrates distillation signal strengths in the form of word clouds. Qwen3-1.7B and Qwen3-4B are used for student and teacher, respectively, on 10k math questions. Compared with OPD, $\Delta$ emphasizes reasoning-connective words such as *hence*, *however*, and *instead*, while suppressing exploratory and verification-related words such as *see*, *try*, and *verify*.

Token‑level heatmaps (Figure 3) reveal that $\Delta$ assigns more negative rewards to generic reasoning tokens (“There”, “Since”) and to tokens that belong to incorrect reasoning paths, making the signal more aligned with correct logical structure.

**Figure 3.** Token-level distillation signals on simple reasoning examples. Blue and red indicate promoting and suppressing signals, respectively, with values clipped to $\pm 6$. Compared with OPD, $\Delta$ more consistently suppresses tokens associated with incorrect reasoning.

Statistical analysis (Table 1) quantifies these shifts: words such as “hence”, “however”, and “note” are enhanced in $\Delta$, while words like “see”, “try”, and “verify” are suppressed. The effect is consistent across Math, Code, and Science corpora.

**Table.** Words with the highest frequency of signal changes in the Math, Code, and Science domains, categorized by whether the signal is Enhanced or Suppressed.

On-Policy Delta Distillation

We expose and fix the convergence instability of the delta reward in on‑policy distillation.

The delta reward $R_{\Delta}$ ignores the student policy $\pi_{\theta}$ when computing the reward, so training driven solely by $R_{\Delta}$ drives the student toward a one‑hot token that maximizes the reward. Although such a token cannot be reached in realistic strong‑to‑weak distillation, the mismatch can cause unstable gradients during learning.

In practice the expectation in $A^{\text{OPD}}_t$ and $A^{\Delta}_t$ is approximated by averaging over the top‑$k$ tokens (with $k=1024$) of the current policy $\pi_{\theta}$ to keep GPU memory bounded.

The conditional advantage $\text{AD2}_t$ disables updates when $A^{\Delta}_t$ is non‑positive, which occurs when the student already matches the teacher ($\pi_{\theta} = \pi^{*}$) or when the delta signal would push the student away from the teacher’s distribution. This sign‑consistent gating prevents over‑training on the trace signal and keeps gradient magnitudes aligned with the original distillation direction.

Computational Costs

OPD2 adds modest training overhead despite the extra teacher‑base forward pass.

OPD2 introduces extra computation because it must run the teacher‑base model to obtain the delta signal. This adds a wall‑clock training overhead of roughly a quarter for the larger Qwen3 models and about eight percent for Gemma4‑E4B. Nevertheless, the cost remains comparable to the existing ExOPD baseline and is modest in the short‑training regime typical of on‑policy distillation.

**Figure 4.** Training curve for various domains. The figures show the reasoning performance changes during on-policy distillation training. We report the average performance on AIME24 and AIME25 for Math, CodeContests for Code, and GPQA for Science. All OPD variants exhibit similar training dynamics, while OPD$^2$ achieves the strongest performance. Note that, in all tables, we report performance at the final training step, not the peak performance observed during training.

Method Overview

On‑policy distillation trains a student to imitate a teacher token by token.

Standard reinforcement‑learning fine‑tuning requires a handcrafted reward signal, which is costly to design and evaluate. OPD sidesteps this by reusing the teacher’s own probability distribution as a dense, token‑level reward.

OPD lets the student copy the teacher’s answer one token at a time, using the teacher’s log‑probability as a per‑token advantage signal.

How does OPD differ from standard policy‑gradient RL that uses a handcrafted reward?

Standard RL supplies an external scalar reward after the whole episode, whereas OPD derives a dense, token‑wise signal directly from the teacher’s probability distribution. This eliminates the need to design a task‑specific reward and lets the gradient be computed at every timestep.

Compute the teacher’s log‑probability: $\log 0.8 \approx -0.223$.

Compute the student’s log‑probability: $\log 0.5 \approx -0.693$.

Reward $R_t = -0.223 - (-0.693) = 0.470$, a positive advantage for the student.

The score function $\nabla_{\theta}\log \pi_{\theta}(y_t)$ is multiplied by $0.470$, nudging the student’s parameters toward increasing the token’s probability.

The reward is positive because the teacher is more confident; OPD therefore pushes the student to raise its probability for that token.

Because OPD only touches sampled tokens, its computational cost matches that of ordinary on‑policy RL while delivering a teacher‑driven learning signal.

Training Dynamics

OPD² keeps improving while earlier methods stall early in training.

Early training of on‑policy distillation methods quickly lifts performance, but OPD and ExOPD soon hit a ceiling or even regress, leaving a gap that OPD² fills.

The delta signal injects a small corrective term that continuously nudges the student toward the teacher, preventing the early stagnation seen in earlier methods.

Compute raw delta = teacher − student = [0.5, 0.2].

Scale delta: 0.1 × [0.5, 0.2] = [0.05, 0.02].

Add to student logits: [1.5, 0.8] + [0.05, 0.02] = [1.55, 0.82].

Re‑compute softmax probabilities; the corrected student now assigns higher probability to the correct class.

The delta provides a gentle, direction‑aware push that improves the student without destabilizing training.

How does the delta signal differ from simply adding a constant bias to the student’s logits?

A constant bias is static and does not reflect the teacher‑student mismatch; the delta signal is dynamic, computed as the actual difference between teacher and student logits, so it only acts where the gap exists and fades as the student learns.

When we plot the training curves of OPD, ExOPD, and OPD² on the Qwen3‑4B non‑thinking setting, OPD² climbs higher early and stays above the others for the entire run.

**Table 8.** Ablation study for OPD$^2$. We conduct an ablation study for OPD$^2$. Replacing the delta signal shows the most significant impact. The formulas indicate how each variant modifies Eq. 8.

Experiment Overview

We outline the evaluation framework, models, data, and settings used to assess OPD2.

To validate OPD2 we construct a large‑scale on‑policy distillation pipeline that can be applied to any reasoning domain without hand‑crafted rewards.

Instead of pre‑training a student from scratch, OPD2 refines a small reasoning model by repeatedly feeding it the teacher’s answer to the same question and nudging the student toward that behavior.

We mix equal parts of challenging questions from Math, Science, and Code so the student sees a balanced set of reasoning tasks during distillation.

**Table 11.** Common training parameters. Training parameters shared across all models.

Non-Thinking Results

OPD2 delivers the highest non‑thinking averages across all model sizes and tasks.

OPD2 attains the highest overall average score (71.6) across all model sizes and domains in non‑thinking mode.

Table 3 shows Qwen3‑8B + OPD2 reaching an average of 71.6, outperforming all other variants.

**Table 3.** Qwen3 non-thinking results for Code and Science. The table shows non-thinking model results for Qwen3. The best performance is shown in bold, and the second best is underlined.

**Table 1.** Performance comparison of different models and methods across Code and Science benchmarks.

Thinking Results

OPD² delivers the strongest gains across all Qwen3 and Gemma4 benchmarks.

OPD² raises the average Code score of Qwen3‑1.7B by +11.1 points.

Table 5 shows the baseline average of 29.3 points increasing to 40.4 points with OPD².

**Table 5.** Qwen3 thinking results for Code and Science. The table shows thinking mode results for Qwen3. The best performance is shown in bold, and the second best is underlined.

**Table 1.** Performance comparison of Qwen3 models (1.7B, 4B, 8B) across Code and Science benchmarks using different training methods (OPD, ExOPD, OPD^2).

**Table.** Evaluation of methods on Gemma4-E4B-it.

**Table 7.** **Gemma4 for Code and Science.** The table shows on-policy distillation results for Gemma4-E4B-it. The best performance is shown in bold, and the second best is underlined.

Appendix

Provides the hardware, training, and evaluation settings needed to reproduce the experiments.

The paper’s reproducibility hinges on fully disclosed hardware and hyperparameter choices, yet many prior works omit these details, leaving readers unable to verify claims.

All models are trained on NVIDIA H100 GPUs, with small models using a single node and larger models distributed across four nodes to balance compute and memory.

Performance is measured with pass@1, i.e., the probability that a single generated answer matches the reference, averaged over multiple stochastic generations.

This table lists various hyperparameters and their corresponding values used for training or inference settings.

**Table 12.** Evaluation benchmark configs. For each benchmark, we report pass@1 (single-sample accuracy), averaged across the listed number of repetitions.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers