On-Policy Self-Adaptation

Yi Ding, Ruqi Zhang

On-Policy Self-Adaptation (OPSA) improves LLM reasoning by suppressing low-probability tokens using entropy-adaptive signals, requiring no teacher.

How can a model improve its reasoning performance using only its own generated trajectories, without needing an external teacher model or verifiable rewards?

On-Policy Distillation (OPD) attempts to improve language models by having a teacher score student-generated reasoning steps, but this supervision is often noisy and unreliable because the student's trajectories are off-policy for the teacher. The authors find that OPD's performance gains actually stem from suppressing low-probability tokens rather than imitating the teacher, and they replace the teacher entirely with an entropy-adaptive negative advantage signal. This method, On-Policy Self-Adaptation (OPSA), more than doubles pass rates on mathematical benchmarks compared to base models while requiring no external supervision, rewards, or hints.

Paper Primer

OPD relies on a teacher model to provide token-level advantages, but the authors demonstrate that this supervision is highly noisy—often assigning positive credit to incorrect answers—and that the student improves regardless of whether this noise is filtered out. By isolating the learning signal, they show that the model's gains are driven by negative advantages applied to low-probability tokens, effectively pruning poor reasoning branches.

OPSA is a supervision-free training algorithm: it identifies the 20% of tokens with the lowest log-probability and assigns them a negative advantage scaled by the token's entropy. This mechanism acts like a gardener pruning dead branches: it suppresses low-probability tail tokens while redistributing probability mass among high-entropy "fork" tokens to encourage diverse, reflective reasoning.

OPSA significantly outperforms standard OPD and base models on mathematical reasoning tasks without requiring any external teacher or ground-truth labels.

On the Qwen3-1.7B model, OPSA improves Avg@32 by 35.41 points on AIME24, a 263% relative gain, and more than doubles Pass@32 across all tested benchmarks.

Why does the teacher's supervision in OPD become noisier as the teacher model scales up?

The authors attribute this to the growing distributional mismatch between the student and teacher policies; as the teacher becomes more capable, the student-generated trajectories become increasingly "off-policy" from the teacher's perspective, leading to unreliable advantage assignments.

Does OPSA's focus on sharpening the probability distribution lead to a collapse in response diversity?

No. While OPSA sharpens the distribution at low-entropy positions, it redistributes probability mass among competing head tokens at high-entropy "fork" positions, which preserves the model's ability to explore alternative reasoning paths.

OPSA demonstrates that fine-grained policy improvement can be achieved through internal token-level uncertainty alone, suggesting that the "distillation" in OPD is largely a byproduct of self-correction rather than knowledge transfer.

Introduction and Motivation

We expose why on‑policy distillation’s gains arise from token suppression rather than teacher supervision.

On‑policy distillation (OPD) promises dense token‑level supervision, addressing the sparsity of reinforcement‑learning‑with‑verifiable‑rewards (RLVR). Yet the teacher evaluates student‑sampled trajectories that are inherently off‑policy, raising doubts about the reliability of its supervision.

OPD trains a student model by having a strong teacher assign token‑wise advantages to trajectories the student itself generates.

Our quantitative analysis uncovers substantial noise in teacher supervision, and this noise fraction grows with teacher scale. Surprisingly, the student’s final performance is largely unchanged whether we keep or discard these noisy signals.

Further investigation shows that OPD’s gains are driven mainly by suppressing low‑log‑probability tokens sampled by the student. Replacing the teacher’s nuanced advantages with a single fixed negative value reproduces most of the performance boost.

These observations motivate On‑Policy Self‑Adaptation (OPSA), a supervision‑free method that assigns entropy‑adaptive negative advantages: stronger signals at high‑entropy positions suppress tail tokens and redistribute probability mass to head tokens.

**Figure 1.** **Left:** Overview of different on-policy reinforcement learning algorithms. Unlike existing methods that derive advantages from external supervision, OPSA enables self-improvement with no external supervision by assigning entropy-adaptive negative advantages to low-probability tokens. This suppresses tail tokens and redistributes their mass among head tokens, sharpening low-entropy positions while preserving diversity at high-entropy forks. **Right:** Training dynamics and performance of OPSA. OPSA eliminates the teacher supervision used in OPD, and further outperforms OPD by 12% in Avg@4. On Qwen3-1.7B, OPSA improves Avg@32 by 263-307% across mathematical reasoning benchmarks and more than doubles Pass@32.

The key shift is moving from teacher‑dependent distillation to self‑adaptation that requires no external supervision.

Teacher Supervision and Noise

Analyzes why noisy teacher signals still yield student improvements.

OPD relies on token‑level advantage signals from a teacher, but those signals are often contradictory to the true answer quality, creating a noisy supervision problem that the paper must explain.

When a teacher’s advantage $A_i$ on a token has the opposite sign of the verifiable reward for that token, the supervision is misleading.

Empirically, all teacher sizes exhibit substantial noise: the 4B teacher yields 20.4 % negative advantages on correct answers and 40.8 % positive advantages on incorrect answers (overall 30.6 % noise). Larger teachers increase this effect, reaching 34.7 % and 50.6 % noise for 30B‑A3B and 235B‑A22B respectively, with the biggest model assigning negative advantages to 97.8 % of correct answer tokens and 96.6 % of incorrect ones.

Despite this high noise, students trained with OPD converge to similar performance regardless of whether the trajectories contain noisy signals or not, indicating an unexpected robustness to misleading teacher supervision.

Research Question: If OPD’s gains are not driven by accurate knowledge transfer, what underlying mechanism enables the student to improve?

Disentangling Improvement Sources

We isolate which tokens and signals drive OPD’s gains via targeted ablations.

We probe OPD by ablating token subsets and advantage signals to see what truly drives improvement. These controlled experiments reveal that low‑probability tokens and negative advantages are the primary sources.

Training only on high‑log‑p tokens yields no performance change.

Experiments restricting training to the student’s top‑log‑p tokens, using either the original OPD advantages or random values drawn from [‑1, 1], leave AIME24 scores unchanged.

Replacing OPD advantages with a fixed negative value does not degrade performance.

Using a constant $-0.5$ advantage on the 20 % of tokens with lowest student log‑p matches full‑token OPD performance, whereas a constant $+0.2$ advantage causes rapid policy collapse.

The OPSA Mechanism

We turn the negative‑signal insight into a self‑adaptation algorithm that learns without any teacher.

Section 3 showed that OPD’s gains survive the removal of the teacher and of positive advantages, leaving only a negative signal on low‑logp tokens as the driver of improvement. The next question is how much negative signal each such token should receive.

A negative advantage penalises a token that the model assigned a low probability, nudging the policy to reduce its likelihood.

Compute $H_{\min}=0.5$ and $H_{\max}=1.2$.

For token 2: $r_2 = 2\frac{1.2-0.5}{1.2-0.5} - 1 = 1$.

For token 4: $r_4 = 2\frac{1.0-0.5}{1.2-0.5} - 1 \approx 0.43$.

With $\delta=1$ and $A_i^{\text{fix}}=4$, the dynamic advantages are $A_2^{\text{dyn}} = 4 - \tfrac14\cdot1 = 3.75$ and $A_4^{\text{dyn}} = 4 - \tfrac14\cdot0.43 \approx 3.89$.

Higher‑entropy tokens receive a slightly larger penalty, concentrating learning signal where the model is most uncertain.

Why not simply use a fixed negative advantage for all low‑logp tokens?

Because low‑logp tokens can arise from two distinct situations: high uncertainty (high entropy) and over‑confident but wrong predictions (low entropy). Scaling by entropy lets the algorithm focus stronger updates on the genuinely uncertain cases while leaving confident mistakes untouched.

OPSA updates only the lowest‑logp tokens, scaling the negative advantage by each token’s entropy, so the model self‑adapts without any external teacher.

Identify $H_{\min}=H_{\max}=1.4$ (only one token), so $r_2 = 2\frac{1.4-1.4}{1.4-1.4} - 1 = -1$ (by definition the normalized score collapses to –1).

With $\delta=1$ and $A_i^{\text{fix}}=4$, compute $A_2^{\text{dyn}} = 4 - \tfrac14\cdot(-1) = 4.25$.

Plug into the loss: $L_{\text{OPSA}} = - (4.25) \log \pi_\theta(y_2|x; y_1)$.

The gradient pushes the model to reduce the probability of the low‑confidence token 2, while leaving the other two tokens untouched.

Because only the worst‑scoring token is updated, OPSA preserves high‑confidence predictions and avoids collapsing diversity.

How does OPSA differ from standard on‑policy reinforcement learning that also uses advantage estimates?

Standard on‑policy RL treats the advantage as a scalar reward signal for the whole trajectory, whereas OPSA computes a token‑level negative advantage only for the lowest‑logp tokens and scales it by entropy. This fine‑grained, entropy‑aware penalty is what enables self‑adaptation without any external reward.

**Figure 4.** Training dynamics of standard OPD and on-policy training with fixed advantages.

**Figure 5.** Training dynamics of on-policy training with dynamic negative advantages.

**Figure 6.** Examples of OPSA updates. (a) At high-entropy positions, OPSA suppresses sampled tail tokens and reallocates probability to head tokens. (b) At high-entropy positions, OPSA redistributes probability among head tokens, with little effect on the tail. (c) OPSA avoids sampling tail tokens at low-entropy positions. (d) OPSA preserves high-confidence predictions.

Why OPSA works can be traced to three effects: (1) it suppresses unlikely tail tokens that would otherwise lead the model down erroneous reasoning paths; (2) it redistributes probability among competing head tokens at high‑entropy positions, keeping diversity alive; and (3) it leaves high‑confidence low‑entropy predictions untouched, preserving precision.

Experimental Setup

We detail models, datasets, baselines, and implementation specifics for the experiments.

We evaluate Qwen3‑1.7B, Qwen3‑4B, and Qwen3.5‑9B models, trained on the DAPO‑17k question‑only dataset without any label supervision.

Evaluation covers three in‑domain math reasoning suites (AIME24, AIME25, HMMT25) and two out‑of‑domain tasks: code generation on MBPP+ and general Q&A on GPQA‑Diamond.

Table 1 contrasts supervision signals of on‑policy methods, showing that OPSA provides dense token‑level signals while requiring no verifiable rewards, external teachers, or auxiliary hints.

Table 2 reports OPSA’s gains across models, with average improvements ranging from +1.2 % to +307 % on the listed benchmarks.

We compare OPSA against RLVR, TTRL, standard OPD, and OPSD, each representing a different combination of reward signals and teacher reliance.

OPSA is implemented in the slime framework, optimizing the objective from Eq. 5 on the lowest‑probability 20 % of sampled tokens, and all experiments run on eight NVIDIA H100/H200 GPUs.

Main Results

OPSA delivers large performance gains across models and benchmarks, with analysis of its behavior.

Recall that OPSA enables self‑adaptation without any external teacher, building on the OPD premise that on‑policy distillation can improve models despite noisy signals.

OPSA achieves a 307 % relative improvement on HMMT25 for the Qwen3‑1.7B model.

Table 2 reports the boost, far exceeding the base model’s scores.

**Table 2.** Performance of OPSA across different models on in-domain mathematical and out-of-domain code generation and general Q&A tasks. All models are evaluated in non-thinking mode.

**Table 3.** Comparison of different on-policy RL methods on Qwen3-1.7B.

**Figure 7.** OPSA elicits long-form reasoning by generating more reflective tokens compared to base models. Moreover, AIME24 Avg@4 performance increases positively with response length.

**Figure 8.** Training dynamics of OPSA when masking fork tokens.

**Figure 9.** Diversity analysis of OPSA. (a) As reasoning length increases, more opportunities for branching arise. At each fork, OPSA distributes probability more evenly across alternative tokens, allowing multiple sampling to explore different reasoning paths and thereby increasing response diversity. (b–c) Jaccard-distance comparisons on AIME24 before and after OPSA training, using 32 sampled responses per problem.

**Figure 10.** Ablation study of OPSA with different training token ratios.

Additional Results and Comparisons

Additional results further validate OPSA’s gains across baselines and token budgets.

NSR treats tokens from incorrect trajectories as negative samples, assigning a fixed negative advantage to each, which encourages exploration while keeping policy entropy stable.

**Table 7.** Comparison of different on-policy RL methods on Qwen3-1.7B.

**Figure 11.** Training dynamics of different methods on Qwen3-1.7B and pass@32 performance.

**Table C.3.** Performance Comparison Under Similar Token Budgets

**Figure 12.** Training dynamics of GRPO training started from the OPSA cold-starting 4B model

Related Work and Conclusion

We situate OPSA among prior distillation and label‑free methods, highlighting their supervision signals.

On‑policy distillation (OPD) has spawned a variety of extensions that modify the supervision signal. Parallel lines of work explore label‑free training, where the model generates its own learning signals without an external teacher.

GKD extends classic distillation by mixing student‑generated trajectories with supervised ones and allowing flexible divergence objectives.

Applies reverse‑KL distillation to large language models, aiming to reduce exposure bias by encouraging the student to match the teacher’s distribution in a way that penalizes over‑confident predictions.

A variance‑reduced estimator for on‑policy distillation that improves efficiency on reasoning tasks by better handling low‑probability tokens.

Replaces the external teacher with the same policy conditioned on additional hints, preserving the OPD training paradigm while removing the need for an external supervisor.

Leverage an LLM‑as‑a‑Judge to score the model’s own outputs, turning those scores into reward signals for reinforcement learning.

Infers pseudo‑gold answers from GRPO rollouts via majority voting or clustering, then uses those answers as rewards for the policy.

Generates pseudo‑gold answers by clustering rollout outputs, then assigns rewards based on cluster proximity.

Uses trajectory‑level entropy as a self‑certainty signal, rewarding the policy for producing low‑entropy (confident) outputs.

Our analysis shows that OPSA reproduces OPD’s gains without any teacher, by assigning entropy‑adaptive negative advantages to low‑probability tokens. This suggests that the core benefit of many on‑policy distillation tricks lies in reshaping the policy’s own token‑level distribution.

Experimental Details

Details the hyperparameters, evaluation settings, and performance tables for the experiments.

Section B.1 enumerates the training hyperparameters: framework, engines, GPU count, learning rate, batch sizes, and decoding settings for both rollout and evaluation phases.

**Table B.1.** Training Configuration

Section B.2 describes the evaluation configuration, specifying the rollout engine, temperature, top‑k, top‑p, token limit, and the metrics reported.

**Table 5.** Evaluation hyperparameters for OPSA

During OPSA training we first identify the 20 % of token positions with the lowest student log‑probabilities. If any of the top‑5 candidate tokens belongs to the reflective word set, the position is marked as a reflective fork and excluded from the loss.

Response diversity is measured with the pairwise Jaccard distance $d_J(r_i, r_j; L)$ over token‑level 4‑grams. For each response we keep its first $L$ tokens, where $L \in \{512, 1024, 2048, 4096, \text{Full}\}$, and compute the macro‑average $D_J(L)$ across 30 problems and 32 sampled responses per problem.

The improvement over the base model is reported as $\Delta D_J(L)=D_J^{\text{OPSA}}(L)-D_J^{\text{Base}}(L)$, where a larger value indicates greater diversity introduced by OPSA.

Section B.3 presents the raw training and inference metrics for each method, highlighting token counts and Avg@32 scores.

**Table 6.** Training and inference overhead of OPSA compared to baselines.

Section B.4 aggregates benchmark results on AIME24, AIME25, and HMMT25, reporting both Avg@32 and Pass@32 metrics for each method.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers