On-Policy Self-Distillation without Any Supervision

Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, Nuno Vasconcelos

U-OPSD replaces ground-truth labels in self-distillation with majority-vote consensus from the model's own rollouts.

Can we improve LLM reasoning performance through self-distillation without using any external ground-truth labels or environment feedback?

On-policy self-distillation (OPSD) typically requires ground-truth solutions to guide a model's learning, limiting its use to domains where such labels are available and reliable. U-OPSD removes this dependency by using the model's own majority-vote consensus to construct a "pseudo-solution" for each problem. The model then distills its own teacher distribution—conditioned on this consensus—into its student policy specifically on trajectories where it initially disagreed with itself. This unsupervised approach consistently outperforms supervised OPSD and reinforcement learning baselines, improving base model performance by up to 10.7% on mathematical reasoning benchmarks.

Paper Primer

U-OPSD functions as a self-correcting loop: it samples multiple rollouts, identifies the majority answer as a pseudo-solution, and forces the model to align its inconsistent reasoning paths with that consensus. The method is essentially a filter that identifies the model's "competence frontier"—prompts where it is confident enough to vote but fallible enough to produce conflicting traces—and applies dense token-level supervision only to those disagreements.

U-OPSD matches or exceeds the performance of supervised methods that rely on ground-truth labels.

Across five math benchmarks, U-OPSD outperformed OPSD (which uses ground-truth) by 3.2% and 2.3% for 4B and 8B models in non-thinking mode. 8.5–10.7% improvement over base models in non-thinking mode.

Consensus-based distillation is more effective than scalar reinforcement learning rewards.

U-OPSD outperformed label-free reinforcement learning baselines (like GRPO) by an average of 7.0–11.3% in non-thinking mode.

Why is this approach better than simply training on the majority-vote answer?

Standard self-training often just imitates the final answer, whereas U-OPSD transfers the entire solution-conditioned next-token distribution. This provides dense, corrective supervision along the reasoning path, which the authors found is essential for performance gains.

What is the primary limitation of this method?

The method is bounded by the base model's initial competence; because it relies on majority voting, it can only improve if the model is already capable of producing the correct answer more often than any single incorrect one.

Motivation and Problem Framing

We expose the supervision bottleneck of on‑policy distillation and propose unsupervised self‑distillation via internal consensus.

Current on‑policy distillation pipelines still depend on external supervision such as ground‑truth answers, environment feedback, or a stronger teacher model. This reliance creates a bottleneck: the model can only improve when an outside signal supplies the privileged context, which limits applicability to unlabeled or costly domains.

On‑policy self‑distillation promises a model that teaches itself, but in practice the “teacher” still looks outside the model for correct answers, so the process is not truly self‑contained.

OPSD lets a single LLM play both teacher and student by conditioning the teacher on a privileged context (e.g., a ground‑truth solution) while the student sees only the problem.

**Figure 1:** Comparison between OPSD / SDFT with ground-truth solution or ICLs (left), SDPO with rich feedback from the environment (middle) and our U-OPSD without any supervision.

The key shift is replacing external supervision with internal consensus, enabling truly unsupervised self‑distillation.

The U-OPSD Framework

U‑OPSD replaces external labels with model‑generated consensus to self‑distill on‑policy behavior.

On‑policy distillation yields dense supervision but still relies on external signals such as gold answers or a stronger teacher. This dependence limits scalability and introduces a supervision bottleneck. U‑OPSD removes that bottleneck by letting the model generate its own pseudo‑labels from internal agreement.

U‑OPSD lets a model teach itself: it runs several rollouts, takes the most common answer as a pseudo‑solution, and then forces the student to match the teacher’s next‑token distribution on the rollouts that disagree with that consensus.

Majority‑vote answer \tilde{a}(x) = 4 (appears twice).

Agreeing set $Y$⁺ contains the two rollouts that output 4; disagreeing set $Y$⁻ contains the single rollout that output 5.

Confidence c(x) = (2/3) ≈ 0.67 ≥ $\tau$ = 0.5, so the sample is kept.

The loss is computed only on the disagreeing rollout, aligning the student’s token‑level distribution with the teacher conditioned on the pseudo‑solution (4).

U‑OPSD concentrates learning on the exact points where the model’s own predictions diverge, driving improvement where uncertainty remains.

Think of the model as a panel of reviewers each writing an answer; the answer that appears most often becomes the pseudo‑label that the model will later try to emulate.

Algorithm 1 U‑OPSD: Unsupervised On‑policy Self‑distillation (one prompt)

Performance Evaluation

U‑OPSD sets new performance peaks on both model sizes.

The experiments evaluate four Qwen3 variants across five math‑reasoning benchmarks, comparing U‑OPSD against supervised and label‑free baselines.

U‑OPSD attains the highest average score of 86.50 on Qwen3‑4B non‑thinking mode, surpassing the next best method by 1.65 points.

Table 1 shows U‑OPSD’s average of 86.50 versus SFT’s 84.85 and GRPO’s 86.20.

GRPO treats each training example as a group and optimizes the policy to improve the average reward across the group rather than individual samples.

SFT fine‑tunes the model on gold‑solution pairs, directly teaching it the correct answer for each problem.

**Table 1.** Non-thinking mode results for Qwen3-4B and Qwen3-8B models across various benchmarks (AIME24, AIME25, HMMT25, MATH500, AMC23) and methods (Base, SFT, GRPO, OPSD, TTRL, RENT, Intuitor, U-OPSD).

**Figure 3.** Training curves of the Qwen3-4B thinking (left) and non-thinking mode (right) runs on AIME24, AIME25 and MATH500. Top: U-OPSD against the supervised methods. Bottom: U-OPSD against the label-free methods.

Extended Benchmarking

U‑OPSD’s extra gains over baselines in non‑thinking mode.

U‑OPSD outperforms all supervised baselines, beating OPSD by 3.2% on Qwen3‑4B and 2.3% on Qwen3‑8B.

Table 1 shows U‑OPSD’s scores exceed those of SFT, GRPO, and OPSD for both model sizes.

In contrast, label‑free RL baselines improve the base models by at most 1.5% under the same rollout budget, highlighting the advantage of consensus‑derived solutions as privileged teacher context for token‑level distillation rather than reducing them to scalar rewards.

Ablation Studies and Mechanism Analysis

Ablation studies dissect the impact of each U‑OPSD component on accuracy.

U‑OPSD replaces external supervision with the model’s own majority‑vote consensus, letting the teacher be built from self‑generated rollouts.

86.7% of pseudo‑labels match the gold answer.

On 64 training prompts (G=8, $\tau$=0.5), 96.3% of rollouts are parsable and 94.0% of prompts receive a pseudo‑label.

Increasing rollouts per prompt to G = 12 improves accuracy by 4.7% over the default.

G = 4 and G = 8 yield 56.99 and 57.10 respectively; G = 12 adds a 4.7% gain.

Further raising G to 16 yields roughly half the gain (≈2.3%).

G = 16 returns half the improvement of G = 12.

EMA teacher update with decay 0.995 gains 2.4% over the frozen teacher at the best checkpoint.

Best EMA (0.995) outperforms the frozen baseline.

At step 150, EMA decay 0.995 improves accuracy by 4.1% relative to the frozen teacher.

Step‑150 comparison shows a 4.1% advantage.

Removing the reference to the boxed pseudo‑label drops performance by 10.3% to 15.8%, pushing all label‑only variants below the base model’s 43.57 %.

Forward KL yields an average score of 20.00.

Table 5 under U‑OPSD.

Jensen–Shannon divergence drops performance to an average of 43.34 (13.8% loss).

Table 5 comparison.

**Figure 4.** Configuration ablations, on Qwen3-8B non-thinking. The vertical axis is accuracy averaged over the five evaluation benchmarks. **Left:** self-consistency threshold $\tau$, the fraction of valid rollouts that must agree for a prompt to receive a pseudo-label. **Middle:** rollouts per prompt $G$, at $\tau=0.5$. **Right:** how the teacher is updated; "fixed" freezes it at the initial policy and is our default, while the EMA rows let it track the student at the given decay.

**Figure 5.** Teacher reference against distillation target, on Qwen3-8B non-thinking, $G=8$, $\tau=0.5$, $k=1$; each point is the five-benchmark average. (a) What the teacher is conditioned on. (b) Which disagreeing rollout is distilled.

Limitations: experiments are confined to Qwen3 models on competition‑math benchmarks; gains shrink in the “thinking” regime; performance is bounded by the base model’s competence; and variance estimates are pending.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers