Rethinking On-Policy Distillation of Large Language Models II: One Training Example

Zixuan Fu, Bingxiang He, Yuxin Zuo, Haohuan Huang, Jinqian Zhang, Ruhang Xiao, Cheng Qian, Qinyu Luo, Huan-ang Gao, Yudong Wang, Zhiyuan Liu, Ning Ding, Chaojun Xiao

On-policy distillation is data-overfed: a single query provides enough state coverage to match full-data performance.

Why does on-policy distillation (OPD) achieve most of its performance gains from a single training query, and what does this reveal about the data-efficiency of the process?

On-policy distillation (OPD) is standard for aligning frontier LLMs, but its reliance on massive datasets remains poorly understood. Researchers often assume that more queries are necessary to provide sufficient supervision for the student model. This paper demonstrates that OPD is data-overfed: a single query generates enough diverse reasoning states to recover most of the performance gains seen in full-data training. The student model improves for hundreds of steps because the algorithm is rate-limited by its own slow absorption of supervision, not by a lack of training data. Training on just 16 semantically diverse queries per domain is sufficient to match the performance of full-data OPD across multiple task domains and model families.

Paper Primer

OPD works by having a student model sample its own rollouts and align its token-level predictions with a teacher. The authors find that this process is "algorithm-starved": the student's ability to absorb the teacher's supervision is the bottleneck, while the data supply is effectively inexhaustible even from a single query.

The core mechanism is state coverage: a single query induces a vast space of reasoning states (prefixes) that the student visits during its rollouts. Because the teacher provides dense, token-level supervision at every visited state, a single query provides enough breadth to cover 71.5% of the states visited by full-data training.

One-shot OPD recovers the majority of full-data performance.

Across math, code, and instruction-following benchmarks, training on a single query recovers 64% to 73% of the teacher-student gap. 16 semantically diverse queries reach 98.9% state coverage and match full-data OPD performance.

The duration of training is a property of the algorithm, not the data.

The rate at which the student aligns with the teacher (absorption rate) declines at the same pace whether training on one query or 17,000 queries. Alignment slows by a similar factor across all data scales, requiring hundreds of steps to converge.

Why does the model keep learning for hundreds of steps if it only has one query?

The student continuously visits new reasoning states through its own rollouts, and the algorithm's absorption rate—the proportion of the teacher-student gap closed per update—diminishes steadily over time, stretching the learning process.

Does the difficulty of the chosen query matter for the final performance?

No; the one-shot gain is robust to query difficulty, response length, and sampling temperature. Even queries the student never solves during training are as effective as those it solves consistently.

This paper suggests that future post-training efforts should focus on "state-level" data curation—selecting queries based on the reasoning states they induce—rather than simply scaling the number of training examples.

Researchers can achieve near-optimal distillation results with minimal data by prioritizing semantic diversity in query selection, effectively shifting the focus from data quantity to state-space coverage.

Introduction and Motivation

We expose why OPD learns from minimal data and challenge the need for massive datasets.

On‑policy distillation (OPD) merges student‑generated rollouts with dense token‑level supervision from a teacher. Prior work has focused on algorithmic dynamics, leaving the influence of training data largely unexplored. This section frames the central question: why does OPD keep improving when trained on just one query?

OPD lets a student model generate its own contexts and receive token‑level guidance from a teacher, turning each rollout prefix into a dense supervision point.

How does OPD differ from standard supervised fine‑tuning?

Supervised fine‑tuning learns from a fixed dataset of input‑output pairs, while OPD generates its own rollouts and receives token‑level teacher predictions at every prefix, yielding far richer, on‑policy supervision.

**Figure 1.** A single query recovers most of full-data OPD's gain on mathematical reasoning (left), and the effect holds across domains and model families. Under multi-teacher OPD, 16 queries per domain match full-data training (right).

Table 1 enumerates the student–teacher model pairs across four task domains, illustrating the breadth of experiments supporting the analysis.

OPD does not require massive datasets; a handful of queries can supply most of the needed supervision.

Notation and Setup

Defines the symbols, OPD objective, and the metrics used to track progress.

We denote a training example by $x \sim D$ and its response by $y = (y_1,\dots,y_L)$. The prefix up to position $i$ is $y_{<i} = (y_1,\dots,y_{i-1})$, and the autoregressive context (state) is $s_i = (x, y_{<i})$. A trajectory is the pair $(x, y)$ generated by sampling $y$ autoregressively from a policy.

On‑policy distillation (OPD) samples trajectories from the student model $\pi_\theta$ and aligns its token‑level distribution to that of a teacher $\pi_T$ on the visited states. The objective can be written as the expected KL divergence $\text{LOPD}(\theta) = \mathbb{E}_{x\sim D,\, y\sim \pi_\theta}\big[ \text{KL}\big(\pi_\theta(\cdot|s_i) \,\|\, \pi_T(\cdot|s_i)\big) \big]$, which in practice is estimated via the per‑token advantage $A^{\text{OPD}}_i = \log \pi_T(y_i|s) - \log \pi_\theta(y_i|s)$. A top‑$k$ variant replaces the single sampled token with the student’s $k$ most probable tokens, reducing variance.

Because raw score changes depend on the initial teacher–student gap and on how much data is used, we report progress as a proportion of those baselines and also measure token‑set agreement.

The One-Shot Phenomenon

One-shot OPD recovers most of full-data performance and remains robust across tasks.

We evaluate whether a single query can capture the bulk of the teacher–student gap that full‑data On‑Policy Distillation (OPD) closes. Across five task domains we train a student on rollouts generated from one query and compare against the standard full‑data OPD baseline.

One‑Shot OPD trains the student on rollouts produced from a single query, yet the student still learns to mimic the teacher on the vast majority of states visited by full‑data OPD.

How does One‑Shot OPD differ from standard OPD that uses many queries?

Standard OPD samples a fresh query each update, continuously expanding the set of visited states. One‑Shot OPD fixes the query, so the state set stops growing after the first few steps; the student therefore learns from a static distribution rather than an ever‑changing one.

**Figure 2 | OPD is data-overfed but algorithm-starved.** Data: one query already covers most of the states full-data OPD visits. Algorithm: the student absorbs an ever smaller proportion of the remaining teacher–student gap, no matter how many queries it trains on.

One‑Shot OPD recovers most of the full‑data OPD gain on mathematical reasoning.

At step 300 the one‑shot model reaches 68.5 % accuracy versus 69.8 % for full‑data OPD, closing 87 % of the teacher–student gap.

**Figure 3.** Validation accuracy and Training dynamics of one-shot OPD on mathematical reasoning.

**Figure 4** | Validation accuracy of one-shot OPD across 3 student–teacher families, on mathematical reasoning averaged over MATH-500 and AMC 2023. AIME 2025 is excluded because the Llama pair scores near zero on it.

**Figure 5.** Validation accuracy of one-shot OPD across three task domains.

**Figure 6.** Validation Accuracy of one-shot OPD under varying query difficulty, response-length cap on the hard query, and sampling temperature on the medium query.

**Table 2.** Mathematical queries used in the query-difficulty experiment, labelled by the student's pass rate before OPD training.

One‑shot OPD recovers the majority of full‑data performance while being insensitive to query difficulty, response length, and temperature.

Why One Query Suffices

We quantify how many distinct states a single query can generate and how diversity expands coverage.

Building on the one‑shot phenomenon, we now ask how a single query can supply enough supervision to approach full‑data performance, and whether adding a few diverse queries can close the remaining gap.

State coverage measures how broadly the rollouts of a training run explore the space of teacher‑generated states, regardless of how many queries are used.

Collect all teacher signatures $h_T(s)$ from the rollouts of the configuration under test.

Project each signature onto the PCA basis learned from the full‑data reference set.

Assign each projected vector to its nearest of the $K=200$ centroids obtained by K‑means.

Count the number of distinct centroids occupied by the configuration’s states.

Divide that count by $K$ to obtain the coverage fraction.

One‑shot OPD reaches 71.5 % state coverage.

Figure 7 (left plot) shows the cumulative coverage curve plateauing at roughly 0.715 for the one‑shot run.

One‑shot OPD improves validation accuracy to 66.9 % (full‑data OPD reaches 70.8 %).

Figure 7 (right plot) reports the one‑shot model’s accuracy curve ending at 66.9 % versus 70.8 % for the full‑data baseline.

Having established that a single query already explores most of the state space, we next vary data diversity to see how many additional queries are needed to approach full‑data performance.

Sixteen‑shot OPD attains 98.9 % state coverage.

Figure 16 (bar chart) reports a coverage of 0.989 for the 16‑shot condition.

**Figure 15** | Fraction of the 64 rollouts per step that solve the training query itself, for the three queries of Table 2.

**Table 5.** Training-state coverage under alternative cluster constructions, at $K = 200$.

**Figure 16** | Training-state coverage under alternative state-extraction positions, at $K = 200$. Error bars show one standard deviation over rollout subsamples.

**Figure 7** | State coverage and validation accuracy of one-shot OPD on the medium query, against full-data OPD. Coverage is measured against the state space full-data OPD reaches, which is therefore the 1.0 level rather than a separate curve.

**Figure 8** | Validation accuracy and state coverage as the training data is made more diverse, on the response side by off-policy training on distinct trajectories and on the query side by on-policy training on distinct queries.

A single query already supplies the majority of useful states; a handful of semantically diverse queries are enough to reach near‑full‑data coverage.

Alignment Dynamics

We examine why alignment slows over many steps and how fixed states affect OPD dynamics.

Having shown that a single query can supply enough states, we now ask why the student continues to improve for hundreds of steps instead of converging quickly.

Alignment measures how far the student policy deviates from the teacher on the states it actually visits, and how quickly that gap shrinks.

Is the distance $d_t$ the same as the training loss?

No. $d_t$ is an evaluation metric computed on the student’s own rollout states; the training loss is a gradient‑based objective that may include regularization and is evaluated on the same states but not averaged in the same magnitude‑preserving way.

“Data‑overfed” means the student sees more examples than its update rule can effectively consume, while “algorithm‑starved” means the learning algorithm updates too slowly to make use of the abundant data.

Why doesn’t providing more queries speed up alignment?

Because each update can only absorb a limited fraction of the remaining distance; extra queries merely repeat similar states, so the absorption rate $v_t$ declines in the same way.

Across 1, 4, 16, and full‑data runs the absorption rate $v_t$ drops steadily, producing a characteristic flattening of the distance curve on a log‑scale. By step 300 each run has removed roughly 78 %–84 % of its step‑30 distance, and the slowdown is indistinguishable across query counts.

**Table 7.** Alignment dynamics over steps 30–300 for on-policy OPD, at the default learning rate.

**Figure 9** | Teacher–student alignment for OPD trained on 1, 4, 16, and all DAPO-Math-17k queries. Distance (left) and absorption rate (right) are both on logarithmic vertical axes against linear steps.

To test whether a steady supply of fresh states is necessary, we fixed the training rollouts (64 trajectories sampled once from the initial student) and ran an off‑policy OPD variant. The off‑policy run still improved for ~200 steps before plateauing, and its alignment metrics followed the same slow‑down pattern as the on‑policy baseline.

**Figure 10** | Validation accuracy and alignment metrics for the on-policy and off-policy one-shot runs.

**Figure 14** | Validation accuracy and training dynamics for one-shot OPD and one-shot RLVR on the same query. Both take a batch of 64 queries per step. RLVR draws 8 rollouts per query and scores each by outcome under GRPO. OPD draws 1 and scores every sampled token against the teacher. Training accuracy is the fraction of rollouts that solve the training query.

Alignment in OPD is fundamentally algorithm‑starved: the per‑step absorption rate decays regardless of data volume, and even a fixed set of states sustains learning for hundreds of updates.

Implications and Synthesis

We reveal how input content and query scheduling shape hidden state coverage in OPD.

The discussion reframes OPD as a state‑generator: any input that provokes useful rollouts can guide the student, even if the input itself carries little task content. We therefore probe how much of the input’s semantics are truly needed and how query scheduling influences learning dynamics.

**Figure 17.** Two 16-shot query sets that differ only in diversity, against full-data OPD.

**Figure 18** | Effect of query scheduling in 2-shot OPD. The vertical dotted line marks the query switch in the sequential schedule.

**Figure 20.** Validation accuracy and actor entropy for code-generation OPD.

**Figure 11.** Validation accuracy of one-shot, 16-shot, and full-data MOPD.

**Figure 13** | Validation accuracy and actor entropy for mathematics OPD trained on the real training set, on a content-light template, or on general-domain WildChat queries.

Content‑light inputs can achieve near‑full state coverage, showing that task‑specific content is not required for effective OPD.

Figure 17’s “16‑shot From 16 Clusters” line reaches state coverage ≈ 1.0, matching the full‑data curve.

Appendix Overview

Appendix lists query‑difficulty items and three supplemental tables summarizing experiments.

The appendix compiles the full set of query‑difficulty items—three difficulty levels with identifiers t59, t22, and t56—followed by several illustrative mathematical puzzles used to probe student understanding.

It also presents three tables: Table 3 enumerates the hyperparameters and their values; Table 4 contrasts two reference‑construction strategies across 1‑shot, 4‑shot, 16‑shot, and full‑data regimes; Table 6 reports exceedance, mean distance, and AUROC for the same settings.

The table lists various hyperparameters and their corresponding values used in the training process, including batch sizes, coefficients, strategies, and optimization parameters.

The table compares two reference construction methods ("Full-data anchored" and "Equal-size pooled") across four data settings: 1-shot, 4-shot, 16-shot, and Full data (held-out).

The provided image contains a table without a formal label or caption. The table displays performance metrics across four settings: 1-shot, 4-shot, 16-shot, and Full data (all). The columns are "Setting", "Exceedance (%)", "Mean distance", and "AUROC".

Experimental Details

Supplementary experimental details and additional analyses.

We fine‑tune a DeepSeek‑R1‑Distill‑Qwen‑1.5B model on synthetic instruction‑following tasks. The teacher is the step‑600 checkpoint of this model, and we generate four completions per prompt at temperature 0.9, with a maximum prompt length of 4096 tokens and a maximum response length of 16384 tokens. Optimization uses AdamW with learning rate $1\times10^{-6}$, batch size 16, and a clip ratio of 0.2.

Table 3 lists the default hyperparameters for On‑Policy Distillation (OPD). Notable entries include a rollout batch size of 64, a top‑K strategy of “Student Top‑K”, a learning rate of $1\mathrm{e}{-6}$, and a training temperature of 1.0. All experiments in this appendix use these defaults unless explicitly noted.

Training‑state coverage is measured by collecting rollouts at every optimization step from 1 to 500, taking five rollouts per step (three for reference clusters and two held‑out). This yields 12{,}000 reference states and 8{,}000 held‑out states, which are reduced to $50$ dimensions by PCA (retaining $71.9\%$ variance) before $K$‑means clustering.

Coverage improves as the number of queries increases. With $K=50$, 1‑shot OPD attains $0.755$ coverage, 4‑shot $0.829$, and 16‑shot reaches full data ($1.000$). Similar trends hold for $K=100$, $200$, and $500$, though absolute coverage declines for larger $K$.

When clusters are built from pooled rollouts (instead of full‑data alone) with $K=200$, 1‑shot coverage rises to $0.750$, 4‑shot to $0.845$, and 16‑shot to $0.958$, indicating that a more diverse reference set boosts early coverage.

In the 16‑shot OPD setting, each query originates from a distinct semantic cluster, raising both diversity and count. Holding the count at 16 but drawing all queries from a single cluster yields $70.9\%$ validation accuracy at step 300, compared to $69.9\%$ for the single‑cluster baseline.

For 2‑shot OPD we compare a sequential schedule (hard query first, then medium) against a mixed schedule (32 copies of each query per batch). Both reach $68.8\%$–$69.1\%$ accuracy at step 600, slightly above the $67.4\%$ one‑shot baseline, showing that query ordering has modest impact relative to the query content itself.

Section 5.2 measures absorption rates using three conventions: a sliding window $[t/1.4,\,1.4t]$, a normalization $d_t/d_{30}$, and a start‑at‑step 30 rule that excludes the initial gradient‑clipping phase. These choices affect how the slope of $\log d$ is estimated and why rates are reported at steps 50 and 200 rather than at step 300.

Table 7 reports the proportion of distance remaining at step 300 and the factor by which the absorption rate declines between steps 50 and 200. The numbers illustrate that the rate drops substantially as training progresses, consistent across the three measurement conventions.

Varying the learning rate (half, default, double) rescales the horizontal step axis without altering the slowdown curve shape. When the step axis is normalized as $u = (\text{lr}/10^{-6})\,t$, the three runs collapse onto a single trajectory, confirming that learning rate only changes how quickly a run traverses the same underlying curve.

Hyperparameters and Additional Results

Appendix B provides hyperparameter tables and rollout details for on‑policy distillation experiments.

Table 8 lists the default hyperparameters for multi‑teacher on‑policy distillation (MOPD), including three teachers (math, code, IF), a rollout batch size of 64, learning rate $1e-6$, and AdamW optimizer with $β_1$=0.9, $β_2$=0.999, $wd$=0.01.

Table 7 reports alignment dynamics for on‑policy OPD between steps 30 and 300 at the default learning rate, showing $d_{300}/d_{30}$ values of 0.16, 0.17, 0.18, 0.22 and $v_{50}/v_{200}$ values of 5.6, 5.2, 4.2, 6.4 across the four query settings.

The experimental setup uses student R1‑Distill‑1.5B, teacher DeepCoder‑1.5B‑Preview, and a baseline full‑data OPD on TACO; the base, system, and WildChat templates are identical to those in Section 7.1, with the system template’s instruction taken from Figure 12b.

Evaluation is performed on LiveCodeBench v6 using the avg@3 metric, which averages the top‑3 generated solutions per problem.

Across the four conditions the final scores differ by only 1.7 points, but token accounting reveals a far larger gap: over 500 steps the baseline consumes 277 M rollout tokens whereas the base template uses just 18 M, with the other two conditions falling in between.

Actor entropy ranks the conditions exactly as in the mathematical experiments, placing WildChat highest and the system template lowest; unlike the math setting, no condition lags behind in code generation, though WildChat incurs about a one‑point cost in the mathematical view.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers