Scaling Properties of Same-Family On-Policy Distillation
Yuntai Bao, Qinfeng Li, Guoqing Jiang, Liwei Chen, Zhiheng Qin, Xuanping Li, Wenqi Zhang, Xuhong Zhang
On-policy distillation (OPD) dynamics follow predictable scaling laws, allowing performance estimation before training.
How do student and teacher model scales, along with specific training design choices, predictably govern the performance gains achieved through on-policy distillation?
Reinforcement learning (RL) transfers reasoning expertise between models, but the scaling behavior of on-policy distillation (OPD)—where a teacher supervises a student's rollouts—is poorly understood. Practitioners currently rely on trial-and-error to determine if a specific teacher will improve a student model. The authors characterize OPD training as a two-phase process: an initial "useful-transfer" regime where accuracy rises linearly with the square root of token-level KL divergence, followed by a noisy tail. They fit joint power laws to predict peak accuracy and transfer rates based on student size, teacher size, and teacher performance. These laws allow researchers to forecast the outcome of an OPD run before training, revealing that smaller teachers often transfer better than larger ones at matched performance levels.
Paper Primer
OPD optimizes a student policy against a teacher-induced token reward, effectively treating the teacher as a proxy for ground-truth reasoning. The authors map this optimization trajectory using the square root of token-mean KL divergence ($d$) as a coordinate for training progress, revealing a consistent linear gain phase before saturation or regression occurs.
Peak student accuracy is predictable via joint power laws in student and teacher scale.
Leave-one-scale-out cross-validation shows the joint law extrapolates to held-out model sizes within 0.7 accuracy points for Vanilla-OPD. Conditioning on teacher score halves the prediction error compared to using parameter counts alone.
Smaller teachers are more efficient supervisors than larger ones at matched performance levels.
The fitted power laws show a negative exponent for teacher size at a fixed gold score, meaning larger teachers transfer capability more slowly. The 3B teacher outperforms the 7B teacher when distilling into a 3B student, demonstrating that teacher quality is not synonymous with teacher size.
Why does the paper focus on "weak-to-strong" distillation?
It allows for the amortization of expensive RL-based task expertise across an entire model family, using a single small expert to train larger, more capable students.
How does the choice of distillation objective (Vanilla vs. Delta) affect the results?
Delta-OPD, which uses the policy shift acquired by the teacher during RL as the reward, generally achieves higher local transfer slopes and peak gains than Vanilla-OPD, particularly in weak-to-strong configurations.
Introduction and Motivation
We expose how reasoning transfers across model sizes via on‑policy distillation.
Reinforcement learning endows large language models with reasoning abilities, yet it is unclear how these abilities transfer when scaling models via on‑policy distillation.
OPD lets a student model learn from a teacher’s token‑level distribution by sampling rollouts with the student’s own policy, so the student receives guidance that is directly relevant to its behavior.
**Figure 1** OPD transfers expertise from an RL-trained teacher to a student policy. **Left**: the three teacher–student setups, with post-RL experts as teachers and SFT bases as students. **Center**: gold score rises linearly with KL-indexed training progress $d$ during the useful-transfer phase, after which dynamics turn noisy. **Right**: the fitted joint power law predicts peak gold error from teacher and student scale.
The paper aims to quantify how reasoning capabilities transfer across model scales during distillation.
Formalizing Distillation Objectives
Formal notation and objectives for on‑policy distillation variants.
This section fixes the notation used throughout and formalizes the three on‑policy distillation objectives that the experiments will compare.
Vanilla‑OPD trains the student to imitate the teacher token‑by‑token by minimizing the reverse KL divergence between their conditional distributions at each generated prefix.
How does Vanilla‑OPD differ from ordinary behavior cloning?
Behavior cloning minimizes a forward KL (student ‖ teacher) on a fixed dataset, which encourages the student to cover the teacher’s support but can over‑penalize low‑probability teacher actions. Vanilla‑OPD uses the reverse KL, so the student is driven to match the teacher’s high‑probability choices while tolerating the teacher’s low‑probability tails.
Delta‑OPD replaces the student‑teacher token reward with the improvement the teacher achieved over its own base policy, thereby transferring the *change* in behavior rather than the absolute policy.
Why not simply distill the teacher’s final policy instead of the delta over its base?
Distilling the final policy (Vanilla‑OPD) ignores *why* the teacher changed during RL; the delta captures the *direction* of improvement, which can be more informative when the base already encodes much of the knowledge.
OffPD trains the student on trajectories generated by the teacher, turning the problem into a standard supervised learning task where the student maximizes the likelihood of the teacher’s demonstrations.
Gold Score is the benchmark performance obtained by an optimal policy on the task; it serves as the upper bound that distillation methods aim to approach.
Experimental Setup
We detail the models, training steps, and metrics used to assess OPD performance.
We evaluate OPD across five model scales using a consistent training pipeline.
Initialize Qwen2.5 Base models at 0.5B, 1.5B, 3B, 7B, and 14B parameters.
Run an initial supervised fine‑tuning (SFT) phase on a subset of the data.
Apply Dolci‑SFT to teach basic instruction‑following under a chat template.
Train teacher models via GRPO RL on the mixed GSM8K + MATH split (14.8 K examples).
Pair each teacher with a student for OPD; each run lasts up to ten epochs (580 updates) with periodic held‑out evaluation.
Measure Gold Score (accuracy on the held‑out test set) for both teachers and students.
Record the token‑mean reverse KL estimator $k_3$ and its square‑root $d = \sqrt{k_3}$ as proxies for training progress.
Compute $\delta_1 = \log 0.8 - \log 0.5 \approx 0.470$.
Evaluate $\exp(\delta_1) - \delta_1 - 1 \approx 1.60 - 0.470 - 1 = 0.13$.
Since $|y|=1$, $k_3 = 0.13$.
Take the square root: $d = \sqrt{0.13} \approx 0.36$.
The token‑mean KL captures per‑token divergence, and its square root provides a smooth signal for tracking training progress.
Characterizing Transfer Dynamics
We map how on‑policy distillation transfers capability over training, identifying a linear early regime and its limits.
To understand how capability moves from teacher to student we examine the full training trajectories of 25 vanilla‑OPD runs, spanning weak‑to‑strong, same‑base, and strong‑to‑weak configurations.
Early in OPD training the student’s gold score climbs roughly linearly with the square‑root KL distance $d$, like filling a bucket at a steady rate $m$, until the bucket overflows and the dynamics become noisy.
How does this transfer dynamics differ from a standard learning‑curve analysis?
Standard curves plot performance versus wall‑clock time or epochs, assuming a monotonic improvement. Here the independent variable is the square‑root KL distance $d$, which directly measures how far the student’s policy has moved from the teacher’s distribution, and the dynamics explicitly split into a predictable linear phase and a later noisy phase.
**Figure 2.** Gold score for the 0.5B, 3B, and 14B students as teacher scale varies. Dashed lines show linear fits to the first 30 observations. Full results are in Figures 10 and 11.
**Figure 3** Fitted initial slope $m$ across teacher scale, including only teachers no larger than each fixed student.
Scaling Laws for Distillation
We fit power‑law models for peak gold score and transfer rate, showing they beat scale‑only baselines.
Joint power‑law fits reduce prediction error for peak gold score compared with a scale‑only baseline.
Table 2 shows a leave‑one‑scale‑out RMSE improvement of 0.7 accuracy points for Vanilla‑OPD.
Joint power‑law fits also improve peak‑score prediction for Delta‑OPD.
Table 2 reports a 0.4 accuracy‑point RMSE reduction for Delta‑OPD.
**Figure 4** Peak gold score across model scale. Left: the complete 25-cell Vanilla grid. Middle: fixed-student Vanilla trends using teachers no larger than the student, with color-matched dashed levels marking each student's direct-RL endpoint. Right: Vanilla peaks grouped by teacher, together with the direct-RL endpoint and the SFT baseline, on the same y axis as the middle panel. Parameter axes are logarithmic and all curves are one-run descriptions.
**Figure 5.** Trajectories of the intermediate-teacher validation. Gold score against $d$ for the 7B student distilled from the 1.5B RL endpoint, the score-matched intermediate 3B checkpoint at step 58, and the 3B RL endpoint. Stars mark peaks and dashed levels each teacher's own gold score.
**Figure 6.** Observed peak gold score and initial transfer rate against full-fit predictions of the joint laws of Equation (6). Red diamonds mark cells whose teachers are OPD products of the bootstrapped chains (§6.3).
Impact of Design Choices
How design choices reshape OPD transfer performance across variants and supervision levels.
We now answer the third research question: how do concrete design choices—OPD variant, supervision degree, and bootstrapping—affect transfer performance. Each factor is varied while holding teacher and student scales constant.
Bootstrapping builds a chain of intermediate student models, each trained via OPD on the previous model, to gradually scale up capability.
How does bootstrapping differ from ordinary fine‑tuning?
Fine‑tuning adapts a single pretrained model directly to a new task, whereas bootstrapping creates a sequence of OPD steps, each using the previous model as a teacher, to progressively enlarge the model while preserving learned behavior.
Delta‑OPD, the alternative objective introduced in §2, is compared against Vanilla‑OPD across matched teacher‑student pairs. Gains are measured relative to each run’s own initialization.
We next vary the degree of on‑policy supervision: pure OPD, OPD with an off‑policy cold‑start, and pure off‑policy OPD. All share the same teachers and evaluation protocol.
Finally we assess bootstrapping weak‑to‑strong OPD chains. Five Vanilla‑OPD and three Delta‑OPD chains are built, stepping through intermediate sizes.
**Figure 18** Gold-score dynamics of the five Vanilla-OPD and three Delta-OPD bootstrap chains. Stars mark the observed peak.
**Figure 19** Peak gold score under three degrees of on-policy supervision, per student, over weak-to-strong and same-based cells. Dash-dotted levels mark SFT initializations.
**Figure 7** Peak gold score against student scale for Delta-OPD, Vanilla-OPD, and direct RL, grouped by teacher.
**Figure 8** Peak gold score under three degrees of on-policy supervision (0.5B expert as teacher).
**Figure 9** Full 0.5B→14B bootstrapped chains against direct baselines.
Related Work
We situate our work among prior on‑policy distillation and related scaling analyses.
On‑policy distillation (OPD) lets a teacher policy provide token‑level supervision on rollouts drawn from the student, sidestepping the exposure bias of offline teacher data. Recent variants reshape the token reward and objective—OPD2, Direct‑OPD, and W2S‑OPD supervise the student with contrasts between post‑RL and pre‑RL teacher checkpoints, a family instantiated by Delta‑OPD, while OPRD rescales the student’s verifier‑driven RL gradient along this shift.
Reward‑model overoptimization treats the gold reward as a function of KL divergence from the initial policy, a view extended to direct alignment and surveyed as reward hacking. Distillation inherits this proxy‑optimization hazard: fitting a fixed offline teacher repeatedly can induce teacher hacking, whereas noisy‑expert theory favors online interaction to mitigate the risk.
Weak‑to‑strong generalization shows that weak supervision can elicit capabilities beyond the supervisor’s own, with feature‑learning theory attributing gains to knowledge latent in the strong model. In pretraining distillation, small teachers improve larger students, though gains saturate or reverse as teachers grow, motivating our effective‑teacher cap.
Scaling laws have been used to predict pretraining loss from model size, data, and compute, and later extended to fine‑tuning, pretraining distillation, and RL compute. Those laws forecast endpoints from resource budgets, whereas our work targets KL‑indexed trajectories to predict peak performance and transfer rates.
Extended Capability Dynamics
Limits of OPD scaling: teacher quality, KL budget, and theoretical assumptions.
On‑policy distillation transfers reasoning from RL‑trained teachers to students via token‑level supervision; this section examines where that transfer breaks down.
Capability dynamics are indexed by KL. Across the Qwen2.5 grid every trajectory starts with useful transfer, but later checkpoints show attenuated improvement, saturation, or even regression. Both `d_transfer` and `d_peak` depend on the teacher–student pair, revealing that a fixed optimizer‑step budget cannot capture the full picture.
Teacher quality is not solely a function of size. While 7B and 14B students follow monotone peak trends, smaller scales produce counterexamples. Because teacher size and competence are collinear (r = ‑0.9999) in the RL‑endpoint grid, bootstrapped chains with OPD‑product teachers break this collinearity, allowing the joint law (Equation 6) to attribute peak error mainly to teacher remaining error and to reveal a negative size exponent at matched score (Table 11).
A held‑out comparison (Table 3) confirms the joint law’s prediction: an intermediate 3B teacher checkpoint, scoring slightly above the 1.5B RL endpoint, teaches the 7B student worse. The same dissociation appears at the trajectory level: the 1.5B OPD teacher outperforms the 0.5B RL expert yet yields poorer transfer to a 3B student.
Prediction connects the two regimes. Early observations (first 30 Vanilla‑OPD and first 40 Delta‑OPD steps) estimate a local transfer rate. Across scale, remaining peak error follows a stable joint power law in student scale and teacher score, while transfer extent stays within a narrow band. Thus peak capability scales with size and teacher quality, whereas the regular regime behaves as an approximately scale‑free KL budget.
Regression and over‑optimization are distinct. Gold degradation can stem from over‑imitating a teacher that lacks certain capabilities. Implicit‑reward over‑optimization additionally requires fixed‑prompt evidence that P continues to improve as G falls. Collapse denotes an abrupt optimization failure, and the diagnostics separate three testable mechanisms.
Implications for OPD design follow from the peak and rate laws. The peak law forecasts attainable capability from student scale and measured teacher quality; the rate law quantifies how quickly KL is spent to approach that peak. Empirically, Delta‑OPD converts token‑mean KL into local gold gain more rapidly than Vanilla‑OPD in 15 of 17 shared cells and improves peak gain in 12 cells.
Beyond human supervision, large reasoning models may keep improving as supervision recedes. Our results show that a compact RL expert can transfer capability to much larger students at rates predictable from scale, and that returns to teacher scaling taper off near the student’s own scale. Consequently, supervision competence need not grow with the policy it trains.
Scope of the evidence is limited to the Qwen2.5 family (0.5B–14B), a single math‑reasoning mixture, one canonical run per teacher–student pair, trajectories capped at 2K tokens, and intentionally stopped rollouts. Evaluation aggregates across prompts, lacking prompt‑bootstrap or seed‑level granularity, and cell‑bootstrap intervals capture only scale‑pair variation.
Several recipe constants remain underspecified. Rollout length is fixed at 2K tokens, though prior work shows it affects OPD stability. Other fixed factors include the shared SFT recipe, teacher RL checkpoint, KL estimator, validation frequency, and same‑step logging convention. The token‑mean coordinate weights tokens rather than full responses, and the archive lacks the full‑sequence k₃ estimator, leaving robustness untested.
The local expansion in Appendix D explains why initial transfer is linear in d, but only its leading exponent. The extent of the linear window, heterogeneous tails, and power‑law fitted coefficients remain empirical regularities without a mechanistic theory. Delta‑OPD’s endpoint law has only five observed departures, and the transfer‑rate power law does not outperform a raw‑affine baseline under leave‑one‑scale‑out RMSE.
Early termination and finite evaluation frequency can right‑censor `d_transfer` and place `d_peak` at the observation boundary. Multiple seeds, denser evaluation, and prompt‑level resampling would quantify both sources of uncertainty.
OPD variants have proliferated (e.g., Generalized Knowledge Distillation, ExOPD, relaxed OPD, W2S‑OPD, Direct‑OPD, OPD2, OPRD). Many combine token‑level distillation with outcome‑level correctness signals, either via separate teachers or hindsight verification. Our experiments use the contrast family (Delta‑OPD) and compare methods at matched KL rather than matched compute.
Proxy over‑optimization and reward quality studies (e.g., Gao et al., 2023; Rafailov et al., 2024) motivate our proxy diagnostics for interpreting regressing OPD trajectories.
Offline teacher hacking studies (Tiapkin et al., 2025) differ from our on‑policy rollouts but highlight the importance of online data diversity, which our design leverages.
Weak‑to‑strong generalization work (Burns et al., 2024; Awano & Suzuki, 2026) shows that strong students can surpass weak supervisors, aligning with our finding that peak gold score improves with teacher scale only up to the student’s own scale.
Weakly supervised learning literature (Zhou, 2018; Ratner et al., 2017) frames OPD as a case where supervision is incomplete and evolves with the student policy, distinguishing it from semi‑supervised learning where ground‑truth labels exist.
Scaling‑law studies for LLM pretraining and adaptation (Kaplan et al., 2020; Hoffmann et al., 2022; Zhang et al., 2024) predict performance across resource axes, but our trajectories use token‑mean KL as the optimization coordinate, making direct transfer of those laws inappropriate.
Our targets are teacher scale, student scale, and the OPD objective shape: how much capability transfers and how fast. Leave‑one‑scale‑out prediction determines whether cross‑pair relationships constitute scaling laws, following the methodology of Choshen et al., 2025.
**Table 3.** Out-of-sample validation of the Vanilla-OPD peak laws with an intermediate 3B teacher checkpoint, in gold-score points. Each row is a 7B-student cell; predictions use the full-fit coefficients of the scale-only (Equation (7)), score-only (Equation (19)), and joint (Equation (6)) laws. The intermediate-teacher cell is held out from every fit; only the joint law orders it correctly relative to the 1.5B endpoint teacher.
Appendix D derives Equation (8), the population counterpart of the token‑mean estimator (Equation 5). The derivation shows that both objectives share the same leading exponent, with slopes differing by the Fisher‑normalized alignment of the update direction with the gold gradient.
D.1 Statement: expanding KL locally yields G(d) = G₀ + m d + O(d²), where m = (g₀ᵀh)/(hᵀ`F_tok` h). Gold score changes to first order, KL to second order, so both share the leading exponent.
D.2 A local square‑root‑KL law: for a differentiable training path $\theta$($\tau$) = $\theta$₀ + $\tau$h + O($\tau$²), the token‑weighted Fisher matrix yields `D_tok`($\theta$($\tau$)) = ($\tau$²/2) hᵀ`F_tok` h + O($\tau$³). Inverting the relation gives Equation (8), linking KL distance d to $\tau$.
D.3 Method‑specific directions and slopes: setting r = $r_V$ or r = r_$\Delta$ selects the Vanilla‑OPD or Delta‑OPD direction h without changing the leading power of d. The local slope measures the Fisher‑normalized alignment of this direction with the gold gradient.
D.4 Assumptions and scope: the derivation assumes common support, differentiable log probabilities, a locally differentiable generated‑prefix distribution with finite moments, a non‑zero Fisher norm, and a smooth expected gold utility. These conditions hold for small early updates but are empirically testable.
Training Hyperparameters
Lists the hyperparameter configurations for SFT, RL teachers, OPD, and evaluation across Qwen2.5 models.
This section enumerates the hyperparameter settings used for supervised fine‑tuning (SFT), reinforcement‑learning (RL) teachers, and on‑policy distillation (OPD) across the Qwen2.5 model family.
The SFT learning rate follows $\\text{LR}(N)=2\\times10^{-5}\\bigl(\\tfrac{N}{0.5\\text{B}}\\bigr)^{-1/4}$, scaling inversely with the fourth root of the model parameter count.
**Figure 10** Qwen2.5 mathematics trajectory grid with token-mean-KL ticks on square-root-spaced axes. Diamonds mark transfer endpoints (hollow when right-censored), and stars mark global gold-score maxima. Dotted purple curves show the teacher-induced proxy reward on right-hand axes for the pairs where it was logged. Rows share the student scale and columns share the teacher scale; panels within a row share axis ranges.
**Figure 11.** Delta-OPD trajectory grid over weak-to-strong and same-base cells with token-mean-KL ticks on square-root-spaced axes. Diamonds mark transfer endpoints (hollow when right-censored), and stars mark global gold-score maxima. Dotted purple curves show the token-mean Delta-OPD reward on right-hand axes, with the same ticks as Figure 10. Rows share the student scale and columns share the teacher scale; panels within a row share axis ranges.
All experiments use a single random seed per configuration to keep compute costs tractable; this mirrors standard practice in scaling‑law studies where trends are inferred from many distinct configurations rather than repeated runs.
Supplementary Results
Additional appendices detail Delta‑OPD results, scaling analyses, and lessons learned.
Table 9 summarizes per‑pair Delta‑OPD statistics, showing peak gains in percentage points and indicating whether each endpoint is observed (Obs.) or right‑censored (Cens.). Five observed transfer endpoints appear in weak‑to‑strong cells and one same‑base cell; the remaining twelve are censored.
**Figure 13.** Student-scale SFT baseline. Points show equally weighted Vanilla step-zero means with within-scale standard deviations. The solid curve is the five-scale remaining-error fit in Equation (18); the dashed curve is refit through 7B and its open square is the 14B outer-scale prediction.
Table 10 compares three functional forms—multiplicative, weighted additive, and equal additive—using leave‑one‑scale‑out RMSE (S/T) and AICc for both Vanilla and Delta methods; the multiplicative law achieves the lowest RMSE and most favorable AICc.
**Figure 14** Peak-performance scaling under the multiplicative and weighted-additive laws. Rows separate Vanilla-OPD and Delta-OPD. The left column fixes student scale and retains teachers no larger than the student; the right column fixes teacher scale. Points are observed peaks, solid curves are multiplicative-law fits, and dashed curves are weighted-additive fits over each series’ support.
Table 11 reports teacher‑size and teacher‑performance fits, revealing strong collinearity (≈ −0.9999) that is reduced by adding bootstrap‑teacher cells, yielding identifiable joint laws with exponents $\zeta$≈0.95 for Vanilla and $\zeta$≈1.01 for Delta.
**Figure 15.** Teacher size versus teacher performance, fitted on the grid plus the auxiliary bootstrap-teacher cells; rows are Vanilla-OPD (top) and Delta-OPD (bottom). Left two columns: leave-one-scale-out RMSE of the size, performance, and joint laws for the peak and rate targets. Third column: bootstrap joint-law teacher exponents for both targets, concentrated away from zero on anticorrelated ridges. Right: capped teacher size against capped-teacher error; red diamonds mark OPD-product teachers, which sit off the RL-endpoint size trend and break the collinearity.
Table 12 presents outer‑scale peak extrapolation errors for various hold‑out scenarios, indicating that the joint teacher law generally reduces RMSE compared to size‑only or weighted‑additive baselines.
Table 13 validates the scaling laws under the joint teacher laws, reporting peak S/T errors of 0.55 / 0.68 for Vanilla‑OPD and 0.20 / 0.32 for Delta‑OPD, and corresponding rate S/T errors.
Table 14 lists power‑law coefficients with 95 % bootstrap intervals for peak error, rate, and extent targets, showing modest differences between Vanilla and Delta methods.