More Methods, Better Agents? a Reality Check on Self-Distillation in Agentic Reinforcement Learning

Linhao Zhong, Bozhen Fang, Xuehao Wang, Duochao Shi, Zekai Luo, Guangming Yao, Yi Yuan, Hao Chen, Jingdong Chen, Chunhua Shen

A systematic benchmark of self-distillation in agentic RL reveals that reported gains are often fragile, unstable, and computationally inefficient.

Do self-distillation methods for agentic reinforcement learning actually outperform a standard baseline (GRPO) when evaluated under consistent, rigorous conditions?

The rapid proliferation of self-distillation methods for training language agents has outpaced the development of a common evaluation protocol, leaving it unclear which techniques actually improve performance. The authors establish a unified benchmark comparing ten representative methods against a Group Relative Policy Optimization (GRPO) baseline across four diverse environments and three model scales. The study finds that no method consistently outperforms GRPO, and that strong final scores often mask unstable training trajectories and poor computational efficiency.

Paper Primer

The paper evaluates whether auxiliary distillation objectives, advantage modulation, and reward shaping actually provide reliable improvements over outcome-only reinforcement learning. By standardizing the evaluation across 132 training runs and 12,124 GPU-hours, the authors isolate methodological progress from favorable sampling or inconsistent experimental conditions.

Self-distillation methods do not provide uniform performance gains over the GRPO baseline.

Across twelve model-task settings, only two methods (PCSD and AgentOPSD) exceed GRPO's aggregate mean performance, and no single method dominates across all benchmarks. PCSD leads with a +4.58 point aggregate improvement, but it fails to outperform GRPO in half of the individual settings.

Final performance metrics frequently obscure severe training instability and late-stage regression.

Several methods exhibit "recovery" patterns where performance collapses to near-zero during training before rebounding, or they finish significantly below their observed validation peak. Nine runs finished at least 20 points below their validation peak, indicating that high final scores are not reliable indicators of stable optimization.

Why is a new benchmark necessary if these methods have already reported gains in their respective papers?

Existing studies often reuse baseline scores from different experimental conditions and rarely report repeated evaluations or training stability metrics, making it impossible to distinguish genuine methodological progress from favorable variance.

Does the additional computational cost of these distillation methods pay off in performance?

No; 112 of the 120 tested runs required more training time than GRPO, yet 58 of those runs achieved lower mean performance, suggesting that the overhead of these methods is often not justified by the results.

The Reality of Self-Distillation

We expose inconsistent evaluation in agentic self‑distillation and introduce a unified benchmark against GRPO.

Large language model (LLM) agents act in environments by emitting sequences of actions and observations. On‑policy self‑distillation (OPSD) supplies denser supervision for student trajectories using a teacher that sees privileged training information. Existing evaluations frequently reuse published baseline scores under mismatched experimental conditions, provide few direct comparisons among contemporaneous methods, and overlook training stability and computational cost, which together obscure the true source of reported gains.

A student policy learns from its own behavior while a parallel teacher, which has access to extra training‑time information, generates richer targets for the student to imitate.

GRPO optimizes a policy by comparing each agent’s return to the average return of its peer group, encouraging relative improvement rather than absolute performance.

We evaluate ten representative self‑distillation methods against the GRPO baseline on four benchmarks—ALFWorld, ScienceWorld, Search‑QA, and WebShop—using three model scales ($2\text{B}$, $4\text{B}$, $9\text{B}$). The study comprises $132$ training runs, totaling $12{,}124.4$ H800 GPU‑hours, with each pretrained backbone and each of the eleven trained agents evaluated over eight rollout seeds. Beyond final task scores we also track training progress, stability, and measured computational cost to assess whether reported gains are robust and efficient.

**Figure 1** Motivation and overview of our benchmark for self-distillation in agentic reinforcement learning. Differences in experimental conditions, reused baseline scores, and incomplete evaluation motivate a common comparison of ten representative methods against GRPO across four benchmarks and three model sizes. We examine the breadth of gains, training stability, and computational cost.

Inconsistent evaluation masks the true performance of agentic RL methods, so independent, comprehensive benchmarking is essential.

Interactive Agents and Learning

We survey prior agentic self‑distillation work and its evaluation shortcomings.

Language agents solve tasks through sequential actions and observations (Shridhar et al., 2021; Yao et al., 2022; Xi et al., 2025). Inference‑time methods interleave reasoning and action (Yao et al., 2023) or refine decisions through verbal feedback (Shinn et al., 2023; Madaan et al., 2023), without parameter updates. Training methods learn from filtered tool‑use annotations (Schick et al., 2023) and curated interaction trajectories (Qin et al., 2024; Chen et al., 2023; Zeng et al., 2024).

Reward‑based optimization learns from sampled behavior (Shao et al., 2024), while online curricula support continued policy improvement (Qi et al., 2025). Multi‑turn RL can exhibit unstable updates and degenerate behavior, motivating careful reward design and rollout sampling (Wang et al., 2025). Our benchmark focuses on capabilities acquired through parameter updates, not on improvements obtained by modifying an agent’s inference procedure.

Distillation transfers teacher predictions, generated sequences, or policy behavior to a student (Hinton et al., 2015; Kim and Rush, 2016; Rusu et al., 2016). Training‑only privileged information (Lopez‑Paz et al., 2016) and reasoning rationales (Hsieh et al., 2023) enrich supervision beyond target labels. Privileged supervision therefore seeks to improve the student’s policy without requiring access to the teacher’s additional information at deployment.

Supervising learner‑visited states or student‑generated sequences helps reduce distribution mismatch (Ross et al., 2011; Lin et al., 2020; Agarwal et al., 2024).

Many studies report gains, but few compare methods fairly. Studies reuse baseline scores across papers, yet prior baseline scores are often missing, making it difficult to rely on reported gains. Fair comparisons require costly reproduction.

Key evaluation evidence is often missing (1/10 repeated evaluation, 5/10 training failure analysis, 3/10 measured cost). Table 1 summarizes coverage of key evaluation dimensions in prior work.

MiniLLM and DistiLLM study divergence objectives and sampling strategies (Gu et al., 2024; Ko et al., 2024), while iterative self‑training learns from selected, self‑generated reasoning or interaction traces (Zelikman et al., 2022; Aksitov et al., 2023). Supervision content, sampling distribution, and optimization objective offer complementary perspectives for comparing distillation‑based agent training.

Agentic self‑distillation derives teacher supervision from privileged context available only during training. The evaluated self‑distillation methods use four mechanisms. Skill‑SD, GRPO+OPSD, SDAR, ICSD, and PCSD add auxiliary objectives (Wang et al., 2026a; Lu et al., 2026; Lan et al., 2026; Lv et al., 2026), whereas RLSD, OCSD, and AgentOPSD modulate policy advantages through teacher–student comparisons (Yang et al., 2026a,c; Wang et al., 2026b).

ADRS shapes rewards and advantages (Zhang et al., 2026), while OVCSD adds environment‑verified teacher continuations (Xia et al., 2026). Recent analyses identify supervision inconsistency and privileged‑information bias as potential failure modes (Yang et al., 2026b; Harne et al., 2026). Our benchmark compares these mechanisms’ performance, training stability, and computational cost under a common evaluation protocol. Appendices D.2, D.3, D.5, and D.4 detail the four mechanisms, respectively.

Evaluating Evaluation Methods

We define a unified benchmark to compare self‑distillation methods across tasks, models, and metrics.

**Table 1.** Coverage of key evaluation dimensions in prior work.

Our benchmark evaluates self‑distillation across four diverse text‑agent tasks using a common GRPO reference, ensuring that method differences reflect true algorithmic impact rather than task‑specific quirks.

Define the four tasks (ALFWorld, ScienceWorld, WebShop, Search‑QA) and pair each with the three Qwen model sizes (2B, 4B, 9B), yielding twelve model–task settings.

Select eleven methods, including the outcome‑only GRPO baseline and ten variants with auxiliary objectives or privileged signals.

For every method‑setting pair, train for 100 global steps, validating every five steps, using the hyperparameters reported by the original papers.

Run eight rollout seeds per trained model at temperature 0.4, then compute mean and sample standard deviation of the task‑specific metrics.

Aggregate results across all twelve settings, weighting each benchmark’s metric equally, and compute validation‑curve AUC, peak‑to‑final decline, and low‑performing outcome statistics.

Step 1: Initialize the environment and the agent’s policy network.

Step 2: For each of the 100 global steps, collect trajectories, compute the GRPO objective, and update the policy.

Step 3: Every five steps, evaluate on the validation set, recording success rate.

Step 4: After training, run eight rollout seeds, compute the mean success rate and its standard deviation.

Step 5: Calculate the validation‑curve AUC from the recorded validation scores.

This concrete trial illustrates how the benchmark ties together task definition, method implementation, and evaluation metrics in a reproducible pipeline.

Performance Under Common Evaluation

Comparative performance of training methods against the GRPO baseline.

Only PCSD exceeds Group Relative Policy Optimization (GRPO)'s twelve‑setting mean of 49.68.

PCSD mean 54.26 (+4.58 points) across twelve settings.

Only AgentOPSD exceeds GRPO's twelve‑setting mean of 49.68.

AgentOPSD mean 51.34 (+1.66 points) across twelve settings.

No method dominates across settings; the leading method depends on both the benchmark and model size. For ALFWorld the top performer shifts from Skill‑SD (2B) to RLSD (4B) to ICSD (9B), while GRPO+OPSD leads WebShop at all scales, and Search‑QA is led by ICSD (2B, 4B) and OCSD (4B).

Win counts can conceal severe regressions. ICSD wins seven of twelve settings but its mean of 44.75 remains below GRPO's 49.68, and its 4B WebShop run drops to 0 % success, illustrating that a simple win tally hides large failures.

**Table 2.** Results for pretrained models and eleven training methods on ALFWorld and ScienceWorld. Entries report mean±SD over eight rollout seeds at T = 0.4; trained models are evaluated after 100 global steps. ALFWorld reports success (%), and ScienceWorld reports task score. Teal bold and underlining mark the highest and second-highest means within each model and metric, including ties. Overall columns are shaded.

**Table 3.** Results on Search-QA and WebShop: mean±SD over eight rollout seeds at T = 0.4, after 100 steps for trained models. We report exact-match accuracy (%) for Search-QA and task score (Score) and success rate (%, Acc.) for WebShop. Highlighting follows Table 2.

Most methods do not consistently outperform GRPO across all settings.

Stability and Learning Dynamics

Re‑evaluating training trajectories shows that AUC rankings differ from final scores.

OCSD attains the highest mean validation AUC of 39.88 across the twelve settings, surpassing the GRPO baseline by 2.99 points.

Table 3 reports OCSD’s AUC as 39.88, while GRPO’s is 36.89.

Stability measures how consistently a method improves during training; regression captures drops after early peaks.

How does training stability differ from simply looking at final performance?

Stability considers the entire validation trajectory, revealing early plateaus, mid‑training collapses, or late regressions that a single endpoint cannot capture.

**Figure 2.** Validation trajectories of eleven methods across four benchmarks and three model sizes, covering 132 training runs and 2,772 validation sweeps. Curves show validation scores recorded throughout training. Colored markers distinguish overlapping curves on 2B ScienceWorld.

Computational Efficiency Analysis

Extra compute rarely boosts performance, and step‑100 gains often vanish when matched to equal training time.

Additional compute rarely yields higher performance, and fixed‑step gains often vanish under equal‑time budgets.

112 of 120 method‑GRPO comparisons need more training time; 58 of those underperform, while only PCSD and AgentOPSD improve with 1.17–1.19× the baseline time.

Additional compute often yields no performance improvement over the baseline.

Methodological Objectives

Defines the shared training loop, objectives, and method‑specific variations used in experiments.

Sample a batch of B tasks.

For each task, rollout G trajectories with at most H interaction turns, using temperature $\tau$₍gen₎.

Unfold trajectories into a micro‑batch D, recording prompts, responses, masks, and IDs.

Assign sparse rewards r₍bt₎ and compute per‑row returns R₍b₎ = ∑ₜ r₍bt₎.

Score the old policy $\theta$₍old₎ and the fixed reference ϕ on D to obtain o and f.

Compute GRPO advantages A via group‑normalized returns (Eq. 1) and broadcast to tokens A₍bt₎.

Apply the selected method’s MethodStep to D, A, and auxiliary quantities, updating $\theta$.

Subtracts the mean return of a task group and rescales by its variability, yielding a stable advantage signal for policy updates.

Why normalize advantages by group statistics instead of using raw returns?

Raw returns differ wildly between tasks (e.g., some tasks are easier), so their scale would dominate the policy gradient. Subtracting the group mean centers the signal, and dividing by the group’s spread makes the magnitude comparable, preventing any single task from overwhelming learning.

Clamps the policy‑ratio term and additionally caps the negative‑advantage branch, limiting how far the update can move when the advantage is adverse.

How does dual‑clipping differ from the standard PPO clipping used elsewhere?

Standard PPO only clips the surrogate u to keep the ratio $\rho$ near 1. Dual‑clipping adds a second safeguard: if the advantage is negative, it also limits u by −c₍dual₎ A, so the update cannot overshoot in the direction that would decrease the policy’s expected return.

L₀ aggregates the policy‑gradient term, a reference‑KL penalty, and an entropy regularizer into a single scalar that drives the student update.

Compute row returns R = [1,1,1,1] and group statistics $\mu$ = 1, sg ≈ 0.

Form advantages A₍bt₎ = (R − $\mu$)/(sg+$\epsilon$₍norm₎) ≈ 0 for all tokens.

Calculate $\rho$ = exp(p − o) = 1.2, clip to [0.8, 1.2] → 1.2.

Apply dual‑clipping: since A = 0, ℓ₍PG₎ = 0.

Evaluate $k_{ref}$ ≈ 0.1 and H($\pi$₍$\theta$₎) ≈ 0.8.

Combine terms per Eq. (3) to obtain L₀ ≈ 2 × 10⁻⁴.

This toy run shows that when returns are identical across a group, the normalized advantage vanishes, leaving the KL and entropy terms to drive learning; the dual‑clipping mechanism does not interfere because the advantage is zero.

**Table S3.** Benchmark-specific settings and method-specific overrides across four benchmarks, shared across the three model sizes. Configuration sources and benchmark extensions are described in Appendix C.1. Unlisted optimization settings retain the defaults in Table S2. SD denotes standard deviation.

**Table S4.** Algorithm-specific hyperparameters shared across all four benchmarks and three model sizes. Sources and implementation choices are described in Appendix C.1; benchmark-dependent values appear in Table S3. Symbols follow Appendix D; $s$ denotes the global training step.

Cross-Setting Performance

Cross‑setting experiments reveal modest AUC improvements while keeping core training dynamics unchanged.

The cross‑setting experiments show that adding calibrated token weighting and persistent teacher support yields a small but consistent increase in training AUC, while the underlying PPO dynamics remain unchanged.

Algorithm S6 ICSD computes calibrated token weights $g$ and $h$ via the CDFs $ \text{CalCDF}_{\xi_{\text{trust}}}$ and $\text{CalCDF}_{\xi_{\text{inf}}}$, multiplies them to obtain $c$, and normalizes over the eligible set $C_b$; the auxiliary loss $L_{\text{aux}}$ is scaled by $\lambda_{\text{ic}}$ and the weights $c$ are treated as constants during back‑propagation.

Algorithm S7 PCSD smooths the detached teacher gap $\Delta = q - p$ over short and long windows ($N_{\text{min}}$, $N_{\text{max}}$) using a decaying mean, interpolates between them based on the variance $v$, and applies a sigmoid‑scaled factor $w$ to produce the final detached weight $w$ that modulates the auxiliary loss $\lambda_{\text{pc}}\langle w (q-p) \rangle_M$.

Algorithm S8 RLSD reweights advantages by first computing a step‑wise schedule $\lambda_s$, then clipping the product $\operatorname{esign}(A)(q-o)$ around $1$ with tolerance $\delta_r$, and finally mixing the original advantage with the reweighted version $\tilde{A}$ before the update.

Algorithm S9 OCSD selects high‑NLL future evidence rows via $F_b$, extracts the top‑$k$ rows with `Top(E; n)`, builds full and ablated prompts, scores them with $q_{\text{full}}$ and $q_{\text{abl}}$, and uses the bounded contrast $z$ scaled by $\lambda_{\text{oc}}$ to modulate the advantage.

Training Dynamics

We detail the stepwise training pipeline that selects evidence, scores it, and updates advantages.

The training dynamics consist of a deterministic sequence of operations that first select evidence pairs, then compute full and ablated scores, and finally use these signals to adjust the policy‑gradient advantage.

For each trajectory $i$, construct the evidence set $E_i = \{b \in B_i : F_b > 0,\; |A_b| > \epsilon_A\}$, discarding empty sets.

Compute the subset size $k_i = \min\bigl(|E_i|,\; \max\bigl(1,\lceil \alpha_{\text{sel}}|E_i|\rceil\bigr)\bigr)$ and select the top‑$k_i$ elements $J_i$ according to the score $n$.

For every selected element $b \in J_i$, generate an evidence pair $(D^+_b, D^-_b)$ via the $\text{EvidencePair}$ routine.

Evaluate the full and ablated scores: $q^{\text{full}}_b = \text{Score}(\theta_{\text{old}}, D^+_b)$ and $q^{\text{abl}}_b = \text{Score}(\theta_{\text{old}}, D^-_b)$.

Compute the signed difference and pass it through a hyperbolic tangent: $z_{b,t} = \tanh\!\bigl(q^{\text{full}}_{b,t} - q^{\text{abl}}_{b,t}\bigr)$.

Standardize the incremental advantage: $v_b = V_b - V_{\text{prev}}$, then $z_b = \dfrac{v_b - \bar v_i}{\text{SD}_{\text{pop}}(B_i) + \varepsilon_{\text{norm}}}$ where $\bar v_i$ and $\text{SD}_{\text{pop}}$ are the batch mean and population standard deviation.

Form a bounded multiplier $m_b = 1 + \delta_{\text{ag}}\,\text{clip}\bigl(\text{sign}(A_b)z_b,\,-1,\,1\bigr)$ and mix it with the GRPO advantage: $bA_{b,t} \leftarrow A_b\bigl[(1-\lambda_{\text{ag}}) + \lambda_{\text{ag}}m_b\bigr]M_{b,t}$.

OVCSD identifies eligible groups (no successful trajectory, low return‑std, at least one prefix) and builds candidate prefixes $C$ via the $\text{Prefixes}$ routine.

AgentOPSD iterates over task groups, updates a signed turn credit $c$, revises the belief $V_b$, and applies the same multiplier logic as in step 7.

Suffix distillation computes a KL divergence $\text{KL}\bigl(Q_{b,t}\,\|\,P_{I}\bigr)$ over valid suffix tokens, weighted by $\lambda_{\text{suf}}$.

ADRS converts teacher scores into token rewards $h_{b,t} = \eta\,\sigma\bigl(\beta_{\text{TVA}}e_{T,b}\bigr)u_{b,t}$ and a clipped advantage correction $w_{b,t}$, then adds $w_{b,t}$ to the advantage.

Finally, the total advantage becomes $bA_{b,t} \leftarrow bA_{b,t} + w_{b,t}M_{b,t}$.

Compute $\lceil \alpha_{\text{sel}}|E_i|\rceil = \lceil 0.3 \times 10\rceil = 3$.

Apply the inner max: $\max(1, 3) = 3$.

Take the outer min with $|E_i|$: $k_i = \min(10, 3) = 3$.

Select the top‑3 elements $J_i$ according to their score $n$.

This concrete calculation shows how the hyperparameter $\alpha_{\text{sel}}$ caps the proportion of evidence that can influence the update, preventing noisy large‑scale swings.

**Algorithm S9** OCSD: full/ablated future-evidence replay **Require:** $D, A, o, F_b; \alpha_{\text{sel}}, L_f, \lambda_{\text{oc}}$ 1: $z \leftarrow 0$ 2: **for** trajectory $i$ **do** 3: $\quad E_i \leftarrow \{b \in B_i : F_b > 0, |A_b| > \epsilon_A\}$ 4: $\quad$ **if** $E_i = \varnothing$ **then** 5: $\quad \quad$ **continue** 6: $\quad$ **end if** 7: $\quad n_b \leftarrow -\frac{\sum_t M_{bt}o_{bt}}{\max(1, \sum_t M_{bt})}$ for $b \in E_i$ 8: $\quad k_i \leftarrow \min(|E_i|, \max(1, \lceil \alpha_{\text{sel}}|E_i| \rceil))$ 9: $\quad J_i \leftarrow \text{Top}_{k_i}(E_i; n)$ 10: $\quad$ **for** $b \in J_i$ **do** 11: $\quad \quad (D_b^+, D_b^-) \leftarrow \text{EvidencePair}(D_b, L_f)$ 12: $\quad \quad q_b^{\text{full}} \leftarrow \text{Score}(\theta_{\text{old}}, D_b^+)$ 13: $\quad \quad q_b^{\text{abl}} \leftarrow \text{Score}(\theta_{\text{old}}, D_b^-)$ 14: $\quad \quad z_{bt} \leftarrow \tanh[(q_{bt}^{\text{full}} - q_{bt}^{\text{abl}})/2]$ 15: $\quad$ **end for** 16: **end for** 17: $\widehat{A} \leftarrow \text{sg}\{A[1 + \lambda_{\text{oc}} \text{sign}(A)z]\}$ 18: $\text{Update}_D(\widehat{A}, 0)$

Aggregate Training Progress

Aggregated training progress reveals method rankings by AUC and final evaluation across all settings.

OCSD achieves the highest aggregate AUC across the twelve settings, while PCSD attains the best final evaluation score.

Figure S1 shows OCSD leading by AUC and PCSD leading at the endpoint; Tables S6 and S7 list the rankings.

Table S6 aggregates AUC and final scores for the twelve equally weighted model–benchmark settings, using the same trajectory summaries applied to all 132 runs across the four benchmarks. Metrics include success % for ALFWorld and WebShop, exact‑match % for Search‑QA, and task scores (including negatives) for ScienceWorld, each weighted equally in the aggregate AUC, stage‑wise, and final‑score comparisons. These aggregate values summarize reported metrics rather than a pooled task‑success probability, and Tables S6 and S7 list the resulting rankings, low‑performance counts, and stage‑wise means for all eleven methods.

Instability and Regression

Instability can persist despite high final scores, so intermediate performance must be examined.

Across the 4B ALFWorld runs, methods that appear stable by final score can still suffer severe intermediate collapses. AgentOPSD, for example, recovers after six consecutive validations at or below 4.3 %, while GRPO+OPSD loses its early gains and finishes near zero. OCSD reaches 85.0 % before declining to 51.4 % with a final‑three‑validation mean of 50.5 %.

Relying solely on a near‑zero final success criterion would identify GRPO+OPSD but miss both AgentOPSD’s temporary collapse and OCSD’s sustained regression. Therefore, stability must be judged by the depth and persistence of intermediate declines, even when the final model remains capable.

Training‑time validation uses greedy decoding ($T = 0$), whereas final evaluation samples at $T = 0.4$ over eight rollout seeds; both report exact‑match accuracy on the same 3,125 questions. For 2B OVCSD, greedy accuracy falls from 38.02 % (step 70) to 1.34 % (step 100), a 36.67‑point drop, yet the sampled endpoint still scores 33.21 % ± 0.33 %, comparable to GRPO’s 32.38 % ± 0.28 %.

ICSD exhibits a milder greedy regression: its 2B greedy accuracy peaks at 39.52 % (step 65), drops to 21.73 % (step 90), and partially recovers to 31.49 % (step 100), yielding a final‑three‑validation mean of 26.31 %.

**Figure S1.** (a) Rankings across all twelve settings by mean training-curve AUC and eight-seed final performance from Tables 2 and 3. AUC integrates each benchmark's validation metric and normalizes by the training horizon. (b) Equally weighted mean validation trajectories, highlighting GRPO and the AUC and final-performance leaders, OCSD and PCSD; the other eight methods are shown in gray.

**Table S5** Equal-weight means across all twelve model–benchmark settings, using ALFWorld/WebShop success (%), Search-QA exact-match accuracy (%), and ScienceWorld task score. $\Delta$ and Worst are the mean and worst paired differences from GRPO; W/L/T counts wins/losses/ties of reported means.

Early Stagnation

Early runs often plateau then drop, revealing hidden performance collapse.

Table S8 reports training profiles for all 33 ALFWorld runs, measuring success (%) at several milestones: the initial peak, the step of that peak, the final validation at step 100, and the average over the last three steps (Tail‑3). It also records the drop from peak to final, the normalized area‑under‑the‑curve (AUC), and both pure training time and total wall‑clock time.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers