PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Chunji Lv, Yangguang Wei, Junlin Liu, Yang Gao, Ming Liu, Xinming Wang, Jinyang Wu, Guoren Wang, Changsheng Li

PCSD improves agentic RL by weighting teacher-distillation signals based on their local persistence rather than isolated token discrepancies.

How can we improve LLM agent reinforcement learning by selectively distilling teacher knowledge across multi-turn trajectories?

Large language model agents often struggle with sparse rewards, where a single outcome signal must guide hundreds of tokens. While on-policy self-distillation provides dense supervision, existing methods either rely on noisy token-level discrepancies or coarse step-level averages that ignore positional variation. Persistent Consistency Self-Distillation (PCSD) solves this by estimating teacher credibility through the local persistence of teacher-favoring signals. It uses adaptive windows and trend-aware modulation to filter out transient noise while retaining fine-grained positional guidance. On the ALFWorld benchmark, PCSD outperforms outcome-only GRPO by up to 15.6 points and existing distillation baselines by up to 6.2 points, demonstrating robust generalization to unseen scenarios.

Paper Primer

PCSD treats teacher guidance as a local signal-processing problem: it assumes that reliable teacher advice persists across a neighborhood of tokens, whereas sampling noise appears as isolated spikes. The method computes a continuous distillation weight for each token by aggregating teacher-student log-probability gaps using an adaptive, exponentially decayed window.

The core mechanism hinges on three steps: adaptive windowing that expands in noisy regions and shrinks in stable ones, exponential decay to prioritize local context, and trend-aware modulation that suppresses weights where teacher support is actively declining. This produces a smooth, gated weight that allows the student to selectively learn from the teacher's most consistent advice.

PCSD significantly improves success rates in long-horizon agentic tasks compared to standard GRPO and existing distillation methods.

On ALFWorld, PCSD achieved an overall success rate of 90.6% (Qwen2.5-3B), outperforming GRPO by 15.6 percentage points and the SDAR baseline by 6.2 points.

The method generalizes effectively to unseen environment configurations without requiring privileged skills during evaluation.

PCSD achieved an 86.7% success rate on the ALFWorld unseen-split, compared to 70.9% for GRPO and 72.7% for SDAR. 14.0 percentage point gain over GRPO on unseen splits.

Why is this approach superior to simply averaging teacher-student discrepancies over an entire step?

Step-level aggregation is too coarse and obscures substantial variation in teacher credibility across different token positions within a single step. PCSD’s adaptive windowing preserves fine-grained positional resolution while maintaining robustness to local noise.

Does PCSD require the teacher to be available during inference?

No. The privileged teacher is used only during training to provide dense supervision. At inference time, the student policy operates independently, which is why PCSD remains competitive or superior to methods that rely on external skill retrieval.

PCSD demonstrates that token-level distillation is most effective when it accounts for the local temporal structure of teacher support. Researchers working on agentic RL should prioritize persistent-consistency metrics over isolated discrepancy measures to improve credit assignment in long-horizon tasks.

Introduction and Motivation

Sparse rewards cripple RL for LLM agents; PCSD injects persistent token‑level guidance.

Reinforcement learning for large‑language‑model agents suffers from extreme reward sparsity: a multi‑turn episode may contain hundreds of generated tokens yet yield only a single scalar outcome at the end. This makes credit assignment noisy and slows learning dramatically.

Standard RL cannot reliably learn from a lone end‑of‑episode signal, so we need dense guidance that tells the agent which token choices are trustworthy as it acts.

The sparse reward problem in multi‑turn agent RL demands dense, trustworthy token‑level guidance.

Related Work

We survey prior RL approaches for LLM agents, highlighting sparse‑reward challenges and the evolution of token‑level guidance techniques.

Reinforcement learning now underlies most post‑training LLM agents, but long‑horizon tasks still suffer from sparse, delayed rewards that make credit assignment difficult.

RL fine‑tunes large language models by treating generated tokens as actions and optimizing a reward signal derived from downstream tasks.

Recent work expands RL‑driven LLMs to concrete interaction domains such as code generation, tool use, GUI manipulation, and web navigation.

GRPO replaces explicit value‑function critics with a group‑relative reward that compares an agent’s return to that of a peer group, improving scalability.

OPD augments sparse trajectory rewards with dense teacher signals sampled from the current policy; OPSD uses a frozen, privileged‑context copy of the student as its own teacher to avoid training a separate model.

Token‑level methods compute a divergence or probability‑ratio per token to weight teacher feedback; step‑level methods aggregate divergence over an entire reasoning or action step and assign a uniform weight to all tokens within that step.

PCSD evaluates the stability of a privileged teacher’s preference for a particular action across a sliding temporal window, using that consistency as a weight for token‑level distillation.

Agent RL Preliminaries

We formalize multi‑turn RL interactions and introduce on‑policy self‑distillation as dense token‑level guidance.

On‑policy self‑distillation (OPSD) augments sparse environmental rewards with dense teacher guidance sampled from the current student policy, using a privileged‑context teacher that sees more information than the student.

The agent repeatedly observes a state, emits a response, and receives the next state, forming a chain of interactions across turns.

How does multi‑turn RL differ from standard single‑step RL?

In single‑step RL the agent takes one action and receives a reward, whereas in multi‑turn RL the agent repeatedly observes states, emits token‑level responses, and receives new states, so credit must be distributed across many intermediate decisions rather than a single action.

Turn 1: sample $y_{1,1}$ and $y_{1,2}$ from $\pi_\theta(\cdot\mid s_1)$ → response “yes”.

Environment reacts, producing $s_2$ = “feedback”.

Turn 2: sample $y_{2,1}$ from $\pi_\theta(\cdot\mid s_2)$ → response “thanks”.

Trajectory $\tau = (s_1, y_1, s_2, y_2)$ = (“question”, “yes”, “feedback”, “thanks”).

This tiny example shows how each token belongs to a specific turn, so any token‑level weight must respect the turn boundaries.

The PCSD Method

PCSD computes token‑level distillation weights by measuring how consistently the teacher favors each token across nearby positions.

On‑policy RL for LLM agents receives only a sparse trajectory‑level reward, so the student gets little signal about which individual tokens were helpful. A privileged teacher can supply token‑level advice, but its usefulness varies across positions. PCSD addresses this by weighting teacher supervision according to how consistently the teacher favors each token over neighboring steps.

PCSD turns the raw teacher‑student log‑probability gap into a smooth, per‑token weight that reflects how persistently the teacher supports each sampled token across a local temporal window.

How is PCSD different from ordinary token‑level self‑distillation that simply uses a fixed teacher weight?

Ordinary self‑distillation applies the same scalar to every token, assuming the teacher is equally reliable everywhere. PCSD instead measures the teacher’s support over a sliding window, adapts the smoothing scale to local variance, and suppresses weights when the support is dropping, so each token receives a weight that reflects its own consistency.

Compute the teacher‑student log‑probability gap $\delta_{k,i}$ for each token in the sampled trajectory.

Form an exponentially weighted estimate $\bar{\delta}^{(N)}_{k,i}$ over a forward window of size $N$ (short and long windows are computed separately).

Calculate the local mean $\mu_{k,i}^{(N_{\text{max}})}$ and variance $\sigma_{k,i}^{2}$ of gaps within the maximum window $N_{\text{max}}$.

Map the variance to an interpolation coefficient $r_{k,i}$ and blend the short‑ and long‑window estimates into $\bar{\delta}^{\text{adaptive}}_{k,i}$.

Estimate the OLS slope over the maximum window, derive the trend factor $\eta_{k,i}$, and combine it with the sigmoid‑gated adaptive estimate to obtain $w_{k,i}$.

Scale the token‑level gap by $w_{k,i}$, average over all valid tokens to get $\mathcal{L}_{\text{PCSD}}$, and add $\lambda_{\text{PCSD}}\mathcal{L}_{\text{PCSD}}$ to the GRPO loss.

Short‑window estimate at position 2: $\bar{\delta}^{(2)}_{2} = \frac{0.5^{0}\cdot0.2 + 0.5^{1}\cdot0.6}{0.5^{0}+0.5^{1}} = 0.33$.

Long‑window estimate at position 2: $\bar{\delta}^{(4)}_{2} = \frac{0.5^{0}\cdot0.2 + 0.5^{1}\cdot0.6 + 0.5^{2}\cdot0.1}{0.5^{0}+0.5^{1}+0.5^{2}} \approx 0.28$.

Local variance over the full window: $\sigma^{2}=0.082$, so $r_{2}= \operatorname{clip}\!\big((0.082-0.01)/(0.05-0.01),0,1\big)=1$ (maxed out).

Adaptive estimate: $\bar{\delta}^{\text{adaptive}}_{2}= (1-1)\cdot0.33 + 1\cdot0.28 = 0.28$.

OLS slope over the window is negative (≈ −0.25); with $\gamma=0.6$ and $\delta_{\text{scale}}=0.425$, the trend factor $\eta_{2}= \operatorname{clip}(1-0.6\cdot\text{ReLU}(0.25/0.425),0,1)=0.65$.

Final weight: $w_{2}= \sigma(4\cdot0.28)\times0.65 \approx 0.71\times0.65 \approx 0.46$.

The weight drops because the gap at position 2 is followed by a sharp decline, illustrating how trend modulation prevents a single large gap from dominating the distillation signal.

**Figure 2.** Framework of PCSD. The student collects on-policy trajectories via multi-turn interaction, while a frozen, skill-augmented teacher scores the student-generated tokens. PCSD aggregates teacher–student log-probability gaps across multiple time scales with exponential decay, adaptively weights them by local variability, and applies trend modulation and continuous gating to derive token-level distillation weights, which are jointly optimized with the trajectory-reward-based GRPO objective.

Experimental Results

PCSD sets new performance records on both WebShop and ALFWorld benchmarks.

Recall that PCSD uses a privileged teacher to give token‑level guidance weighted by the teacher’s consistency across time, addressing the sparse‑reward problem of standard RL for LLM agents.

PCSD achieves 90.6% overall success on ALFWorld, surpassing the next‑best method by 3.9%.

Table 1 shows PCSD at 90.6% versus GRPO at 86.7% and SDAR at 72.7%.

Across both benchmarks, PCSD consistently leads: 85.0% on WebShop and 90.6% overall on ALFWorld, while the closest competitor lags by several points.

**Figure 1.** Overall performance comparison. Main results on WebShop and ALFWorld with Qwen2.5-3B-Instruct. Left: WebShop Score and Acc; Middle: ALFWorld sub-task radar; Right: ALFWorld Overall success rate.

**Table 1.** Performance on ALFWorld and WebShop. For ALFWorld, we report category-wise and instance-level Overall success rates (%). For WebShop, we report the normalized Score and task success rate(Acc, %) on 128 validation tasks. * denotes evaluation with skills. Best and second-best are highlighted.

**Figure 3.** Training dynamics. Average teacher–student gap (left) and gate activation ratio (right) for Qwen2.5-3B-Instruct on ALFWorld. Translucent and solid curves show raw and smoothed values, respectively.

**Figure 4.** Generalization to unseen ALFWorld environments. Category-wise and Overall success rates (%) of SDAR, GRPO and PCSD on the ALFWorld unseen split.

**Table 3.** Sensitivity to the distillation coefficient. We report category-wise and overall success rates (%) on ALFWorld using Qwen2.5-3B-Instruct.

**Table 2.** Component ablation on ALFWorld. Success rates (%) using Qwen2.5-3B-Instruct. $N = 1$ denotes pointwise weighting, $N = 4$ a fixed local window, and Overall the success rate across all evaluation instances.

Robustness and Analysis

Additional implementation details and ablation insights for PCSD.

Recall that PCSD mitigates sparse‑reward RL by letting a frozen privileged teacher provide token‑level guidance weighted by its persistent consistency across time.

All local‑aggregation and gating hyperparameters are fixed throughout training, which cleanly isolates the impact of the Persistent‑Consistency weighting from any co‑adaptation between the policy and the teacher.

The downside of this rigidity is that the agent cannot adapt its aggregation window or gating thresholds to shifting trajectory statistics or to fluctuations in teacher reliability.

Future work may replace the static settings with context‑dependent parameters learned from trajectory statistics, teacher uncertainty estimates, and environmental feedback.

An even more ambitious direction is self‑evolving distillation, where the student, teacher, credibility estimator, and skill repository co‑evolve online.

The privileged teacher $\pi_{T}$ is instantiated as a frozen copy of the initial student checkpoint; no gradients flow through it and it is excluded from the optimizer.

For each sampled trajectory the teacher evaluates the student’s tokens via teacher‑forcing and returns token‑level conditional probabilities used by the PCSD loss, while all gradients affect only the student policy.

The skill repository supplies general interaction rules and task‑type‑specific procedural knowledge (e.g., object manipulation or navigation), but never instance‑level solutions, expert trajectories, target‑object locations, future observations, or hidden environment states.

The repository is built without any access to validation trajectories; ALFWorld’s splits and WebShop’s fixed task set are used only for evaluation.

**Table 6.** A complete successful rollout on an unseen ALFWorld environment. The agent recovers from an unsuccessful initial search in the fridge, locates the potato on a countertop, preserves the required acquire–heat–place ordering, and completes the task in eight valid actions.

**Figure 5.** Relationship between teacher-quality changes and weight changes under skill-removal and shuffled-skill perturbations. Dashed and solid lines show method-specific linear fits, while the annotations report Spearman correlations. PCSD exhibits weak nonnegative associations, whereas SDAR shows inverse associations under both perturbations.

**Figure 6.** Robustness of token-weight rankings under isolated-gap perturbations. Spearman correlation is computed between the original and perturbed token-weight rankings. PCSD remains comparable to SDAR under light perturbation and yields higher mean rank preservation under the 5% and 10% perturbations. Error bars indicate variability across repeated perturbation draws.

Training Setup

Defines the privileged teacher input and the GRPO update rule that normalizes advantages across trajectory groups.

Policy‑gradient updates for LLM agents are notoriously noisy because each rollout yields a single scalar reward, yet the loss is applied token‑by‑token. This variance makes the policy ratio $\rho$ unstable and forces aggressive clipping, which in turn slows learning.

GRPO rescales every token’s advantage by how its trajectory’s reward compares to the mean and spread of a batch of $G$ sampled trajectories, so the update reflects “is this rollout better than its peers?” rather than an absolute reward.

How does GRPO differ from the standard PPO clipping that operates on whole‑trajectory returns?

PPO clips the ratio of new to old policy probabilities at the trajectory level, treating each rollout as a single data point. GRPO first converts each trajectory’s return into a z‑score $A_g$, then applies the same clipping to every token’s importance ratio $\rho_{g,i}$, so the update respects both per‑token likelihood changes and the relative quality of the trajectory.

Compute the mean $\mu_R = (10+4)/2 = 7$ and standard deviation $\sigma_R = \sqrt{[(10-7)^2 + (4-7)^2]/2} = \sqrt{(9+9)/2}=3$.

Group‑relative advantages: $A_1 = (10-7)/(3+ \epsilon_{\text{num}}) \approx 1.0$, $A_2 = (4-7)/(3+ \epsilon_{\text{num}}) \approx -1.0$.

Assume the current policy probabilities for a particular token are $\pi_{\theta}(y_{1,i})=0.6$, $\pi_{\theta}^{\text{old}}(y_{1,i})=0.5$ (so $\rho_{1,i}=e^{\log0.6-\log0.5}\approx1.2$) and similarly $\rho_{2,i}=0.8$ for trajectory 2.

Clip with $\epsilon_{\text{clip}}=0.2$: $\bar{\rho}_{1,i}=1.2$ (already within $[0.8,1.2]$), $\bar{\rho}_{2,i}=0.8$.

Token‑level contribution for trajectory 1: $\min(1.2\times1.0,\,1.2\times1.0)=1.2$; for trajectory 2: $\min(0.8\times(-1.0),\,0.8\times(-1.0))=-0.8$.

Average over the six tokens (mask $m_{g,i}=1$) gives the advantage part $(-0.8+1.2)/2 = 0.2$; adding a tiny KL term yields the final $L_{\text{GRPO}}$.

Normalizing advantages across trajectories turns a raw reward gap of 6 into a symmetric +1/‑1 signal, dramatically lowering gradient variance while still rewarding better rollouts.

Training Procedure

Describes the step‑by‑step training loop and the PCSD weighting mechanism that stabilizes token‑level distillation.

Standard RL for LLM agents suffers from extremely sparse episode rewards, making token‑level credit assignment noisy. The PCSD method injects a privileged teacher that supplies fine‑grained guidance, but that guidance itself fluctuates across time, so we need a stable weighting scheme.

The student policy is rolled out to collect trajectories, the frozen teacher evaluates the same tokens, and a variance‑aware weight turns the teacher‑student log‑probability gap into a smooth per‑token loss.

How does this differ from ordinary teacher‑forcing in supervised fine‑tuning?

In teacher‑forcing the teacher’s output is taken as the ground‑truth label and the student is trained directly on it. Here the teacher only provides a *gap* signal that is weighted by a variance‑aware estimator; the student still optimizes its own policy via RL (GRPO) and the teacher never supplies a hard label.

Copy the current student parameters to the rollout policy.

Generate two trajectories: $[a_1,a_2,a_3]$ and $[b_1,b_2,b_3]$.

Normalize the raw rewards of each trajectory within the task group to obtain GRPO advantages $A_1$ and $A_2$.

Feed the three student tokens to the frozen teacher; obtain log‑probability gaps $\delta_{1},\delta_{2},\delta_{3}$.

Compute PCSD weights $w_1,w_2,w_3$ (see the aggregation formula below) and form the weighted NLL loss.

Back‑propagate only through the student’s log‑probs; update student parameters.

The loop keeps the teacher static, so the only moving part is the student; this isolates the variance‑aware weighting as the sole source of token‑level guidance.

Instead of using a single fixed window, PCSD blends a short and a long exponential window, letting low‑variance regions keep fine detail while high‑variance regions rely on a smoother estimate.

Why not simply fix a single window size for all tokens?

A fixed window forces the same bias‑variance trade‑off everywhere: short windows preserve detail but amplify noise, while long windows smooth out noise but erase rapid teacher signal changes. The gated interpolation lets the algorithm adapt locally based on the observed variance.

Compute short‑window weights: $a^{(3)}_{t,0}=0.45$, $a^{(3)}_{t,1}=0.36$, $a^{(3)}_{t,2}=0.19$.

Short‑window estimate $\bar{\delta}^{(3)}_t = 0.45\cdot0.2 + 0.36\cdot0.5 + 0.19\cdot0.1 = 0.33$.

Compute long‑window weights (normalized): $a^{(5)}_{t,0}=0.38$, $a^{(5)}_{t,1}=0.30$, $a^{(5)}_{t,2}=0.24$, $a^{(5)}_{t,3}=0.19$, $a^{(5)}_{t,4}=0.15$.

Long‑window estimate $\bar{\delta}^{(5)}_t = 0.38\cdot0.2 + 0.30\cdot0.5 + 0.24\cdot0.1 + 0.19\cdot0.4 + 0.15\cdot0.3 = 0.34$.

Assume variance estimate $V_t=0.12$, thresholds $\tau_{\text{low}}=0.05$, $\tau_{\text{high}}=0.15$, giving $r_t=\operatorname{clip}((0.12-0.05)/(0.15-0.05),0,1)=0.7$.

Interpolated weight $\delta_t = (1-0.7)\cdot0.33 + 0.7\cdot0.34 = 0.339$.

The gate leans heavily toward the long window because the variance is relatively high, yet the final weight stays close to the short‑window estimate, preserving detail while gaining stability.

Further analysis shows that exponential aggregation acts as a denoising filter: it reduces variance proportionally to an effective sample size and attenuates isolated spikes exponentially with distance.

Weight Robustness Analysis

We test how PCSD’s token‑weighting holds up to noise and teacher‑quality changes.

Recall that PCSD leverages a privileged teacher to supply token‑level guidance weighted by the teacher’s consistency across multiple temporal scales.

We test whether isolated spikes in the teacher‑student log‑prob gap disrupt the global ranking of distillation weights. Random gaps are perturbed by adding a magnitude‑3.0 spike to 1 %, 5 % or 10 % of entries, and we recompute the Spearman correlation between original and perturbed rankings for PCSD and the baseline SDAR.

Next we ask whether changes in teacher confidence translate into proportional changes in token‑level weights. For each state‑action pair we compute $ΔQ$ and $ΔW$ under two manipulations: removing the retrieved skill context and shuffling the skill. A consistent weighting rule should yield a non‑negative correlation between $ΔQ$ and $ΔW$.

Prompt Construction

Describes how prompts isolate information and the ALFWorld prompt template.

The prompt construction isolates information by limiting the student’s input to the original task instruction, observable environment feedback, and its causal interaction history. The teacher sees the same observable prefix plus any retrieved privileged skills and scores the exact tokens the student generates, without sampling alternative actions.

For ALFWorld, each interaction step provides the agent with the task objective, a bounded history of recent observations and actions, the current textual observation, and the set of admissible actions. The prompt format mandates that the agent first emits a step‑by‑step reasoning block before proposing the next action.

Algorithm Details

Describes the PCSD training loop and analyzes its weighting mechanisms.

We begin by sampling a batch of tasks $\{x_b\}_B$ and copying the current policy parameters to $\theta_{\text{old}}$. For each update step $u=1,\dots,K$ we retrieve task‑level skills $S_b$ and draw $G$ trajectories $\tau_{b,g}$ from the frozen student policy $\pi_{\theta_{\text{old}}}$ without any skill conditioning.

From each trajectory we compute returns $R_{b,g}$ and group‑normalized advantages $A_{b,g}$. Then, for every token position we evaluate the student log‑probability $\ell_S$ and the teacher log‑probability $\ell_T$, forming a detached gap $\delta_{b,g,t,i}= \text{sg}(\ell_T^{b,g,t,i}) - \ell_S^{b,g,t,i}$.

Next we estimate a local variance $V_{b,g,t,i}$ over a window of size $N_{\max}$, clip it to obtain a reliability ratio $r_{b,g,t,i}$, and combine an exponentially weighted gap $\bar{\delta}_{\text{adaptive}}^{b,g,t,i}$ with a window‑averaged gap $\bar{\delta}^{b,g,t,i}_{N_{\max}}$. An OLS slope $s_{b,g,t,i}$ over the same window yields a one‑sided trend factor $\eta_{b,g,t,i}$, which together define the final token weight $w_{b,g,t,i}$.

The exponential aggregation captures the average level of teacher support while suppressing isolated spikes, and the trend factor ensures that a consistently decreasing support cannot inflate the weight. Positive correlation between teacher and student gaps reduces the attainable variance reduction, yet the aggregation still eliminates the uncorrelated component of local noise.

Hyperparameters

All hyperparameter choices used across methods and the specific PCSD settings.

This section enumerates the hyperparameters that define each method’s optimization behavior and details the PCSD‑specific settings that shape token‑level distillation.

Table 5 lists the PCSD‑specific hyperparameters that control the token‑level distillation weight and its temporal modulation.

The asymmetric design of the trend estimator only attenuates the distillation weight when the local slope is negative; it never creates additional credit, preserving the original teacher‑student gap magnitude.

All trainable methods use AdamW with a learning rate of $1\times10^{-6}$, a KL‑penalty of $0.01$, gradient clipping at norm $1.0$, and run for $150$ update steps. Batches contain $16$ tasks with eight rollouts each (total $128$ trajectories), and prompt lengths are capped at $2{,}048$ tokens for ALFWorld and $4{,}096$ for WebShop.

Table 6 shows a full successful rollout on an unseen ALFWorld task, illustrating the required acquire–heat–place ordering and confirming that the agent can recover from an initial mis‑search.

Benchmark Protocols

Defines training and evaluation setups for ALFWorld and WebShop benchmarks.

ALFWorld follows the GiGPO setup: training uses the official train split, while in‑distribution validation runs on *`valid_seen`* and out‑of‑distribution validation on *`valid_unseen`* (unseen environments and layouts). Each update samples 16 tasks with eight rollouts per task (128 trajectories), and each periodic validation evaluates 128 episodes from *`valid_seen`*. Episodes are capped at 50 environment actions; prompts may be up to 2,048 tokens and responses up to 512 tokens. A binary *won* signal determines success, yielding a reward of 10 for successful trajectories and 0 otherwise (reward scaling is used only during training).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers