Agentopsd: Recursive Self-Distillation for Agentic Reinforcement Learning

Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang

AgentOPSD assigns turn-level credit in RL by recursively updating a Bayesian belief state based on self-distillation gaps.

How can we assign credit to individual turns in long-horizon agentic RL when only the final outcome is verifiable?

Standard reinforcement learning methods for agents broadcast a single trajectory-level reward across all actions, failing to distinguish pivotal decisions from routine ones in long-horizon tasks. AgentOPSD transforms local teacher-student log-probability gaps into turn-level evidence, recursively updating a Bayesian belief state to measure how each action revises the probability of success. This approach consistently outperforms standard group-relative policy optimization, achieving an 89.1% success rate on ALFWorld with Qwen2.5-7B.

Paper Primer

The core challenge is that sparse terminal rewards provide no signal for intermediate decisions, and uniform credit assignment becomes increasingly inaccurate as interaction horizons grow. AgentOPSD addresses this by treating the self-distillation gap as evidence that updates a running belief state: it aggregates token-level gaps into turn-level evidence, then recursively propagates this through a success-belief accumulator to identify which turns meaningfully shift the trajectory toward an outcome.

AgentOPSD provides more precise credit assignment than uniform trajectory-level methods, particularly in long-horizon interactions.

In ALFWorld, uniform-credit methods like RLSD and GRPO degrade by -3.59 and -2.91 success points per additional turn, respectively, while AgentOPSD degrades by only -0.54 points per turn.

Recursive belief revision and turn-level aggregation are the primary drivers of performance gains.

Ablations show that replacing recursive belief revision with raw local gaps drops success rates from 89.1% to 82.8%, and replacing turn-level aggregation with per-token accumulation drops it to 78.9%.

Why is a local self-distillation gap insufficient for credit assignment on its own?

A local gap scores each turn in isolation, failing to account for whether the evidence is redundant given the preceding history. AgentOPSD’s recursive update ensures that credit is assigned based on how much a turn revises the current belief state, rather than its absolute local signal.

Does this method require additional training overhead like a learned critic or extra rollouts?

No. AgentOPSD is critic-free and requires no additional rollouts; it relies on a single forward pass of the teacher branch to estimate the evidence proxy during the standard policy optimization loop.

The Credit Assignment Problem in Agentic RL

We expose why trajectory‑level rewards miss pivotal decisions and motivate turn‑level credit.

Agentic reinforcement learning relies on rewards that are only verifiable after an entire trajectory finishes. Because the learning signal is broadcast uniformly across all steps, the method cannot isolate the few pivotal decisions that actually drive success. This gap motivates a turn‑level credit‑assignment approach.

Verifiable rewards are outcome signals that can be checked only after the full episode ends, providing a binary indication of success or failure.

Trajectory-level advantage is a scalar derived from the final reward that is added to every action in the episode, treating the whole trajectory as equally responsible.

Token‑level on‑policy distillation (OPSD) produces dense supervision for each generated token, but in agentic settings actions span multiple tokens and the environment only reacts at turn boundaries. Consequently, the token‑level gaps are misaligned with the points where credit should be assigned. A turn‑level credit mechanism must aggregate these token signals and respect the interaction history.

Trajectory‑level rewards fail to isolate pivotal decisions, making turn‑level credit essential.

AgentOPSD: Core Mechanism

AgentOPSD turns sequence‑level rewards into per‑turn credits through three concrete steps.

GRPO computes a single sequence‑level advantage `A_seq` and copies it to every token, leaving the contribution of each turn ambiguous—a fatal credit‑assignment bottleneck for long‑horizon tasks.

**Figure 2.** **Overview of AgentOPSD.** Left: the agent loop, interacting with the environment over turns $1, \dots, K$. Middle: AgentOPSD converts GRPO's single sequence-level advantage into turn-level reshaped advantages in three steps: (1) aggregate the token-level teacher-student gaps $\delta_{k,t}$ within a turn into a turn-level gap $e_k$; (2) recursively update a belief state $B_k$ (initialized from the group success rate) and read off its marginal revision $\Delta B_k = B_k - B_{k-1}$; (3) reshape the sequence-level advantage $A_{seq}^{(i)}$ per turn into $\tilde{A}_k^{(i)}$. Right: vanilla GRPO instead broadcasts the same $A_{seq}^{(i)}$ to every token/turn. Each token in turn $k$ inherits $\tilde{A}_k$.

AgentOPSD treats each turn like a weather forecast: it gathers local evidence (the teacher‑student gap), updates a belief about eventual success, and then distributes the overall reward proportionally to the belief shift.

Update cumulative evidence: cₖ = 0.9 · 0.2 + 0.3 = 0.48.

Logit update: ℓₖ = 0.61 + 0.48 = 1.09.

Belief: Bₖ = $\sigma$(1.09) ≈ 0.75.

Marginal revision: $\Delta$Bₖ = 0.75 − 0.65 = 0.10.

Assume `A_seq` = +1.2; sign = +1, so qₖ = +0.10.

Across a three‑turn trajectory the mean of q is 0.08, std ≈ 0.02; zₖ = (0.10‑0.08)/0.02 = 1.0.

Clip with b = 0.2: wₖ = clip(1 + 0.2·1.0, 0.8, 1.2) = 1.2.

Final turn advantage with $\lambda$ = 0.5: ~Aₖ = 1.2 · [(1‑0.5)+0.5·1.2] ≈ 1.32.

The belief update amplifies a turn’s credit only when its evidence pushes the group‑level confidence upward; a negative eₖ would produce a negative $\Delta$Bₖ and thus reduce the turn’s share of the advantage.

Algorithm 1 AgentOPSD: Recursive state updates for turn‑level credit

How does AgentOPSD differ from the naïve GRPO approach that simply broadcasts `A_seq` to every token?

GRPO treats all turns as equally responsible for the final outcome, which discards any temporal structure. AgentOPSD first measures each turn’s local evidence (eₖ), then lets that evidence shift a belief state (Bₖ). The marginal belief change $\Delta$Bₖ acts as a turn‑specific weighting factor, so turns that actually contributed to success receive a larger share of `A_seq`, while unhelpful turns may even receive a negative credit.

The conversion of a single sequence‑level advantage into turn‑level credits enables the policy gradient to receive fine‑grained feedback, dramatically improving credit assignment in long‑horizon tasks where early decisions are otherwise invisible.

Recursive Belief Updates

Recursive belief update turns raw turn scores into decayed support estimates and reshapes advantages for credit assignment.

Turn‑level scores eₖ alone cannot tell whether a turn adds new decisive evidence or merely repeats what earlier turns already established.

We keep a running belief about how supportive the trajectory is, letting each new turn adjust it while older evidence fades geometrically.

Compute the prior log‑odds: logit(0.6)=\ln\frac{0.6}{0.4}\approx0.405.

c₁ = 0.5·0 + 0.2 = 0.2 → ℓ₁ = 0.405 + 0.2 = 0.605 → B₁ = $\sigma$(0.605)≈0.647.

c₂ = 0.5·0.2 + (‑0.1) = 0.0 → ℓ₂ = 0.405 + 0.0 = 0.405 → B₂ = $\sigma$(0.405)≈0.600.

c₃ = 0.5·0.0 + 0.3 = 0.3 → ℓ₃ = 0.405 + 0.3 = 0.705 → B₃ = $\sigma$(0.705)≈0.669.

The decay $\gamma$ quickly reduces the influence of early evidence, allowing later turns to shift the belief even if earlier scores were strong.

How does this differ from simply summing the turn scores?

A plain sum treats every past turn equally; the recursive update discounts older scores with $\gamma$ and retains the fixed prior log‑odds, preventing early turns from permanently dominating the belief.

Having a calibrated belief per turn, we now translate its revision into a credit that reshapes the advantage used for policy updates.

Turn‑level credits are first centered and scaled within each trajectory, then clipped to a modest range before modulating the verifier‑derived advantage.

Mean $\mu$ = (0.2‑0.1+0.4+0)/4 = 0.125.

Std $\sigma$ = sqrt(((0.2‑0.125)²+(‑0.1‑0.125)²+(0.4‑0.125)²+(0‑0.125)²)/4) ≈ 0.208.

Normalized deviations: $q^{norm}_{1}$=0.36, $q^{norm}_{2}$=‑1.08, $q^{norm}_{3}$=1.32, $q^{norm}_{4}$=‑0.60.

Unclipped multipliers: 1 + b·$q^{norm}$ give [1.11, 0.46, 1.40, 0.82]; after clipping to [0.7, 1.3] they become [1.11, 0.70, 1.30, 0.82].

Bounded credits \tilde{q}_{i,k}=$z_{i,k}$·$q^{norm}_{i,k}$+$\epsilon$₀ ≈ [0.40, ‑0.76, 1.69, ‑0.49].

Even a large positive raw credit (0.4) cannot increase the multiplier beyond 1 + b = 1.3, preventing runaway advantage scaling.

Why clip the normalized credit instead of using it directly?

Clipping caps the per‑turn multiplier to a narrow band [1‑b, 1+b], so no single turn can dominate the advantage update, which stabilizes gradients and avoids extreme policy swings.

The reshaped advantage $A_i$ is finally fed into the surrogate objective that drives the policy update.

Empirical Performance

AgentOPSD delivers higher success and far better horizon robustness than GRPO.

We evaluate on three environments—ALFWorld, Search‑QA, and WebShop—using Qwen2.5‑3B and Qwen2.5‑7B instruction‑tuned models trained on 8 × H800 GPUs. Privileged skills are retrieved from the SkillBank during training only; inference runs without external skills.

AgentOPSD loses only 0.54 success points per additional turn, far less than uniform‑credit baselines.

Figure 1(b) regresses per‑sub‑task success against mean successful episode length on ALFWorld (Qwen2.5‑7B).

**Figure 1.** **Training dynamics and horizon-robustness of AgentOPSD on Qwen2.5-7B-Instruct / ALFWorld.** (a) Validation success rate over training. (b) Sensitivity to task horizon: success points lost per extra turn (OLS slope of per-sub-task success against the measured mean turns of successful episodes). (c) Policy entropy over training.

AgentOPSD consistently outperforms GRPO baselines across benchmarks.

Mechanism Ablation and Sensitivity

We test each design choice and hyperparameter to see how much they affect success on ALFWorld.

AgentOPSD improves long‑horizon credit assignment by recursively distilling turn‑level advantages into belief updates. This section asks whether each ingredient of that pipeline is truly needed.

Turn‑level Credit Assignment attributes reward to each decision turn, while Recursive Self‑Distillation repeatedly refines the agent’s belief using its own past predictions.

Replacing turn‑level granularity with per‑token accumulation drops success to 85.9 %.

Full method 89.1 % → per‑token 85.9 % (−3.2 %).

Using the raw local gap $e_k$ instead of recursive revision $\Delta B_k$ reduces success to 82.8 %.

Full method 89.1 % → $e_k$ ablation 82.8 % (−6.3 %).

Dropping the signed direction (keeping only $|\Delta B_k|$) lowers success to 80.5 %.

Full method 89.1 % → sign‑less 80.5 % (−8.6 %).

Removing the empirical prior $B_0$ anchor cuts success to 78.9 %.

Full method 89.1 % → no anchor 78.9 % (−10.2 %).

Reducing the reshaping weight $\lambda$ from 0.5 to 0.25 drops success by 4.7 % (to 84.4 %).

Best $\lambda\!=\!0.5$ yields 89.1 %; $\lambda\!=\!0.25$ yields 84.4 %.

Varying the evidence decay $\gamma$ moves success within a 5.5 % window (82.0 %–87.5 %).

Across $\gamma\in\{1.0,0.95,0.9,0.8\}$ the rate ranges from 82.0 % to 87.5 %.

**Figure 4.** Teacher–Student Gap $\bar{\delta}$ when training Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop and Search-QA.

**Figure 5.** Reward Curve when training Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop and Search-QA.

**Figure 3.** Hyperparameter sensitivity of AgentOPSD. Rows: ALFWorld (Qwen2.5-7B), Search-QA (Qwen2.5-3B), and ALFWorld (Qwen2.5-3B). Columns sweep one knob ($\lambda$, $\gamma$, $\epsilon_{high}$) with the others held at our setting. Curves are rolling means; shaded bands show the local $\pm 1$ standard deviation.

**Table 2.** Component ablation of AgentOPSD on ALFWorld with Qwen2.5-7B (success rate, %). Each row removes or replaces a single mechanism while holding all other settings fixed. The signed direction and the state prior anchor have the largest impact on performance, while the recursive state revision and turn-level granularity provide smaller but consistent improvements.

Related Work

We situate AgentOPSD among prior credit‑assignment and self‑distillation approaches.

Credit assignment for long‑horizon agents has been tackled from several angles: verifiable‑reward post‑training, on‑policy self‑distillation, and classic return‑decomposition methods.

GRPO treats a batch of trajectories as a group, computes a shared advantage, and broadcasts that same signal to every turn in each trajectory.

GiGPO extends GRPO by mixing the group‑level signal with step‑level estimates derived from repeated anchor states across trajectories.

Combines trajectory‑level advantages with step‑level advantages estimated from anchor‑state repetitions, improving side‑credit assignment.

AgentOPSD, the focus of this paper, derives turn‑level evidence from privileged teacher–student likelihood gaps and updates a belief state recursively.

Recursively distills teacher–student log‑probability gaps into turn‑level advantages, using a belief state to propagate credit across turns without a learned critic.

StepOPSD aggregates the same teacher–student signal over action‑centered step spans but still scores each span locally.

Aggregates teacher–student log‑ratio over fixed step windows and assigns credit per window, without turn‑level recursion.

Classic RL baselines such as PPO with GAE provide per‑step temporal‑difference signals, while return‑decomposition methods like RUDDER and VinePPO recover step‑wise structure at additional computational cost.

Policy gradient method that clips policy updates to stay within a trust region, often paired with a learned value function.

Computes a weighted sum of temporal‑difference residuals to produce low‑variance, low‑bias advantage estimates.

Redistributes sparse terminal rewards to earlier steps by learning a return‑decomposition model.

Monte‑Carlo credit method that augments PPO with additional rollouts or a learned scorer to estimate intermediate values.

Theoretical Analysis and Details

Formal definitions, theoretical properties, and the training pseudocode for AgentOPSD.

We formalize the turn‑level credit signal used by AgentOPSD and derive its theoretical guarantees.

Because the correction term in (15) is monotone in $B_k$, the sign of $e_k$ matches the sign of $B_k$, guaranteeing that the ranking of turns by evidential strength is unchanged.

We clip a sign‑adjusted, standardized belief signal to keep the per‑turn advantage within a bounded multiplicative band.

Proposition 1 (Boundedness). Because $m_k\in[1-b,1+b]$, the deviation $|\tilde A_k - A(i)|$ is at most $\lambda b|A(i)|$, guaranteeing $(1-\lambda b)|A(i)| \le |\tilde A_k| \le (1+\lambda b)|A(i)|$.

Proposition 2 (Sign Preservation). The factor $(1-\lambda)+\lambda m_k$ is strictly positive ($\lambda\le1$, $b<1$), so $\operatorname{sign}(\tilde A_k)=\operatorname{sign}(A(i))$ for every turn.

Proposition 3 (Recovery of GRPO). Setting $\lambda=0$ makes $\tilde A_k = A(i)$ for all turns, reducing the AgentOPSD gradient to the original GRPO gradient.

Proposition 4 (First‑order decomposition). With $c_k = \gamma c_{k-1} + e_k$, $\ell_k = \operatorname{logit}(B_0) + c_k$, and $\Delta\ell_k = e_k - (1-\gamma)c_{k-1}$, the belief revision satisfies $\Delta B_k = B_{k-1}(1-B_{k-1})\,\Delta\ell_k + O((\Delta\ell_k)^2)$.

Proposition 5 (Exact budget). Summing the incremental Bayes factors telescopes: $\sum_{k=1}^{K}\Delta B_k = B_K - B_0$.

Proposition 6 (Non‑identifiability). Two trajectories can share the same final reward yet have different per‑turn contributions, showing that trajectory‑level returns alone cannot identify turn‑level credit.

Proposition 7 (Interpretation of $B_0$). For a group of $G$ trajectories with $S$ successes, the maximum‑likelihood estimate of the success probability $\theta_x$ is $\bar R = S/G$. AgentOPSD clips this estimate to $[\epsilon_0,1-\epsilon_0]$ before applying $\operatorname{logit}$, with $\epsilon_0=10^{-4}$.

**Table 3.** Hyperparameters for AgentOPSD.

**Table 4.** Per-environment training configuration. Optimization settings shared across all environments are listed in Table 3; the environment-specific settings are below.

Algorithm 1 presents a single AgentOPSD training iteration at turn‑level granularity. The only additions over GRPO are a teacher forward pass per turn and the belief‑reshaping block.

AgentOPSD training iteration (high‑level pseudocode)

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers