SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Jinyang Wu, Shuo Yang, Zhengxi Lu, Fan Zhang, Yuhao Shen, Lang Feng, Haoran Luo, Zheng Lian, Shuai Zhang, Zhengqi Wen, Jianhua Tao

SEED uses self-generated hindsight skills to provide dense, evolving supervision for agentic reinforcement learning.

How can we improve agentic reinforcement learning by distilling hindsight information from completed trajectories into a self-evolving policy?

Outcome-based reinforcement learning (RL) for agents is often inefficient because sparse, trajectory-level rewards fail to identify which intermediate actions were correct or flawed. SEED (Self-Evolving On-Policy Distillation) addresses this by using the current policy to analyze its own completed trajectories, extracting natural-language "hindsight skills" that describe reusable strategies or failure-avoidance rules. The model then re-scores its own sampled actions under these skill-augmented contexts, creating a dense, token-level distillation signal that is optimized jointly with the environment reward.

Paper Primer

SEED operates as a self-evolving loop where the policy serves two roles: an actor collecting trajectories and an analyzer extracting hindsight. By synchronizing these roles, the hindsight supervision adapts to the policy's current capabilities and failure modes, ensuring the auxiliary training signal remains relevant as the agent improves.

SEED significantly improves task performance and sample efficiency across diverse agentic domains.

In ALFWorld, SEED outperformed outcome-only RL (GRPO) by 14.9–45.9 points and surpassed static skill-distillation baselines by up to 38.1 points. SEED achieved higher success rates with 60% of the training data than standard RL methods achieved with 100%.

The core mechanism is a confidence-gated distillation objective: the model compares the log-probability of an action under an ordinary history versus a skill-augmented history. This shift acts as a dense, token-level "hindsight" signal that guides the policy toward actions endorsed by its own successful (or corrected) past experiences.

Why is it necessary for the analyzer to be "self-evolving" rather than using a fixed, pre-trained analyzer?

As the policy improves, it encounters new states and failure modes; a static analyzer would eventually provide stale or mismatched supervision. Synchronizing the analyzer with the latest policy checkpoint ensures that hindsight guidance evolves alongside the agent's decision-making capabilities.

Does the agent require these hindsight skills during deployment?

No. The skills are used exclusively as training-time privileged supervision. At inference time, the agent acts solely based on the ordinary interaction history, requiring no external memory, skill banks, or additional prompts.

Motivation and Problem Framing

We expose why outcome‑based RL struggles with sparse rewards and outline SEED’s self‑evolving solution.

Outcome‑based RL optimizes agents only with episode‑level rewards, which are sparse and give little guidance for the many intermediate decisions a long‑horizon agent must make. This creates a supervision gap: the policy sees only the final success or failure, not which observations or actions were decisive. Sparse reward is like searching for a needle in a haystack, while dense hindsight supervision is like having a map that points directly to the needle.

Agentic RL equips a language model with the ability to iteratively reason, act, use tools, and incorporate feedback, turning a static generator into an interactive decision‑making system.

Outcome‑based RL treats the final task result—success or failure—as the sole learning signal, ignoring the rich intermediate signals that occur during an episode.

The inefficiency of sparse‑reward outcome‑based RL stems from its inability to provide dense, decision‑level guidance.

The SEED Framework

SEED trains a policy to extract and distill hindsight skills, closing the granularity gap in long‑horizon RL.

Outcome‑based RL suffers from a granularity mismatch: the environment returns a single scalar after an entire trajectory, yet the policy must learn from thousands of token‑level decisions. SEED bridges this gap by extracting reusable “hindsight skills” from completed trajectories and distilling their effect back into the policy.

After a trajectory finishes, the analyzer reads the whole episode and writes down a short natural‑language recipe that captures what worked (or what failed) – a reusable piece of knowledge the policy can later imitate.

Trajectory 1 succeeds: actions = [open‑door, walk‑to‑key, pick‑key]; outcome $R=1$.

The analyzer reads the full sequence and writes the skill “open the door, then go to the key”.

Trajectory 2 fails: actions = [walk‑to‑key, open‑door]; outcome $R=0$.

The analyzer produces the skill “open the door before moving toward the key”.

Even with the same sparse binary reward, the analyzer extracts distinct procedural advice that the policy can later imitate.

During each update the current policy acts, the same policy (frozen as $\pi_{\theta_{\text{old}}}$) analyzes its own trajectories, and the resulting skill‑augmented context is used to re‑score the exact actions the policy just took. The “teacher” sees the hindsight skill, the “student” sees only the ordinary history; the student is nudged toward the teacher’s preferences.

How does OPD differ from standard knowledge distillation, which also aligns a student to a teacher?

In ordinary distillation the teacher is a separate, often larger model that sees the same inputs as the student. Here the teacher sees *more* information—the hindsight skill—while the student sees only the original history. Moreover, the teacher’s outputs are detached, so the loss only nudges the student’s own logits rather than copying the teacher’s parameters.

Training loop for SEED (Algorithm 1)

**Figure 2.** Overview of SEED. Stage 1 (Hindsight Skill SFT) equips the policy to extract hindsight skills from completed trajectories. Stage 2 (Self-Evolving On-Policy Distillation) jointly optimizes outcome-based RL and skill-conditioned OPD in a self-evolving agentic loop.

Empirical Performance

SEED beats GRPO on every benchmark, showing stronger performance and sample efficiency.

We evaluate SEED on three long‑horizon agentic benchmarks and compare it against a suite of outcome‑based and skill‑aware baselines.

GRPO normalizes rewards across a batch of trajectories, letting the policy chase the best‑performing group rather than a single episode’s outcome.

ReAct augments the language model with a simple reasoning‑then‑action loop, letting it generate an intermediate thought before issuing an environment command.

SEED improves the ALFWorld macro‑average by up to 45.9 percentage points over GRPO.

Table 1 reports the per‑task and macro‑average success rates; the largest gap appears on the Heat task (+35.0) and the overall macro‑average gains reach 45.9 points.

**Table 1.** Main results across different model scales and agentic domains.

**Figure 3.** **Training dynamics on ALFWorld.** We compare SEED and GRPO using Qwen2.5-3B-Instruct as the backbone. Translucent curves show raw measurements, while solid curves show 13-point centered moving averages.

**Figure 4.** Sample efficiency analysis. SEED consistently outperforms GRPO across different data fraction settings and surpasses full-data GRPO using only 60% of the training data.

**Figure 5.** Cross-domain generalizability on ALFWorld Unseen. SEED generally outperforms GRPO across unseen task types, demonstrating stronger cross-domain generalizability.

**Figure.** A comparison of agent trajectories between GRPO and SEED for the task "put a candle in toilet."

SEED consistently outperforms GRPO across benchmarks, delivering higher success rates and better sample efficiency.

Theoretical Analysis

Formal analysis of SEED’s three core theoretical properties.

Outcome‑based RL treats each completed trajectory as a single scalar signal, discarding the rich behavioral patterns that occur inside the rollout; SEED recovers those patterns as “hindsight skills” and distills them back into the policy.

This section formalizes three SEED design requirements: (i) on‑policy hindsight yields an occupancy‑matched adaptive target, (ii) the OPD term supplies dense, decision‑specific credit even when group rewards are tied, and (iii) refreshing the shared analyzer prevents stale supervision.

Proposition 1 shows that, when trajectories are collected by the current policy, the OPD gradient is a KL‑divergence toward a re‑weighted target distribution that matches the policy’s own state‑action occupancy.

Proposition 2 proves that the OPD term remains non‑zero whenever the hindsight gate varies across actions, even if all trajectories share the same terminal reward.

Proposition 3 bounds the drift of the OPD gradient caused by using an outdated analyzer, showing that the error scales with the sigmoid’s Lipschitz constant and the average change in the skill‑induced log‑probability shift.

The table summarizes the dataset statistics for three domains: Embodied Reasoning, Web Navigation, and Search-Augmented QA, detailing the number of SFT training samples, RL training samples, and test samples for each.

We evaluate SEED on three long‑horizon domains: ALFWorld (embodied household reasoning), WebShop (interactive e‑commerce navigation), and a composite Search‑Augmented QA suite covering seven QA datasets.

Baselines span prompting‑only methods, outcome‑based RL (GRPO and its skill‑augmented variants), and self‑distillation approaches (OPSD, Skill‑SD, RLSD, SDAR), all trained with the same backbone and rollout budget for a fair comparison.

SEED: Self‑Evolving On‑Policy Distillation (high‑level pseudocode)

Ablations and Implementation Details

Key ablations reveal which SEED components drive its performance gains.

Outcome‑based RL struggles because it ignores the behavioral patterns hidden in completed trajectories; SEED extracts those patterns as “hindsight skills” and distills them into the policy to guide future exploration.

Algorithm 1 alternates between a frozen snapshot $\pi$$\theta_{o}$ld that collects on-policy rollouts and extracts hindsight skills, and the trainable policy $\pi$$\theta$ that rescoring the same tokens under ordinary and skill‑augmented contexts; gradients flow only through the ordinary branch while the OPD loss aligns $\pi$$\theta$ with the extracted skills.

**Figure 7.** A representative trajectory on Sokoban. The sequence shows six consecutive actions executed by the agent. Arrows indicate the temporal progression of the trajectory, and the action taken at each step is displayed below the corresponding observation.

Table 2 evaluates SEED against three ablations: removing the hindsight‑skill fine‑tuning stage, disabling the self‑evolving OPD gate, and omitting the on‑policy skill branch. In every task the full SEED model outperforms each stripped variant, confirming that each component contributes positively.

Table 4 lists concrete episode‑level skills extracted from successful and failed rollouts across ALFWorld, WebShop, and Search‑based QA, demonstrating that SEED can capture both reusable workflows and failure‑avoidance rules.

On the ALFWorld Unseen benchmark SEED improves average success by +15.3 points over GRPO.

Table 7 shows SEED’s average of 86.2 % versus GRPO’s 70.9 %.

**Figure 8.** Success rates across three backbones and three domains. Success rates increase over training in all nine settings, showing consistent learning across model scales and agentic tasks.

**Figure 9.** OPD loss dynamics. The loss generally decreases and stabilizes during training, indicating that the policy progressively internalizes the behavioral guidance provided by hindsight skills.

**Figure 10.** Prompt of analyzer.

Every SEED component—hindsight‑skill supervision, self‑evolving OPD gating, and on‑policy skill injection—provides a measurable boost; removing any of them degrades performance, confirming that the full self‑evolving pipeline is essential for the observed gains.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers