Progressive Agent Skill Generation via Reinforcement Learning
Junhao Shen, Zhanqiu Zhang, Yiwen Guo, Hong Cheng
Skill-$\alpha$ uses reinforcement learning to progressively generate agent skills by treating skill construction as a sequence of evaluable local edits.
How can we progressively generate reusable agent skills from raw experience trajectories using reinforcement learning?
Large language model agents often rely on external skills to guide complex tasks, but automatically generating these skills from documents or experience is difficult because they lack a direct supervision signal for quality. Skill-$\alpha$ treats skill generation as a progressive editing process, using reinforcement learning to train a generator that makes local, evaluable changes to a skill document. This approach improves downstream success rates by up to 6.7 points on standard benchmarks compared to heuristic or pipeline-based methods.
Paper Primer
Existing methods for skill generation rely on rigid heuristics or pipeline-style consolidation that must be custom-designed for specific evidence sources. Because a skill's quality is only visible through its effect on downstream task performance, these methods struggle to assign credit to specific pieces of evidence or procedural instructions.
Skill-$\alpha$ reformulates skill generation as a sequential decision-making problem: the generator reads evidence units one by one and applies local edits—such as creating, updating, or pruning rules—to a growing skill document. The core mechanism is the rollback reward: a local edit is evaluated by comparing the worker agent's performance on an anchored query before and after the edit, providing an execution-grounded signal for reinforcement learning.
Skill-$\alpha$ significantly outperforms heuristic and pipeline-based skill generation baselines across multiple benchmarks.
Evaluated using a GPT-4o worker agent on CL-Bench and tau2-bench. Achieved a 3.3-point improvement on CL-Bench and a 6.7-point improvement on tau2-bench over the strongest baseline.
Why is reinforcement learning preferred over standard supervised fine-tuning for this task?
Skills lack a natural supervision signal based on factual correctness; their value is defined by how they change agent behavior. RL allows the model to learn from execution-grounded feedback (rollback rewards) rather than relying on static, potentially suboptimal human-written examples.
Does this method require a different setup for document-based versus experience-based evidence?
No. Skill-$\alpha$ uses a unified serialization format that allows the same generator to process heterogeneous evidence sources, treating both documents and execution traces as sequential inputs for the editing policy.
By decomposing skill construction into local, reward-driven edits, Skill-$\alpha$ enables a unified, scalable approach to agent skill acquisition that adapts to diverse evidence sources without manual pipeline engineering.
Abstract
The abstract motivates skill generation, introduces Skill‑$\alpha$, and highlights its performance gains.
Large language model agents often rely on external skills as modular procedural units, making the automatic generation of high‑quality skills from documents or experience a pressing challenge. Existing skill‑generation pipelines depend on heuristics or hand‑crafted consolidation steps that must be tailored to each evidence source, while learning‑based approaches lack a natural supervision signal tied to downstream usefulness. To address this, we introduce Skill‑$\alpha$, a reinforcement‑learning method that treats skill generation as a progressive editing process and employs a rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query.
The Skill-α Framework
Skill‑$\alpha$ progressively edits skills with rollback reward, turning skill generation into a learnable edit process.
LLM agents struggle because skill generation offers no natural supervision; without a clear signal, a fluent skill can still be redundant or harmful.
Supervised Fine‑Tuning (SFT) is the standard baseline where a model is trained on static demonstration data before any skill‑editing is applied.
Skill‑$\alpha$ treats skill generation as a sequence of local edits, letting each edit be judged by execution feedback rather than a single monolithic output.
How does Skill‑$\alpha$ differ from a conventional one‑shot skill generator?
One‑shot generators produce a complete skill in a single forward pass, which makes it hard to assign credit to any particular part of the input. Skill‑$\alpha$ instead breaks generation into atomic edits, each receiving its own rollback reward, so the learning signal is directly tied to the edit that caused the improvement.
Step 1 (CREATE): add a rule “open file” → $z_1 = \{ \text{open file} \}$.
Step 2 (UPDATE): refine the rule to “open file with read mode” → $z_2 = \{ \text{open file (read)} \}$.
Step 3 (MERGE): combine with an existing “close file” rule → $z_3 = \{ \text{open file (read)}, \text{close file} \}$.
Step 4 (PRUNE): remove a redundant “open file” duplicate → $z_4 = \{ \text{open file (read)}, \text{close file} \}$ (no change).
Step 5 (NOOP): evidence $x_3$ adds nothing, so $z_5 = z_4$.
The example shows how each atomic action incrementally shapes the skill, and how rollback reward can evaluate the benefit of each edit independently.
The edit action space defines the atomic operations the generator can perform on a skill artifact.
Why not use a single generic INSERT action instead of the four specialized edits?
INSERT would conflate adding, fixing, merging, and deleting, making it impossible for the rollback reward to know which semantic effect caused a performance change. The four actions keep the semantics explicit, so the credit signal can be correctly attributed.
Rollback reward assigns credit by comparing the worker’s answer before and after an edit on the same anchored query.
What would happen if we used the raw task reward instead of rollback reward?
The raw task reward reflects the final outcome after many edits, making it impossible to tell which individual edit contributed positively or negatively. The learning signal would be noisy, and the policy could reinforce harmful edits that happen to coincide with a lucky final answer.
**Figure 1.** An overview of Skill-$\alpha$. Left: during inference, Skill-$\alpha$ reads evidence sequentially and applies a sequence of local edit actions to progressively generate the skill from the initial state $z_0$ to the final skill $z_T$. Right: during training, the skill generator samples a group of candidate actions from the current skill state, constructs the corresponding edited skills, and evaluates them on the same evidence-related anchored query with a benchmark-specific verifier; the resulting rollback rewards are then used for the GRPO update.
Progressive editing enables structured refinement of skills, turning a vague generation problem into a sequence of learnable decisions.
The edit action space (CREATE, UPDATE, MERGE, PRUNE) precisely defines what the generator can change, making the learning problem well‑posed and the rollback reward meaningful.
Empirical Results and Benchmarks
Skill‑$\alpha$ outperforms all baselines across CL‑Bench, SpreadsheetBench, and tau2‑bench.
Skill‑$\alpha$ achieves the highest pass rate on tau2‑bench.
55.83 % pass rate, beating the next‑best method by 2.5 percentage points.
Across the three benchmarks, Skill‑$\alpha$ consistently exceeds every baseline, including the strong SFT and prompt‑only approaches. The advantage is most pronounced on the domain‑wise tau2‑bench, where the progressive editing process yields a clear margin. This demonstrates that the RL‑driven refinement of skills generalizes beyond the training contexts.
CL‑Bench measures how well a generated skill can solve a set of held‑out tasks after being trained on a small collection of source contexts.
SpreadsheetBench evaluates a model’s ability to acquire spreadsheet‑operation skills from execution trajectories.
tau2‑bench assesses domain‑workflow skill reuse by training on trajectories from three distinct domains.
**Table 3.** Experience-to-skill results on SpreadsheetBench and tau2-bench. tau2-bench Avg. is the unweighted average over Airline, Retail, and Telecom. All numbers are pass rates (%), and $\uparrow$ indicates that higher is better. Skills are generated from the shared GPT-4o trajectory pool and then reused on held-out tasks under both worker backbones; Claude-Sonnet-4.5 is used only as a transfer worker. Best and second-best results within each backbone block are in **bold** and <u>underlined</u>.
**Figure 2.** Training dynamics and rollout action distribution under the main ablations. Left: training reward over RL steps for Skill-$\alpha$ and its RL ablations. Right: action distribution accumulated over rollout steps. "SFT only" is omitted because it has no RL rollout trajectory.
**Table 4.** Ablation study. CL-Bench Avg. averages the four CL-Bench categories, and tau2-bench Avg. averages Airline, Retail, and Telecom. All numbers are pass rates (%).
Skill‑$\alpha$ consistently outperforms SFT baselines across document and trajectory tasks.
Training and Evaluation Protocols
Defines the core system prompt that guides Skill‑$\alpha$ to edit SKILL.md for downstream agents.
This prompt specifies how the model receives a SKILL.md document and evidence, then chooses a single edit action to improve future agents.
Core system prompt for Skill‑$\alpha$
Limitations and Caveats
We outline three key limitations of Skill‑$\alpha$.
Skill‑$\alpha$ treats skill generation as a progressive, RL‑driven editing process that refines artifacts over time.
First, Appendix A only yields a local ranking of candidate edits under the paper’s assumptions, offering a single rollback comparison that is a noisy binary signal on one anchored query.
This local signal neither estimates an expected score difference nor guarantees that repeated local updates improve the final skill across its task family.
Second, the reward interface still depends on benchmark‑specific verifier design, meaning that moving Skill‑$\alpha$ to new environments requires extra work on evaluation protocols and comparison criteria.
Third, the action space and skill representation are deliberately simple and text‑centric, suitable for SKILL.md‑style procedural guidance but not yet able to handle richer structured, multimodal, or executable skill artifacts.
Questions & answers
What is the main contribution of the Skill-α paper?
Skill-α introduces a framework that treats skill generation for large language model agents as a sequential decision-making problem, using reinforcement learning to train a generator that applies local, evaluable edits to a skill document rather than producing a skill in a single pass.
What problem does Skill-α address?
Skill-α addresses the difficulty of automatically generating high-quality procedural skills for LLM agents from documents or experience, a task where existing heuristic and pipeline-based methods struggle because there is no natural supervision signal tied to skill quality.
Why is there no natural supervision signal for skill generation?
A skill's quality is not defined by factual correctness but by how it changes downstream agent behavior, so standard supervised learning on static examples cannot reliably capture whether a skill is useful, redundant, or harmful.
Why does Skill-α use reinforcement learning instead of supervised fine-tuning?
RL allows the model to learn from execution-grounded feedback via rollback rewards rather than relying on static, potentially suboptimal human-written examples, because a skill's value is defined by its effect on agent behavior rather than factual correctness.
How does the progressive editing mechanism in Skill-α work?
The generator reads evidence units one by one and applies one of four atomic edit actions—CREATE, UPDATE, MERGE, or PRUNE—to a growing skill document (SKILL.md), with each edit receiving its own rollback reward based on the worker agent's performance before and after the edit on an anchored query.
What is the rollback reward in Skill-α?
The rollback reward is a local evaluation signal computed by comparing a worker agent's performance on an anchored query before and after a single edit, allowing credit to be attributed directly to the specific edit that caused a performance change.
Why does Skill-α use four specialized edit actions instead of a single generic insert action?
A single INSERT action would conflate adding, fixing, merging, and deleting, making it impossible for the rollback reward to identify which semantic effect caused a performance change; the four distinct actions (CREATE, UPDATE, MERGE, PRUNE) keep semantics explicit so credit can be correctly attributed.
What would happen if the raw task reward were used instead of the rollback reward?
The raw task reward reflects the final outcome after many edits, making it impossible to identify which individual edit contributed positively or negatively, resulting in a noisy learning signal that could reinforce harmful edits that coincidentally preceded a correct final answer.
Does Skill-α require separate setups for document-based versus experience-based evidence?
No. Skill-α uses a unified serialization format that allows the same generator to process heterogeneous evidence sources, treating both documents and execution traces as sequential inputs for the editing policy.
What benchmarks and datasets were used to evaluate Skill-α?
The paper evaluates Skill-α on three benchmarks, including the domain-wise tau2-bench; the paper does not provide further detail on the other two benchmarks in the excerpted content.
What are the key quantitative results reported for Skill-α?
Skill-α improves downstream task success rates by up to 6.7 percentage points over heuristic or pipeline-based methods across the three benchmarks, with the most pronounced advantage on the domain-wise tau2-bench.
How does Skill-α compare to supervised fine-tuning and prompt-only baselines?
Skill-α consistently outperforms both SFT and prompt-only baselines across all three benchmarks, demonstrating that RL-driven skill refinement generalizes beyond training contexts.
What are the acknowledged limitations of Skill-α?
The paper identifies three main limitations: (1) the rollback reward is a noisy binary signal on a single anchored query and does not guarantee improvement across the full task family; (2) the reward interface depends on benchmark-specific verifier design, requiring extra work to transfer to new environments; and (3) the action space and skill representation are text-centric and not yet capable of handling structured, multimodal, or executable skill artifacts.
Is the rollback reward guaranteed to improve the final skill over repeated updates?
No. The paper explicitly states that the local rollback signal neither estimates an expected score difference nor guarantees that repeated local updates improve the final skill across its task family.
What is the skill representation format used in Skill-α?
Skill-α uses a text-centric SKILL.md document format for procedural guidance, which the paper notes is not yet able to handle richer structured, multimodal, or executable skill artifacts.
How is the training prompt structured in Skill-α?
The training prompt specifies how the model receives a SKILL.md document and evidence, then chooses a single edit action to improve future agent performance, as described in the paper's Training and Evaluation Protocols section.
Who are the authors of Skill-α, and where was it published?
The paper does not specify the authors' names or the publication venue in the provided content; the arXiv identifier is 2608.01678.
Key terms
- Skill-α
- The proposed framework that treats LLM agent skill generation as a reinforcement-learning-driven progressive editing process over a skill document.
- skill
- A modular procedural unit used by LLM agents to guide behavior on complex tasks, stored as a document of rules or instructions.
- SKILL.md
- The text-based document format used in Skill-α to represent a skill as a set of procedural rules that can be progressively edited.
- rollback reward
- A local training signal computed by comparing agent performance on an anchored query before and after a single edit, used to attribute credit to individual edits rather than final outcomes.
- progressive editing
- The process of incrementally refining a skill document through a sequence of atomic, individually evaluated edit operations rather than generating the skill all at once.
- edit action space
- The set of four discrete operations—CREATE, UPDATE, MERGE, and PRUNE—that the Skill-α generator can apply to a skill document at each step.
- CREATE
- An edit action that adds a new rule or entry to the skill document.
- UPDATE
- An edit action that modifies an existing rule in the skill document.
- MERGE
- An edit action that combines two or more existing rules in the skill document into one.
- PRUNE
- An edit action that removes a rule from the skill document, typically to eliminate redundant or harmful content.
- anchored query
- A fixed evaluation query used as a reference point to measure the effect of a single edit on agent performance when computing the rollback reward.
- worker agent
- The downstream LLM agent whose task performance is used to evaluate the quality of a generated or edited skill.
- Supervised Fine-Tuning (SFT)
- A standard training approach where a model is trained on static demonstration data, used as a baseline in the Skill-α experiments.
- tau2-bench
- A domain-wise benchmark used in the paper's evaluation on which Skill-α shows its most pronounced performance advantage over baselines.
- evidence unit
- A single piece of input—such as a document excerpt or execution trace step—that the Skill-α generator processes sequentially to decide on an edit action.
- unified serialization format
- A common input representation in Skill-α that encodes both document-based and experience-based evidence as sequential text, enabling the same generator to handle heterogeneous sources.
- credit assignment
- The problem of determining which specific action or decision in a sequence was responsible for a change in outcome, a core challenge that Skill-α addresses through rollback rewards.