Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills

Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni, Shijie Zhou, Ziyi Yang, Gangwei Jiang, Mengyu Zhou, Yu Cheng, Xiaoxi Jiang, Guanjun Jiang

Skill Self-Play (Skill-SP) uses an evolving library of modular skills to guide LLM self-evolution, balancing task diversity with rigorous verification.

How can LLMs autonomously generate diverse, verifiable training tasks to improve their own reasoning and tool-use capabilities without relying on manual data?

Self-evolution methods for LLMs often collapse into either narrow, environment-bound domains or chaotic, unverified synthetic noise. Skill Self-Play (Skill-SP) resolves this by organizing expertise into a library of modular, co-evolving skill packages that act as structural interfaces for task generation. This framework consistently pushes performance ceilings, yielding absolute gains of up to 42.9 points on tool-use benchmarks and 12.0 points on logical reasoning tasks.

Paper Primer

Skill-SP functions as a bi-level optimization loop: a Proposer synthesizes tasks conditioned on sampled skills, a Solver learns to execute them, and a Controller updates the skill library based on execution feedback. The skill library acts as a mail sorter: it reads the task requirements, routes them to the appropriate structural template, and restricts the generator to valid, frontier-targeted patterns.

Skill-SP significantly outperforms unguided self-play and rescues misaligned models.

Evaluations on API-Bank, BFCL, and ZebraLogic benchmarks across five backbones (3B–14B parameters). Up to +42.9 absolute points on tool-use and +12.0 points on logical reasoning.

The framework maintains a dual-stream generation process: a skill-conditioned stream provides structural priors for rigorous verification, while an open-ended exploration stream induces new skills to prevent mode collapse. This prevents the "context bloat" of raw trajectory aggregation by distilling scattered experiences into compact, reusable units.

Why is a skill-based interface superior to simply prompting the model with historical trajectories?

Historical trajectories lead to context bloat and dilute structural control; in contrast, a skill interface distills experience into a compact, reusable unit that provides explicit structural priors and programmatic validators for the task generator.

What is the role of the "Controller" in this loop?

The Controller manages the library's lifecycle by refining existing skills based on execution feedback, pruning obsolete ones that yield trivial tasks, and inducing new skills from successful open-ended exploration.

By replacing passive post-hoc filtering with proactive, skill-conditioned task synthesis, Skill-SP provides a sustainable architecture for autonomous model improvement that scales across diverse, verifiable domains.

The Self-Evolution Dilemma

Introducing Skill Self‑Play to reconcile task diversity with reliable verification.

Self‑evolutionary LLM training must balance two opposing forces: expanding the task space while keeping verification trustworthy. Existing approaches either lock the model into narrow, environment‑bound domains with perfect feedback, or open the floodgates to unguided generation that relies on weak post‑hoc filters and quickly degrades data quality. Skill Self‑Play (Skill‑SP) resolves this dilemma by coupling a dynamically evolving skill library with a proposer that generates tasks conditioned on those skills, guaranteeing both breadth and verifiable execution.

Verification reliability measures how confidently a system can confirm that a generated task was executed correctly, without ambiguity or hidden errors.

How does verification reliability differ from simply checking output format?

Format checks only ensure syntactic conformity (e.g., JSON well‑formedness). Verification reliability goes further by confirming semantic correctness—e.g., that a program actually produces the intended result or that a proof logically follows.

Unguided self‑play lets an LLM freely invent tasks and solutions, then applies a passive filter to discard obviously bad examples.

Why can’t a stronger post‑hoc filter replace the need for guided generation?

A stronger filter can reject more bad tasks, but it still reacts after the fact; it never influences the proposer’s search direction. Consequently, the generator wastes effort exploring unproductive regions of the task space, and the overall diversity remains limited.

**Figure 1 |** Motivation of Skill Self-play. Left: *Environment-bound methods* provide reliable verification but severely restrict the task space. Middle: *Unguided generation* broadens coverage but depends on passive post-hoc filtering, allowing residual errors to degrade data fidelity. Right: *Skill Self-Play* uses a proactive Skill Orchestrator with dual streams for skill-guided generation and open-ended exploration, enabling iterative skill evolution with both broad coverage and high fidelity.

The core challenge of self‑evolution is balancing task diversity against verification reliability.

Skill Self-Play Framework

Methodology details the Skill Self‑Play loop that co‑evolves tasks, skills, and policies.

Self‑evolution of large models stalls when generated tasks become either too easy or impossible to verify, leaving the learner with noisy signals.

Skill Self‑Play turns raw self‑play into a guided curriculum builder: a proposer creates tasks, a skill library supplies reusable structural priors and validators, and only tasks that pass a binary quality filter are handed to the solver.

Sample a skill proportionally to $\sigma$; s₁ is chosen with probability 3/4.

Conditioned on s₁, the proposer emits a prompt $x=$ “Sort the list [3,1,2]” and a contract $c$ that asserts the output list is ordered.

Run K=2 verification rollouts of the current solver on $x$; it returns y₁=[1,2,3] (pass) and y₂=[3,2,1] (fail), giving v₍solve₎≈0.5.

Apply the validity mask: the contract is syntactically correct and the rollouts are consistent, so the mask equals 1.

Compute the gated reward: $1-2|0.5-0.5| = 1$; the task is accepted and added to the curriculum.

The example shows how a skill injects a concrete structure (sorting) and a validator, turning a raw prompt into a verifiable task that automatically lands on the solver’s learning frontier.

How does Skill Self‑Play differ from classic self‑play where two agents simply compete?

Classic self‑play lets agents generate each other’s experiences without any external notion of “task quality”. Skill‑SP inserts a reusable skill library that supplies structural priors and a hard validity filter, so generated tasks are guaranteed to be well‑formed and calibrated to the solver’s current ability.

The proposer is a conditional generator: given a sampled skill s, it emits a prompt x and a hidden verification contract c that together define a candidate task.

Why not train the proposer directly on the raw verification reward instead of the gated frontier score?

Optimizing only the raw reward would push the proposer toward tasks that are trivially easy (reward ≈ 1) or impossible (reward ≈ 0), both of which provide little learning signal. The gated frontier score forces the proposer to aim for a 50 % success probability, i.e., the sweet spot where the solver learns most efficiently.

A skill is a packaged recipe that bundles generation hints, example demonstrations, and a validator, plus bookkeeping statistics that guide how often the skill is reused.

What prevents the skill library from growing without bound as new tasks are discovered?

Skill‑SP periodically prunes skills whose expected frontier reward falls below a threshold $\gamma$₍prune₎ (Eq 8) and only adds newly induced skills after they pass integrity and novelty checks (Eq 10). This keeps the library compact and focused on high‑yield priors.

Sample a skill s ∼ S using the usage‑biased distribution $\sigma$.

Condition the proposer on s and generate a candidate task (x, c) via the skill stream.

Run the binary validity filter (Eq 4) on (x, c); discard if any condition fails.

Estimate the solver’s success probability v₍solve₎ using a few verification rollouts.

Compute the gated reward R₍propose₎ (Eq 2) and feed it to GRPO to update $\pi$₍propose₎ (Eq 5).

In parallel, generate open‑ended candidates $\pi$₍propose₎(· | ∅) to keep the skill space diverse.

**Figure 3 | Overview of Skill Self-Play.** An evolving skill library routes task-generation guidance to the proposer, which generates candidate tasks that undergo validity verification. Valid candidates are ranked by frontier reward to construct the solver curriculum. The resulting signals drive co-evolutionary proposer and solver updates, while validation failures, novel samples, and task-level statistics trigger skill refinement, pruning, and induction, distilling training feedback into reusable skills for subsequent task generation.

Empirical Evaluation

Skill‑SP delivers large, consistent gains over unguided baselines on both tool‑call and reasoning tasks.

Skill‑SP improves tool‑call prediction by up to +6.5 points over the base models and far outperforms Unguided SP.

Table 1 shows average gains of 2.8–6.5 points across five backbones, with a 42.9‑point jump on Ministral‑3‑8B, while Unguided SP yields smaller or negative changes.

The experimental setup evaluates two task families—tool‑call prediction (API‑Bank Levels 1‑3 and BFCL categories) and logical reasoning (ZebraLogic puzzles of four size scales). Five backbones from 3 B to 14 B parameters are initialized identically for proposer and solver, and all runs use five self‑play iterations with GRPO updates.

Table 2 confirms that Skill‑SP also raises logical‑reasoning accuracy, delivering up to +12.0 points overall and more than +35 points on the smallest puzzles, while Unguided SP cannot bootstrap this benchmark at all.

Data‑Loop Diagnostics (Section 4.4) show that Skill‑SP’s mixed pool keeps task difficulty near the solver’s learning frontier (mean vₛₒₗᵥₑ≈0.57) and expands the skill library by ~20 new packages per iteration.

**Figure 2.** Performance footprint. Skill-SP broadly expands Qwen3-4B-Ins capabilities across tool-calling and logical reasoning. Notably, Unguided SP is absent from the latter as it fails to synthesize valid puzzles.

**Figure 4.** **Tool-call ablations.** Skill-SP improves more steadily than Unguided SP, while skill-only data underperforms the mixed pool.

Skill‑SP consistently outperforms unguided baselines across tool‑call and logic benchmarks.

Dynamic Skill Routing

Dynamic Skill Routing steers the proposer toward reliable, diverse tasks via adaptive skill sampling.

The proposer must generate tasks that expose structural priors, yet naïvely sampling skills uniformly wastes compute on low‑yield attempts. Dynamic Skill Routing solves this by biasing selection toward skills that have historically produced verifiable, high‑quality tasks while still giving under‑tested skills a chance to prove themselves.

Think of a restaurant host who assigns tables to waiters based on each waiter’s recent service rating—high‑rated waiters get more tables, but the host still gives a few new waiters a chance to prove they can handle a table.

Counts: $a_1=2$, $a_2=1$, $a_3=0$; successes $n_{\text{ver}}=(1,0,0)$, $n_{\text{con}}=(1,0,0)$, $n_{\text{fd}}=(0,0,0)$.

Smoothed rates (Eq. 11): for $s_1$, $r_{\text{ver},1}=(1+1)/(2+2)=0.5$, $r_{\text{con},1}=0.5$, $r_{\text{fd},1}=0.25$; for $s_2$, $r_{\text{ver},2}=0.25$, $r_{\text{con},2}=0.25$, $r_{\text{fd},2}=0.25$; for $s_3$, all $r=0.25$ (pure prior).

Composite scores (Eq. 12): $v_{\text{skill}}(s_1)=0.5\cdot0.5+0.25\cdot0.5+0.25\cdot0.25=0.406$, $v_{\text{skill}}(s_2)=0.25$, $v_{\text{skill}}(s_3)=0.25$.

Clipping (no effect, all within $[0.1,1.0]$) and exploration bonus: $w(s_1)=0.406\,(1+1\cdot e^{-2/5})\approx0.406\cdot1.67=0.68$, $w(s_2)=0.25\,(1+e^{-1/5})\approx0.25\cdot1.82=0.46$, $w(s_3)=0.25\,(1+e^{0})=0.25\cdot2=0.50$.

Normalised probabilities (Eq. 13): sum $=1.64$, so $P(s_1)=0.41$, $P(s_2)=0.28$, $P(s_3)=0.30$.

The under‑tested skill $s_3$ receives a sizable probability despite having no successes, because the exploration bonus compensates for its zero count, preventing it from being starved.

How does Dynamic Skill Routing differ from a simple $\epsilon$‑greedy bandit that adds a fixed exploration probability?

$\epsilon$‑greedy injects a constant chance of picking a random skill regardless of its history, which can waste samples on clearly poor skills. Dynamic Skill Routing uses a *decaying* exploration bonus that is strongest when a skill has been tried few times and fades as evidence accumulates, and it also smooths early success estimates with a Beta prior to avoid over‑reacting to single successes.

**Figure.** Ministral-3-8B. Held-out tool-call (left) and logical-reasoning (right) trajectories.

**Figure 11.** Ministral-3-14B. Held-out tool-call (left) and logical-reasoning (right) trajectories.

Benchmark Analysis

We detail the benchmarks used to evaluate tool‑call and logical‑reasoning performance.

Our evaluation suite combines two tool‑call benchmarks—API‑Bank and BFCL—with a logical‑reasoning benchmark, ZebraLogic. Together they probe two verifiable agent behaviours: executing grounded actions from dialogue and solving deterministic constraint‑satisfaction puzzles.

API‑Bank (Li et al., 2023) presents a normalized record containing a system prompt, tool descriptions, a user dialogue, and a hidden reference answer with the correct tool name and arguments. The solver sees only the system and user messages and must emit a single block; evaluation exact‑matches the normalized tool name and parameters, reporting accuracy across Levels 1–3 to separate easy routing from harder argument‑propagation cases.

BFCL (Patil et al., 2025) tests generalization across distinct function‑schema conventions, type systems, and argument‑normalization rules. Four categories—simple JavaScript, Python, Java, and live‑simple—stress different schema conventions; the benchmark‑specific scorer normalizes predictions before comparison, making BFCL a transfer check for learned tool‑use behavior.

Algorithm 2 (Skill Self‑Play) is shown for reference; the section does not re‑introduce its mechanics, which were defined earlier in the methodology.

The overall pipeline runtime is dominated by solver‑curriculum construction (57 %) and proposer policy optimization (24.3 %). Other stages—skill‑stream generation, exploration‑stream generation, and solver policy optimization—share the remaining wall‑clock share, with skill‑library evolution accounting for only 6.5 % of total time.

**Table 7.** Evaluation benchmark roles. API-Bank and BFCL evaluate tool-call prediction from different benchmark sources, while ZebraLogic evaluates a separate logical-reasoning task family with deterministic verification.

ZebraLogic (Lin et al., 2025) poses grid‑style puzzles whose unique solution must be fully reconstructed. Evaluation checks both grid‑level correctness (entire puzzle solved) and cell‑level correctness (partial progress), with four scale tiers (Small to X‑Large) that expose whether Skill‑SP improves deep constraint tracking beyond surface formatting.

Tool-Call Case Studies

Limits of Skill‑Self‑Play and concrete tool‑call case studies illustrate practical constraints.

Skill‑Self‑Play works well for autonomous self‑evolution, but it still relies on a minimal base capability to bootstrap valid learning signals, and its current heuristics—static mixing ratio $α$ and preset difficulty bounds—often need retuning when new task families appear.

**Table 8.** Tool-call case studies from the induced skill library. All packages were created from self-play trajectories and saved with `source=skill_induction`. Each package binds prior evidence to a next-action rule and checks that the resulting call remains grounded and schema-valid. The final line in each row compactly renders the verified call from its package example; filtered/routed reports the package statistics accumulated during routing.

**Routing Metadata and Curriculum State (m, $\sigma$)** | Field | Content | | :--- | :--- | | **Package** | `skill_038` — `book_table_from_discovered_restaurant`; added in iteration 2 with source=`skill_induction`; `induction_novelty`=specialization. | | **Description** | When a user requests a dinner reservation with a prior restaurant discovery and specifies guests, date, and time, the system should book a table at the top-rated restaurant from the discovery list. | | **Induction Reason** | The dialogue state shows a clear pattern of task progression from restaurant discovery to table booking, grounded in user intent, prior observations, and explicit constraints. The tool selection is logically derived from the sequence of user requests and available observations, with parameters filled from prior findings and normalized into valid formats. | | **Statistics** | 82 attempts; 82 consistent; 74 boundary and verified records; 8 too-easy records; no too-hard or inconsistent records; mean $v_{solve} = 0.572$; last updated in iteration 4. |

Extract the restaurant identifier TRT‑8842 from the prior observation.

Normalize the requested date “Saturday” to ISO format 2023‑09‑16 and the time “7 PM” to 19:00.

Copy the guest count 4 from the user request.

Assemble the tool call `book_table`(`restaurant_id`=TRT‑8842, date=2023‑09‑16, time=19:00, guests=4).

Validator checks that all parameters are present, correctly typed, and ISO‑compliant; the call passes.

This example shows how the proposer copies concrete identifiers and normalizes temporal arguments directly from prior observations, guaranteeing that the generated tool call is both grounded and schema‑valid.

Detect the confirmed availability slot (10:30–11:30) in the observation.

Copy participants “Sarah” and “Alex” and the topic “project updates” from the user request.

Normalize the slot to ISO timestamps (2023‑09‑16T10:30, 2023‑09‑16T11:30).

Construct the tool call ScheduleMeeting(participants=[Sarah, Alex], start=2023‑09‑16T10:30, end=2023‑09‑16T11:30, topic=“project updates”).

Validator enforces ISO format, non‑empty participant list, and that the slot lies within the originally requested window; the call succeeds.

By preferring the evidence‑grounded slot over the user‑stated window, the proposer respects verified availability while still satisfying the original request constraints.

Logical Reasoning Cases I

Package‑level metadata, construction rules, and concrete examples for `skill_042` and `zebra_skill_022`.

This section expands two induced packages—`skill_042`, a tool‑call pattern for meeting scheduling, and `zebra_skill_022`, a logical‑reasoning pattern that resolves positional dependencies via a fixed‑anchor adjacency bridge.

Trigger: the user request contains participants, a time window, and a topic; the observation supplies a concrete slot.

Tool rule selects

Parameter rule copies participants, normalizes the confirmed slot to ISO 8601 timestamps (2023‑10‑12T10:30:00 / 2023‑10‑12T11:30:00), and copies the topic “project updates”.

Answer check verifies that all required fields are present, correctly typed, and grounded in the dialogue.

Verified call:

The pattern binds the tool call to an observed slot rather than merely echoing the user‑provided window, guaranteeing that the executed action is verifiable against prior evidence.

Logical‑reasoning packages encode reusable constraint topologies. `zebra_skill_022` introduces a fixed‑anchor adjacency bridge: a fixed entity, a `next_to` relation, and a `same_house` pair jointly resolve a non‑local positional dependency.

Assign the cat to house 2 (fixed anchor).

Place “golf” in an adjacent house (house 1 or 3) because of the

Choose a house for the

Check all‑different constraints: each house now has a unique person, pet, and hobby.

Resulting assignment satisfies the bridge: the cat’s location, the adjacent golf hobby, and the co‑located bird‑painting pair uniquely determine the full puzzle.

The adjacency bridge forces a local relation (cat ↔ golf) while the `same_house` pair creates a global coupling, eliminating ambiguous placements that would survive with isolated constraints.

Logical Reasoning Cases II

Concrete logical‑reasoning skill instances expose the structural limits of the Skill Self‑Play pipeline.

This section walks through three concrete logic‑puzzle skills and the tables that record their generation, validation, and routing metadata, highlighting where the pipeline’s assumptions become bottlenecks.

**Proposer Hint, Validator, and Offline Oracle (h, v)** | Category | Description | | :--- | :--- | | **Generator hint** | Vary participants (for example, John and Lisa), time ranges (for example, 9 AM to 5 PM), and topics (for example, Q3 review or budget planning), while ensuring that a confirmed time slot is provided in an observation. | | **Validator** | Tool name must be `ScheduleMeeting`; participants must be a non-empty string array; `start_time` and `end_time` must be ISO 8601 timestamps; start must precede end; both timestamps must lie within the requested window; the topic must be non-empty; and no parameter may be invented. | | **Offline recognition** | Look for requests involving “schedule a meeting” or “plan a meeting” with participants and time constraints. | | **Offline evidence extraction** | Extract the participants, the requested start and end range, and the topic. | | **Offline decision rule** | If an observation confirms a specific available slot within the requested window, select `ScheduleMeeting`. | | **Offline parameter recovery** | Normalize times to ISO 8601; copy participants and topic directly; use the confirmed slot as the start and end times. | | **Offline final check** | Ensure that all parameters are present, valid, derived from dialogue or observation, and free of schema violations. |

**Table 11.** Induced logical-reasoning skill patterns. Both packages are saved with source=`zebra_skill_induction`. Counts are accepted or rejected structural-evidence records, not solver scores. Every accepted record reported here has valid schema and constraints, a unique puzzle, and a capped solution count of one.

**Proposer Example (e) and Accepted Offline Instance** * **Stored package example**: Two houses; Name={Elena, Felix}, Nationality={German, Japanese}, Drink={Coffee, Tea}, Hobby={Gardening, Reading}, and FavoriteSport={Badminton, Soccer}. Clues: (1) Elena directly left of Felix; (2) Felix in House 2; (3) Elena left of Felix; (4) German same house as Soccer; (5) Tea left of Coffee; (6) Elena next to Badminton; (7) Gardening not in House 1. * **Example rationale**: Felix fixed in House 2 and Elena next to Badminton force Elena to House 1 and Badminton to House 2; the `same_house` component then resolves the remaining positional bridge. * **Accepted blueprint**: Three houses; Name={Leo, Mia, Ryan}, Nationality={French, Indian, Swedish}, Drink={Latte, GreenTea, Cappuccino}, Hobby={Photography, Cooking, Reading}, and FavoriteSport={Tennis, Badminton, Volleyball}; difficulty medium; theme City Apartment Residents. * **Compiled clues**: (1) Leo in House 2; (2) Leo same house as Indian; (3) Mia same house as Swedish; (4) Indian same house as Cappuccino; (5) Swedish next to Cappuccino; (6) Ryan same house as French; (7) GreenTea same house as Volleyball; (8) Swedish next to Tennis; (9) Photography left of Reading; (10) Volleyball directly left of Indian; (11) French same house as Cooking; (12) Photography left of Cooking. * **Unique solution**: House 1: GreenTea, Volleyball, Photography, Mia, Swedish; House 2: Cappuccino, Tennis, Reading, Leo, Indian; House 3: Latte, Badminton, Cooking, Ryan, French. * **Structural checks**: `logical_generation_mode`=blueprint; `prompt_valid`=1; `schema_valid`=1; `constraints_valid`=1; `puzzle_unique`=1; `solution_count_capped`=1.

**Routing Metadata and Induction Record (m, $\sigma$)** | Field | Content | | :--- | :--- | | **Package** | `zebra_skill_031` — Chain-Anchor-Bridge Pattern; added in iteration 2 with source=`zebra_skill_induction`; `induction_novelty` is a hybrid chain-solving structure. | | **Description** | A directed adjacency chain is anchored by a fixed value and reinforced with cross-attribute `same_house` and `next_to` bridges, so deduction propagates through value-sharing and adjacency rather than position alone. | | **Induction Reason** | The inducing trace contains a linear `directly_left_of` chain (Elara–Gita–Leo), anchored by Leo in House 3 and extended through cross-attribute links. The pattern adds `same_house` and `next_to` bridges that the source directed-chain skill does not model. | | **Evidence Summary** | The exported stats.json has no populated routing counters (the $\sigma$ fields are zero). The saved refinement evidence contains 178 accepted and 2 rejected records. Every accepted record has `constraints_valid`=1, `schema_valid`=1, `puzzle_unique`=1, and `solution_count_capped`=1. |

**Proposer Construction Resource (r): Declarative Compiler** (`zebra_skill_031`) **RUNTIME SPEC** version=1; `fixed_limit`=1; preferred relation types are `directly_left_of`, `same_house`, and `next_to`. **SEED PROGRAM** (1) `directed_chain`(length=3, `anchor_end`=true); (2) `same_house_chain`(width=2); (3) `component_bridge`(`component_width`=2, relation=`next_to`). **COMPILER GUARDS** prune=true and `ensure_relation_diversity`=true. **LINEAGE** `derived_from`=`zebra_skill_002`.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers