Demystifying Agent Skills: Why They Work—Until They Don’t

Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao, Rui Cao, Mengdi Wang, Shilong Liu, Yijiang Li

Skills stabilize agent execution by acting as procedural anchors rather than just injecting missing facts.

Why do LLM agent skills sometimes improve task success and sometimes degrade it, and what specific mechanisms drive this variance?

LLM agents often struggle to reuse prior experience, frequently rediscovering the same procedural details across tasks. Existing evaluations treat skills as black boxes, measuring only aggregate success without explaining why they help or where they fail. The authors use a contrastive trajectory-analysis methodology to compare agent performance with and without skill access. They find that skills function primarily as procedural anchors that stabilize execution, rather than as simple knowledge injections. Skills improve task success over raw execution and direct workflow memory, particularly by reducing operational failures like environment setup errors and command corruption.

Paper Primer

The paper addresses the "black box" nature of agent skills by decomposing the skill-use pipeline into representation, retrieval, and invocation. By normalizing 8,135 trial records and performing paired trajectory analysis, the authors map how skills transform noisy prior experience into standardized procedural artifacts.

Skills are procedural anchors: they provide a stable template for setup, tool sequences, and verification checks. This mechanism is distinct from knowledge injection, which accounts for only a small fraction of skill utility.

Skills stabilize agent execution by reducing operational fragility.

Skill-augmented agents show a significant reduction in execution-layer failures, such as environment setup errors and output-format mismatches, compared to raw execution. Environment-infrastructure failures dropped from 5.3% in raw execution to 0.2% with skill usage.

Exact ground-truth skill retrieval is neither sufficient nor necessary for task success.

As skill pools grow from 5 to 100, actual-use precision for ground-truth skills collapses, yet downstream task success remains stable. Actual-use precision falls from 29.6% to 3.3% in some configurations, while success rates remain comparatively flat.

Why do skills outperform direct workflow memory?

Workflow memory often preserves verbose process noise, failed branches, and irrelevant exploration from past runs. Skills distill this experience into compact, standardized procedural guidance, which reduces agent timeout and drift.

What is the primary failure mode for skills?

Skills fail when they are misapplied or followed too rigidly. Agents may retrieve a plausible skill but fail to adapt it to the current context, or apply it mechanically when it is no longer applicable.

The authors emphasize that skill-based self-improvement is a lifecycle problem. Success depends on the agent's ability to generate, retrieve, and—crucially—adapt procedural abstractions to the specific requirements of the current task.

Researchers should shift from optimizing aggregate success rates to analyzing the procedural mechanisms of skill use. Future agent design must prioritize the agent's ability to judge the applicability of retrieved skills rather than just increasing retrieval accuracy.

Abstract

A concise overview of when LLM agent skills succeed and why they fail.

The authors examine why injected LLM agent skills boost performance in some trajectories but break down when the skill‑retrieval mechanism or source trajectory composition is misaligned.

The Mechanics of Agent Skills

We expose why skills help, when they fail, and how trajectory mixtures shape their impact.

LLM agents increasingly rely on reusable “skills” to boost performance, yet existing evaluations only ask whether a skill improves aggregate success.

Skills are not a uniform plug‑in; their benefit swings dramatically with the mix of successful and failed source trajectories and with how the agent retrieves them.

The “Raw” condition gives the agent no prior procedural aid; it must discover everything from scratch on each task.

We categorize every observed skill interaction into three high‑level categories (SC1–SC3) and twelve concrete modes, capturing both successes and failure mechanisms.

**Figure 1.** **Experimental pipelines.** Top: skill versus procedural memory. We execute each task in a fixed Docker environment, collect successful and failed trajectories, and form a fixed-budget composition grid. The same trace pool is distilled either into Workflow Memory or into a reusable SKILL.md, which are then evaluated on matched tasks under the same protocol. Bottom: skill retrieval. Each task is paired with a candidate pool containing its ground-truth skill and $k-1$ real distractors (random, similar, or dissimilar). Matched pools are evaluated independently in three procedures: (A) embedding-based ranking without task execution, (B) explicit agent selection without Docker execution or verification, and (C) full-pool real execution with skill-use parsing after task verification. Outputs from (A) and (B) are not passed to (C).

**Figure 2.** Taxonomy label distribution across trajectory mixtures and experimental arms. Stacked bars show trajectory-level labels for Raw, Workflow Memory, and Skill across the six source-trajectory mixtures from $5s0f$ to $0s5f$. Labels are grouped into three high-level categories; per-mode percentages are reported in Appendix Table 11.

Skills are not universally beneficial; their impact is contingent on the source trajectory mixture.

Skill Construction and Prompting

Defines the prompt templates that turn raw traces into reusable skills and structured analyses.

This section introduces the family of prompts that the paper uses to extract, label, and aggregate agent skills from execution traces.

A structured template that asks a language model to synthesize a reusable skill file from one or more execution traces.

Detect the repeatable process: copying a file.

List the single step “run `cp $SOURCE $DEST`”.

Identify the tool: the Unix `cp` command.

Record the failure mode: permission errors, mitigated by checking write permissions before copying.

Fill the markdown template, e.g. name: `file_copy`, description: copy a file safely.

The example shows how the prompt turns raw command logs into a reusable, self‑documenting skill.

How does the Skill Creator Prompt differ from simply asking the model to “summarize the traces”?

Summarization yields free‑form text without a fixed structure, so downstream tools cannot reliably extract preconditions, steps, or failure modes. The prompt’s markdown sections enforce a deterministic schema that can be parsed and executed automatically.

A variant of the Skill Creator Prompt that omits explicit hints about failure signals, requiring the model to infer them from the traces alone.

Detect the repeatable process: load‑process‑save.

Infer the failure signal “missing input file” from the exception text.

Generate a skill file that includes a precondition “input file exists” and a step “handle FileNotFoundError by aborting early”.

The model must extract the failure mode without being told to look for it, demonstrating the prompt’s stricter requirement.

Why might the No‑Hint version be harder for the model to satisfy?

Because the model cannot rely on a prescribed “failure modes” section; it must detect and articulate errors from raw logs, which demands deeper reasoning about the trace semantics.

A JSON‑based prompt that asks the model to explain why a trial succeeded or failed, linking outcomes to the injected skill or workflow.

Identify that the outcome is success, so `primary_mode_candidate` = “`clean_execution`”.

Since no skill is present, set `skill_effect_judgment` = “`not_applicable`”.

`Evidence_span` pulls the line “All tests passed” from the codex output.

This illustrates how the prompt maps a simple successful run to a concise, machine‑readable label.

What is the difference between the `primary_mode_candidate` and `secondary_factors` fields?

`primary_mode_candidate` captures the dominant cause of the outcome (a single noun phrase), while `secondary_factors` lists additional, less decisive contributors that may have influenced the result.

A prompt that takes a batch of labeled trajectories and synthesizes a compact set of canonical failure or success modes.

Merge the two “`missing_python_dependency`” entries into a single mode.

Define three modes total: `missing_python_dependency`, `api_misuse`, `clean_execution`.

Assign each id to its mode with a one‑sentence reason (e.g., “dependency error prevented import”).

The example shows how the prompt consolidates raw labels into a manageable taxonomy.

How does the prompt avoid creating overly fine‑grained modes?

It explicitly asks for 8–14 modes and requires each mode to cover multiple input ids, encouraging the model to merge closely related labels rather than keeping every nuance separate.

A prompt that unifies multiple batch‑level mode taxonomies into a single canonical set.

Recognize that all three refer to the same underlying problem.

Create a unified mode named “`missing_python_dependency`”.

Map each original batch mode to the unified name in the `batch_mode_map`.

This demonstrates the prompt’s ability to collapse synonymous entries while retaining traceability.

What risk does merging modes pose, and how does the prompt mitigate it?

Over‑merging can hide distinct failure patterns; the prompt mitigates this by requiring a concise definition for each unified mode and by preserving the list of original batch modes that contributed to it.

A JSON‑driven prompt that compares three agent trajectories (raw, workflow‑injected, skill‑injected) on the same task and reports differences.

Assign raw mode = “timeout”, workflow mode = “`clean_execution`”, skill mode = “`missing_dependency`”.

Compute `workflow_vs_raw` delta: `what_changed` = “added workflow script”, `net_effect` = “fixed”, `fixed_mode` = [“timeout”], `introduced_mode` = [] .

Compute `skill_vs_raw` delta: `what_changed` = “injected skill”, `net_effect` = “regressed”, `fixed_mode` = [], `introduced_mode` = [“`missing_dependency`”].

Set `skill_mechanism` = “`knowledge_injection`” (skill provided knowledge but was missing a dependency), `workflow_mechanism` = “`procedural_anchor`”.

Assign confidence = “high”.

The example shows how the prompt captures both quantitative changes and qualitative mechanisms.

How does the prompt represent arms that are absent in the data?

If an arm is missing, its entry is set to null and any delta involving that arm receives `net_effect` = “`not_comparable`”, preserving the integrity of the comparison without fabricating results.

These prompts together form the “lens” through which the paper evaluates skill creation, labeling, aggregation, and comparative impact.

Ablation and Retrieval Analysis

Quantitative ablations reveal how retrieval pool composition and outcome annotation affect downstream success.

This section reports the full ablation suite: retrieval‑pool effects (Table 14), selection vs. execution performance (Table 15), and the impact of exposing or hiding source‑trajectory outcomes (Table 16).

During skill construction the system records whether the originating trajectory succeeded, and later uses that label to bias skill selection.

How does “no‑hint” differ from the standard “normal” setting?

In “normal” the agent sees a binary success flag attached to each skill, enabling it to filter out skills that historically failed. “No‑hint” strips that flag, so the agent must decide based only on the skill’s embedding, which removes a useful supervisory signal.

**Figure b.** Arm 3 skill-use precision and downstream success.

Exposing source‑trajectory outcomes (“normal”) improves downstream success by up to 30 % over the “no‑hint” condition across all trajectory mixtures.

Table 16 reports success rates such as 0.7548 (normal) vs. 0.4231 (no‑hint) for the 5s0f mixture on TB2, a gap of 0.3317 ≈ 30 % absolute.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers