Resource2Skill: Distilling Executable Skills from Human-Created Resources for Software Agents

Yijia Fan, Zonglin Di, Zimo Wen, Yifan Yang, Mingxi Cheng, Qi Dai, Bei Liu, Kai Qiu, Yue Dong, Ji Li, Chong Luo

Resource2Skill distills multimodal human tutorials into a hierarchical, executable skill library for software agents.

How can we automatically distill human-created tutorials and resources into a structured, executable library of skills for software agents?

Software agents often struggle to perform complex authoring tasks because they lack reusable procedural knowledge, relying instead on isolated factual recall or trial-and-error. While human tutorials—like screen-recorded videos—contain the necessary temporal and visual cues, they are too unstructured for agents to use directly. Resource2Skill solves this by distilling these multimodal resources into a hierarchical "Skill Wiki," where each entry bundles structured text, executable code, and visual examples. Agents navigate this taxonomy to retrieve and compose relevant skills, or use the same construction operator to acquire new ones online when they encounter gaps. Across seven authoring domains, this approach improves agent performance by an average of 11.9 percentage points over agents without skill access, consistently outperforming strong baseline harnesses.

Paper Primer

The core mechanism is a unified construction operator that maps raw resources—videos, code, and articles—into a standardized tuple containing metadata, prose, visual thumbnails, and executable code. This design treats skill construction and usage as a single pipeline: the agent uses a hierarchical browse-select-execute interface to narrow the search space before composing the most relevant skills.

Resource2Skill significantly boosts agent performance across diverse software-authoring domains.

The system outperformed no-skill agents by +11.9 percentage points on average and beat strong agentic-harness baselines in 26 of 28 domain-model cells.

Ablations reveal that video is the non-substitutable source of procedural signal, particularly for tasks requiring temporal sequencing. Furthermore, the hierarchical "MetaBrowse" selection strategy consistently outperforms flat retrieval methods (like standard dense or lexical search) by leveraging the domain taxonomy to narrow the candidate pool before the language model selects skills for execution.

Why is a hierarchical wiki better than a flat list of retrieved documents?

A hierarchical structure encodes domain-specific conventions and narrows the search space, allowing the agent to focus on topically relevant subtrees rather than treating the entire library as a flat, undifferentiated list.

What happens if the offline library lacks a skill required for a specific task?

The system uses the same construction operator online to search for new resources, distill them into temporary candidates, and validate them for immediate use, allowing the library to grow incrementally.

For developers building agentic systems, this paper demonstrates that multimodal human tutorials are a high-value, underused asset that can be systematically converted into executable procedural memory.

Introduction and Motivation

We expose why agents need automatically distilled procedural knowledge and outline our proposed solution.

Software agents currently rely on hand‑written or trace‑derived skill libraries, which are text‑centric and ignore the rich procedural signals in tutorial videos and other multimodal resources. This leaves a large portion of human expertise untapped, creating a gap between what agents can do and what humans routinely demonstrate. Resource2Skill addresses this gap by automatically distilling those multimodal resources into reusable, executable skills.

Agents need a scalable way to capture and reuse the step‑by‑step knowledge that humans convey in videos, code snippets, and articles, because hand‑crafted skill collections cannot keep up with the breadth of available expertise.

**Figure 1.** Resource2Skill distills multimodal resources into a hierarchical Skill Wiki across seven creative software domains.

The shift from hand‑written skills to automated distillation unlocks scalable, multimodal procedural knowledge for agents.

The Resource2Skill Pipeline

Four stages turn raw resources into a hierarchical, multimodal Skill Wiki and enable agents to browse, select, and execute skills.

The Resource2Skill pipeline consists of construction, wiki organization, selection, and execution stages, all coordinated through a Model Context Protocol (MCP) that mediates browse‑select‑execute interactions.

Construction: distill raw multimodal resources into structured skill entries.

Wiki organization: store entries in a hierarchical, multimodal Skill Wiki.

Selection: use MetaBrowse to retrieve and compose a subset of skills for a query.

Execution: apply the selected skill code via the MCP interface.

A Skill Wiki is a hierarchical, multimodal repository of executable skill entries that agents can browse and invoke.

Path $p$ places the skill under the “Slide → Title” subtree.

Text view $x_{\text{text}}$ spells out the action in natural language.

Visual view $x_{\text{visual}}$ shows the rendered slide with the new title.

Code view $x_{\text{code}}$ provides a VBA macro that creates the text box.

Metadata $m$ records provenance and versioning.

The hierarchical path $p$ lets MetaBrowse quickly focus on the “Title” subtree, avoiding a flat‑library scan.

How does a Skill Wiki differ from a flat skill library?

In a flat library every entry is a lone item, so retrieval must compare against the whole set. The Skill Wiki groups entries by taxonomy paths, allowing the agent to first narrow to a relevant subtree before examining individual skills, which dramatically reduces search cost.

The Construction Operator $f_{\theta}$ distills raw multimodal resources into structured Skill Wiki entries, enforcing quality via a domain‑specific predicate.

Key frames are extracted (e.g., the moment the chart appears).

OCR reads the on‑screen menu labels.

A language model generates a textual description of the steps.

A VBA macro is synthesized from the visual cues.

The tuple is normalized and passed through $AD$, which verifies that the code runs and that all modalities are present.

The operator fuses visual, textual, and code evidence into a single coherent skill entry, guaranteeing executability before the skill reaches the wiki.

Is the Construction Operator just standard knowledge distillation?

No. Traditional knowledge distillation compresses a teacher model into a student model. Here $f_{\theta}$ actively extracts multimodal evidence from heterogeneous resources, normalizes it into a structured schema, and applies domain‑specific validity checks—operations far beyond simple model compression.

MetaBrowse first narrows candidates with a lexical BM25 scorer that respects the wiki hierarchy, then a language model composes a subset of skills for execution.

BM25 scores each skill using its name, tags, applicability text, and path; the “Slide → Title” subtree receives the highest scores.

$CK(q)$ returns three candidate skills: (a) insert title text, (b) insert logo image, (c) set slide layout.

The language model $\pi_{\phi}$ evaluates the three candidates and selects (a) and (b) as a coherent pair.

The selected code snippets are concatenated and sent to the MCP for execution.

Because the taxonomy path $p(s)$ is part of the BM25 score, MetaBrowse automatically prefers skills that live in the “Title” subtree, aligning retrieval with the user’s intent.

How does METABROWSE differ from a simple nearest‑neighbor retrieval?

Nearest‑neighbor retrieval would compare the query only to flat text embeddings, ignoring the hierarchical organization. METABROWSE augments lexical similarity with the taxonomy path $p(s)$, then lets a language model reason over multimodal views before committing to a subset, providing both structural bias and semantic composability.

**Figure 2.** Resource2Skill pipeline. A construction operator ($f_{\theta}, A_{D}$) distills resources into the hierarchical Skill Wiki; METABROWSE retrieves candidates and the language model selects from text/visual/code views, applied through MCP to a domain backend. The same operator is reused online when the offline pool is insufficient.

Main Empirical Results

Resource2Skill outperforms baselines across seven domains and multiple agent backends.

Web domain overall score reaches 82.4 % with the full skill pool.

Figure 4 shows the scaling curve for the Web domain, where the score climbs to 82.4 % at the FULL skill pool.

An off‑the‑shelf agent interface that runs raw code via a CLI tool without a reusable skill library.

How does an Agentic Harness differ from Resource2Skill’s Skill Wiki?

The harness executes fresh code for every task, while the Skill Wiki supplies pre‑distilled, reusable skill sequences that can be invoked repeatedly, reducing synthesis overhead and improving consistency.

**Figure a.** Skill-pool scaling. The final tick, FULL, is the complete skill pool used by RESOURCE2SKILL; the table lists plotted percentages.

**Table 1.** Main comparison, overall score (%). Avg. is the unweighted mean over all seven domain columns. Bold marks the best system per column within each backend group. Per-cell paired outcome counts and Wilcoxon p-values are tabulated in Appendix G.

Resource2Skill consistently outperforms baseline agentic harnesses across all domains.

Component Ablations

We isolate each component to see how much performance drops when it is removed.

We run a series of controlled ablations to answer the question “does removing this component hurt?”. Each experiment holds everything else constant so the drop isolates the component’s contribution.

**Figure b.** Wiki organization. The full wiki interface is compared with no skills and flat pure-text skill access; the table lists plotted percentages.

Our Wiki beats the flat‑text library by up to 8.2 percentage points, with the largest gains in Excel, Web, and Blender.

Panel b of Figure 3 reports the three arms; the numeric gaps range from 2.5 pp to 8.2 pp.

Removing the Video source family drops the average score from 68.9 % to 59.4 %, a loss of 9.5 pp.

Table 3’s top row (Video out) shows the 59.4 % average; the full‑source row shows 68.9 %.

Adding both Visual and Code fields to the Text baseline yields a combined gain of 3.9 pp over Text‑only.

Table 4 reports Text = 65.0 % and Full = 68.9 %; the incremental contributions of Visual (+1.9 pp) and Code (+2.0 pp) sum to 3.9 pp.

Our hierarchy‑then‑LM MetaBrowse selector outperforms the BM25 baseline by 2.9 pp on average.

Table 5 shows Ours = 68.9 % average versus BM25 ≈ 66.0 % (derived from the per‑domain numbers).

Table 4 details the modality ablation: Text alone reaches 65.0 %, Visual adds +1.9 pp, Code adds +2.0 pp, and the full multimodal entry attains the overall 68.9 % peak.

Table 5 compares six selection strategies; the MetaBrowse hierarchy‑then‑LM policy wins every domain, confirming that combining structural browsing with language‑model reranking captures signals missed by pure BM25 or dense retrieval.

Online Skill Acquisition

Ablation of offline vs. online skill acquisition shows where distilled skills help or fail.

The system first queries a fixed offline Skill Wiki; if no suitable skill is found, it falls back to an online search that can add up to $100$ freshly distilled skills for the current request.

How does this online acquisition differ from standard on‑the‑fly retrieval?

Standard on‑the‑fly retrieval simply fetches existing knowledge at inference time, whereas our online arm actually distills new procedural skills within a strict $100$‑skill budget and stores them in an isolated pool that does not alter the baseline wiki.

We ask whether the online acquisition component yields a measurable benefit over a purely offline Skill Wiki. By fixing the online budget, the ablation isolates the coverage gain attributable to freshly distilled skills.

Qualitative case studies reveal two recurring failure modes. First, partial grounding occurs when a retrieved skill’s surface pattern is used but key parameters remain unresolved (e.g., Web tabletop, Excel #NAME?, Blender perfume). Second, conservative composition happens when the skill arm sticks to a single pattern, losing the variation a less‑anchored agent would have introduced (e.g., PPT placeholder, Reaper less‑dynamic).

**Table 2.** Offline and online skill acquisition, overall score (%). $T_{standard}$ is the regular benchmark; $T_{novel}$ targets capabilities missing from the offline pool. Bold marks the better configuration within each task set.

**Figure 3.** Web success. w SKILLS (left) vs. w/o SKILLS (right).

**Figure 4.** Web failure. w SKILLS (left) vs. W/O SKILLS (right).

**Figure 5.** PPT success. w SKILLS (left) vs. w/o SKILLS (right).

**Figure 6.** PPT failure. w SKILLS (left) vs. w/o SKILLS (right).

**Figure 7** Excel success. w SKILLS (left) vs. w/o SKILLS (right).

**Figure 8.** Excel failure. w SKILLS (left) vs. w/o SKILLS (right).

**Figure 9.** Blender success. w SKILLS (left) vs. w/o SKILLS (right).

**Figure 10** Blender failure. w SKILLS (left) vs. w/o SKILLS (right).

**Figure 11** Reaper success. Spectrogram and waveform of w SKILLS (top) vs. w/o SKILLS (bottom).

**Figure 12** Reaper failure. Spectrogram and waveform of w SKILLS (top) vs. w/o SKILLS (bottom).

Harness Baseline Configurations

This appendix details the baseline harness configurations, the construction operator’s components, and the deterministic acceptance checks.

Both baseline harnesses—ClaudeCode‑H and Codex‑H—are off‑the‑shelf CLIs that we run without any custom tooling or Skill Wiki integration, preserving their default planning and tool‑use loops for a fair practitioner‑level comparison.

Reproducibility is ensured by publishing the exact per‑brief invocation contracts, raw stdout/stderr traces, agent logs, and JSON score files that allow regeneration of every cell in Table 1 without re‑running any agent.

The operator turns a raw multimodal resource into a structured JSON skill entry by issuing a single LM call and then applying a fixed series of deterministic clean‑up steps.

$AD$ filters candidate skills through five independent deterministic checks, discarding any entry that fails any single gate.

Run logs for each (brief, condition) pair contain the BM25 shortlist (top $K\!=\!20$), the LM selector’s transcript with the $n\!=\!5$ chosen skill IDs and justifications, the execution trace of invoked skills, per‑call tool success flags, and the final scored artifact with judge JSON.

Because the library, agent, judge, decoding seed, and candidate budget are held constant across selection strategies, any drop in the overall score directly reflects poorer retrieval or composition; this is evident in the lower scores of the retrieval‑only baselines (Embed 60 %, BM25 + Embed 64.2 %, Random‑FullPool 58 %) versus our full system (68.9 %).

Composition is a single ordered application of up to $n\!=\!5$ selected skills; no multi‑pass replanning occurs, so conflicts appear as tool‑level “not‑applicable” returns or as degraded artifact scores, both of which are recorded in the run directory.

Judge reliability was measured on 17 task–artifact pairs; Spearman $\rho$ between the automatic judge and the median of three human raters was 0.71 and ICC(2,1) was 0.66, with the utility axis showing the greatest human‑judge disagreement.

Repeating the judge on the same 17 pairs yielded $\rho\!=\!0.83$ between runs, indicating that judge variance is substantially lower than the variance between judge and human ratings.

Limitations and Schema

Limits of the Skill Wiki approach and its evaluation are outlined.

The Skill Wiki is stored as a simple directory tree — `skills_wiki`/<domain>/<`skill_id`>/ — containing subfolders source/, text/, visual/, code/ and a meta.json file. The metadata records the skill identifier, name, domain, category path, tags, applicability, source type, provenance, and validation status, mirroring the semantic sections of the main paper. This layout implements the abstract tuple $s = (p, x_{\text{text}}, x_{\text{visual}}, x_{\text{code}}, m)$ introduced in Section 3.1.

Scores are obtained via $GPT\text{-}5.4$ vision on rendered artifacts, with judge‑human agreement on the 17‑task subsample (Appendix F) and a blinded human A/B study (five raters per pair across seven domains, Appendix K) confirming the reported preference direction. The authors do not claim the system generalizes to domains that lack a programmatic tool interface or a public stream of procedural content. Online acquisition introduces additional search, distillation, and validation latency; it is evaluated only on a fixed online pool against Tnovel to isolate coverage gains, not to conflate library‑scaling effects with uncontrolled test‑time context expansion. Retrieval‑style baselines (Embed, BM25+Embed) operate over the distilled Skill Wiki rather than the raw resource corpus, and a same‑budget raw‑resource comparison is left for future work.

Human Preference Study

Human evaluators prefer Skill‑enhanced artifacts, confirming the automatic judge.

Human participants chose artifacts generated with the Skill Wiki over those without it in 85.5 % of non‑tie comparisons.

Across 200 blinded pairwise judgments, 136 votes favored with‑Skills, 23 favored without, and 41 were ties; excluding ties the win rate is 85.5 %.

The per‑domain win rates mirror the ranking produced by the automatic judge’s overall‑score deltas, reinforcing its reliability as the primary evaluation signal.

**Table 8.** Per-domain human A/B study against w/o Skills. Five raters per pair on anonymized side-by-side renderings. Aggregation is a micro-average over 200 individual votes. Win rate excl. ties excludes ties from the denominator.

Implementation Details

Appendix details domain backends, skill entry sizes, and inference token budgets.

The conclusion reports that Resource2Skill raises artifact quality by +11.9 points over no‑skill agents and outperforms two agentic‑harness baselines in 26 of 28 aggregate cells.

Seven authoring domains are evaluated: Web, Excel, PowerPoint (PPT), Blender, Reaper, CAD, and UE5; ablations focus on the core five (Web, Excel, Reaper, PPT, Blender) while CAD and UE5 remain held‑out.

Each skill entry bundles front‑matter, prose, and executable code; token estimates use a conservative four‑character‑per‑token rule applied to the concatenated text.

At inference, MetaBrowse builds a BM25 shortlist of $K = 20$ candidates, then selects $n = 5$ entries to expose fully; the selection step consumes roughly 4–10 k tokens, while the five full entries add about 22 k tokens, yielding a total context of ≈ 26–32 k tokens.

Benchmark Construction Protocol

This appendix details the benchmark construction, run orchestration, rendering, scoring, and reproducibility procedures.

Benchmark items are task briefs that specify a domain, a stable `brief_id`, a slug, a natural‑language request, and the required output path and artifact type. They contain no skill identifiers, source URLs, or solution steps.

Candidate briefs are produced from domain‑specific prompt templates with fixed random seeds, then manually screened for clarity, feasibility on the domain backend, and non‑overlap with the resource corpora used to build the Skill Wiki. After screening, each domain retains a pool of $80$ briefs.

The main comparison, scaling study, and online/offline study all use a fixed matched subset of $N=80$ briefs. Ablation experiments use separate fixed subsets of $N=40$ sampled from the same $80$‑brief pool to stay within the run budget.

The online/offline study adds a stress‑test set $T_{\text{novel}}$, drawn from capability regions that the offline pool fails to cover during preliminary screening.

A run is one cell in the Cartesian product of domain, brief, and condition. All runs share the same agent loop and the same Model Context Protocol (MCP) server; only the skill‑access policy, wiki root, pool whitelist, source set, entry format, or harness baseline differ between conditions.

Each run follows a four‑stage loop: (1) plan the task, (2) browse or retrieve candidate skills when permitted, (3) execute the plan through domain‑specific tools, and (4) verify the result by rendering the artifact.

Artifacts are rendered before scoring: Web pages via Playwright screenshots, Excel workbooks via LibreOffice‑generated contact sheets, Blender scenes via final PNG or headless hero‑shot, PPT decks via video/frame capture (or static slide sheets), Reaper projects to WAV then MP3, and CAD/UE5 outputs following the same artifact‑first rule.

Non‑audio artifacts are judged by GPT‑5.4 with low reasoning effort; audio artifacts are judged by a GPT‑4o‑series model. Judges see only the brief, the rendered artifact, and the domain rubric, and they assign integer scores $0\!-\!10$ on five axes, which are multiplied by $10$ for reporting.

The overall metric is the unweighted arithmetic mean of the five axis scores. Any missing, unopenable, too‑small, or silent artifact receives an overall score of $0$ and is counted in the denominator when computing condition means.

For matched comparisons, scores are aggregated over the same brief IDs within each domain and condition, guaranteeing paired deltas by construction. Each score JSON records artifact path, render path, rubric breakdown, overall score, and judge metadata to enable later re‑aggregation without rerunning agents.

Brief generation is wiki‑blind: the generator receives only the domain name, a hand‑curated per‑domain guidance string enumerating artifact types, and a fixed structural template. No skill identifiers, wiki listings, taxonomy data, or prior benchmark statistics are provided.

Resource‑collection queries are driven solely by per‑domain category lists in the hand‑designed taxonomy; they do not consume brief text, slugs, or any benchmark‑side identifier.

The acceptance predicate (AD) thresholds—minimum schema fields, deduplication hash policy, executability gate, and per‑domain size constants—were tuned on a small development sample drawn before brief generation and frozen before any benchmark cell was scored.

Benchmark results (failure rates, scores, and qualitative error analyses) are stored alongside run artifacts but are never fed back into the brief generator, resource collector, taxonomy, or acceptance predicate; the library snapshot used for all studies remains unchanged.

Statistical Outcome Counts

Outcome tables quantify the advantage of w Skills over w/o Skills across domains.

Table 6 reports per‑cell win/tie counts and paired Wilcoxon signed‑rank $p$‑values for the $w\,$Skills versus $w/o\,$Skills comparison on $N=80$ matched briefs per cell.

**Table 6.** Per-cell paired outcome counts and paired Wilcoxon $p$-values for the w SKILLS vs. w/o SKILLS delta on matched $N=80$ briefs per cell.

Table 7 breaks down the GPT‑5.4 comparison per evaluation axis, reporting raw scores (multiplied by 10) and the matched percentage‑point difference $\Delta$ for each domain.

**Table 7.** Per-axis details for the GPT-5.4 main comparison (%). $\Delta$ is the matched difference of w SKILLS over w/o SKILLS.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers