Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
Yuqi Huang, Hui Li, Yifei Cao, Zikun Cai, Tianci Xie, Yangfan Gao, Pinxue Guo, Chaoyun Zhang, Bingxuan Li, Rui Zheng, Hang Yan, Ning Liao
Work2Skill captures human GUI demonstrations into reusable procedural skills via agentic evidence inspection.
How can we convert raw human GUI demonstrations into reusable, model-agnostic procedural skills for GUI agents?
Computer-use agents struggle with unfamiliar workflows because they lack the domain-specific procedural knowledge—such as internal portal navigation or required receipt checks—that humans naturally apply. Work2Skill addresses this by capturing human GUI demonstrations and using an induction agent to iteratively inspect synchronized video, accessibility trees, and input logs to write reusable procedural skills. On standard benchmarks, this approach consistently outperforms baseline methods, improving mean evaluation scores by up to 10.54 points and increasing success rates across multiple execution models.
Paper Primer
The framework consists of three parts: WorkRecorder, which aligns multimodal evidence (video, mouse/keyboard events, and accessibility trees) into a common timeline; WorkDemo-350, a curated dataset of 350 demonstrations; and an agentic induction method that interleaves evidence inspection with skill writing. The induction agent acts as a researcher: it samples video and structured UI data on-demand to resolve ambiguities, then generalizes the demonstration into a reusable skill file that captures dependencies and completion checks without needing the original recording.
Work2Skill significantly improves downstream GUI task success compared to No Skill and existing demonstration-based baselines.
Evaluations on MyPCBench Bounded Action and OSWorld-Verified tasks across four execution models (GPT-5.5, Opus-4.8, and two Qwen variants). Mean score improvements of 4.67–10.54 points on MyPCBench and 3.11–7.50 points on OSWorld-Verified.
Induced skills generalize to new task instances and transfer across different execution models.
Testing on MyPCBench Jitter (tasks with changed parameters/constraints) and cross-model induction-execution pairings. Score improvements of 3.93–8.64 points on Jitter tasks; skills remain effective even when induced by one model and executed by another.
Why is an agentic, on-demand inspection approach better than simply feeding the entire recording into the model?
Loading long recordings creates context pressure and risks omitting critical details. By interleaving inspection with skill writing, the agent can revisit specific timestamps to verify transitions or clarify ambiguous clicks only when needed, leading to more robust procedural guidance.
What is the scope of this method — does it work for any desktop application?
The method relies on the availability of accessibility-tree (AX) information and visual grounding. While it supports both local macOS and remote Linux workflows, its effectiveness depends on the application's accessibility support and the quality of the captured multimodal evidence.
Work2Skill demonstrates that reusable agent skills are best induced by treating human demonstrations as an interactive evidence resource rather than a static dataset, allowing agents to extract procedural rules that survive changes in task parameters.
Introduction to Work2Skill
Introducing Work2Skill, a framework that turns human GUI demos into reusable, model‑agnostic skills.
Current GUI agents excel at low‑level interaction but stumble when a task requires domain‑specific procedures that are not spelled out in the user request. Human demonstrations contain exactly the missing procedural knowledge because they operate in the same GUI space, yet extracting reusable rules from raw recordings is non‑trivial. Work2Skill bridges this gap by recording, aligning, and inductively converting demonstrations into reusable skills that any agent can invoke.
It treats a human demonstration as raw evidence, aligns it with structured UI metadata, and then distills a reusable skill that any agent can call without re‑watching the whole video.
It aligns low‑level interaction logs with visual frames and AX metadata so that each recorded action is anchored to a concrete UI element.
**Figure 1.** Work2Skill framework: WorkRecorder, WorkDemo-350, and Agentic Skill Induction.
**Figure 2.** Overview of Work2Skill. WorkRecorder aligns human GUI actions with visual and available accessibility evidence to build WorkDemo-350. An agent inspects evidence on demand and revises reusable skills for downstream GUI execution. Downstream evaluation compares these skills with guidance from the fixed workflows of ShowUI-Aloha and Demo2Tutorial.
The key shift is from directly copying demonstrations to an agentic skill‑induction process that queries evidence on demand.
Related Work and Data Collection
Contextualizes prior GUI learning efforts and introduces the WorkRecorder data pipeline.
Prior work has shown that human GUI demonstrations can supply rich supervision for agents across desktop, web, and mobile platforms, enabling data collection, grounding, and model training.
Separate lines of research focus on skill induction and active evidence acquisition, where reusable procedural guidance is distilled from demonstrations or from autonomous interaction with the environment.
**Figure 3.** Human GUI experience collection with WorkRecorder. A demonstrated workflow (left) is captured and reviewed through the recorder interface (center), then organized into an evidence episode with task metadata, action traces, and available multimodal evidence (right).
The WorkDemo-350 Benchmark
WorkDemo-350 supplies synchronized multimodal recordings of GUI tasks for skill induction.
Existing GUI agents lack procedural knowledge, making complex tasks hard to automate. A shared repository of human demonstrations can provide the missing procedural evidence.
It is a curated set of 350 human‑recorded GUI episodes, each pairing raw video with synchronized action and UI metadata, giving agents a concrete evidence episode to imitate.
Episode 1: timestamps [0 s, 5 s, 10 s]; mouse clicks [click A, click B, click C].
Episode 2: timestamps [0 s, 4 s, 9 s]; mouse clicks [click D, click E, click F].
Alignment: at 5 s in Episode 1 the video frame shows the same UI state as the click B event, enabling a direct mapping between visual context and action.
Aggregating across episodes yields a lookup table of (timestamp → UI element → action) triples.
Even with just a handful of synchronized timestamps and clicks, an agent can infer the procedural relationship between visual context and the required interaction.
The table provides a statistical comparison between two benchmarks: MyPCBench and OSWorld-Verified.
With these synchronized multimodal recordings, agents can query any modality at any point, enabling robust skill induction across diverse GUI environments.
Agentic Skill Induction Mechanism
Agentic Skill Induction converts evidence episodes into reusable procedural skills.
Skill induction bridges raw demonstration recordings and reusable procedural guidance for GUI agents. It leverages the NexAU framework to inspect evidence and iteratively author skill files.
Think of a chef watching a cooking video, noting each ingredient and step, then writing a clean recipe that anyone can follow without re‑watching the video.
How does this induction differ from a simple demo‑to‑code transcription?
Transcription copies low‑level UI events verbatim, while induction abstracts the demonstration into a high‑level procedural description, generalizes inputs, and adds explicit completion and recovery checks, making the resulting skill reusable across contexts.
Load the evidence episode: read task.json, logs, screenshots, and video.
Inspect initial evidence to identify the task goal and relevant UI controls.
Draft a
Validate the draft against the evidence (check that all controls and states are covered).
If validation fails, return to the specific evidence fragment and revise the skill.
When all checks pass, finalize the skill file and record provenance.
The agent reads
It opens
It samples a 0.5‑second segment from
Using this information, the agent writes a
It runs the built‑in validation: the screenshot and video frame both show the button, so the skill passes all checks.
Even with minimal, complementary evidence the agent can synthesize a complete, reusable skill by stitching together file, image, and video cues.
Performance Evaluation
We evaluate induced skills against baselines, showing top scores and success rates.
Recall that Work2Skill records human demonstrations and distills them into reusable procedural skills, addressing the lack of procedural knowledge in GUI agents. This section quantifies how well those induced skills perform compared with established baselines.
This baseline runs the GUI task without any learned skill, relying solely on the agent’s generic prompting ability.
ShowUI‑Aloha adapts the original ShowUI approach to text‑only interaction, feeding the agent a recorded UI trace as a prompt.
D2T (Demonstration‑to‑Text) converts a UI demonstration into a concise textual recipe that the agent follows.
Work2Skill attains the highest Score and Success Rate (SR) among all methods across both benchmarks and all four executors.
Table 2 shows Work2Skill leading in every model column for both MyPCBench and OSWorld‑Verified, with bolded scores and SR values.
**Table.** Performance comparison of Work2Skill against demonstration-based baselines for each execution model.
**Figure 4.** Cross-model skill transfer on 64 MyPCBench tasks. Rows denote induction models; columns denote executors. $\Delta$ denotes the gain over the same executor's No Skill baseline, in Score points or SR percentage points, calculated from displayed two-decimal values.
Generalization to New Tasks
Work2Skill generalizes to new task variants, boosting scores and success rates across models.
Work2Skill retains its advantage on jittered tasks, raising average success‑rate by up to +9.38 % over the No Skill baseline.
Table 3 shows consistent gains across four execution models; Figure 5 confirms the advantage persists under multiple ablation settings.
MyPCBench Jitter provides a controlled test of skill reuse: each task variant changes only the destination, date, or other parameters, leaving the underlying UI interaction unchanged. GPT‑5.6‑sol rewrites the task instructions to match the new requirements, and human annotators verify correctness (see Appendix E.1).
**Table 3.** In Table 3, Work2Skill improves the average score over No Skill by 3.93–8.64 points and the success rate by 6.25–9.38 percentage points. Compared with the stronger demonstration-based baseline for each executor, the Score advantage is 4.28–6.25 points. Work2Skill also achieves the highest success rate for three executors, while tying D2T for Qwen3.5-9B. Average execution steps also decrease by 2.49–5.44 relative to No Skill across the four execution models. These gains across all four
**Figure 5.** Ablation and pipeline-prompt comparison on 64 MyPCBench tasks using GPT-5.5 with high reasoning effort. In (a), Full Evidence uses all available modalities; UI-Grounded Action Text Only combines recorded actions with target UI descriptions. w/o KeyLogs also removes overlays from indexed images. Score, SR, and Steps measure downstream execution; Tokens reports mean skill-induction token usage per task in units of $10^4$.
Transferability and Ablation Studies
We quantify how skill transfer, evidence, and prompt design affect downstream performance.
We now examine two ablation questions: whether induced skills transfer across models, and how evidence and prompt design shape skill utility.
Induced skills transfer across models, with GPT‑5.5 induction yielding up to +9.46 score points over the No‑Skill baseline.
All four induction‑execution pairings improve on the No‑Skill baseline; the GPT‑5.5 + GPT‑5.5 pairing achieves a 9.46‑point score gain and a 9.38 pp success‑rate increase.
Work2Skill prompt raises the task score by +3.97 points compared to the Pipeline Prompt.
Using identical evidence, tools, and authoring policy, the Work2Skill prompt attains a 3.97‑point higher score.
Work2Skill prompt boosts success rate by +6.25 percentage points over the Pipeline Prompt.
The same experimental setup shows a 6.25 pp increase in success rate.
Work2Skill incurs +12.7 % higher induction token usage than the Pipeline Prompt.
Average token count per skill induction rises by 12.7 % under the Work2Skill prompt.
Work2Skill reduces average execution steps from 48.98 to 47.38, a drop of -1.6 steps.
Measured across 64 tasks, the average steps decrease by 1.6.
**Figure 6.** Execution trajectories (top) and rubric outcomes (bottom) for a TableFind reservation with fallbacks, using GPT-5.5 with each method's extracted guidance. Both baselines leave fallback handling and persistence checks incomplete (55/100); Work2Skill satisfies all eight requirements (100/100). Appendix F.2 provides the detailed rubric and execution analysis.
Case Study and Induction Example
Limits of the generated skill and the evidence supporting its design.
Recall that Work2Skill records human demonstrations and then induces reusable procedural skills, addressing the lack of procedural knowledge in GUI agents.
F.1 presents the raw induction trace for the “MyPCBench Bounded Action” reservation task (hard app‑f016). The GPT‑5.5 agent runs with high reasoning effort, makes 30 tool calls, inspects 22 evidence images in two batches, and samples a 156.47‑second video in two calls.
**Figure 7.** A paired SprintBoard task and its evaluation rubric before (left) and after (right) Jitter perturbation. The variant changes the target board and the required card transition, with rubric items revised to match the new requirements.
F.2 analyzes execution trajectories (Figure 6 in the main text) and highlights where baseline agents miss secondary subgoals: they fail to inspect all three fallback restaurants (R3–R4), to favorite the top two (R5), and to preserve state after a page refresh (R7). Their overall rubric score is 55 / 100.
**Figure 8.** Category distribution of the 286 retained OSWorld-Verified tasks. The legend gives the task count and percentage for each category.
**Table 8.** Our skill-induction settings.
**Table 9.** Our downstream-execution settings.
**Table 10.** Evidence and visual tools available for skill induction in each modality condition. Task instructions and common file-reading and skill-authoring tools remain available. UI annotations denote metadata and descriptions derived from each action's target element in the AX tree.
F.3 reproduces the complete generated skill in Markdown form. The procedure explicitly carries the reservation date, time, and party size across primary and fallback searches, checks for field resets, excludes the primary restaurant from the fallback set, and defers final confirmation until after the fallback steps.
Limitations: the induction trace is a single illustrative run and does not include a downstream execution score; the skill assumes the GUI layout remains stable (e.g., the “Search” link and “Favorite” toggle are present); and the high‑effort configuration may be infeasible for real‑time agents.
Recorder Implementation Details
Details of the WorkRecorder implementation, keyframe policy, and evidence alignment.
WorkRecorder comprises a macOS screen‑capture app, a local coordination service, and optional Linux accessibility providers.
On macOS the app records a user‑selected region as H.264 video and taps CoreGraphics for mouse/keyboard events, optionally aggregating typing into semantic operations.
For remote Linux sessions the AT‑SPI collector runs inside the target desktop, streams accessibility trees over an SSH tunnel, and maps remote pointer coordinates to the Linux screen.
Accessibility evidence is sampled every two seconds, limited to six tree levels and 400 nodes; each snapshot provides element names, roles, and bounds for later click resolution.
When a click is resolved, candidate nodes are ranked by confidence, interactive role, and an area penalty that prefers smaller controls, with the chosen node recorded alongside its evidence.
Keyframe selection centers on operation boundaries (start/stop/pause/resume) and explicit user marks; additional triggers include clicks, drag releases, scroll completions, hotkeys, command submissions, and window/tab switches.
For actions like clicks or hotkeys the recorder may reuse a preceding keyframe as before‑action context, provided it occurred within 1.5 s; otherwise no context is attached.
Result frames wait for interface stability, defined as two consecutive stable samples or an adaptive provider signal, with a default timeout that either captures the best‑available state or suppresses the frame.
Duplicate triggers are collapsed, a minimum 500 ms gap separates plans, and at most two result‑related keyframes are kept per action, respecting a session budget of 64 keyframes.
Time alignment maps native input timestamps (monotonic) to video frames using the first‑frame anchor; remote Linux events use wall‑clock timestamps projected onto the same timeline, with pauses omitted.
The Linux launcher aborts recording if host‑clock drift exceeds two seconds, a deployment safeguard rather than a measured alignment error.
The exported evidence schema links each recorded probe result to its triggering action via monotonic timestamps, with a 2.5 s nearest‑action fallback and a 3 s backfill window for missing targets.
Episode File Structure
Details of episode files, action logs, and evidence counts used for skill induction.
The induction agent reads a per‑task episode directory that holds the task definition and all recorded evidence. The benchmark converter always creates three core files: task.json, episode.json, and trace/actions.jsonl. Optional logs, video, summaries, and images are added when available (see Table 4).
task.json must contain the authoritative instruction and task inputs; a placeholder containing only an inferred task ID does not suffice for induction. The converter deliberately omits the grading field from this file.
episode.json serves as a manifest: it records the task ID, episode duration, counts of evidence items, and indexes the files that are present. Its source entry records the original export location for provenance, but this location is not added as an extra input root for the agent.
**Table 4.** Files exposed to the induction agent in a converted episode. The first three files form the core structure; the remaining artifacts are included when available.
Each line of trace/actions.jsonl records a single action with fields such as `action_id`, `sequence_index`, kind, `media_time_ms`, caption, and `target_label`. The first action of the hard app‑f016 episode looks like:
{ "`sequence_index`": 0, "`action_id`": "`act_0JYEZK480Ta_81AR`", "kind": "click", "`media_time_ms`": 3908, "caption": "Left Click push button - Calendar", "`target_label`": "push button - Calendar" }
When present, evidence/manifest.json and evidence/`image_actions`.jsonl map images to actions. In this episode the first image‑index record references the same action ID, with a frame time of 3908 ms and an action‑delta of 0 ms, indicating an exact temporal match. A non‑zero delta would signal an approximate match.
Event frames capture the interface state near the input event; verifying the effect of an action may require inspecting a subsequent frame.
The schema does not guarantee that every modality (video, keylog, images) is present for every demonstration. In Linux mode the continuous tree sequence lives in the Linux evidence directory, while the Mac export can preserve snapshot images.
The induction agent can only inspect the evidence that resides in its exposed input directory. Table 1 (referenced in the main text) reports average counts per episode: the hard app‑f016 episode contains 37 action records, 23 evidence images, 176 input events, and 29 recorded keyframes.
Image export follows specific grouping rules: double‑clicks are merged, adjacent typing actions are combined into a single image with the last frame, and scroll records do not generate images. The converter writes counts.keyframes from the recording’s keyframe count (or falls back to loaded records) and counts.`evidence_images` from the exported images. Exporting all keyframes creates an additional evidence/`all_keyframes`/ directory with its own manifest.
Induction Tool Definitions
Defines the Evidence‑Exploration tools used for skill induction.
This appendix enumerates the concrete tools the agent can invoke when exploring evidence for skill induction.
All tools resolve paths relative to the root “.” for reads and output/ for writes, and write operations cannot overwrite runtime‑owned logs.
Example Exploration Trace
Appendix D details the illustrative exploration trace and the prompts that drive skill extraction.
Skill Induction is the process of converting a recorded Evidence Episode into a reusable, model‑agnostic procedural skill.
Load the skill‑creator tool, locate the authoritative
List the directory and glob for evidence files, then read the relevant traces and image indices.
Inspect representative initial, critical, and final‑state images using
If a transition remains unclear, call
Write a skill with
Read the final
The Induction System Prompt defines the task, evidence‑access boundaries, and completion requirements for extracting reusable NexAU skills.
The NexAU Agent Configuration creates a single agent with the system prompt, registers the required tools, and enforces the skill‑creation policy without prescribing a separate controller or an explicit information‑gain objective.
The Pipeline Prompt baseline mirrors the Work2Skill setup but expands the bootstrap prompt into an explicit workflow that orders artifact inspection, frame selection, and skill authoring.
Pipeline Prompt Details
This appendix details the operational protocols for evidence inspection and skill extraction within the Work2Skill framework.
The extraction process relies on event-aligned evidence frames rather than arbitrary screenshots. These frames are decoded from episode videos at precise timestamps, allowing for the reconstruction of UI state changes and user interactions without relying on fixed coordinates or session-specific identifiers.
Locate and read the authoritative task.json to understand the goal before interpreting the trajectory.
Inventory input files recursively and read textual artifacts using the read file tool.
Inspect representative initial, critical, and final event frames using read visual evidence.
Use get video metadata and watch video for coarse sampling, extracting frames at exact action timestamps when fine-grained detail is required.
Infer reusable procedures, semantic target-selection rules, and completion checks, separating these from episode-specific values.
Write each skill as a SKILL.md file and perform a self-review against the quality rules before finalizing.
Skill-Creator transforms raw task evidence into portable, operationally complete procedures by abstracting away instance-specific details while retaining the underlying logic.
**Table 6.** Method-specific contents inserted into the shared <`task_skills`> wrapper at the end of the system prompt. No Skill adds no block.
**Table 7.** Representative changes in the paired MyPCBench Jitter tasks. Descriptions summarize the changed requirements.
Tool Usage Statistics
Tool usage statistics and the full OSWorld task ID list are presented.
Figure 9 shows the mean number of tool calls per task for four models on the MyPCBench Bounded Action benchmark (64 tasks) and the OSWorld‑Verified subset (286 tasks). The heatmap highlights that GPT‑5.5 relies heavily on “watch video” and “extract video frames”, while Qwen3.5‑35B‑A3B achieves the lowest total calls per task.
**Figure 9.** Mean tool calls per task during skill induction, using a shared linear color scale. Values are rounded to two decimals; positive values that would round to zero are shown as < 0.01. Horizontal rules separate tool groups; the bold row reports mean total calls per task. Qwen-9B and Qwen-35B denote Qwen3.5-9B and Qwen3.5-35B-A3B, respectively.
Table 12 enumerates the complete set of 286 OSWorld‑Verified task identifiers used for demonstration collection and evaluation. The IDs are listed in full, sorted lexicographically, and read left‑to‑right across each row.
Task ID Reference
Complete list of task identifiers for the OSWorld‑Verified subset.
Table 12 enumerates all 286 task IDs that comprise the OSWorld‑Verified subset used in the experiments.