UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Zihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke, Mengcheng Lan, Weixian Lei, Hanxuan Li, Honglin Li, Xiyun Li, Zaitang Li, Leowei Liang, Xin Luo, Haozhe Ma, Jiayi Mao, Zhoujie Pan, Can Qin, Tianyuan Qu, Weiqi Wang, Wenkai Wang, Yonglin Wang, Yuxin Wang, Chenxu Wu, Yingchen Yu, Chenyu Zhang, Yuhao Zheng

UI-Mate uses environment-grounded training and in-context demonstration learning to improve GUI agent reliability.

How can we improve GUI agent performance on complex, long-horizon tasks by integrating in-context demonstrations into open-weight foundation models?

Foundation GUI agents often fail because they lack reliable training data and struggle to resolve the procedural ambiguities inherent in user instructions. UI-Mate addresses this by combining a closed-loop, environment-grounded training pipeline with DemoCUA, a mechanism that treats demonstrations as flexible, subtask-level workflows rather than rigid action sequences. On the OSWorkerBench benchmark, UI-Mate-27B achieves a 41.0% strict success rate, significantly outperforming its base model while demonstrating that in-context guidance substantially improves long-horizon reliability.

Paper Primer

The core challenge is that GUI interaction data is inseparable from the environment; without executable verifiers and diverse task coverage, agents learn narrow, brittle execution patterns. UI-Mate solves this with a closed-loop pipeline that jointly constructs tasks and environments, filters trajectories, and uses a hierarchical capability tree to rebalance data coverage for supervised fine-tuning and online reinforcement learning.

DemoCUA enables procedural adaptation: it converts demonstrations into subtask-level workflows that guide the agent without forcing rigid replay. The agent treats the demonstration as a prior, not a target: the live screenshot remains authoritative, allowing the model to skip irrelevant steps or re-plan when the interface diverges from the recording.

Demonstrations significantly boost long-horizon task reliability.

On the 33-task OSWorkerBench self-demo subset, providing a single demonstration increased strict success from 17.2% to 35.4%. A 18.2 percentage point improvement in strict success.

UI-Mate establishes a new state-of-the-art for open-weight GUI agents.

UI-Mate-27B scored 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena, outperforming its Qwen3.6-27B base model by 17.7 points on OSWorkerBench. Significant gains in both general computer-use benchmarks and long-horizon office tasks.

Why is a new benchmark like OSWorkerBench necessary if existing ones like OSWorld already exist?

Existing benchmarks often rely on instruction-only protocols that fail to measure an agent's ability to adapt to procedural guidance. OSWorkerBench introduces a paired protocol—evaluating the same task with and without a demonstration—to isolate the value of procedural transfer and long-horizon information reuse.

How does the agent avoid the "shortcut" of blindly copying demonstration steps?

The training data includes cases where the demonstration is fully aligned, partially misaligned, or irrelevant to the current interface. This teaches the model to use the workflow as advisory guidance while retaining the ability to veto demonstration steps based on the live screenshot.

The primary assumption is that task success can be captured by an executable verifier that inspects the final environment state. This allows the agent to optimize for outcomes rather than just imitating human trajectories, which is critical for recovering from execution errors.

For researchers, UI-Mate demonstrates that scaling GUI agents requires moving beyond trajectory collection toward environment-grounded, verifiable data loops. For practitioners, it provides a path to reliable delegation by using multimodal demonstrations to resolve procedural ambiguity.

Introduction and Motivation

We expose why current GUI agents falter on long‑horizon tasks and outline UI‑Mate’s three core fixes.

GUI agents, i.e., Computer‑Use Agents (CUAs) that act on screens, still stumble on long‑horizon tasks because instructions leave procedural details ambiguous and the models lack workflow context. Two bottlenecks underlie this failure: a training bottleneck—scarce, environment‑grounded data and distributional bias—and an interaction bottleneck—prompt ambiguity that leads to unreliable execution across runs. UI‑Mate tackles both by (1) a scalable environment‑grounded training stack, (2) in‑context demonstration learning that turns multimodal demos into flexible subtask workflows, and (3) the OSWorkerBench benchmark exposing long‑horizon challenges.

**Figure 1.** UI-Mate combines strong general computer-use capabilities with demonstration-guided execution. *Top*: For an underspecified cross-application task, an optional multimodal demonstration is distilled into a subtask-level workflow. During execution, the harness provides the current subtask and key milestones; the agent grounds them in the live interface, acts, and advances the workflow upon completion. *Bottom*: In the general CUA evaluation setting, UI-Mate-27B is competitive with leading open- and closed-weight systems across OSWorld-Verified, OSWorkerBench, and WindowsAgentArena. In the reported self-demo evaluation, one same-task demonstration raises task scores from 76.8 to 81.2 on GameDev, 40.3 to 65.8 on the OSWorld subset, and 67.9 to 81.1 on the 33-task OSWorkerBench subset.

The gap between instruction‑only agents and workflow‑aware agents is that the former miss procedural context, which demonstrations can supply to improve reliability.

System Overview

Defines the task formulation and outlines UI‑Mate’s system components.

Long‑horizon computer‑use tasks suffer from ambiguous instructions and a lack of workflow context, which makes it hard for an agent to plan over many steps.

We treat a computer‑use task as a pair $(x, E)$ and let the agent act in discrete decision turns, each conditioned on the instruction, recent history, and the current screenshot.

Step 1: $o_1$ shows the desktop; the policy samples $y_1 = (r_1, a_1)$ with $K_1=2$ actions: $a_1^{(1)}=$ click Notepad icon, $a_1^{(2)}=$ wait for launch.

Step 2: After the launch, $o_2$ shows the Notepad window; $y_2$ contains $K_2=1$ action $a_2^{(1)}=$ type “Hello”.

Step 3: No further actions are needed, so the policy emits the terminal action and terminates.

Bounding the history $h_t$ means the model never sees more than the last two responses, yet it can still orchestrate a three‑step workflow.

Why does the response $y_t$ sometimes contain multiple actions instead of a single one?

Grouping atomic actions that naturally belong together (e.g., a click followed by a drag) reduces the number of observation‑action cycles, which speeds up execution and keeps the history window small.

When a demonstration is available, it is split into ordered subtasks; the agent tracks a pointer $n_t$ and conditions on the current subtask view $g_t$ while still obeying the verifier.

Initially $n_0=1$; $g_0$ shows the goal “open Mail” and its verification predicate $v_1$.

After clicking the Mail icon, the observation $o_t$ satisfies $v_1$, so the agent emits \texttt{subtask\_complete} and $n_{t+1}=2$.

Now $g_{t+1}$ exposes the second subtask’s goal “type body” and its predicate $v_2$; the agent types the email text.

When $v_2$is satisfied, another \texttt{subtask\_complete} is emitted and the pointer caps at$N=2$, ending the demonstration.

The pointer advances only when the live screen confirms the subtask, preventing the agent from blindly following a stale script.

How does the pointer $n_t$ differ from a simple step counter?

Unlike a fixed counter, $n_t$ moves forward only when the observation $o_t$ fulfills the current subtask’s verification predicate $v_{n_t}$, so progress is tied to actual UI state rather than an arbitrary schedule.

UI‑Mate’s environment‑grounded data pipeline builds executable training trajectories by constructing applications, initializing files, rolling out agents, and pruning invalid runs.

The training stack combines supervised fine‑tuning (SFT) with agentic reinforcement learning to teach interaction protocols, visual grounding, and long‑horizon planning.

DemoCUA converts recorded human or agent rollouts into structured workflows of subtasks, allowing the policy to follow, skip, or revise demonstrated steps during execution.

OSWorkerBench provides 100 long‑horizon office tasks across 41 applications, evaluating both instruction‑only and demonstration‑guided performance.

The unified harness orchestrates the observation–reasoning–action loop, supporting local desktops, virtual machines, and sandboxed environments for both deployment and evaluation.

The Data Flywheel

How the UI‑Mate data flywheel turns diverse instructions into a continuously improving training distribution.

Long‑horizon GUI agents stall when the training distribution is narrow or noisy; the section therefore describes the pipeline that supplies a steady stream of realistic, diverse, and verifiable data.

The flywheel continuously recycles four modules—Instruction Curation, Environment Construction, Trainable Task Generation, and Rollout Infrastructure—so that each new batch of data both expands coverage and fixes gaps discovered in the previous cycle.

Instruction Curation selects the 4 instructions and records their frequencies (each appears once).

Environment Construction creates two Ubuntu VMs, randomising wallpaper and task‑bar position for each, then copies the PDF into VM 1 and the spreadsheet into VM 2.

Trainable Task Generation produces SFT bundles for “open‑email” (using the PDF) and “copy‑cell” (using the spreadsheet), attaching simple success predicates.

Rollout Infrastructure runs the bundles, yielding 3 successful rollouts and 1 failure (insert‑image on a VM lacking an image editor).

Diagnostics flag the missing image‑editor capability; the next curation cycle adds “install‑image‑editor” to the instruction pool.

The example shows how a single failure propagates back to instruction curation, turning a short‑term loss into a long‑term capability gain.

How does the Data Flywheel differ from a conventional data‑pipeline that simply aggregates raw logs?

A conventional pipeline is a one‑way street: data flows in, is stored, and never returns. The Data Flywheel is a closed loop—diagnostics from rollouts actively reshape the instruction pool, environment randomisation, and task generation, ensuring the distribution continuously evolves rather than stagnates.

**Figure 2.** Overview of the UI-Mate data flywheel. The pipeline jointly scales instruction curation, environment construction, trainable task generation, and rollout infrastructure. SFT data combines filtered agent rollouts with validated and repaired human trajectories; RL data is built from verifiable task bundles whose evaluators are refined using complementary probes and rollout feedback. Coverage and outcome diagnostics are fed back to rebalance capabilities and continuously improve the diversity, difficulty, and reliability of the training distribution.

Beyond raw instructions, the system organizes capabilities into a hierarchical tree that guides both data generation and diagnostic reporting.

**Figure 3** Representative subset of the collected GUI capability tree. The hierarchy proceeds from application domains to fine-grained capabilities and representative workflows; sector sizes are schematic.

Training Recipe

UI‑Mate learns GUI interaction via supervised fine‑tuning then agentic reinforcement learning.

GUI agents often collapse on long‑horizon tasks because instructions are vague and the model lacks a sense of workflow; UI‑Mate tackles this by first teaching basic screen‑action mapping and then refining the policy with online, task‑level rewards.

SFT teaches the model to generate a correct GUI response given the current screenshot, interaction history, and the natural‑language instruction.

Why does SFT use a token‑level prediction loss instead of directly supervising the final GUI action?

Token‑level loss provides a dense learning signal for every word of the reasoning trace, allowing the model to learn the step‑by‑step reasoning that leads to the correct action, whereas action‑only supervision would be extremely sparse.

Starting from the SFT policy, RL lets UI‑Mate improve by interacting with verifiable GUI environments and receiving a binary success signal per trajectory.

GRPO treats trajectories that share the same task and policy snapshot as a comparison group, computing a relative advantage from the binary success signal.

Compute weighted sum of successes: $4\cdot1 + 3\cdot0 + 5\cdot1 = 9$.

Compute total turns: $4+3+5 = 12$.

Baseline $\mu_{\text{turn}} = 9/12 = 0.75$.

Base advantage for $\tau_1$: $A^{\text{base}}_{1,t}=1-0.75=0.25$ for each of its 4 turns.

Base advantage for $\tau_2$: $A^{\text{base}}_{2,t}=0-0.75=-0.75$ for each of its 3 turns.

Base advantage for $\tau_3$: $A^{\text{base}}_{3,t}=1-0.75=0.25$ for each of its 5 turns.

Weighting by turn count prevents short, lucky successes from overwhelming longer, harder failures, yielding a more balanced learning signal.

Why are groups where every trajectory has the same outcome excluded from GRPO?

Because such groups provide no relative information—there is nothing to distinguish one trajectory from another—so the advantage estimate would be zero for all members, offering no gradient signal to improve the policy.

PCM injects fine‑grained supervision by up‑weighting decision turns that the verifier marks as critical and down‑weighting irrelevant or harmful steps.

How does PCM avoid over‑emphasizing noisy process annotations?

By clipping the raw weight to a fixed maximum $w_{\max}$ and adding a base offset $b$, PCM limits any single annotation from dominating; the subsequent normalization further guarantees that the average influence per trajectory stays at one.

Instead of waiting for every rollout in a group to finish, the learner begins updating as soon as enough completed trajectories are buffered, reducing idle time.

What would happen if we omitted the IcePop filter?

Without IcePop, isolated tokens with extreme likelihood ratios could produce large, noisy gradients that destabilize training, especially when the rollout policy is stale.

The sampler reallocates rollout budget toward domains where the current policy underperforms, keeping the learning signal informative as the model improves.

Why keep a base sampling component instead of allocating all budget to weak domains?

Base sampling maintains coverage of well‑solved domains, preventing catastrophic forgetting and ensuring the model continues to see a broad distribution of tasks.

**Figure 4** Agentic RL system of UI-Mate. Our RL Pipeline can be divided into rollout stage (left) and update stage (right). Adaptive sampler drives online sandbox rollouts under a fixed policy snapshot; completed trajectories are pooled per task group for outcome verification, optional process credit assignment, and asynchronous GRPO updates.

UI‑Mate’s training pipeline first grounds basic GUI actions with SFT and then leverages group‑relative RL, process‑credit weighting, asynchronous updates, and adaptive curriculum to acquire robust long‑horizon planning.

DemoCUA: In-Context Demonstrations

In‑context demos guide the agent while forcing it to infer missing low‑level actions.

Long‑horizon GUI tasks expose a common failure: agents learn to copy the recorded workflow verbatim, ignoring the live screen and thus breaking when the interface changes.

DemoCUA treats a recorded demonstration like a GPS that tells you the next major waypoint but hides the turn‑by‑turn instructions, forcing the driver to read road signs (the live screenshot) to navigate correctly.

Training input shows only the three high‑level subtask goals and the live screenshot after each goal.

The model must infer the missing mouse‑down/up pairs from the screenshot (e.g., locate the “File” button).

During inference the harness provides the full six‑action list, allowing the model to execute the exact clicks once it has inferred their locations.

By hiding the low‑level clicks during training, the agent cannot simply copy the workflow; it learns to read the screen and generate the missing primitive actions.

Context‑management routine that folds or truncates interaction history before each inference call.

**Figure 5. Demonstration representation.** Offline (top), a human recording is annotated and segmented into subtasks with goals and verifiable completion criteria. Online (bottom), the agent follows the current subtask using live screenshots and advances upon completion.

OSWorkerBench Benchmark

OSWorkerBench evaluates CUA performance on realistic, long‑horizon office workflows with multimodal demos.

OSWorkerBench includes 100 realistic office tasks covering 41 applications and 10 job families

Section reports 100 tasks spanning 41 normalized applications and 10 consolidated job families

OSWorkerBench provides a more realistic, long‑horizon evaluation than previous benchmarks.

Evaluation Setup

Key performance numbers for UI‑Mate on public computer‑use benchmarks.

We evaluate UI‑Mate on three public benchmarks—OSWorld‑Verified, WindowsAgentArena, and OSWorkerBench—using the same instruction‑only protocol and a $200$‑step interaction budget per task.

UI‑Mate‑27B attains a $77.0\%$ average score on OSWorld‑Verified, surpassing all open‑weight baselines.

Table 1 reports UI‑Mate‑27B at $77.0\%$, while the best open‑weight baseline (GPT‑5.5) reaches $78.7\%$.

UI‑Mate‑27B achieves a $66.2\%$ success rate on WindowsAgentArena, the highest among open‑weight agents.

Table 3 shows UI‑Mate‑27B at $66.2\%$, beating the next best open‑weight model (Kimi‑K2.6) at $63.3\%$.

On OSWorkerBench, UI‑Mate‑27B records a $41.00\%$ overall binary success and a $76.86\%$ progress score.

Table 2 lists UI‑Mate‑27B at $41.00\%$ binary success and $76.86\%$ progress, outperforming its Qwen3.6‑27B base by $17.67$ and $24.51$ points respectively.

DemoCUA Performance Analysis

Ablation study quantifies how in‑context demonstrations affect UI‑Mate across benchmarks.

OSWorkerBench evaluates extended workflows; Figure 10a shows trajectory‑length histograms and Figure 10b the cumulative distributions, confirming that many tasks require dozens of decision turns.

GPT‑5.6‑Sol’s shorter turn count stems from action batching—about 3.83 action records per turn—so the raw workflow length is comparable to the other agents.

**Table 15.** Trajectory lengths and task scores with and without demonstrations on OSWorld-Subset 30 problems with UI-Mate-27B. Human Demo reports the number of steps in the human demonstration, while No-Demo and Demo report the average model steps and normalized task scores over five runs. Demonstrations make the model trajectory length track the human demonstration much more closely (rank correlation with the human step count rises from 0.66 to 0.86).

**Figure c.** Most frequent normalized applications across the benchmark.

**Figure 10.** Decision-turn distributions for GPT-5.6-Sol, Kimi-K2.6, and UI-Mate-27B on the same 100 OSWorkerBench tasks. A decision turn is one observation-to-action model response. A response containing multiple UI actions still counts as one turn because no new observation occurs within the batch; terminal responses and duplicate log records are excluded. Trajectory-length histograms using 20-turn bins show the prevalence of extended interactions, while empirical cumulative distributions summarize the full trajectory-length profiles; open triangles mark trajectories censored by the turn limit or technical failures. Each system has one rollout per task, and decision-turn counts characterize end-to-end interaction patterns rather than atomic UI-operation counts.

**Table 4.** GameDev.

Demonstrations raise UI‑Mate‑27B’s average GameDev score from 76.76 % to 81.15 %, a gain of +4.39 pp.

Table 4 reports per‑task improvements, with the largest deltas on godot‑04 (+18.89 pp), godot‑07 (+17.89 pp), and qgis‑01 (+11.25 pp).

**Table 5.** OSWorkerBench-Subset. Table 5 presents the self-demo results on the more demanding OSWorkerBench subset. Demonstrations improve the average normalized task score from 67.85% to 81.14%, a gain of 13.29 percentage points, with improvements on 28 of the 33 tasks. The number of tasks receiving a perfect score also increases from one to five. At the same time, the average trajectory length increases from 173.3 to 216.0 steps (Table 16). This increase does not simply indicate lower execution efficiency: OSWorkerBench tasks contain repetitive and branching subtasks, and agents without demonstrations frequently terminate after

On OSWorkerBench‑Subset, demonstrations increase the average normalized task score from 67.85 % to 81.14 %, a gain of +13.29 pp.

Table 5 shows improvements on 28 of 33 tasks and five perfect‑score completions versus one without demos.

On OSWorld‑Subset, demonstrations boost the average score from 40.27 % to 65.75 %, a gain of +25.48 pp.

Table 5 reports improvements on 18 of 30 tasks; four previously unsolvable tasks achieve perfect scores.

**Figure 11.** Demonstrations help the agent modify the correct underlying object. The reference demonstration and demo-conditioned run modify the reusable Player template, whereas the no-demo run modifies only one Player copy inside a game level. The two operations look similar in the interface, but only the former persists in the intended source object.

Case study 1 (GameDev‑Godot) demonstrates that the demo‑conditioned agent edits the reusable Player template, while the no‑demo agent mistakenly edits a level‑local copy, leading to a failed artifact check.

Case study 2 (OSWorld2‑Visa) shows the demo‑conditioned agent correctly maximizes the PDF viewer, adjusts the viewport, and submits the required \$18{,}000 certificate, achieving a 99.5 % score versus 24.5 % without demos.

UI-Mate Application

UI-Mate lets any vision‑language model control a desktop by separating inference from interaction.

UI‑Mate provides a thin desktop front‑end that delegates all inference to an external endpoint, turning any vision‑language model into a fully fledged Computer‑Use Agent.

The app contains no model; a lightweight harness talks to a remote endpoint while a platform bridge executes the resulting actions.

Capture the screen at native resolution (e.g., 1920 × 1080) and downscale to the $1000 \times 1000$ grid.

Apply Retina scaling factor (2× on a Retina display) to map $(250, 400)$ to $(500, 800)$ in device pixels.

Translate the scaled coordinates to the global screen origin, yielding screen coordinates $(1500, 2200)$.

Invoke the macOS AX API; it finds the UI element at $(1500, 2200)$ and triggers a click.

If AX fails, fall back to a synthetic click at the same screen coordinates.

The bridge’s two‑stage mapping (virtual grid → device pixels → global screen) isolates the model from display quirks while keeping the latency budget low.

How does UI‑Mate differ from agents that embed a model locally on the desktop?

UI‑Mate never ships a model with the client; the desktop front‑end only sends screenshots and receives action tokens. This decoupling lets any OpenAI‑compatible endpoint be swapped in, avoids GPU requirements on the user machine, and isolates model updates from the native bridge.

In a general CUA run the agent receives a natural‑language goal, iterates through observation‑reasoning‑action cycles, and logs each step as a JSONL trajectory.

**Figure 12.** Demonstrations teach reliable viewport control for document-grounded reasoning. The reference demonstration and demo-conditioned run maximize the PDF viewer, locate and cross-check the critical values, and submit consistent form entries and supporting evidence. The no-demo run enlarges the document view without adequately repositioning the viewport, leading to incorrect visual/OCR readings and erroneous downstream decisions.

Users record a one‑time demonstration of a complex desktop workflow; the system extracts a structured workflow that later guides autonomous runs.

Recorder captures a click at $(120, 250)$ opening the PDF viewer.

Bridge confirms the window title “PDF Viewer” via AX and reports success.

Next action is a scroll command; the bridge translates “scroll down 2 pages” into a wheel event.

Bridge waits for the view to settle, then captures a screenshot of page 3.

VLM receives the screenshot and the prompt “extract the table”; it returns the table text.

The saved sequence encodes both high‑level intent (open PDF, zoom, extract) and low‑level UI primitives, enabling reliable replay without re‑learning the visual cues.

**Figure 13** UI-Mate App’s four-layer architecture: the frontend controls runs and demonstrations, the backend configures requests, the harness drives the agent loop, and the bridge provides macOS observation and input.

This table compares the performance of different model deployment configurations in terms of step time.

Related Work and Discussion

Contextualizes prior work and outlines future directions for computer‑use agents.

Computer Use Agents (CUAs) are models that operate a computer by observing the screen and issuing keyboard and mouse inputs. Early CUA systems combined separate planning, grounding, and reflection modules around general vision–language models, while newer native agents integrate perception, grounding, and action end‑to‑end.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers