Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia, Gongjie Zhang, Jianan Zhang, Long Li, Long Chen, Lei Wang, Gaole Dai, Pengxiang Li, Liangyu Chen, Yue Wang, Steven Hoi

A foundation GUI agent using hybrid GUI+CLI actions and real-device RL to bridge the simulation-to-reality gap.

How can we build a reliable, cross-platform GUI agent capable of real-world device operation through integrated environment infrastructure and iterative reinforcement learning?

GUI agents often fail in real-world deployment because they are optimized for sanitized, simulated environments that lack the interruptions, permission constraints, and dynamic states of physical devices. Qwen-UI-Agent addresses this by training on a massive real-device mobile runtime and using a unified action space that interleaves GUI operations with CLI commands and batched execution. This approach achieves state-of-the-art performance on real-device mobile benchmarks, reaching a 92.2% success rate on MobileWorld-Real and 97.5% on AndroidDaily.

Paper Primer

The system hinges on a "data flywheel" and a hybrid action space. It treats GUI interaction as a universal interface for visual tasks, while delegating structured operations like file manipulation or data processing to a bash-based CLI, allowing the model to batch these actions to reduce inference cycles.

Qwen-UI-Agent outperforms frontier models on real-device mobile tasks.

On MobileWorld-Real, it achieves a 92.2% success rate, surpassing Seed 2.1 Pro (88.7%) and Gemini 3.1 Pro (86.2%). 6.0 to 7.5 percentage point lead over top-tier proprietary models.

Hybrid GUI+CLI execution significantly improves efficiency in computer-use tasks.

On OSWorld-v2, the agent requires 135.8 steps per task, substantially fewer than competing models like MiniMax M3. Over 40% of action outputs are batched, reducing the total number of observation-reasoning-execution cycles.

Why does the paper emphasize "real-device" training over standard simulation?

Simulated environments exclude common real-world disruptions like CAPTCHAs, unexpected pop-ups, and network instability. The authors found that models trained only in simulation fail to recover from these interruptions, often entering repetitive action loops or misreading interface states.

What is the role of the "harness layer" in this architecture?

The harness layer acts as a proactive service coordinator that monitors digital signals (like notifications), maintains user context, and manages cross-platform workflows, allowing the agent to initiate tasks without waiting for explicit user instructions.

Introduction: Toward Real-World GUI Agents

We expose why existing GUI agents fall short and outline Qwen‑UI‑Agent’s real‑world‑centric solution.

Current GUI agents excel on simulated benchmarks but stumble on real devices, where UI layouts, interruptions, and permission handling are far more complex.

To bridge this gap we envision agents that reliably run on actual hardware, span multiple platforms, blend GUI and CLI actions, handle long‑horizon workflows, act proactively, and continuously improve with minimal human effort.

The core problem is that simulated environments do not capture the messiness of real‑world GUIs, so agents trained only there cannot be trusted in practice.

We identify six concrete transitions needed to move from sandbox‑only agents to real‑world utility.

1) From simulated environments to real‑device execution; 2) From isolated domains to cross‑platform workflows; 3) From GUI‑only actions to hybrid GUI + CLI with batched actions; 4) From short‑horizon to reliable long‑horizon task completion; 5) From human‑intensive pipelines to AutoResearch‑style automation; 6) From reactive execution to proactive service initiation.

Qwen‑UI‑Agent implements these transitions via (i) a real‑device mobile foundation with >100 phones and 150 apps, (ii) a unified hybrid action space that interleaves GUI clicks and CLI commands and batches compatible actions, (iii) an AutoResearch data flywheel that generates tasks, diagnoses failures, and iterates, (iv) scalable online RL that trains on >100‑step trajectories across 10 k concurrent environments, and (v) a lightweight harness layer that enables proactive, stateful cross‑device services.

**Figure 1.** Qwen-UI-Agent demonstrates leading or competitive performance across diverse GUI settings.

**Figure 2.** An illustrative trajectory of Qwen-UI-Agent for proactive cross-platform task execution. The left panel summarizes the system capabilities underlying the trajectory, including cross-platform execution, a hybrid action space spanning GUI, CLI, and API operations, and environments ranging from large-scale sandboxes to real devices. The right panel illustrates a travel-recovery scenario triggered by a flight-cancellation notification. After identifying the affected tasks and commitments, the agent searches for alternative flights through an API, requests user approval before rebooking via a mobile GUI, updates the affected meeting schedule through desktop GUI and CLI actions, and sends the revised artifact to the relevant recipients.

The key shift is moving from sandbox‑only evaluation to reliable operation on real‑world devices.

System Architecture and Methodology

Methodology details the unified environment, hybrid actions, and data flywheel that enable robust real‑world GUI agents.

Existing GUI agents falter in the wild because they train in narrow simulators and rely solely on point‑and‑click interactions, which makes them brittle when faced with diverse devices, flaky system states, or tasks that require command‑line shortcuts.

The infrastructure stitches together scalable sandboxes, real‑device fleets, a hybrid GUI + CLI action layer, and a unified cross‑platform interface so the agent can perceive, act, and learn everywhere a user works.

How does this infrastructure differ from a typical simulator‑only setup?

Typical simulators provide only virtual screens and deterministic resets; our infrastructure adds real‑device execution with health monitoring, a hybrid GUI + CLI action repertoire, and a single API that lets the agent treat all platforms uniformly.

**Figure 3.** The environment infrastructure of Qwen-UI-Agent. (a) Scalable sandbox environments spanning mobile-use, computer-use, browser-use, and DeepSearch provide controllability and repeatable evaluation for data synthesis and training. (b) A sim-to-real bridge extends agents to real devices with real applications, networks, and account states, supporting user takeover for login, payment, permission, and confirmation. (c) A hybrid GUI+CLI action space interleaves graphical operations with direct command execution. (d) A unified interface standardizes the thinking-action-observation loop across heterogeneous environments.

Instead of forcing every interaction through clicks, the agent can pick the most efficient modality—GUI for visual tasks, CLI for file or system operations—within a single plan.

GUI action

CLI action

Observation stream now contains a screenshot of the editor and a structured CLI output.

By batching the three steps into a single action tuple, the agent avoids three separate perception‑decision cycles, halving latency for simple scripts.

Why isn’t the hybrid space just “GUI + CLI” as two separate agents?

Because a single policy can reason over both modalities jointly, choosing the cheapest path at each step and preserving a coherent history ($h_t$) that spans visual and textual feedback.

The flywheel closes the loop: the agent generates failures, those failures drive targeted task synthesis, new trajectories enrich the training set, and the improved model produces fewer failures.

How does the flywheel actually reduce human effort?

Humans only review high‑level failure diagnostics and approve newly synthesized tasks; the bulk of trajectory collection, filtering, and labeling is automated by the VLM judge and the agent itself.

**Figure 5.** The data flywheel of Qwen-UI-Agent. Domain capability bootstrapping initializes training, iterative refinement loop identifies capability weaknesses and generate targeted tasks, and the resulting data improve the next training iteration.

Training builds on the flywheel data: first a supervised fine‑tuning (SFT) pass, then Action‑aware RL to fix recurring action errors, and finally online RL with the Group Relative Policy Optimization (GRPO) objective for long‑horizon planning.

acquire() – lease a sandbox or real‑device session and obtain a session identifier.

reset(session) – return the environment to a clean start state (clear caches, reinstall apps).

step(session, action) – execute a GUI, CLI, or API action; capture screenshot, CLI output, and API response.

evaluate(session, trajectory) – run the VLM judge to label each step as correct, partial, or failed.

release(session) – return the leased resources to the pool and update health metrics.

Health‑aware scheduler selects an eligible device, leases it, and falls back on a replacement if the task fails.

The health‑aware scheduler guarantees continuous throughput despite flaky devices, automatically rerouting tasks away from unhealthy phones and re‑validating them after repair, which is essential for scaling real‑device training.

**Figure 4.** Real-device mobile runtime with closed-loop environment governance. The health-aware scheduler routes each task to an eligible phone, App/account, and display; unhealthy targets remain blacklisted until repair and revalidation. Virtual displays allow one phone to run multiple Apps concurrently. Evidence-based review separates task success, model failure, and environment failure from the complete trajectory, and confirmed environment issues are fed back to the scheduler.

The table lists various actions categorized into GUI Actions, CLI Actions, API Actions, and Interaction and Control Actions, along with their respective definitions.

The harness layer (Section 2.5) ties the agent’s internal state to user context, parses mobile notifications into proactive tasks, and orchestrates cross‑platform workflows by planning on a global device graph.

The core trick is a tightly coupled environment‑action‑data loop: a scalable, health‑aware infrastructure supplies diverse, realistic experiences; a hybrid GUI + CLI action space lets the policy choose the cheapest modality; and an automated data flywheel continuously turns failure into fresh training data.

Experimental Results

Qwen‑UI‑Agent sets new success‑rate records on mobile and desktop GUI benchmarks.

Qwen‑UI‑Agent‑27B reaches 82.1% success on the GUI‑only subset of MobileWorld, a new state of the art.

Table 2 shows 82.1% versus the previous best of 70.1%.

A real‑device benchmark that runs human‑written mobile tasks on live Android phones, exposing the agent to changing apps, accounts, and network conditions.

How does MobileWorld‑Real differ from typical sandbox mobile benchmarks?

Sandbox benchmarks run in resettable emulators where the app state is fully controllable; MobileWorld‑Real runs on physical devices with live accounts, so the agent must handle unpredictable UI changes, network latency, and authentication failures.

A desktop‑use benchmark that evaluates long‑horizon workflows on real operating‑system environments, reporting both partial progress and binary task completion.

Why does the paper report both partial and binary scores on OSWorld‑v2?

Partial scores capture how far the agent progressed on a multi‑step task even if it did not finish, while binary scores reflect full task success; together they reveal planning quality and robustness.

**Figure 7.** Overview of MobileWorld-Real, a real-device benchmark with human-written tasks that reflect the breadth of everyday mobile use. Representative examples and aggregate statistics show broad domain coverage and a long-tailed App distribution. Matched-model results further show lower success rates and longer trajectories than on AndroidDaily, highlighting the challenge of real-world mobile interaction.

**Figure 8.** Demonstration of real-device mobile GUI execution. The trajectory is rendered as key frames with the executed action annotated beneath each frame. In this cross-app task, the agent looks up the target address on Amap, finds the most popular café nearby on Dianping, and posts a summary of the findings on RedNote.

Qwen‑UI‑Agent delivers large performance gains on both mobile and computer‑use benchmarks while keeping inference costs modest.

Analysis of Performance

Qwen‑UI‑Agent tops the benchmarks, closing the performance gap to humans.

Qwen‑UI‑Agent‑27B outperforms the strongest baseline by +5.4 % on the DeepSearch success‑rate metric.

Table 7 shows the 27B model at 73.6 % versus the best baseline at 68.2 %.

On the five GUI‑grounding benchmarks, the 27B variant captures the top spot on every metric.

Table 8 lists the 27B model as bolded in all columns, with the second‑best underlined.

**Table 7.** Performance comparison on DeepSearch benchmarks: BrowseComp (BC) and BrowseComp-ZH (BC-ZH). “–” indicates the result is not reported.

**Table 6.** Performance comparison of various models on BC and BC-ZH benchmarks.

**Figure 10.** Demonstration of DeepSearch-assisted GUI execution. The upper panels show the multi-round search process, including keyword-based DeepSearch, targeted web fetches with intermediate thoughts, and the final search summary; the lower row shows the subsequent GUI trajectory. DeepSearch resolves the knowledge and reasoning problem before GUI execution, converting the vague cross-source request into an explicit target: Qwen-UI-Agent identifies the largest comeback in the World Cup knockout stage through DeepSearch, then opens RedNote and navigates directly to the highest-engagement related post from the past week without blind in-app searching.

**Table 8.** Performance comparison on GUI grounding benchmarks. ScreenSpot-Pro reports no-zoom results, with zoom-in results shown in parentheses when available. Baseline results marked with * are obtained from our own evaluation.

**Table.** Performance comparison of Qwen-UI-Agent against other models across various benchmarks, categorized into General capabilities and Agentic capabilities.

**Figure 11.** **Demonstration of proactive service based on mobile notifications.** The trajectory is organized into highlighted stages: proactive task initiation from a detected notification, proactive execution phases, and a decision-ready result, with the executed action annotated beneath each key frame. Flight-cancellation recovery: when the user's next-morning flight is canceled and a 14:00 demo is at risk, the harness proactively searches alternative flights and high-speed trains, evaluates which options arrive on time, and presents a decision-ready travel recovery plan.

**Figure 12.** Demonstration of cross-platform task execution. In this workflow, mobile subtasks run on virtual screens of the physical device, so execution does not block the user's own actions. Parallel multi-app search: the agent searches sushi restaurants on Dianping, Meituan, and Amap concurrently through multiple virtual screens, consolidates their ratings, and summarizes the top three options in a local report opened on the computer.

Behavioral Analysis and RL Impact

We dissect how Qwen‑UI‑Agent’s execution changes across real‑device failures, hybrid actions, and RL training.

Qwen‑UI‑Agent tackles the unreliability of GUI agents by pairing a scalable data‑flywheel with a hybrid action space and online RL that closes the verification loop.

Action RL lets the agent refine its low‑level action choices by rewarding successful primitives and penalising repeats, turning raw interaction steps into a learned policy.

How does Action RL differ from simply adding more supervised fine‑tuning data?

Supervised fine‑tuning only teaches the model to predict the next action from a static dataset, while Action RL continuously evaluates the effect of each primitive on the actual device state and updates the policy to prefer actions that demonstrably advance the task.

Execution‑capability limitations explain 40.3 % of Qwen 3.7 Plus failures on real devices.

Table 10 aggregates failure patterns across MobileWorld‑Real and AndroidDaily.

Real‑world scenario challenges account for 52.0 % of failures.

Table 10 shows the split between capability and scenario dimensions.

Exploration failure alone contributes 19.5 % of all failures.

Figure 13 visualises typical exploration dead‑ends.

Erroneous action loops appear in 14.3 % of trajectories.

Figure 13 (b) illustrates a loop where the same click is repeated.

Lost execution state is observed in 6.5 % of long‑horizon tasks.

Figure 13 (c) shows the agent restarting a sub‑task after an app switch.

UI misreading causes 24.7 % of real‑world failures.

Figure 13 (d) highlights a placeholder that the model treats as user input.

Pop‑up interference is responsible for 18.2 % of failures.

Figure 13 (e) shows an ad overlay that blocks the intended button.

Physical widget control errors account for 9.1 % of failures.

Figure 13 (f) demonstrates overshooting a date‑picker slider.

CLI actions increase from 40.7 % to 55.1 % of all actions on OSWorld‑v2 (+14.4 pp).

Table 11 reports the per‑action breakdown for OSWorld‑Verified vs. OSWorld‑v2.

Batched actions appear in 62.1 % of tasks on OSWorld‑Verified and 88.9 % on OSWorld‑v2 (+26.8 pp).

Table 11 (a) shows the task‑level usage difference.

Capture a screenshot of the target page.

Run OCR to locate the search box.

Click the box, type the query, press Enter.

Scroll down until the desired result appears.

Click the result – all five primitives are emitted as a single batch.

Action RL raises overall task success by more than 7 %.

Aggregate results in Section 4.3 show a 7 % lift after RL.

Reasoning‑token consumption drops by 21.3 % after Action RL.

Section 4.3 reports a 21.3 % reduction.

Average interaction steps rise by 8.4 % with Action RL.

Section 4.3 notes the step‑count increase.

Grounding of confusable elements improves by 6.3 % after Action RL.

Table 12, column “Confusable‑Element Grounding”.

Sorting & ranking success rises by 3.8 %.

Table 12, column “Sorting and Ranking”.

Multi‑target completeness gains 4.4 %.

Table 12, column “Multi‑Target Completeness”.

Premature completion errors drop, yielding a 5.2 % lift.

Table 12, column “Premature Completion”.

Reward for long‑tail actions climbs by 6.4 % after Action RL.

Table 13 compares rewards before/after RL.

Verification actions increase by 14.7 % post‑online RL.

Section 4.4 reports the verification‑action fraction.

False‑stop rate drops by 11.2 % after online RL.

Section 4.4 quantifies the reduction.

GUI‑action share grows by 6 % under online RL.

Section 4.4 notes the modality shift.

Mixed GUI+CLI trajectories rise by 10.6 % after online RL.

Section 4.4 reports the increase.

Constraint‑satisfaction improves by 8.6 % on OSWorld.

Section 4.4 compares pre‑ and post‑online RL constraints.

Constraint‑satisfaction improves by 7.5 % on BrowseComp‑ZH.

Section 4.4 reports the same metric on a second benchmark.

**Table 10.** Failure-pattern distribution over all failed Qwen 3.7 Plus trajectories on real devices.

**Figure 13.** Representative real-device failure patterns of Qwen 3.7 Plus. Execution capability limitations: (a) exploration failure, (b) erroneous action loops, (c) lost execution state. Real-world scenario challenges: (d) UI misreading, (e) pop-up interference, (f) physical widget control. Speech bubbles quote the model's abridged thinking, with the critical fallacy in purple.

**Table 11.** GUI+CLI usage and batched-execution statistics on OSWorld-Verified and OSWorld-v2. Panel (a) reports CLI and batched-action usage at the action and task levels. Panel (b) reports the composition of batched outputs. Mean batch size counts the number of primitive actions in a batch. Differences are computed as OSWorld-v2 minus OSWorld-Verified. Differences are reported in percentage points (pp).

**Figure 14.** **Representative GUI interaction patterns.** The panels show (a) filling structured spreadsheet content, (b) finding content with in-page search, (c) visual navigation through scrolling, (d) use of a native media-timeline feature, (e) continuous spatial interaction, and (f) zooming in for fine-grained inspection. Each panel pairs the rendered action with its resulting application state.

**Figure 15.** **Representative CLI interaction patterns.** The model uses CLI tools to (a) retrieve relevant documents from a mixed corpus, (b) parse machine-readable artifact structure, (c) compress many visual candidates into one labeled comparison, (d) transform an artifact programmatically, (e) execute a repeated computation and materialize its outputs, and (f) verify exact output and process postconditions. Each panel juxtaposes the application context with a command excerpt and its task-specific purpose.

**Figure 16.** Representative batched-action patterns. (a) a compact GUI macro for in-page search, (b) a 21-action dependent form sequence that stops at the next uncertain dialog, and (c) a GUI–CLI handoff that verifies a spatial manipulation through structured task state. Each row pairs the rendered batch with the first post-batch observation.

**Figure 17.** Representative trajectory of tightly coupled GUI–CLI collaboration. The task requires Qwen-UI-Agent to achieve a score of at least 100 in a browser-based dinosaur game without using DevTools or CDP. GUI observations provide evidence for measuring jump dynamics, diagnosing failed control policies, and recognizing changes between day and night modes, while CLI code iteratively implements stateful key control and adaptive obstacle detection.

**Table 12.** Performance on five error-pattern-specific test sets before and after action RL training.

**Figure 18.** Action RL case study. The SFT model navigates to the account page and repeatedly clicks the share icon before navigating back, becoming trapped in an ineffective local loop. In contrast, the model after action RL recognizes that it is on the wrong page, switches to exploring Mastodon's web interface, locates the correct entry point, and successfully configures the target invite link with the required settings.

**Table 13.** Performance comparison between frequent and long-tail actions.

**Figure 19.** **Online RL elicits verification and self-correction.** On the same task, the online RL policy (bottom) succeeds where the SFT policy (top) fails, because it reopens the application to inspect the actual result and corrects the problem it finds before finishing. The SFT policy trusts a superficial script check that only counts the chart, never looks at the rendered output, and ships an empty chart with no data series, whereas the RL policy reopens the file to verify, finds that a leftover application instance would overwrite the chart, removes it so the chart persists, and confirms that the full clustered chart is actually rendered before terminating.

**Figure 20.** Online RL elicits emergent cross-modal action. On the same task, the online RL policy (bottom) succeeds where the SFT policy (top) fails, because it grounds its decisions in what it visually observes and verifies the result before finishing. The SFT policy reads the receipt from OCR text alone, mistakes a Cash Out for income, and declares success without checking, whereas the RL policy opens the receipt image to visually read it, records it correctly as an expense, and reads the spreadsheet back to verify before terminating.

**Figure 21.** Online RL improves long-horizon search and verification. Without online RL, the model follows incorrect entity associations and reaches the unsupported answer Guangzhou. With online RL, it identifies the key chain of Hai Yan, Nirvana in Fire, Huang Weide, and Chengdu.

Related Work

Survey of recent GUI agents across mobile, browser, and desktop domains.

Recent mobile‑agent research moves beyond screenshot‑grounded action prediction toward models that incorporate memory, reflection, and reinforcement learning. Works such as Mobile‑Agent‑v3.5 (Xu et al., 2026) and UI‑Venus‑1.5 (Gao et al., 2026) demonstrate unified GUI modeling across platforms, while Step‑GUI (Yan et al., 2025) and UI‑TARS‑2 (Wang et al., 2025a) explore data‑flywheel‑driven multi‑turn RL. Evaluation has expanded from pure grounding (e.g., ScreenSpot‑Pro) to realistic daily‑use scenarios like MobileWorld, AndroidDaily, and RealMobile.

Browser agents can exploit structured artifacts such as DOM and accessibility trees, enabling richer interaction than pure visual cues. Early systems like WebVoyager (He et al., 2024) and SeeAct (Zheng et al., 2024) combined multimodal LLMs with browser APIs, while newer efforts—OpenAI Operator (2025), Google Project Mariner (2025), and UI‑TARS (Seed, 2025)—push toward autonomous, long‑horizon web workflows. Recent work (WebWorld, WebEvolver) adds world‑model components to improve planning across multiple sites.

Computer‑use agents face a vastly larger action space and longer horizons, demanding reliable grounding and verification. Systems such as UI‑TARS (Qin et al., 2025), DART‑GUI (Li et al., 2025b), OpenCUA (Wang et al., 2025b), UltraCUA (Yang et al., 2025b), and EvoCUA (Huang et al., 2026) scale foundation models with multimodal pretraining and agentic reasoning. Benchmarks like OSWorld‑Verified and OSWorld‑v2 now emphasize robustness, scalability, and strict success verification for real‑world desktop tasks.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers