UI-Venus-2 Technical Report

Venus Team, Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Yusong Hu, Jianrong Jiang, Jianguo Li, Runze Li, Jinzhen Lin, Zhenyu Ma, Changhua Meng, Han Peng, Xinyu Qiu, Shuheng Shen, Zhongyi Shui, Weiqiang Wang, Ming Wen, Zhuoer Xu, Hang Yan, Kaiwen Yang, Ruilin Yao, Nanjun Yu, Zhengwen Zeng, Lianrui Zhang, Yunzhu Zhang, Zhe Zhao, Beitong Zhou

UI-Venus-2 is a foundation GUI agent that scales environment coverage, task generation, and verification for reliable computer use.

How can a unified foundation model be trained to reliably operate across mobile, web, and desktop GUI environments?

GUI agents often fail in real-world deployment because they rely on brittle, task-homogeneous benchmarks and coarse reward signals that confuse superficial progress with actual task completion. UI-Venus-2 addresses this by jointly scaling environment coverage to 170+ mobile apps and desktop OS, using a deep-research pipeline for function-grounded task generation, and implementing multi-model, trace-level verification to ensure reliable reinforcement learning. The model achieves state-of-the-art performance across multiple mobile and computer-use benchmarks, significantly outperforming prior GUI-specific models in realistic, long-horizon scenarios.

Paper Primer

The core mechanism is a unified closed-loop reasoning–action framework trained via Multi-teacher On-policy Distillation (MOPD). To ensure the agent learns executable behaviors rather than just reasoning traces, the system uses structured action-aware supervision that suppresses distillation signals for correct actions and emphasizes correction only when the action type or parameters deviate from the teacher's hint.

UI-Venus-2-27B achieves superior performance on realistic, user-centric mobile benchmarks compared to both general-purpose VLMs and prior GUI-specific agents.

On VenusBench-Mobile, the 27B model reaches 48.7%, outperforming the strongest baseline, Claude-Opus-4.6 (36.5%). 12.2 percentage point improvement over the leading general-purpose baseline.

The agent demonstrates robust memory-intensive capabilities, essential for cross-session and long-horizon tasks.

On the MemGUI benchmark, UI-Venus-2-27B achieves a 70.3% success rate, surpassing the strongest baseline, Seed2.0 Pro (65.6%).

Why is the new verification mechanism more reliable than previous approaches?

Previous methods relied on coarse, holistic judgments of final interface states. UI-Venus-2 uses trace-level verification that evaluates semantic completion against task-relevant visual keypoints and aggregates multi-model voting to reduce single-judge bias and reward hacking.

How does the agent handle the complexity of desktop operating systems compared to mobile or web environments?

Desktop tasks are serialized as TaskSpecs with materialization plans for environment setup and hierarchical segmentation for long-horizon workflows, allowing the agent to manage multi-session dependencies and stateful complexity.

Introduction and Motivation

We expose why current GUI agents fail and introduce UI‑Venus‑2 as a unified, general-purpose solution.

Current GUI agents suffer from limited environment coverage, brittle task construction, and unreliable reward verification, which prevents deployment beyond narrow benchmarks.

GUI agents cannot operate reliably in real‑world settings because they are trained on narrow, benchmark‑specific data, leading to poor coverage and fragile task execution.

**Figure 1.** Performances of UI-Venus-2 on GUI-agent benchmarks. Each panel compares UI-Venus-2-27B and UI-Venus-2-9B with some selected strong baselines. We favor standalone end-to-end systems evaluated on the closest available task subset and step budget; source-reported action scaffolds may still differ. MobileWorld uses GUI-only success rate on 17 tasks with 50 steps, WebVoyager uses the refreshed 595-task split, Odysseys uses average rubric score over 200 tasks, VenusBench-CAPTCHA uses micro Pass@1 over all 219 examples, and VenusBench-GD uses English-instruction micro-average accuracy. “*” denotes the results are reproduced by us.

The shift from benchmark‑specific models to general‑purpose GUI agents is the key advance of UI‑Venus‑2.

System Architecture and Training

Methodology details the three-stage training pipeline and unified action design that enable GUI agents to learn across platforms.

UI‑Venus‑2 learns to act across mobile apps, web pages, and desktop software by chaining three training stages: large‑scale trajectory‑based mid‑training, domain‑specific offline RL, and a final multi‑teacher on‑policy distillation.

Instead of stopping at a generic vision‑language stage, we keep training on real GUI interaction trajectories so the model learns concrete action patterns early.

Task 1 (mobile): Click (10, 20) → Type ("OK") → Swipe (up) → Wait (1 s).

Task 2 (web): Click (150, 80) → Scroll (down) → Click (150, 300) → Submit (form).

Task 3 (desktop): Open ("Notepad") → Type ("Hello") → Save ("file.txt") → Close ("Notepad").

Training on these concrete traces forces the model to align language intent with precise GUI actions, which pure image‑caption pre‑training cannot provide.

How does Trajectory‑based Mid‑training differ from standard pre‑training on image‑text pairs?

Standard pre‑training only teaches the model to associate images with descriptive text; Trajectory‑based Mid‑training adds a sequence of executable actions, teaching the model to predict the exact GUI commands that realize a task.

All GUI environments share a single vocabulary of action types and parameter schemas, so the same token sequence can drive a click on a phone screen or a menu selection on a desktop.

Token 1:

Token 2:

During inference on a mobile app, Token 1 triggers a tap at (120, 45); Token 2 would be ignored because the mobile decoder expects only Click actions.

During inference on a desktop, Token 1 is mapped to a mouse click, and Token 2 inserts the string "Save" into the active window.

The same token sequence can be repurposed across platforms, demonstrating the power of a shared action schema.

Why does a unified action space help more than maintaining separate action vocabularies for each platform?

Separate vocabularies force the model to learn duplicated semantics for similar operations (e.g., clicking on mobile vs. desktop). A unified space lets the model reuse the same representation, so improvements in one domain directly benefit the others and the distillation stage can align tokens globally.

Stage II applies offline reinforcement learning on the mid‑trained model, providing step‑level supervision for navigation tasks in mobile, web, and desktop environments, as well as specialized supervision for CAPTCHA and grounding.

Stage III consolidates the domain‑specific experts via Multi‑teacher On‑policy Distillation (MOPD), preserving the broad multimodal reasoning while integrating the fine‑grained skills learned in the earlier stages.

Structured Action‑Aware Distillation recognizes that only the short action span of a GUI response interacts with the environment, so we weight the distillation signal toward correct action types and their parameters rather than treating all tokens uniformly.

Teacher‑Side Action‑Type Conditioning augments the teacher’s prompt with a hidden hint about the correct action type, allowing the teacher to score the student’s sampled tokens more precisely without exposing the hint to the student at inference time.

**Figure 2.** System Overview of UI-Venus-2. The figure illustrates the task generation and trajectory collection process of UI-Venus-2. Diverse tasks are constructed to form a multi-domain task pool, and interaction trajectories are collected across mobile, browser, and computer environments, covering a broad range of real-world applications, websites, and desktop software.

**Figure 3** The Three-Stage Pipeline of UI-Venus-2. Following the overall training recipe of UI-Venus-1.5, UI-Venus-2 starts with large-scale trajectory-based mid-training to inject GUI interaction knowledge. The resulting model is then optimized independently for each domain using step-level Offline-RL, covering Grounding, CAPTCHA, Mobile, Web, and Computer tasks. Finally, the domain-specialized models are consolidated into the final UI-Venus-2 model via multi-teacher on-policy distillation.

Data Generation and Verification

Data generation unifies capability catalog, task synthesis, and trajectory collection into a closed‑loop pipeline.

GUI agents suffer from limited environment coverage because their training data are confined to narrow benchmarks. A unified data pipeline can expand diversity while guaranteeing executability. This section details that pipeline and its verification components.

The pipeline iteratively expands task diversity and execution reliability by tightly coupling a capability catalog, task synthesis, and trajectory collection.

Step 1: Catalog entry for

Step 2: Task “Open the settings file” is synthesized, passes the validity gate, and is dispatched.

Step 3: Execution clicks the file icon, succeeds, and logs the observed UI state back into the catalog.

Step 4: Task “Click the submit button” fails because the button is disabled; the failure case is recorded.

Step 5: The catalog updates its

The example shows how a single failure instantly creates a new constraint that reshapes future sampling, preventing repeat mistakes.

How does this pipeline differ from a standard data‑augmentation routine?

Standard augmentation treats data as static and applies random transforms, whereas the pipeline actively updates the capability catalog based on execution feedback, guaranteeing that newly generated tasks remain executable and that the data distribution reflects real‑world constraints.

Domain‑specific extensions instantiate the core pipeline for web navigation, desktop computer use, synthetic GUI grounding, and synthetic CAPTCHA generation. Each domain adds bespoke validation and sanitization steps while feeding results back into the shared catalog.

**Figure 4** Examples of synthesized GUI Grounding training data. Our pipeline generates diverse and realistic interface screenshots spanning desktop (macOS, Windows), mobile (iOS), and web platforms, covering both professional software and consumer applications. All interfaces are rendered in a real headless Chromium browser via Playwright, ensuring high-fidelity visual output that closely mirrors authentic user environments.

Robust reinforcement learning also requires fine‑grained verification of trajectories. We therefore introduce a two‑level verification framework that assesses both the overall semantic outcome and the correctness of individual actions.

SGV treats the full execution trace as a semantic object and uses a Vision‑Language Model as a judge to decide whether the trace satisfies the task objective.

Why use a VLM as a judge instead of purely rule‑based checks?

Rule‑based checks can verify exact states (e.g., file contents) but cannot assess open‑ended semantic goals such as “post a message in the correct group.” A VLM can reason about visual context and language jointly, enabling verification of nuanced, high‑level objectives.

Before each action is executed, the verifier checks whether the proposed interaction aligns with the current screenshot, declared intent, and overall task goal.

How does sample‑level verification interact with trace‑level verification?

Sample‑level labels provide fine‑grained supervision that aggregates into the trace‑level categories. For example, a trajectory with many Correct and Exploratory steps will be classified as Partial, while one dominated by Ineffective or Incorrect steps becomes Failed. This hierarchy supplies both granular and holistic reward signals.

Experimental Setup

UI‑Venus‑2‑27B achieves the highest mobile GUI success rate at 84.0%.

UI‑Venus‑2‑27B attains the top mobile GUI success rate across all evaluated benchmarks.

Table 1 shows UI‑Venus‑2‑27B reaching 84.0% on MobileWorld (50‑step) and leading scores on the other six benchmarks.

Implementation details: we set the sampling temperature to 1.0 and keep Qwen3.5’s default visual resolution. For general agentic tasks, reasoning mode (“think”) is enabled and the full reasoning history is fed back to preserve coherence. For GUI grounding tasks, reasoning mode is disabled and temperature is set to 0 to enable direct, efficient localization.

Benchmarks span six domains: Mobile Use (six mobile‑GUI suites), Computer Use (three desktop‑computer suites), Web Navigation (four live‑web suites), GUI Grounding (four fine‑grained visual‑grounding suites), CAPTCHA Solving (five diverse CAPTCHA suites), and GUI Agent Safety (two safety suites reporting Attack Success Rate).

We evaluate a broad set of models: general Vision‑Language Models (Qwen3.5‑9B, Qwen3.6‑27B, Claude‑Opus‑4.6, etc.), GUI‑specific models (UI‑Venus‑1.5‑8B, GUI‑Owl‑1.5‑32B‑Instruct, etc.), and our UI‑Venus‑2 variants (9B and 27B).

**Table 1.** Performance comparison on various mobile GUI benchmarks. VenusBench-Mobile reports success rate on its 149-task primary pool. For MobileWorld, we report GUI-only success rate on 117 tasks under the 50-step setting; values in parentheses, when available, use 100 steps. MemGUI reports Main Results pass@1. “*” denotes the baseline results evaluated or reproduced by us.

Mobile GUI Performance

UI‑Venus‑2 sets new state‑of‑the‑art performance across mobile benchmarks.

UI‑Venus‑2‑27B attains the top MobileGym score and leads on every mobile benchmark.

MobileGym: 60.5% vs. strongest baseline 52.0% (8.5% absolute gain); similar leads on VenusBench‑Mobile, AndroidWorld, MobileWorld, KnowUBench, and MemGUI.

Computer Use Performance

UI‑Venus‑2‑27B outperforms baselines on key computer‑use benchmarks.

UI‑Venus‑2‑27B leads the DeskCraft benchmark by 14.1 points over the next best model.

Achieves 55.5 % versus Kimi‑K2.6’s 41.4 %.

OSWorld is a simulated operating‑system environment that presents a full desktop GUI for agents to interact with, enabling systematic evaluation of task completion, navigation, and tool use.

Web Navigation Performance

UI‑Venus‑2 models set new state‑of‑the‑art scores across web navigation and GUI grounding benchmarks.

UI‑Venus‑2‑27B achieves 93.4% task completion on WebVoyager, the highest reported.

Outperforms the strongest GUI baseline Fara1.5‑27B by 4.1 points and GPT‑5 (SoM) by 2.8 points.

On Online‑Mind2Web, UI‑Venus‑2‑27B reaches 78.3% task completion.

The 9B variant attains 74.0%, surpassing the Fara1.5‑27B baseline despite its smaller scale.

UI‑Venus‑2‑27B sets a new state‑of‑the‑art score of 80.2% on the REAL benchmark.

This improves over the strongest baseline (74.4%) by 5.8 points.

On Odysseys, UI‑Venus‑2‑27B obtains an averaged rubric score of 80.4%.

The perfect‑rubric score reaches 66.3%, beating the strongest baseline by 21.8 points.

UI‑Venus‑2‑27B achieves a perfect‑rubric score of 66.3% on Odysseys.

This is 21.8 points higher than the strongest baseline.

UI‑Venus‑2‑27B attains 80.1% accuracy on the VenusBench‑GD grounding benchmark.

The 9B model reaches 77.1%, confirming strong performance at a smaller scale.

UI‑Venus‑2‑27B scores 74.1% on ScreenSpot‑Pro, the second‑best result.

Only Qwen‑UI‑Agent‑27B (76.6%) is higher; the 9B variant records 73.0%.

UI‑Venus‑2‑27B achieves 79.1% on OSWorld‑G‑R, the highest among GUI‑specific models.

The 9B model follows closely at 78.5%.

UI‑Venus‑2‑27B records 66.9% accuracy on UI‑Vision, the second‑best score.

Only Qwen‑UI‑Agent‑27B (70.0%) exceeds it; the 9B model reaches 65.0%.

CAPTCHA Solving Performance

UI‑Venus‑2 dramatically outperforms baselines on CAPTCHA benchmarks.

UI‑Venus‑2‑27B improves Pass@1 by 51.6 percentage points over the best general VLM on VenusBench‑CAPTCHA.

Table 5 shows the best general VLM (Qwen3.6‑27B) at 28.3 % while UI‑Venus‑2‑27B reaches 79.9 %.

Across all five benchmarks, UI‑Venus‑2 models achieve 78–80 % Pass@1, whereas the strongest general VLMs range from 28 % to 55 %.

**Figure 5.** Overview of VenusBench-CAPTCHA. Each panel shows one complete, uncropped representative screenshot. The OCR label gives the target transcription, numbered boxes indicate the required click order, and arrows visualize annotated drag trajectories. These annotations are added for presentation only and are not part of the model input.

**Table 5** Performance Comparison across CAPTCHA Benchmarks. All results are Pass@1 percentages, and higher is better. We evaluate on VenusBench-CAPTCHA and four public benchmarks: MCA-Bench Wu et al. (2026b), Spatial-CAPTCHA-Bench Kharlamova et al. (2026), NextGen-CAPTCHAs Liu et al. (2026b), and Open CaptchaWorld Luo et al. (2025b). We use 1,000 sampled MCA-Bench examples, 15 NextGen-CAPTCHAs task types, and 16 Open CaptchaWorld task types; see the appendix for selection details.

**Table 6.** Safety evaluation on OSHARM and OS-BLIND benchmarks. ASR denotes Attack Success Rate (%, lower is better).

Safety and Related Work

GUI agents struggle with coverage, and UI‑Venus‑2 unifies trajectory collection across platforms.

The community has shifted from isolated, platform‑specific GUI tasks toward agents that operate across mobile, desktop, and web environments. This section surveys recent work in each domain, highlighting the capabilities and challenges that motivate a unified approach.

OSHarm measures how likely a GUI agent’s actions could cause harmful outcomes on the operating system.

Action Space and Implementation Details

Appendix lists the unified action space, prompts, and CAPTCHA benchmark details.

The appendix first defines the full set of actions that UI‑Venus‑2 agents can invoke on any platform.

The grounding prompt asks the agent to output the centre point of a target region as “[x,y]”, or “[-1,-1]” if the task is unrelated to the image.

The mobile prompt frames the agent as a GUI Agent, lists the same actions with explicit coordinate ranges, and adds usage guidelines such as exploring with Swipe or copying text via LongPress then Paste.

CAPTCHA benchmarks evaluate the full perception‑reasoning‑action pipeline, requiring exact action sequences rather than just textual answers.

**Table 7.** All actions and their definitions used in **UI-Venus-2**. We unify the action space and map all the actions in the existing open-source dataset to this space.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers