OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use
Hanzhang Zhou, Long Li, Quyu Kong, Lei Wang, Panrong Tong, Long Chen, Tianyu Xia, Gongjie Zhang, Xu Zhang, Chen Liu, Jianan Zhang, Chenglin Cai, Zhengwen Zeng, Steven Hoi, Yue Wang
OpenGCUA provides an open-source stack for generalist computer-use agents that unify visual, programmatic, and shell-based control.
How can we build a generalist computer-use agent (GCUA) that reliably completes complex tasks by combining code execution with intelligent decision-making?
Computer-use agents are currently fragmented, with frontier systems like GPT-6 Astra remaining closed and existing open-source agents restricted to rigid GUI-only or batched action sequences. OpenGCUA introduces a generalist agent design where the model dynamically decides how to observe, act, and organize its work by writing executable programs that combine GUI, shell, and API operations. On OSWorld-v2, this design improves mean task scores by 19.6 points and reduces steps per task by 53.0% across eight frontier models compared to GUI-only baselines.
Paper Primer
The core mechanism is a unified program-based action space that replaces fixed action lists with executable code. By writing programs that include loops, conditional branches, and assertions, the agent can process intermediate results and verify task states without requiring a new model call for every operation.
The system also implements harness-to-model decision transfer, which delegates workflow control—such as when to request a screenshot or how to track task progress—from rigid harness rules to the model itself. This allows the agent to optimize its own observation frequency and maintain a persistent, model-managed record of its work.
Generalist computer-use agents (GCUA) significantly outperform GUI-only baselines in both task success and interaction efficiency.
Testing on the OSWorld-v2 benchmark across eight frontier models showed a mean partial score increase of 19.6 points and a 53.0% reduction in steps per task. The score gap between the strongest and weakest models was narrowed by more than half.
Code execution provides efficiency gains beyond simple shell access.
Compared to a GUI+CLI hybrid baseline, the full GCUA configuration added 6.1 points to the mean partial score and cut steps per task by 14.8%.
Why is this approach better than simply batching GUI and CLI actions?
Batched actions fix the sequence of operations in advance, whereas code execution allows the model to use conditional logic and loops to respond to intermediate results, effectively consolidating multiple round-trips into a single model call.
What is the primary goal of the harness-to-model decision transfer?
It shifts workflow decisions—like when to request visual feedback—from static rules to the model, allowing the agent to reduce unnecessary context and improve cache reuse by only requesting screenshots when the text output is insufficient.
Abstract and Overview
We introduce OpenGCUA and its key design advances for generalist computer‑use agents.
Computer‑use research is shifting from GUI‑only interaction toward agents that operate through both visual and programmatic interfaces, expanding the kinds of tasks they can perform.
The agent can run snippets of code and feed the results back into its reasoning loop, so later steps can depend on earlier computed values.
Instead of hard‑coded harness rules deciding what to observe or which task state to update, the model itself makes those choices, granting flexibility and adaptability.
On OSWorld‑v2, OpenGCUA raises the mean partial score by +19.6 points across eight evaluated models.
Mean partial score increased from the baseline to the OpenGCUA configuration, as reported in the evaluation table.
OpenGCUA reduces the average number of steps per task by 53.0 % compared to batched GUI‑only execution.
Step counts were measured on the OSWorld‑v2 benchmark; the OpenGCUA configuration required roughly half the steps of the GUI‑only baseline.
In real‑device macOS tasks, Qwen3.8‑Max uses 19.4 × as many reported reasoning tokens as GPT‑6 Astra.
Token logs from the real‑device harness show Qwen3.8‑Max generating 19.4 times more tokens while solving the same tasks.
We release the full OpenGCUA implementation, prompts, evaluation trajectories, a preliminary real‑device harness, and a cross‑device SDK that unifies programming interfaces across computers, mobiles, and browsers.
The Rise of Computer-Use Agents
We define GCUAs, outline the OpenGCUA project, and list its six key contributions.
Computer‑use research has reached a turning point: models now aim to complete work on real machines by any means that work, not just by manipulating screenshots and mouse clicks.
A GCUA is an autonomous system that decides what to observe, how to act, and how to organize its work, using both visual interfaces and programmatic APIs.
We introduce OpenGCUA, an open‑source effort that (1) builds a model unifying computer and browser use with code execution, (2) provides a real‑device harness and cross‑device SDK, (3) releases execution trajectories of frontier models, (4) studies efficiency gains from code execution and harness‑to‑model decision transfer, and (5) creates real‑device evaluation benchmarks.
Defining the GCUA Stack
Defines the GCUA mechanism: three per‑step decisions across four interfaces.
The layer view (Brain, Eyes, Body, Hands, Feet) cannot tell whether an agent picks the most efficient way to act. A GUI‑only agent may click through dozens of screens where a single shell command would suffice. To capture true generality we must define the agent by the decisions it makes at each step.
Step 1: The model selects q₁ = {CLI} (no screenshot), and emits p₁ = “open notes.txt”. Executing p₁ updates s₁ to have the file handle open and yields e₁ = “OK”.
Step 2: The model keeps q₂ = ∅ (no new view) and emits p₂ = “`read_line`”. The execution returns e₂ = “Hello world”. Since q₂ is empty, o₂ = (e₂) only.
Step 3: The model selects q₃ = {GUI} to verify the terminal output visually, requests a screenshot, and emits p₃ = “print e₂”. The screenshot shows “Hello world” on the screen; e₃ is empty. The final observation o₃ combines the empty e₃ with the screenshot view.
This walk‑through shows how the GCUA can interleave headless CLI actions with a single visual check, avoiding unnecessary screenshots while still guaranteeing correctness.
How does a GCUA differ from the earlier GUI + CLI hybrid agents?
The hybrid agents fix three constraints: they always request a screenshot, they emit a predetermined batch of GUI/CLI primitives, and they update their internal state with a hard‑coded rule h. A GCUA lifts all three—qₜ can be empty or include any view, pₜ can be an arbitrary program with loops and branches, and mₜ₊₁ is produced by the learned policy itself, not by a developer‑specified harness.
Agent Taxonomy and Positioning
We compare existing agent families and motivate the shift to code execution.
To situate our approach, we first place GCUA alongside prior agent families. Table 1 summarizes which of the three per‑step decisions—observation ($q_t$), action ($p_t$), and organization—are fixed by the architecture versus chosen dynamically by the model.
OSWorld‑v2 is a simulated desktop environment that exposes both visual screenshots and programmable APIs, letting agents observe pixels, run code, and read execution results.
Why now? 2026 saw a clear shift from pure clicking toward writing short programs. Vendor guides (e.g., OpenAI’s GPT‑6 Astra) start recommending code snippets with libraries like PyAutoGUI, and OSWorld‑v2 benchmarks show agents performing more actions through APIs and file operations. This transition mirrors a broader industry move: code execution lets a single model call embed conditionals, loops, and verification, collapsing what previously required dozens of round‑trips into one call. Like drafting a flexible recipe that can adapt to ingredient availability, code execution adapts at runtime, whereas a fixed batch of clicks is like pre‑packing a lunchbox without knowing the day's menu.
Efficiency matters beyond raw success rate. We measure four efficiency metrics: steps per task (model calls), total output tokens, reasoning tokens (a subset of output), and monetary cost derived from token usage. Fewer steps reduce API overhead; fewer tokens lower latency and price. Reporting these together shows how code execution can cut both time and expense while preserving performance.
Mechanism: Code Execution
Unified code execution lets the agent combine GUI clicks and programmatic checks in one program.
GUI‑only agents stumble when a task requires both clicking and checking programmatic results; the brittle pipeline forces a model call after every GUI action.
pyautogui.click(100,200) — GUI click on the “Open” button.
out = bash("ls /tmp") — shell returns a newline‑separated list of files.
files = out.split("\n")[:2] — keep the first two filenames.
for f in files: assert os.path.exists(f) — loop asserts each file exists.
The loop lets the agent handle an arbitrary number of files without generating a separate click for each, demonstrating how code execution compresses many actions into one model step.
How does this differ from a traditional GUI script that just records a sequence of clicks?
Traditional scripts replay fixed clicks and cannot react to intermediate results; the code‑execution program can inspect command output, make decisions inside loops, and only proceed when conditions are satisfied, reducing the number of model calls and enabling conditional behavior.
**Figure 3.** Programs coordinate actions and checks. (a) GUI clicks alternate with checks of generated contracts and local drafts. (b) A loop visits seven webpage sections and checks each active section through the DOM. (c) A draft comparison determines whether to click Send, without verifying delivery.
**Figure (a) Observation Preparation** and **(b) Executable Verification**. The figure illustrates two processes: (a) shows an automated workflow where a 3D object view is rotated, captured, and tiled into a 5x5 contact sheet using Python code, and (b) shows a verification process where background edits on slides are checked for transparency and content preservation using assertion-based code.
Code execution raises Qwen3.8‑Max’s mean task score by 5.2 points on OSWorld‑v2.
Compared to a hybrid CUA baseline that batches GUI + CLI actions, the unified program improves the average score.
Harness-to-Model Decision Transfer
Dynamic harness rules let the model request observations only when they aid progress.
GUI‑only agents drown the language model in redundant screenshots, inflating context and breaking long‑range reasoning. By moving the “when to capture” decision into the model, we keep the harness reliable while letting the agent prune unnecessary visual input.
The harness supplies fixed tools (screenshot, todo, intent); the model decides when to invoke each, turning static policies into task‑aware actions—like a driver who checks the rear‑view mirror only when a lane change is imminent.
Step 1: Model executes
Step 2: The environment returns text “Saved.” The model evaluates the text and determines the visual state is already clear, so it skips
Step 3: The next input to the language model contains only the text “Saved.” No new image is added to the context.
Skipping the second screenshot cuts the token budget by roughly the size of one image embedding while preserving all needed information.
How does this differ from a static policy that always captures a screenshot after every GUI action?
The static policy forces the model to process every image, even when the preceding textual output already disambiguates the state. The dynamic policy lets the model omit the image, reducing context length and allowing longer reasoning horizons without sacrificing correctness.
**Figure.** (a) Screenshot: Choose Whether to Request a New Image. (b) Todo: maintain a persistent record of progress. (c) Intent: State the intention of the code.
Benchmark Performance
OpenGCUA boosts scores by ~20 points and halves interaction steps across eight models.
OpenGCUA improves task scores and cuts interaction steps for every evaluated model.
Table 2 shows a mean partial‑score increase of +19.6 points and an average step reduction of 53.0 % across eight models.
Table 2 aggregates the per‑model results, confirming that OpenGCUA consistently outperforms the GUI‑only baseline in both task success and interaction efficiency.
**Figure 1.** On the offline set of OSWorld-v2, OpenGCUA improves all eight frontier models, raising mean partial score by 19.6 points and reducing steps per task by 53%. On real-device, the gap between models remains large: Qwen3.8-Max uses 19.4x GPT-6 Astra’s reasoning tokens but completes fewer tasks.
**Figure 6.** Code-execution ablation on OSWorld-v2. Code execution achieves higher task scores and fewer steps than single-action GUI, batched GUI, and batched GUI+CLI setting across all three models.
Ablation and Resource Analysis
Analyzes how each added mechanism impacts score, steps, cache reuse, and how resource limits shift model rankings.
This section quantifies the contribution of each ablation component and examines how monetary and time budgets reshape model rankings.
Code execution improves Qwen3.8‑Max task score by 6.8 points.
Score rises from 57.2 to 64.0 when code execution is enabled.
Code execution reduces Qwen3.8‑Max steps per task by 13.8 %.
Steps drop from the baseline by 13.8 % with code execution.
Code execution improves DeepSeek‑V4.1‑Flash task score by 9.1 points.
Score rises from 55.8 to 64.9 under code execution.
Code execution reduces DeepSeek‑V4.1‑Flash steps per task by 7.6 %.
Steps per task decrease by 7.6 % compared to the batched GUI + CLI baseline.
Adding the screenshot action raises Qwen3.8‑Max cache‑hit rate by 10.5 %.
Cache hit climbs from 75.6 % to 86.1 % when screenshots are omitted when program output suffices.
Adding the screenshot action raises DeepSeek‑V4.1‑Flash cache‑hit rate by 14.4 %.
Cache hit increases from 61.9 % to 76.3 % under the same condition.
The todo interface lifts Qwen3.8‑Max score by 1.6 points.
Score moves from 62.4 to 64.0 when the todo mechanism is enabled.
The todo interface lifts DeepSeek‑V4.1‑Flash score by 1.8 points.
Score moves from 56.7 to 58.5 with the todo mechanism.
Rolling screenshot buffer improves Qwen3.8‑Max cache‑hit rate by 16.0 %.
Cache hit rises from 75.6 % to 91.6 % when older images are removed in batches.
Rolling screenshot buffer improves DeepSeek‑V4.1‑Flash cache‑hit rate by 13.4 %.
Cache hit rises from 61.9 % to 75.3 % under the same policy.
Micro‑compaction reduces Qwen3.8‑Max cache‑hit rate by 7.0 %.
Cache hit falls from 75.6 % to 68.6 % when older tool outputs are replaced by placeholders.
Micro‑compaction reduces DeepSeek‑V4.1‑Flash cache‑hit rate by 3.0 %.
Cache hit drops from 61.9 % to 58.9 % under the same policy.
Qwen3.8‑Flash reaches a score of 60.5 at a budget of 0.38 $ per task.
Applying a spending cap of 0.38 $ per task yields a 60.5 partial score for Qwen.
GPT‑6 Astra reaches a score of 75.7 at a budget of 15.26 $ per task.
When the per‑task spending cap is raised to 15.26 $, GPT‑6 Astra attains the highest observed score of 75.7.
**Table.** Ablation study of the proposed mechanisms for computer use, comparing Qwen3.8-Max and DeepSeek-V4.1-Flash across different configurations.
**Figure 7.** **Task performance under monetary and time constraints.** Mean partial scores on the 82-task offline subset of OSWorld-v2 under increasing per-task spending caps (a) and recorded time limits (b). Panel (a) reports average expenditure, including unsuccessful attempts. Top bars identify the highest-scoring models; shading indicates cache-write cost bounds. Time comparisons are exploratory.
Overall, the ablations demonstrate that code execution and smart context policies drive both effectiveness and efficiency, and that resource‑aware selection can overturn raw benchmark rankings.
Real-Device Deployment Challenges
We examine why benchmark agents falter on users’ computers and introduce a cross‑device SDK with a five‑layer harness.
Generalist computer‑use agents (GCUAs) currently rely on GUI‑only interaction, which is brittle. OpenGCUA improves this by letting models emit executable code and manage observation policies dynamically.
Benchmarks provide a prepared task state and exclusive control, but real devices present a mutable, shared environment.
The user and agent must share control, so the agent cannot assume exclusive access to the pointer or keyboard. The screen changes as windows open, close, and move, invalidating stale coordinates. User requests evolve, requiring the agent to track the overall goal across interruptions.
Observation and control need explicit targets because relevant content may reside in background windows. Tasks start in an existing user environment, mixing relevant and unrelated files, so the agent must identify correct inputs before acting.
It gives the agent a uniform programming interface to coordinate actions across phones, desktops, and browsers, while keeping each platform’s native capabilities explicit.
How does this SDK differ from a single‑device API?
A single‑device API binds all operations to one implicit target, whereas the SDK requires each operation to be scoped to an explicit resource handle, allowing the agent to address multiple devices and windows simultaneously without accidental redirection.
Programs can combine observations and actions from multiple resources, persisting variables across turns, enabling workflows like retrieving a report on a phone, copying it to a computer, and entering results in a browser.
The SDK separates observation capture from delivery, letting the program decide which data to emit to the model, which supports efficient cross‑device execution without a model call for every intermediate step.
Each resource declares its supported operations, so the agent can query capabilities and fall back to compatible methods when a feature is unavailable on a given platform.
Our real‑device harness organizes functionality into five layers—device management, window handling, input routing, observation processing, and action execution—each with clear responsibilities, enabling modular extension to new platforms.
Cross-Device SDK Architecture
The harness links the model, runtime, SDK, driver, and OS to run code on real devices.
The agent writes a Python script, the runtime executes it, the SDK binds abstract IDs to concrete UI elements, the driver turns those bindings into native OS actions, and the OS/app returns observations that close the loop.
Model emits:
Runtime runs the line; no state change yet.
SDK looks up
Driver chooses an accessibility action (least intrusive) and sends a click event to the OS.
OS processes the click, updates the button state, and sends a success flag back to the driver.
SDK packages the flag as an observation
The SDK’s provenance check ensures the ID used in the click came from a fresh observation, preventing the agent from acting on stale or fabricated UI references.
Why not run the script directly on the OS without the SDK and driver?
Direct execution would require hard‑coded screen coordinates and OS‑specific calls, which break when the UI changes or when the agent runs on a different platform. The SDK enforces a clean abstraction layer and lets the driver select the safest action modality.
**Figure 8.** Architecture of the Cross-Device SDK. A common programming interface connects agent programs to computers, mobile devices, and browser environments, enabling cross-device workflows through platform-specific adapters.
**Figure 1.** The architecture of the five-layer harness for interacting with a MAC-OS real device.
To avoid disrupting a user’s active session, the harness prefers element‑level accessibility actions, falls back to window‑relative coordinates, and only uses foreground input as a last resort. When a less intrusive method fails, the agent escalates to the next supported option, coordinating with the user if necessary.
Real-Device Performance
Real-device benchmarks expose large efficiency gaps and observation choices between models.
GPT‑6 Astra achieves the lowest interaction steps and token usage among the evaluated models.
Table 5 shows GPT‑6 Astra averaging 34.5 steps and 12.9 K output tokens per task, outperforming all other configurations.
Other models trade lower monetary cost for higher token usage: Qwen3.8‑Max consumes 5.7× more tokens than GPT‑6 Astra while costing only \$1.49 per task, and Claude Opus 5 and Gemini 3.8 Flash sit between these extremes.
**Table.** Interaction cost and task completion metrics for various models.
**Figure 10.** **Reasoning cost and observation-target selection.** GPT-6 Astra captures the selected Amazon page, while Qwen3.8-Max captures the foreground Taobao page and then switches to Amazon. Qwen3.8-Max uses substantially more reasoning tokens.
**Figure 11.** Task coverage in code execution. GPT-6 Astra fills all 125 required Level answer cells, while Qwen3.8-Max leaves 40 empty.
Future Directions
This roadmap outlines upcoming releases, research priorities, and ways the community can contribute.
Open Generalist GCUA aims to make generalist computer use a shared research effort, aligning models, execution systems, and real‑device evaluation.
Section 5.1 reports empirical validation of the agent design using existing models rather than training a new one, focusing on code execution and harness‑to‑model decision transfer.
The next release will introduce a GCUA model that unifies computer and browser interaction, coordinates visual and programmatic actions, requests observations on demand, and preserves task state, especially for real‑device scenarios.
Section 5.2 outlines three research priorities: more efficient reasoning and execution, reliable operation on users’ devices, and real‑device environments with benchmarks.
Efficient reasoning and execution will tighten model thinking, improve code execution and context handling, and explore new system‑level architectures for speed and resource savings.
Reliability on users’ devices targets extending the harness and SDK across computers, browsers, and mobiles, ensuring robust target selection, failure recovery, and task continuity despite interruptions.
Real‑device environments and benchmarks will create macOS evaluation suites covering diverse applications, interactions, and state changes to measure task completion, time, and resource usage under authentic conditions.
Section 5.3 invites community contributions in three areas: improving the Harness and SDK, building real‑device evaluation infrastructure, and proposing research goals.
Contributors can add platform adapters, enhance observation/control reliability, and optimize execution within the Harness and SDK.
Building real‑device evaluation involves creating realistic tasks, defining evaluation criteria, and establishing benchmark pipelines with execution traces that expose current limitations.
Proposing research goals means identifying new workflows, capabilities, or questions that the released systems and trajectories should address.
Related Work
Survey of prior computer‑use agents, infrastructure, and our Open Generalist CUA release.
OpenAI positions GPT‑6 Astra as a high‑performance computer‑use agent, promoting code execution while keeping a structured tool as a fallback. Anthropic’s Claude Cowork and Claude Code similarly expose precise connectors and a full browser‑mouse‑keyboard interface, with a Dispatch feature for phone‑to‑desktop task handoff.
Both releases omit key implementation details and execution trajectories, hindering reproducibility and systematic study. Consequently, the community cannot verify or extend these systems reliably.
Earlier open CUA models such as UI‑TARS, UI‑TARS‑2, OpenCUA, ScaleCUA, EvoCUA, and Fara focus on GUI‑centric interaction and report strong benchmark results. Recent hybrid approaches—CoAct‑1, ComputerRL, UltraCUA, ToolCUA, and StateAct—combine GUI control with code, APIs, or program‑state operations. We adopt the term GCUA for agents that jointly select observation, action, and organization policies across visual and programmatic interfaces.
Open‑source infrastructure projects like the cua driver suite, the Agent S series, and the MCP‑wrapped Open Computer Use runtime provide cross‑platform drivers, memory, visual grounding, and code‑based execution. These resources complement our Open Generalist CUA stack, which integrates a trained model with a real‑device harness and cross‑device SDK.
Our Open Generalist CUA release bundles the model, prompts, and evaluation trajectories, together with a preliminary real‑device harness and cross‑device SDK. On the offline OSWorld‑v2 subset, the full configuration raises mean partial score by 19.6 points and cuts steps per task by 53 % relative to batched GUI‑only execution. Ablations confirm that code execution adds value beyond GUI and CLI access alone.