AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
Cheng Yang, Yifan Wu, Yutao Huang, Zhaohua Zhang, Beiduo Chen, Muxi Chen, Chenchen Zhao, Hexuan Deng, Haolin Yang, Geyuan Zhu, Sa Zhu, Jianhuan Zhuo, Qiuyong Xiao, Jianhao Ruan, Yiran Peng, Jiayi Zhang, Tian Ye, Xinlei Yu, Tianwen Jiang, Jihong Zhang, Yuyu Luo
AutoGUIWorld synthesizes GUI interaction trajectories using image generators to train agents without running software.
Can we synthesize high-quality, diverse GUI interaction trajectories for training agents by using image generators as visual world models, bypassing the need for live environment execution?
GUI agents require diverse interaction data to learn how software environments respond to actions, but collecting this data is bottlenecked by the high cost of deploying and maintaining complex, specialized software environments. AutoGUIWorld bypasses this by using a meta-planner to define task sequences and a pretrained image generator to iteratively edit screenshots, synthesizing temporally coherent, action-grounded trajectories without ever executing the underlying software. Fine-tuning Qwen3.5-35B-A3B on these synthetic trajectories improves mean task scores on OSWorld from 33.0% to 40.8% and doubles the success rate on ScienceBoard from 14.0% to 32.2%.
Paper Primer
The framework decomposes trajectory generation into two layers: a planning layer that specifies the task logic and intended visual changes, and a visual world-model layer that renders the next screenshot. By conditioning the image generator on the current screenshot and a rendering prompt derived from the planned action, the system maintains visual consistency and layout persistence across multi-step workflows.
Synthetic trajectories significantly boost performance on real-world desktop and scientific benchmarks.
Fine-tuning on AutoGUIWorld data improved the mean task score on OSWorld (+7.8 points) and increased task success on ScienceBoard (+18.2 points) compared to the base model. The four-benchmark mean score increased from 23.6% to 36.5%.
Synthetic data matches the visual feature distribution of real-world datasets.
Qwen embedding distance metrics show that the synthetic-to-real distance is comparable to the variation observed between independently collected real-world GUI datasets. The distance ratio $\rho$ ranges from 0.59 to 1.24 across four domains.
Why is this approach preferred over simply collecting more human demonstrations?
Human demonstrations are limited by the availability and accessibility of specific software environments. AutoGUIWorld expands coverage to specialized scientific and professional software without the overhead of installation, licensing, or environment configuration.
How does the system ensure the generated screenshots are actually usable for training?
The framework employs action grounding to map semantic element descriptions to spatial coordinates and applies transition-level quality filtering to remove defective samples where the generated image is inconsistent with the intended action.
AutoGUIWorld demonstrates that pretrained image generators can function as effective visual world models for GUI agents, providing a scalable path to generating high-quality training data for complex, multi-step software tasks.
Introduction
GUI agents need diverse trajectories, but collecting them is slow and costly.
GUI agents need high‑quality interaction trajectories to learn how software responds to actions, but collecting such data requires deploying and configuring diverse applications, which is slow and costly.
AutoGUIWorld addresses this bottleneck by combining pretrained image generators with a planner to synthesize trajectories without ever running the target software.
A GUI trajectory is a step‑by‑step record of screenshots, the mouse/keyboard actions that produced them, and the resulting visual changes.
**Figure 1.** Overview of AutoGUIWorld as an environment-free GUI data engine. It samples a structured GUI world, realizes a seed screenshot, plans action-conditioned transitions, and uses Image2 as a visual world model to synthesize coherent screenshot-action-screenshot trajectories.
The bottleneck of GUI‑agent training is the scarcity of diverse, high‑quality interaction trajectories.
Formalizing GUI Interaction
We formalize GUI interaction as a POMDP and factorize trajectories for visual world modeling.
GUI interaction is a sequential decision problem where the agent never sees the full system state, only rendered screenshots.
The environment’s complete state is hidden; the agent perceives only screenshots and issues OS‑level actions.
Because a single screenshot cannot capture scroll positions, dialog stacks, or in‑progress edits, we condition the policy on the full history to preserve the information needed for correct decision‑making.
Step 0: Given h₀ = (∅, ∅), the policy yields a₀ = “click Start” with probability $\pi$(a₀ | h₀, $\tau$).
Step 0 transition: P(\tilde{s}_1 | h₀, a₀) produces a screenshot of the Start menu.
Step 1: History h₁ = (\tilde{s}_0, a₀) now includes the desktop screenshot and the click. The policy yields a₁ = “type Notepad” with probability $\pi$(a₁ | h₁, $\tau$).
Step 1 transition: P(\tilde{s}_2 | h₁, a₁) produces a screenshot of the Notepad window.
Joint probability: Multiply the initial joint p($\tau$, \tilde{s}_0) by the two policy probabilities and the two transition probabilities to obtain the full trajectory probability.
This concrete walk‑through shows why conditioning on the full history is essential: the second action depends on the screenshot generated after the first click.
AutoGUIWorld Overview
AutoGUIWorld plans actions then renders visual states to synthesize GUI trajectories without real execution.
Running a GUI agent in a real environment is slow and costly, limiting the diversity of training trajectories.
AutoGUIWorld first plans a sequence of high‑level actions, then asks an image generator to paint each resulting screen, stitching together a full trajectory without ever touching the real OS.
Step 0: start with screenshot $\tilde{s}_{0}$ showing the two buttons.
Step 0 prompt $r_{0}$ encodes “click button 1 → highlight”. $MV_{\phi}$ renders $\tilde{s}_{1}$ where button 1 is highlighted.
Step 1 prompt $r_{1}$ encodes “click button 2 → show settings”. $MV_{\phi}$ renders $\tilde{s}_{2}$ with the settings panel visible.
The example shows how the planner’s abstract actions are turned into concrete visual updates without any real GUI execution.
How does AutoGUIWorld differ from a traditional GUI simulator that runs the actual application?
Traditional simulators execute the real program stack to produce each screen, incurring OS calls and rendering overhead. AutoGUIWorld instead predicts the visual outcome with a learned image model, so it can generate many diverse trajectories instantly while still respecting the intended action order.
Visual World Model Synthesis
The method details describe how AutoGUIWorld builds a visual world model and uses Voyager to render GUI trajectories.
The core challenge is generating realistic GUI screenshots without running the actual applications. AutoGUIWorld solves this by separating planning from visual synthesis.
A visual world model is a generative image system that, given a structured description of a GUI state, produces a realistic screenshot.
How does this visual world model differ from a generic text‑to‑image generator?
Generic generators take an unconstrained text prompt and produce any image, often ignoring platform‑specific UI conventions. The visual world model, by contrast, is conditioned on a structured GUI description that encodes OS constraints, visual style, and concrete UI elements, guaranteeing that the output is a valid screenshot for the target operating system.
Combine the three factors into the description: “Windows desktop, dark theme, one window containing a centered Save button.”
Pass the description to Image2, which renders a 256×256 pixel screenshot showing the dark‑themed window with the Save button.
The resulting image $\tilde{s}_0$ is stored as the seed and later used as the reference for all actions.
The model’s ability to honor platform constraints ensures that downstream tasks (e.g., clicking the Save button) are grounded in a realistic UI layout.
Voyager iteratively observes the current GUI image, decides the next atomic action, and asks Image2 to render the resulting state, forming a closed‑loop trajectory.
Why is Voyager’s closed‑loop rendering not just another image‑to‑image translation pipeline?
Standard image‑to‑image pipelines transform a source image into a target image without explicit reasoning about actions. Voyager, however, reasons over a planned action sequence, generates a textual prompt that describes the intended GUI change, and feeds that prompt back into Image2. This explicit action‑conditioned loop preserves the semantics of the interaction, not merely the visual appearance.
Step 1: Voyager observes $\tilde{s}_0$, receives action “click button”, produces prompt “Add a new document window to the desktop”.
Image2 renders $\tilde{s}_1$, now showing a fresh document window beside the original.
Step 2: Voyager observes $\tilde{s}_1$, receives action “type ‘Hello’”, produces prompt “Insert the word Hello into the new document”.
Image2 renders $\tilde{s}_2$, displaying the document with the word Hello typed.
The example shows how each action’s semantic intent is turned into a concrete visual change, preserving both functionality and appearance.
**Figure 2.** Overview of AutoGUIWorld. The method first samples a structured GUI world from an OS substrate, visual appearance space, and initial GUI state space. It then realizes a seed screenshot, generates seed-conditioned tasks, plans atomic actions, and rolls out a clean sequence of GUI visual states using Image2 as a visual world model.
**Figure 3.** Voyager-guided closed-loop rendering. At each step, Voyager observes the current GUI image and receives the planned action. It produces a first-person thought, an action abstract, and an after-action rendering prompt. Image2 uses the current image as an image-to-image reference and renders the next GUI state, which is returned to Voyager for the next iteration. The bottom trajectory shows how one planned action sequence becomes a temporally consistent chain of rendered GUI states.
Action 1: Type the requested sentence into the blank Notepad document.
Action 2: Press Ctrl+S to open the Save As dialog.
Action 3: Click the Documents folder in the Save As dialog navigation pane.
Action 4: Click the File name text box.
Action 5: Select all text in the File name box.
Action 6: Type `draft.txt` into the File name box.
Action 7: Click the Save button.
Action 8: Terminal report (no further visual change).
Data Feature Overview
AutoGUIWorld expands GUI training coverage and aligns visual features with real data.
AutoGUIWorld covers four OS domains, expanding interface coverage beyond the single‑domain ScaleCUA.
Training set includes Chrome, Ubuntu, Windows, and macOS totaling 79,266 samples.
**Domain composition**
Visual Alignment and Distribution
Key quantitative results on visual alignment, interface coverage, and trajectory complexity.
AutoGUIWorld boosts UI coverage over ScaleCUA by up to 77 % across the four domains.
Relative gains range from 63 %–77 % on Ubuntu, 51 %–75 % on Windows, 49 %–71 % on Web, and 10 %–13 % on macOS.
Beyond raw coverage, the visual features of the synthetic trajectories remain closely aligned with real data, as shown by distribution‑matching metrics.
**Figure 5.** Distribution alignment in the Qwen3.5-9B visual-token space. Panel a reports the matched synthetic–ScaleCUA MMD normalized by the median real–real MMD; horizontal bars give group-bootstrap 95% confidence intervals and $\rho = 1$ marks the empirical real-data scale. Panel b shows every available real–real distance, the matched synthetic–ScaleCUA estimate with its interval, and synthetic distances to the remaining real sources.
**Figure 6.** Exploratory DINOv2 MMD using 500 images per source. Gray and blue spans collect the available real–real and synthetic–real source-pair distances within each domain; single available comparisons appear as points. Ubuntu is the only domain whose synthetic–real range lies wholly above the observed real–real range.
**Figure 7.** Low-level appearance and generation-residual diagnostics. Panel a places each synthetic random-patch statistic against the range of real sources in the same domain; the gray spans show real-source ranges, and OCR denotes recognition confidence. Panel b reports content-matched ROI C2ST AUC; 0.5 is chance. Text and icon patches remain near chance, while Ubuntu flat backgrounds retain the clearest residual.
Table 2 details the synthetic trajectory statistics that underpin the coverage gains.
Ubuntu trajectories average 11.9 steps (P90 = 24) with the highest sample count (1394); Windows averages 11.4 steps (P90 = 25); macOS shows shorter sequences (7.1 steps) but the highest action‑type entropy (0.92).
Table 8 reports random‑patch MMD diagnostics, confirming that Ubuntu’s synthetic‑real discrepancy exceeds the real‑real baseline, whereas the other domains fall within or below it.
Table 10 presents static‑complexity measurements at three confidence thresholds; Ubuntu’s flat‑background complexity (0.65) surpasses Windows (0.54), while text and icon‑edge complexities are comparable.
AutoGUIWorld delivers realistic visual features and substantially richer GUI trajectories, expanding both spatial coverage and interaction complexity while staying within natural visual variation.
Agent Performance Evaluation
Synthetic trajectories boost real GUI task performance.
Recall that AutoGUIWorld synthesizes visual trajectories to replace costly live GUI runs. This section shows how that synthetic pre‑training transfers to real software interaction.
AGW‑35B raises the equal‑weight mean of the four benchmark scores from 23.6 % to 36.5 %.
Measured on OSWorld, macOSWorld, Windows Agent Arena, and ScienceBoard at checkpoint 469.
Compared to the AgentNet run that used 350 k real demonstrations, AGW‑35B (80 k synthetic examples) outperforms the best AgentNet checkpoint on all four benchmarks, with the largest gap on ScienceBoard (32.2 % vs 16.8 %). This demonstrates that high‑quality synthetic data can match or exceed real‑world demonstrations for post‑training.
**Figure 8.** Static interface complexity across OmniParser confidence thresholds. Panel a compares the rasterized union coverage of synthetic and ScaleCUA screenshots; the annotated percentage is the relative gain at the middle threshold, 0.15. Panel b reports the element-count difference, synthetic minus ScaleCUA. Coverage is higher in all twelve comparisons, including Windows where the detector returns fewer synthetic elements.
**Figure 9.** Element detection and OCR examples on Windows interfaces. Each row shows the same full-screen view without annotations, with blue OmniParser boxes at confidence threshold 0.15, and with teal OCR regions. Row summaries report full-image counts and mean OCR confidence.
**Figure 10.** Task execution scores for Qwen3.5-35B-A3B and AGW-35B (checkpoint 469), with all OSWorld and WAA domains. The top Overall averages the four benchmark scores equally. Within each benchmark, Overall averages task scores, retaining partial credit on OSWorld and WAA. Gray shows base scores, blue shows gains, and hatching marks decreases. Blue ticks mark AGW-35B scores. n counts tasks, and gains are computed before rounding.
Related Work
Related work, discussion, and author contributions contextualize our approach.
We situate our work among prior efforts to collect GUI interaction data and to model GUI dynamics, then discuss limitations, future directions, and the team behind the research.
Methods for acquiring sequences of GUI actions and screenshots. Human demonstrations provide direct supervision, record‑based pipelines extract traces from tutorials, and automated agents synthesize new trajectories through exploration and task synthesis.
Predictive models that simulate GUI state transitions. Approaches range from programmatic executables and learned predictors to image‑based generators that render future screens.
ScaleCUA normalizes GUI action costs across environments, allowing a single policy to be evaluated on heterogeneous software without bias.
Image2‑based world models (e.g., Image2) struggle with long‑horizon consistency; generation errors accumulate, yielding hallucinated screens or incorrect action outcomes. Nonetheless, synthesizing trajectories without executing software offers a low‑cost avenue for expanding training data.
**Figure 11.** All task domains on macOSWorld and ScienceBoard. Gray bars show base success rates, blue extensions show gains, and gray hatching marks decreases. Blue ticks identify AGW-35B scores. Overall uses all tasks in each benchmark. ScienceBoard uses the 143-task subset in Appendix C.2.
**Figure 12.** Post-training performance of AGW-35B (blue circles) and the AgentNet-trained model (orange dashed lines and squares). Gray marks the base, shared across all four benchmarks. Curves use actual training steps and panel-specific score ranges.
**Figure 13.** ScreenSpot-Pro accuracy for Qwen3.5-35B-A3B and AGW-35B (checkpoint 469). Gray shows base scores and blue shows training gains. The top group reports overall, icon, and text accuracy; each domain reports icon and text separately. $n$ counts all examples in each group.
Our audit of 42,526 desktop transitions flagged 1,293 (3.04 %) action–image mismatches; after filtering invalid steps, 818 of 39,351 (2.08 %) remain inconsistent, chiefly in text entry, dragging, scrolling, and keyboard interactions.
Future work should train visual world models on domain‑specific interaction data, improving long‑horizon fidelity and enabling more reliable trajectory generation for specialized software environments.
Core contributors: Cheng Yang, Yifan Wu; additional contributors include Yutao Huang, Zhaohua Zhang, and many others. Supervisors: Tianwen Jiang, Jihong Zhang, Yuyu Luo.
Transition Fidelity and Filtering
Evaluating how faithfully Image2 reproduces GUI actions and filtering out low‑quality trajectories.
GUI agents rely on diverse interaction trajectories; AutoGUIWorld generates them at scale by using a Visual World Model as a surrogate environment.
This subsection describes the VLM quality‑control (QC) protocol that audits each desktop transition and filters out steps that violate five image‑action dimensions.
**Structured verdict and deterministic normalization.**
The transition audit flags 1,293 inconsistent steps out of 42,526 evaluated desktop transitions.
1,293 / 42,526 ≈ 3.04 % overall action‑image inconsistency.
After discarding steps with invalid actions or missing targets, the inconsistency drops to 818 out of 39,351 remaining transitions.
818 / 39,351 ≈ 2.08 % filtered failure rate.
The table presents failure rates across different operating systems (Ubuntu, Windows, macOS) for both "Evaluable" and "Filtered" categories. It includes columns for the total count, the number of failures, and the calculated failure rate percentage for each OS, with a final row summarizing the totals for "All" operating systems.
Failure Rate Analysis
Failure analysis reveals action‑dependent error rates and showcases typical trajectory breakdowns.
This section quantifies how often synthesized GUI trajectories fail, highlights which actions and platforms are most error‑prone, and illustrates concrete failure cases.
Text‑entry failures dominate Ubuntu terminal workflows, while Windows exhibits the highest drag, hotkey, and document‑editing error rates; click errors drop sharply after target and action checks, but text‑entry errors remain largely unchanged.
Representative failure modes include exact‑content substitution (replacing a command or value with a plausible but incorrect one), semantic over‑completion (producing a final visual artifact without intermediate formatting steps), and unintended content mutation (e.g., a cursor‑movement shortcut altering persistent code).
Missing or incorrect action effects also appear: drag and scroll actions may leave the region unchanged, clicks can fail to toggle controls, and key presses sometimes trigger unrelated application states.
Qualitative trajectory examples cover 31 Ubuntu, 45 Windows, 16 macOS applications and 67 Chrome websites, each showing an initial state, a key spatial action, and the resulting final state.
Subsections G.1–G.4 present trajectory galleries for Ubuntu desktop, Windows 11, macOS, and Chrome respectively (figures omitted).
Action Space Definitions
Defines the unified action space and coordinate format used for GUI trajectories.
We define a unified `computer_use` interface that enumerates 14 primitive actions, each serialized as ordered tool calls.
Coordinates are expressed as $C(x, y)$ on a resolution‑independent 0–999 integer grid; dragging uses $C1$ and $C2$ for start and end points.
AutoGUIWorld leverages this interface to synthesize trajectories at scale.
Training and Inference Harness
Details the message format, history handling, and training setup for GUI agents.
The harness uses a single JSONL‑based message builder that packs the task description, screenshots, and the rolling interaction history into a multimodal record.
Message construction for one step
The history window keeps the five most recent steps (including the current one) with full screenshots, while the five steps before that are stored as plain‑text action descriptions without images.
Assistant responses follow a strict serialization: they begin with “# Step N:”, then an imperative Action line, and finally one or more blocks containing JSON‑encoded tool arguments.
Inference loop that parses and executes tool calls
During training, only the final assistant response of each example contributes to the loss (`loss_scale`=`last_round`); earlier messages serve purely as context.
Training fine‑tunes Qwen3.5‑35B‑A3B with a mixture‑of‑experts backbone augmented by Gated DeltaNet, running on 32 GPUs (4 nodes) with tensor‑parallel = 2, pipeline = 1, data‑parallel = 16, and expert‑parallel = 8 for 256 experts.
Key training hyper‑parameters
Evaluation Infrastructure
Details of the evaluation setup, task sets, and score aggregation.
We evaluate six checkpoints (80, 157, 235, 314, 392, 469) from the AGW‑35B run, which fine‑tunes Qwen3.5‑35B‑A3B on AutoGUIWorld trajectories. Inference follows the shared message format and `computer_use` interface; responses contain an action description and tool calls without a separate thinking block. Point coordinates lie on a 0–999 grid and are mapped to screen pixels by multiplying by the screenshot dimension and dividing by 1,000.
For comparison we fine‑tune the same base model on ~350 k examples from AgentNet and evaluate at steps 40, 80, 160, 320, 686. AgentNet data comprise human demonstrations on Windows, macOS, and Ubuntu, augmented with generated reasoning annotations, and its harness enables thinking while AGW‑35B uses non‑thinking responses. On the ScienceBoard benchmark the step‑40 score is 15.4 % (22 successful tasks out of the fixed 143‑task set).
All runs are served through vLLM with tensor parallelism of two. OSWorld and WAA execute inside Docker‑managed QEMU VMs, macOSWorld uses Docker‑OSX, and ScienceBoard relies on the OSWorld Docker provider. Benchmarks run with `max_steps` = 100, meaning each step corresponds to one agent turn: generate a response, execute tool calls, receive a fresh screenshot, and repeat.
**Table 4.** Evaluation task sets and scoring units. Each checkpoint uses the same task set within a benchmark. Counts describe the evaluated samples, not model results.
The table lists reported issues, their corresponding task identifiers, and the effect on evaluation.
Data Feature Numeric Results
Appendix D lists the full numeric tables that support the paper’s feature analyses.
This appendix reproduces the raw numeric results referenced in the main text, enabling precise inspection and figure recreation.
**Table 7.** Exploratory DINOv2 MMD using 500 images per source. Ranges collect the available source-pair distances within each domain.
**Table 9.** Content-matched patch C2ST AUC. Values near 0.5 indicate chance-level separation.
Prompt Catalog and Planning
This section catalogs the prompt templates and planning logic used to synthesize GUI interaction trajectories.
AutoGUIWorld synthesizes interaction trajectories through a multi-stage pipeline: seed realization, task generation, action planning, visual transition synthesis, and target annotation. The system relies on structured prompt templates to ensure consistency across diverse operating environments.
The seed describer transforms high-level platform and environment metadata into a detailed, photorealistic prompt that guides the visual world model to render the initial desktop state.
The task generator creates executable GUI operations by balancing semantic diversity with platform-specific constraints. It explicitly supports "trap tasks"—plausible-sounding but infeasible requests—to test an agent's ability to verify UI state before attempting an action.