SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments
Tian Jin, Wenhao Wang, Jing Kang, Haoting Shi, Rui Ye, Yijun Lu, Siheng Chen
SoftwareGym automates the synthesis of diverse, reliable desktop software environments to scale computer-use agent training.
How can we automatically synthesize diverse, reliable, and executable software environments at scale to train computer-use agents?
Training capable computer-use agents requires diverse, reliable interaction data, but manually building and integrating real-world software environments is slow, expensive, and prone to overfitting. SoftwareGym is an automated pipeline that synthesizes standalone desktop applications by generating functional specifications, diverse UI blueprints, and interaction schemes, then iteratively refining them through GUI-based evaluation and repair. This approach produces 182 runnable applications, and training on these trajectories improves out-of-domain OSWorld performance from 41.39% to 56.97% Pass@3.
Paper Primer
SoftwareGym functions as a multi-agent factory: it designs software variants, implements them as executable code, and uses a GUI-interaction loop to verify and repair functionality. The core move is the "evidence gate"—only applications that pass both source-level audits and automated GUI-based functional tests are admitted into the training library.
Synthesized environments provide transferable supervision that improves performance on real-world, out-of-domain tasks.
Models trained exclusively on SoftwareGym trajectories were evaluated on the OSWorld benchmark, which contains entirely different, real-world desktop applications. The model improved OSWorld Pass@3 from 41.39% to 56.97% (a 15.58 percentage point gain).
Iterative GUI-guided repair is essential for creating reliable training environments.
Ablation studies comparing source-only repair to GUI-guided repair showed that GUI feedback significantly boosts the final functional pass rate of synthesized applications. GUI-guided repair reached an 83.1% pass rate, compared to 48.8% for source-only repair.
Why is this approach better than simply using existing browser-based software environments?
Browser-hosted environments are limited by their shared substrate (HTML/DOM), which restricts interaction depth and workflow complexity. SoftwareGym synthesizes standalone desktop applications, preserving the realistic states and complex, persistent workflows required for robust agent training.
Does this pipeline require significant manual effort to scale?
No; the pipeline is designed for scalability. It expands minimal human input into detailed functional specifications and diverse UI designs, automating the entire design, implementation, and verification process to reduce development time and costs compared to manual construction.
Researchers can now bypass the bottleneck of manual environment construction by using SoftwareGym to generate large-scale, diverse, and verified software libraries for training generalist computer-use agents.
Introduction and Motivation
We expose the training bottleneck: lack of diverse, reliable software environments for computer-use agents.
Training capable computer‑use agents (CUAs) is hampered by three intertwined challenges: (1) obtaining diverse, realistic interfaces; (2) guaranteeing functional reliability of generated trajectories; and (3) scaling the creation process without prohibitive engineering effort.
Existing native applications demand extensive manual construction, limiting coverage and risking application‑specific overfitting, while browser‑hosted environments, though easy to deploy, restrict interaction depth and workflow complexity. SoftwareGym addresses these gaps by automatically producing structurally diverse GUI designs, turning each design into a runnable application, and verifying functionality through checkpoint‑based GUI evaluation with iterative repair.
OSWorld is a benchmark that evaluates how well a computer‑use agent can execute a broad set of OS‑level tasks across many software applications.
**Figure 1.** Representative applications synthesized by SoftwareGym. Pic-Editor, VideoPlayer, TableEditor, and CodeEditor represent diverse software categories and GUI designs. Each row traces a representative multi-step workflow, demonstrating substantial interaction depth.
The primary bottleneck in CUA training is the lack of diverse, reliable software environments.
The SoftwareGym Pipeline
We detail how SoftwareGym designs, diversifies, and verifies synthetic applications.
Existing benchmarks rely on a fixed pool of real applications, which limits diversity and forces agents to overfit to a narrow set of GUIs.
The pipeline turns a high‑level software type into a fully verified desktop application, then extracts executable GUI tasks from it.
How does this pipeline differ from simply re‑using existing desktop applications as environments?
Existing environments are static snapshots of real software; SoftwareGym synthesizes new applications from scratch, guaranteeing that every function and UI element is deliberately specified and automatically verified, thus providing unlimited, controllable diversity.
Function design writes a contract that says exactly what a user can do, what the GUI should show, and how the internal state must change.
Function 1 records: entry = “New”, action = click “File → New”, state change = empty document created, GUI response = blank editor appears, checkpoint = editor visible.
Function 2 records: entry = “Edit”, action = click “File → Save”, state change = document written to disk, GUI response = “Save successful” toast, checkpoint = toast displayed.
During evaluation the GUI agent clicks “File → New”, sees the blank editor, then clicks “File → Save” and verifies the toast, satisfying both contracts.
By encoding both the user action and the expected GUI feedback, the contract eliminates ambiguity about whether an implementation behaved correctly.
Style diversification creates multiple UI blueprints that implement the same functions but differ in layout, navigation, and visual identity—like designing houses with the same rooms but distinct floor plans.
Normalize both trees: each consists of three nodes (window, toolbar, editor).
Compute tree‑edit distance: one move operation (swap) yields distance = 1.
Threshold set to 2; distance = 1 < 2, so Blueprint B is regenerated with a different arrangement (e.g., sidebar navigation).
Even a single structural change can push the distance above the threshold, ensuring that the final set of styles is genuinely diverse.
Icon design first creates a style‑independent manifest of semantic roles, then each style‑specific designer renders distinct SVGs for those roles.
**Figure 2.** SoftwareGym synthesizes diverse, reliable, and executable applications through design, implementation, and evaluation. Design produces rich functional specifications and distinct interfaces. Implementation realizes and verifies the applications. Evaluation tests their GUI behavior and returns failures for subsequent implementation repair.
**Figure 3.** SoftwareGym grounds every synthesized task in capabilities verified through GUI interaction. It first extracts evidence-supported capabilities, then assigns coherent combinations to difficulty- and state-controlled slots, and finally realizes each slot as a natural request with an explicit initial state and expected outcome.
Implementation and Verification
Turn design specs into working apps, evaluate via GUI tests, and iteratively repair failures.
Manual coding of GUI applications is error‑prone; a single missing binding can break an entire workflow.
A Verified Design Package is a complete, machine‑readable blueprint that lists every GUI function an app must expose together with the expected visual layout.
CODER creates an Avalonia project skeleton and adds a menu bar.
It generates command bindings for OpenFile and SaveFile and wires them to the menu items.
It inserts placeholder code that opens a file‑picker dialog for OpenFile.
It adds a file‑write routine for SaveFile, respecting the persistence contract.
DEVCODER runs the functional manual, confirms both commands exist and are reachable from the menu.
The package makes the implementation fully traceable: every menu action can be mapped back to a verified function.
When a GUI test fails, the system pinpoints the missing or incorrect code and patches only that part, keeping all previously correct behavior intact.
GUIAGENT executes the Save test, reaches the Save button, and records a failure because the button is disabled.
DEVCODER inspects the functional manual, finds that SaveFile should be bound to the Save button.
It inserts the missing command binding in the source code and recompiles.
GUIAGENT reruns the Save test; the button is now enabled and the file is written successfully.
The repair fixes only the missing binding, leaving the already‑working OpenFile functionality untouched.
How does this repair differ from a typical code refactoring?
Refactoring rewrites larger code regions without concrete failure evidence, whereas GUI‑guided repair is driven by a specific GUI failure trace and patches only the exact missing or incorrect piece, preserving all previously verified behavior.
CODER generates an application from the Verified Design Package.
EVALTASKGENERATEAGENT creates a fixed set of GUI test cases from the functional manual.
GUIAGENT runs the tests, records any failures, and computes the functional pass rate $P_{\text{func}}$.
DEVCODER uses the failure traces to perform localized implementation repairs.
Repeat the evaluate‑repair cycle until all cases pass or the version budget is exhausted.
**Figure 5.** SoftwareGym makes quality-controlled synthesis practical at scale. Bars decompose per-application resource use across design, implementation, and evaluation.
**Figure 6.** Iterative evaluation and implementation repair consistently improve software reliability across the synthesized application library. (a) GUI pass-rate gains (b) Source-code growth (c) Category-level pass rates
Experimental Setup
Key details of software synthesis, training, and the benchmark used for evaluation.
Seed 2.1 Pro achieves perfect performance on at least one SoftwareGym‑Bench category.
On the Pic category, Seed 2.1 Pro reaches 100.0% Pass@3, the highest among all evaluated agents.
SoftwareGym‑Bench is a curated suite of 120 tasks drawn from twelve software categories, designed to test agents’ ability to manipulate GUIs while handling distractors and protected objects.
**Table (a) OSWorld** and **(b) SoftwareGym-Bench**
**Table 2.** Pass@3 (%) on SoftwareGym-Bench. Within each model category, bold and underlined scores denote the best and second-best results, respectively.
SoftwareGym‑Bench covers 12 categories with 120 tasks.
Main Results
SoftwareGym supervision yields large OSWorld performance gains and competitive in‑domain results.
SoftwareGym‑only supervision raises OSWorld overall Pass@3 from 41.39 % to 56.97 %.
Table 1 reports the improvement across all eight OSWorld application domains.
The same trend appears in‑domain on SoftwareGym‑Bench, where the 9 B base model’s Pass@3 climbs from 25.8 % to 40.0 %.
SoftwareGym supervision lifts the 9 B model’s Pass@3 on SoftwareGym‑Bench by 14.2 %.
Table 2 records the increase from 25.8 % to 40.0 %.
**Table.** Performance evaluation of various models across different software applications (Chrome, GIMP, Calc, Impress, Writer, Thunderbird, VLC, VS Code) and an overall metric.
Iterative evaluation‑guided repair raises GUI task pass rate from 54.9 % (v1) to 91.5 % (v3).
Section 4.2 reports the progressive improvements across versions.
SoftwareGym‑only supervision improves OSWorld performance.
Benchmark Construction Details
We isolate the impact of design style, GUI‑guided repair, and training data on benchmark performance.
To understand which parts of the pipeline drive performance, we run three ablations that each remove a key component while keeping everything else constant.
Structured style design dramatically increases structural diversity of generated interfaces.
Full Design raises the macro‑average tree‑edit distance score from 4.9 to 83.6, outperforming the simplified baseline across all categories.
GUI‑guided implementation repair yields a far higher final pass rate than source‑only repair.
Starting from an identical 50.4 % initial pass, GUI‑guided repair reaches 83.1 % while source‑only repair stalls at 48.8 %.
Training solely on SoftwareGym‑generated trajectories attains strong performance on the OSWorld benchmark.
A model trained only on SoftwareGym trajectories achieves 56.97 % on OSWorld, despite never seeing native OSWorld data during training.
Combining SoftwareGym and OSWorld trajectories further improves performance on the SoftwareGym‑Bench.
The mixed‑trajectory model raises the SoftwareGym‑Bench score from 40.0 % (SoftwareGym‑only) to 47.5 % when OSWorld data are added.
**Figure.** A six-step process demonstrating how to use the "Find and Replace" feature in a spreadsheet application. - **Step 1:** Locate "Draft total". - **Step 2:** Open Edit. - **Step 3:** Open Find and Replace. - **Step 4:** Enter "Final total". - **Step 5:** Open File. - **Step 6:** Save with panel open.
**Figure 9.** A successful PicEditor interaction that processes two images as JPEG files at 640-pixel width and quality 72, leaves both queue rows marked Done, and keeps the Batch Queue page open.
**Figure 10.** A successful MusicPlayer interaction that enables the favorites-only library filter and leaves the filtered Library view open.
The image displays a five-step process for uninstalling an extension in a software interface: **Step 1:** Open the View menu. **Step 2:** Open Extensions. **Step 3:** Select rainbow-indent. **Step 4:** Start uninstall.
**Figure 12.** A successful ForumAPP interaction that creates the public, open-join Civic Desk community with the requested summary.
**Figure 13.** A successful ChatAPP interaction that attaches `project_brief`.txt to the Loom Council conversation.
Ablation Studies
Ablations isolate how design style, GUI‑guided repair, and training data source each affect agent performance.
The first ablation tests whether enforcing a rich style‑design pipeline yields more structurally diverse blueprints than a minimal “Simple” pipeline.
Full Design raises the macro‑average structural‑diversity score from 4.9 to 83.6.
Table 4 reports a macro‑average of 83.6 for Full Design versus 4.9 for Simple Design.
**Table 4.** Full Design consistently produces more diverse blueprint structures across all twelve categories. Results are on a 0–100 scale; each category contains one set of ten blueprints per method.
The second ablation asks whether feeding GUI evaluation feedback into the repair loop improves functional reliability.
Evaluation‑guided repair lifts the GUI task pass rate from 50.4 % to 83.1 %, a +32.7 % gain over source‑only repair.
Table 5 shows the evaluation‑guided branch reaching 83.1 % at v3 while the source‑only branch falls to 48.8 %.
**Table 5.** Evaluation-guided implementation repair substantially improves functional reliability, whereas source-only repair remains near the shared starting level. Macro-average GUI task pass rate with 95% hierarchical bootstrap confidence intervals.
The final ablation compares models trained on trajectories from SoftwareGym, from native OSWorld, and from both sources combined.
SoftwareGym‑only training attains 56.97 % Pass@3 on OSWorld, only 4.5 % below the OSWorld‑only model.
Table 6 reports 56.97 % Pass@3 for the SoftwareGym‑only model versus 61.48 % for the OSWorld‑only model.
**Table 6.** Complete OSWorld results for the training-software-source ablation. Run rows report average evaluator scores; Avg@3 averages the three runs, and Pass@3 reports task-level success across them.
**Table 7.** Complete SoftwareGym-Bench results for the training-software-source ablation. Run rows report binary success rates; Avg@3 averages the three runs, and Pass@3 reports task-level success across them.
OSWorld Results
OSWorld per-run scores reveal GPT‑5.4 leads with the highest Pass@3.
GPT‑5.4 achieves the highest Pass@3 score on OSWorld.
Pass@3 of 92.31 % (Run 1) and an overall Pass@3 of 92.31 % across three runs.
**Table.** Performance evaluation of various AI models across different task categories (Pic, Mail, PDF, Music, Code, Forum, Writer, Browser, Video, Slides, Table, Chat) and their overall scores.
SoftwareGym-Bench Scores
SoftwareGym‑Bench run scores and pass rates across models.
Qwen3.5‑9B SFT (Ours) achieves a Pass@3 rate up to 20 percentage points higher than prior baselines.
It reaches 80 % Pass@3 versus 60 % for Seed 2.1 Pro and 40 % for Qwen3.5‑9B (OS‑World).
Other models achieve lower Pass@3 rates: Seed 2.0 Pro and Pic Mail reach 100 % but only on limited categories, while most baselines hover between 40 % and 60 %.
Detailed Performance Metrics
CUA maintains a 60.00 overall average across diverse tasks.
The Computer-use agent (CUA) attains an overall average score of 60.00 across the 11 benchmark tasks.
Table 9 (continued) lists per‑task percentages; the overall column averages to 60.00.
Trajectory Scoring and Filtering
How trajectories are filtered, converted, and split for training.
We filter generated GUI trajectories with a two‑stage scoring pipeline before adding them to the training corpus.
First we verify that the final GUI state satisfies the user request, then we verify that the intermediate actions form a sensible, safe, and efficient interaction.
Perform a structural audit of each generated trajectory.
Run the outcome judge (see Prompt 1) on the final five assistant messages and their screenshots.
Run the process judge (see Prompt 2) on the full message sequence and up to ten sampled screenshots.
Retain only trajectories where both judges output a score ≥ 0.8.
Convert retained trajectories from the teacher schema to Qwen3.5 multimodal format.
Validate the converted trajectories for strict message‑format and consistency; discard any that fail.
Apply source‑specific selection: keep the highest‑scoring qualified attempt per OSWorld task directory; keep all qualified SoftwareGym attempts.
Cap each frozen semantic group at two trajectories for SoftwareGym; balance OSWorld domains to match corpus size.
Split the final set into group‑constrained partitions with a 90/5/5 ratio, ensuring no overlap of frozen groups across splits.
Measures and Definitions
Defines key metrics for evaluating generated UI layouts and their implementation fidelity.
This section formalizes the metrics used throughout the paper. They quantify layout variety, structural differences, visual consistency, and functional completeness of synthesized applications.
Layout‑Archetype Distance measures how many high‑level layout dimensions differ between two style variants.
Structural Blueprint Diversity quantifies how distinct the component hierarchies of generated interfaces are.
Visual Fidelity scores how well an implemented application preserves the design tokens of its UI blueprint.
Functional Coverage measures the proportion of real‑world functions that synthesized applications successfully reproduce.
Model Configurations
Provides the complete performance percentages for each model across all evaluated metrics.
The authors include an exhaustive table of percentage scores for each model on the suite of benchmark tasks; the values range from 0.00 % to 60.00 % and are presented in the raw numeric dump.