Environment-Free Synthetic Data Generation for API-Calling Agents

Seanie Lee, Sanjoy Chowdhury, Chao Jiang, Cheng-Yu Hsieh, Ting-Yao Hu, Alexander T Toshev, Oncel Tuzel, Raviteja Vemulapalli

ESAT generates high-quality API-calling agent trajectories using only API specifications, bypassing the need for executable environments.

How can we generate high-quality, multi-step API-calling trajectories for training LLM agents without needing real, executable environments?

Training API-calling agents typically requires building complex, stateful environments and populating them with realistic data, creating a massive bottleneck for scaling to new domains. The authors introduce ESAT, an environment-free pipeline that uses LLMs as on-the-fly digital world models to simulate API responses and state changes directly from API specifications. Fine-tuning on this synthetic data yields significant performance gains, with smaller models often outperforming much larger baselines and even models trained on real-environment data.

Paper Primer

ESAT reduces agentic data generation to a lightweight specification problem by replacing functional backends with a three-stage pipeline: task synthesis, trajectory generation, and quality filtering. The core mechanism is a state-aware LLM simulator that acts like a mail sorter: it reads the API call arguments and the current interaction history, then routes the agent to the correct simulated state update while maintaining schema compliance.

Synthetic ESAT data enables smaller models to outperform significantly larger, state-of-the-art baselines.

Qwen3.5-27B trained on ESAT data approaches the performance of the 120B-parameter Nemotron model, while smaller 4B–8B models consistently beat GPT-4o on AppWorld and OfficeBench benchmarks. Performance gains reach up to 50.5% on AppWorld and 60.5% on OfficeBench compared to base models.

Synthetic data generated from scratch can outperform data collected from real, executable environments.

On the AppWorld benchmark, models fine-tuned on ESAT-generated trajectories outperformed models trained on trajectories collected from the official AppWorld environment. Consistent improvements of 0.7–15.3% over real-environment training data.

Why is this approach more scalable than existing synthetic data methods?

Most prior methods require fully implemented environments and backend databases to execute API calls. ESAT requires only API specifications, allowing it to generate training data for any new API ecosystem without building bespoke infrastructure.

How does the simulator maintain state consistency across long, multi-step trajectories?

The simulator conditions its responses on a rich context window containing the full API specification, task instructions, and the history of previous API calls. It is further constrained by rules for data consistency and schema compliance, with an LLM judge refining responses that fail validation.

Motivation and Background

We expose the environment-dependent data bottleneck and introduce ESAT, a specification‑only synthetic trajectory pipeline.

Training API‑calling agents stalls because gathering multi‑step interaction data requires fully built, stateful environments. This environment‑dependent data collection forms a major scalability bottleneck.

Training API‑calling agents stalls because gathering multi‑step interaction data requires fully built, stateful environments.

To bypass this bottleneck we propose ESAT, an environment‑free synthetic data generation pipeline that uses only API specifications. The pipeline first synthesizes diverse tasks, then a teacher agent interacts with an LLM‑based simulator that generates coherent API responses on‑the‑fly, and finally an LLM judge filters the trajectories for quality.

Evaluations on the AppWorld and OfficeBench benchmarks show that fine‑tuning on ESAT‑generated data yields up to 50.5 % improvement on AppWorld and 60.5 % on OfficeBench, surpassing models trained on real environment data. These gains demonstrate that high‑quality supervision can be obtained without any executable backend.

```json { "app_name": "<action_item_list_app>", "api_name": "create_task", "path": "/<action_item_list_app>/projects/{project_id}/tasks", "method": "POST", "description": "Create a new task within a project.", "parameters": [ { "name": "project_id", "type": "integer", "required": true, "description": "The ID of the project. If set to 0, the task will be created in your default/inbox project.", "default": null, "constraints": [] }, { "name": "title", "type": "string", "required": true, "description": "The title of the task.", "default": null, "constraints": [ "length >= 1" ] }, { "name": "access_token", "type": "string", "required": true, "description": "Access token obtained from <action_item_list_app> app login.", "default": null, "constraints": [] }, { "name": "section_id", "type": "integer", "required": false, "description": "The ID of the section within the project.", "default": null, "constraints": [] }, { "name": "description", "type": "string", "required": false, "description": "The description of the task.", "default": "", "constraints": [] }, { "name": "due_date", "type": "string", "required": false, "description": "The due date of the task in YYYY-MM-DD format.", "default": null, "constraints": [] }, { "name": "duration", "type": "number", "required": false, "description": "The duration of the task.", "default": null, "constraints": [ "value >= 0.0" ] }, { "name": "duration_unit", "type": "string", "required": false, "description": "The unit of the task duration.", "default": null, "constraints": [ "value in ['minutes', 'hours', 'days']" ] }, { "name": "order_index", "type": "integer" } ] } ```

**Figure 11.** Example trajectory 3. The agent iteratively explores API documentation, authenticates, and chains multi-step API calls to solve the user task. All API responses are generated by the LLM-based simulator.

The primary obstacle to scaling API‑calling agents is the environment‑dependent data collection bottleneck.

The ESAT Pipeline

We describe ESAT, a three‑stage pipeline that creates synthetic API‑calling trajectories without real environments.

Training API‑calling agents traditionally requires live services, sandbox environments, and costly orchestration. Without such infrastructure the data‑collection bottleneck stalls progress.

Instead of executing real APIs, we let a language model play both the user and the backend, stitching together tasks, simulated calls, and a judge to produce a full training trajectory – like a playwright drafting a script using only an outline, without ever staging a rehearsal.

How does ESAT differ from conventional pipelines that record real API interactions?

Conventional pipelines capture logs from live services, which demands deployed back‑ends, authentication, and error handling. ESAT replaces the live back‑end with an LLM that generates responses on‑the‑fly from the API schema, eliminating the need for any executable environment.

**Figure 1** ESAT pipeline that generates agentic trajectories using only API specifications as input. First, a task generator LLM composes tasks based on API specifications which are then filtered by a judge LLM. Then, an agent LLM generates solution trajectories by interacting with an API simulator LLM. Finally, a trajectory judge LLM selects high quality trajectories.

Define a combinatorial grid of configuration buckets (difficulty × action × focus × #apps × #APIs).

Randomly sample a bucket, then prompt the task‑generator LLM with the bucket’s attributes and the full API specs of the sampled apps.

Apply inverse‑frequency sampling to prioritize under‑used apps and APIs.

Judge each generated task for solvability and bucket conformity.

Rewrite accepted tasks into concise intent‑level requests and re‑judge for validity.

Prompt the task‑generator LLM with: difficulty=easy, action=read, app=Calendar, API=CreateEvent.

LLM returns the task “Schedule a meeting tomorrow at 10 am” and lists CreateEvent as required.

The judge LLM verifies the task is well‑formed and solvable, then passes it to the rewriting stage.

Rewriting compresses the procedural phrasing to the concise intent “Create a meeting tomorrow at 10 am”.

The final task is accepted for trajectory synthesis.

Bucketized generation forces the system to produce tasks across all difficulty‑action combos, while inverse‑frequency sampling ensures that rarely used APIs eventually appear in the dataset.

The simulator acts as a mock server that answers API calls purely from the specification, much like a stubbed unit‑test that returns canned values consistent with the contract.

Why not simply call the real API instead of using an LLM simulator?

Real APIs require network access, authentication, and may have rate limits or side effects. The LLM simulator produces deterministic, privacy‑preserving responses from the schema alone, enabling unlimited, fast data generation without any external dependencies.

Simulation quality‑check loop

The simulator receives the call and the schema, then produces: {"`event_id`":"E123","status":"created"}.

Schema validation confirms both fields exist and types match.

The judge LLM checks that the response reflects the requested date and title; it approves.

The simulator respects the schema while also injecting realistic fields (e.g., a generated `event_id`), demonstrating the “data realism” rule.

Trajectory generation stops when the agent invokes the special task‑completion API, exceeds a step limit, or calls an API from an app not present in the original task.

The judge LLM acts as a quality inspector, discarding any trajectory that fails to solve its task, similar to a reviewer rejecting a draft that doesn’t meet the specification.

What happens to trajectories that the judge rejects?

Rejected trajectories are discarded and not used for fine‑tuning; the pipeline may retry synthesis for the same task to obtain a passing example.

Performance on AppWorld

ESAT data dramatically boosts API‑calling performance on the AppWorld benchmark.

ESAT data yields up to 47% improvement on the AppWorld benchmark across model sizes.

Table 1 shows gains of 8.4–47.0% over zero‑shot baselines for ESAT‑S52 and ESAT‑S52‑AW7.

A collection of 90 real‑world API‑calling tasks split into a seen set (Test‑N) and a held‑out set (Test‑C) to evaluate generalization.

How does AppWorld differ from synthetic benchmarks like OfficeBench?

AppWorld uses real API specifications and a verifier that runs in an actual environment, whereas synthetic benchmarks rely on fully generated specs and simulated execution, so AppWorld measures true transfer to real‑world APIs.

**Table 1.** Performance on AppWorld test splits. For ESAT-S52 and ESAT-S52-AW7, the numbers in parenthesis show the gains with respect to the base model zero-shot performance. For ESAT-S52-AW7 + AWT, they show the gains with respect to AWT. For Qwen3.5 models, using AWT data on top of ESAT data did not give any performance gains. Hence, we report ESAT-S52-AW7 results for ESAT-S52-AW7 + AWT.

**Table 3.** AppWorld performance of Qwen3-8B trained on ESAT-S52-AW7 data filtered with different judge models.

ESAT data dramatically improves API‑calling agents on AppWorld, delivering up to 47% gains without any real‑environment interaction.

Performance on OfficeBench

Evaluates LLM judge precision and synthetic data gains, showing Gemini filtering excels.

A benchmark of office‑automation tasks used only for evaluation; it provides no training data, so synthetic trajectories must be generated to train agents.

Gemini‑3.1‑Pro filtering attains 95.2% precision when its judgments are compared to real‑environment execution.

Evaluated on 720 trajectories (8 per task across 90 AppWorld training tasks) with ground‑truth labels from execution‑based verifiers.

In the ablation, GLM‑5.1‑FP8 provided mixed downstream effects—slightly improving one metric while degrading the other—whereas Gemini‑3.1‑Pro consistently boosted performance across both pass‑rate and exact‑match metrics.

**Table 2.** Performance on OfficeBench dataset. The numbers in parenthesis show the gains with respect to the base model zero shot performance.

Simulator Quality Analysis

We assess how well the ESAT simulator generates reliable API responses.

Recall that ESAT sidesteps costly environment execution by letting an LLM simulate API calls directly from specifications.

The ESAT simulator yields valid responses for 93.7 % of API calls.

Evaluated 27 K simulated calls (1 K trajectories) with GPT‑5.1 as a judge.

**Figure 2** Simulator failure rate as a function of output token length, as judged by GPT-5.1.

We found no statistically significant correlation between the length of the input context and the simulator’s failure rate.

Compared to prior synthetic datasets—ToolAlpaca, ToolACE, and Nemotron—ESAT’s stateful simulator delivers markedly higher downstream performance because those baselines lack coordinated read/write over a persistent environment.

Tables 4 and 5 confirm this advantage: across Qwen3.5‑2B/4B/9B and Qwen3‑4B/8B/14B models, the ESAT rows (with or without AWT) achieve the highest Test‑N and Test‑C scores, consistently outpacing the other training data configurations.

Extended Results

Full benchmark tables reveal the top‑performing model and the limited impact of upsampling.

Qwen3.5‑27B zero‑shot reaches 73.2 ± 2.4 % Task Goal Completion on the AppWorld Test‑Normal split, the highest among all evaluated models.

Table 16 reports 73.2 ± 2.4 % for Qwen3.5‑27B (zero‑shot) versus lower scores for all other configurations.

Upsampling trajectories to balance per‑task counts does not consistently improve performance; gains on Qwen3.5‑9B are offset by degradations on other models.

**Table 14.** Distribution of failure modes during trajectory synthesis for OfficeBench.

**Figure 8.** API coverage by frequency threshold for OfficeBench apps.

**Table 15.** Effect of upsampling filtered teacher trajectories to 8 per task.

**Table.** Performance comparison of various models on Test-N and Test-C datasets using TGC and SGC metrics.

Data Synthesis Statistics

Statistical tables detail ESAT‑AW7 task and trajectory yields and failure modes.

The task‑generation pipeline starts with 29,676 candidate tasks; after the initial judge only 14,361 (48.39 %) remain, and the final judge approves 12,242 tasks (41.25 % of the original pool). Table 6 quantifies this attrition at each stage.

**Table 5.** Comparison of ESAT-OB data with existing synthetic API-calling datasets in terms of effectiveness on OfficeBench.

For trajectory synthesis, 14,361 input tasks yield 11,445 completed trajectories (79.70 %), and after judge filtering 9,352 trajectories (65.13 %) are retained. Table 7 reports these yields.

**Table 7.** Trajectory synthesis throughput for ESAT-AW7.

Failure analysis (Table 8) reveals that most synthesis failures stem from simulation errors (45.38 %) and judge rejections (41.78 %). Minor sources include unexpected app usage and agent execution failures.

**Table 6.** Task synthesis throughput for ESAT-AW7.

The appendix also lists analogous statistics for ESAT‑S52, OfficeBench, and related components (sections B–L), but detailed tables are omitted here.

Trajectory Filtering Details

This appendix reports yields, failure breakdowns, and coverage statistics for ESAT‑S52.

The following tables quantify how many tasks and trajectories survive each pipeline stage and where failures occur.

**Table 8.** Distribution of failure modes during trajectory synthesis for ESAT-AW7.

**Table 10.** Trajectory synthesis throughput for ESAT-S52.

**Table 9.** Task synthesis throughput for ESAT-S52.

Failure Mode Analysis

Key statistics on ESAT‑S52 and OfficeBench synthesis, failures, and coverage.

This appendix quantifies the generation pipeline for ESAT‑S52 and the OfficeBench benchmark, reporting throughput, failure modes, attribute balances, and API‑coverage statistics.

**Figure 3.** Task (top row) and trajectory (bottom row) distributions of ESAT-AW7 across task focus, action type, difficulty, and the number of apps.

**Figure 4.** API coverage across the 340 AppWorld APIs, showing the number (and percentage) of APIs called in at least $N$ successful trajectories.

Implementation Details

Implementation details, throughput, failures, and dataset coverage for OfficeBench.

The implementation pipeline processes 11 797 generated tasks, of which 3 465 pass the initial judge, 3 336 are successfully rewritten, and 2 380 survive the final judge, yielding a 20.17 % overall pass rate.

**Figure 5.** Task and trajectory distributions of ESAT-S52 across task focus, action type, difficulty, API range and the number of apps.

**Figure 6.** API coverage across our synthetic app APIs, showing the number (and percentage) of APIs called in at least $N$ successful trajectories.

Table 12 details task‑synthesis throughput: from the initial 11 797 tasks, 3 465 (29.37 %) survive the first judge, 3 336 (28.28 %) are rewritten successfully, and 2 380 (20.17 %) pass the final judge.

Table 13 shows trajectory‑synthesis throughput: of the 3 465 input tasks, 1 889 (54.52 %) yield completed trajectories, and 1 711 (49.38 %) remain after judge filtering.

Table 14 breaks down 1 754 discarded trajectories: 1 358 (77.42 %) fail due to agent‑model execution, 178 (10.15 %) are rejected by the judge, 151 (8.61 %) encounter simulation failures, and 67 (3.82 %) exceed the max‑solve‑step budget.

During trajectory generation, 12 084 simulated API calls were issued; only 919 required a retry (7.61 %), indicating that the LLM‑based simulator produces well‑formed responses on the first attempt in the vast majority of cases.

If a rewritten task is rejected at final validation, the pipeline falls back to the corresponding initial task for trajectory synthesis, preserving throughput while maintaining dataset quality.

Task‑distribution analysis (C.3) reveals near‑uniform coverage across task focus (24–26 %), action type (31–35 %), and difficulty (32–34 %). Two‑app tasks dominate (40 %), with single‑app (35 %) and three‑app (25 %) tasks completing the mix.

Trajectory‑distribution analysis shows balanced difficulty (easy 34 %, medium 32 %, hard 34 %), action types (mixed 43 %, read 30 %, write 27 %), and a slight bias toward iteration (29 %) and open (28 %) task focuses.

Training Configuration

Details of dataset coverage, training pipeline, and evaluation settings for ESAT models.

API coverage is measured by counting distinct OfficeBench APIs that appear in at least one successful synthetic trajectory.

Out of 20 available APIs, 14 (70 %) are invoked; the same 70 % appear at least five times, dropping only to 13 APIs (65 %) at ten occurrences, while 12 APIs (60 %) appear fifty times or more.

Four Excel APIs (`convert_to_pdf`, `create_new_file`, `delete_cell`, `set_cell`) and two PDF APIs (`convert_to_image`, `convert_to_word`) are never generated.

Task synthesis (Stage 1) uses GLM‑4.7‑FP8 as both generator and LLM judge; trajectory synthesis (Stage 2) employs GLM‑5.1‑FP8 as teacher and API simulator, with a step limit of 50 for AppWorld and 30 for OfficeBench.

Task‑focus distribution across the ESAT‑OB dataset is: mixed 2,906 (25 %), read 2,999 (25 %), write 2,774 (24 %), open 3,118 (26 %).

Action‑type distribution is: mixed 4,003 (34 %), read 3,607 (31 %), write 4,187 (35 %).

Difficulty distribution is: easy 3,817 (32 %), medium 3,980 (34 %), hard 4,000 (34 %).

Number‑of‑apps distribution is: 1 app 4,133 (35 %), 2 apps 4,735 (40 %), 3 apps 2,929 (25 %).

Trajectory‑focus distribution mirrors the task split: mixed 322 (19 %), read 409 (24 %), write 502 (29 %), open 478 (28 %).

Trajectory‑action‑type distribution is: mixed 729 (43 %), read 513 (30 %), write 469 (27 %).

Trajectory‑difficulty distribution is: easy 580 (34 %), medium 542 (32 %), hard 589 (34 %).

Trajectory‑apps distribution is: 1 app 603 (35 %), 2 apps 751 (44 %), 3 apps 357 (21 %).

Training fine‑tunes all models on full multi‑turn trajectories; each turn is predicted conditioned on all preceding turns with loss masked over context tokens.

We use AdamW with learning rate $2 \times 10^{-5}$, linear decay, and a 1 % warmup; context length is 32,768 tokens, per‑device batch size is 1.

Models < 14 B parameters train on eight H100 GPUs; models ≥ 14 B use eight B200 GPUs, leveraging PyTorch native FSDP2 and the Transformers library.

Sequence packing concatenates roughly four samples per training sequence; we run 10 epochs by default, but truncate Qwen 3.5 training to 5 epochs to avoid over‑fitting.

Checkpoint selection: stage 1 uses both AppWorld train and dev splits; stage 2 fine‑tunes on the train split alone, selecting the best checkpoint on the dev split.

OfficeBench adopts the identical hyper‑parameters because it lacks a dedicated development set.

Inference runs via vLLM with temperature 1.0; maximum steps are capped at 50 for AppWorld and 30 for OfficeBench, and eight independent samples per task yield the average pass@1 metric.

To assess upsampling, we generate up to eight teacher trajectories per AppWorld training task, filter out incorrect ones, then either keep the remaining set (no upsampling) or duplicate trajectories until each task contributes exactly eight samples (upsampling).

OfficeBench API‑coverage remains stable across frequency thresholds, matching the 70 % coverage reported earlier.

Test‑N results (mean ± std) for Qwen variants: 3.5‑4B w/o upsampling 64.29 ± 2.86, w/ upsampling 64.06 ± 4.09; 3.5‑9B w/o 71.21 ± 2.44, w/ 73.14 ± 2.33; 3‑4B w/o 46.70 ± 1.65, w/ 42.39 ± 1.54; 3‑8B w/o 53.84 ± 1.57, w/ 51.89 ± 1.49.

Test‑C results (mean ± std) for Qwen 3.5‑4B w/o upsampling 46.65 ± 2.91, w/ 46.58 ± 4.28; Qwen 3.5‑9B w/o 28.33 ± 1.41, w/ 28.09 ± 1.26; results for Qwen 3‑4B and 3‑8B are not provided.

**Figure 7.** Task and trajectory distributions of ESAT-OB across task focus, action type, difficulty, API range and the number of apps.

Simulator Implementation

Details of the LLM‑based API simulator used for trajectory synthesis.

Table 17 reports OfficeBench results for several Qwen model sizes, comparing zero‑shot baselines with the ESAT‑OB variant; the ESAT‑OB scores consistently exceed the baselines, with the largest Qwen 3.5‑27B reaching 79.3 ± 2.1 overall.

Task synthesis creates a large, diverse pool of multi‑step user tasks from only API specifications. A single LLM (GLM‑4.7‑FP8) plays three roles—generator, judge, and rewriter—by switching prompts.

Bucketized generation organizes the space of possible tasks into a combinatorial grid of configuration buckets. Each bucket is a tuple (difficulty, action type, task focus, num apps, [min APIs, max APIs]), yielding 360 buckets that are processed in random order, each targeting a fixed number of accepted tasks (Nbucket).

Inverse frequency‑based sampling prevents the generator from over‑using a small set of apps/APIs. Usage counts c(a) and c($\alpha$) are maintained; apps are sampled with probability proportional to 1 / (1 + c(a)), and the top‑10 least‑covered APIs are highlighted as focus APIs in the prompt.

Task judging uses the same API specification and bucket‑specific requirements to ask an LLM judge to reason through the task and output a binary yes/no verdict; only tasks that receive a “yes” proceed.

Task rewriting compresses verbose, procedural generator outputs into concise, intent‑level user requests. The rewriter sees the original task, the same requirements block, and only the APIs actually used, and is guided by three in‑context examples that pair a long procedural description with a short rewrite.

Final re‑judging runs the rewritten task through the judge a second time to catch any dropped entities; only tasks passing both judgments enter the final pool.

Prompt templates (Listings 1–8) assemble the system prompt, the per‑call user prompt, and the various clause insertions (difficulty, action‑type, focus) that drive generation, judging, and rewriting.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers