What Makes Good Agentic Data? an ACE Lens on Data Generation for LLM Agents

Xingshan Zeng, Zishan Xu, Boju Zhang, Yuzhou Wu, Lingzhi Wang, Jianghao Lin, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Weinan Zhang, Yong Yu, Qun Liu, Weiwen Liu

A unified framework for agentic data generation using the Accuracy–Complexity–Diversity (ACE) lens.

What are the essential properties of high-quality data for training LLM agents, and how can we systematically evaluate or generate such data?

Agentic data generation is currently fragmented, with domain-specific methods conflating how data is constructed with how it is verified and allocated. This makes it difficult to compare pipelines across tool-use, software engineering, and embodied domains. The authors introduce a factorized data object $(E, q, \tau, v)$—representing environment, task, interaction, and verifier—and organize generation pipelines by their dependency structure. They propose the Accuracy–Complexity–Diversity (ACE) lens to formalize generation as a constrained distribution-design objective. This framework reveals a shift toward execution-grounded accuracy, learner-relative complexity, and behavioral diversity, providing a common language to evaluate and scale agentic experience.

Paper Primer

The paper defines agentic data as a four-factor object: an environment specification ($E$), a task signal ($q$), an interaction realization ($\tau$), and an optional verifier ($v$). This factorization separates the components of an interactive problem from the specific training paradigm, allowing for a unified taxonomy of generation pipelines.

Generation pipelines are categorized by their primary anchor and dependency structure: forward pipelines ground tasks in existing environments, reverse pipelines derive tasks from observed trajectories, and structure-first pipelines use intermediate blueprints to coordinate factors. The ACE lens acts as a gatekeeper: accuracy establishes the feasible support of valid data, while complexity and diversity shape the distribution of useful learning mass within that support.

Accuracy is a non-negotiable admission condition for agentic data.

The authors define accuracy as a conjunctive validity decision $A(d) = V_E(E) \land V_q(q \mid E) \land V_\tau(\tau \mid E, q) \land V_v(v \mid E, q, \tau)$, where invalid data cannot be compensated for by increased complexity or diversity. The framework mandates that if no candidate passes the accuracy gate, the utility of the generation process is zero.

Why is the ACE framework necessary when existing domains already have their own evaluation metrics?

Existing domain-specific metrics conflate generation mechanisms with application domains, making it impossible to compare, for example, a GUI agent's data pipeline with a scientific discovery agent's pipeline. ACE provides a cross-domain analytical lens that separates the construction of data from the objectives used to evaluate its utility.

How does the paper define "complexity" in a way that is useful for training?

Complexity is defined as the difficulty induced by a grounded task for a specific learner configuration, measured by the verified failure probability. Useful data is not simply "hard," but occupies a learnable band near the model's current capability frontier.

Researchers should shift from domain-specific data generation to a factorized approach, using the ACE lens to explicitly separate the validity of the environment and task from the learner-relative difficulty of the interaction.

The ACE Lens for Agentic Data

We define the ACE framework that structures agentic data generation beyond simple instruction synthesis.

Agentic data generation must keep environment, task, interaction, and verifier signals mutually consistent, unlike ordinary instruction synthesis which only cares about surface text.

ACE organizes the three core requirements—Accuracy, Complexity, Diversity—so that generated agentic data is valid, appropriately challenging, and non‑redundant.

How does ACE differ from typical data‑quality checklists used in instruction‑tuning datasets?

Typical checklists focus on surface properties such as grammar or token‑level correctness. ACE, by contrast, requires four‑way consistency (environment, task, interaction, verifier), a learner‑relative difficulty curve, and explicit non‑redundancy across the full tuple $(E, q, \tau, v)$. This makes the data useful for training agents that must act in the world, not just generate text.

**Figure 1.** The ACE framework for Agentic Data, comprising Accuracy, Complexity, and Diversity, which are influenced by the Environment, Task, Interactions, and Verifier components.

**Figure 1.** The ACE lens views agentic data generation as constrained distribution design: Accuracy defines valid support, while learner-relative Complexity and Diversity shape useful, non-redundant experience over $(E, q, \tau, v)$.

Agentic data differs from instruction‑tuning data because it must be simultaneously valid, appropriately challenging, and uniquely diverse for the learner.

Defining the Agentic Data Object

Formulating agentic data as (E, q, $\tau$, v) unifies environment, task, interaction, and verification across domains.

Representing agentic data as a single object (E, q, $\tau$, v) captures the full ACE factors and enables systematic cross‑domain analysis.

We model tool‑augmented multi‑turn interaction as a partially observable Markov decision process (POMDP) — a standard formalism for sequential decision making under uncertainty. The tuple $M = (U, S, A, O, P, R)$ defines user intent, latent state, actions (including language and tool calls), observations, transition dynamics, and optional reward. At turn $t$, the agent selects $a_t$ based on the observable history $h_t$, and the environment returns the next state and observation.

It bundles the environment specification, task description, interaction trace, and an optional verifier into a single record that fully describes an agentic episode.

How does this data object differ from a plain trajectory log?

A plain log records only the sequence of actions and observations ($\tau$). The Agentic Data Object adds the explicit environment spec ($E$), the task signal ($q$), and an optional verifier ($v$), which together make the record self‑contained and evaluable across the ACE dimensions.

Environment parameterization captures everything needed to instantiate a concrete world for data generation. We define $e = (D, F, P_{\text{rule}}, \Omega, v)$, where $D$ is a state carrier (e.g., a repository), $F$ the available tool set, $P_{\text{rule}}$ the policy and permission constraints, $\Omega$ the observable subset of the latent state, and $v$ an optional success interface.

From interaction to agentic data we isolate three core factors: the environment specification ($E$), the task signal ($q$), and the interaction realization ($\tau$). Their sum constitutes the agentic data object, as expressed by $\text{Agentic Data} = E + q + \tau$ (Equation 5). This decomposition is conceptual, not a serialization requirement.

For the remainder of the survey we adopt the common data object $d = (E, q, \tau, v)$ (Equation 6). The optional verifier $v$ records how consistency or outcome is evaluated—via schema checks, executable tests, or learned judges—allowing the same notation to cover supervised fine‑tuning, reinforcement learning, and online generation pipelines.

Generation Paradigms

We map generation pipelines into forward and reverse paradigms, exposing their trade‑offs.

Forward pipelines dominate by grounding tasks in concrete environments, but they inherit any environment brittleness. Reverse pipelines flip control, enabling task‑centric design at the cost of extra feasibility checks.

Forward generation builds an environment first, then derives tasks conditioned on that environment, and finally obtains interaction trajectories. Sources of environments fall into three overlapping categories: (1) real or curated specifications harvested from APIs, repositories, and applications; (2) LLM‑synthesized tool and rule descriptions that expand beyond collected resources; and (3) programmatic constructions that implement databases, simulators, and stateful arenas. This ordering yields strong grounding because later factors are conditioned on an existing environment, but it also creates a cascading dependence where any brittleness in the environment propagates to tasks and trajectories. Recent pipelines mitigate this risk by inserting verification checks at each stage—environment sanity, task feasibility, and trajectory validity.

Reverse generation departs from the $E\!\to\!q\!\to\!\tau$ order and anchors the pipeline on a different artifact. Task‑first methods start from a target capability or instruction pattern and then synthesize compatible tools, environments, and interactions. Trajectory‑first approaches explore or mine existing workflows before formulating the user‑facing task, ensuring executability but limiting the distribution to discovered behaviors. Structure‑first pipelines first generate an intermediate scaffold—such as a tool graph, dialogue skeleton, or blueprint—and then flesh out environments, tasks, and trajectories. Adaptive and self‑evolving extensions close the loop by continuously revising generation strategies based on verification outcomes and model performance.

**Figure 3.** Generation Paradigms: Forward Generation vs Reverse Generation.

**Table 1.** Representative works in forward agentic data generation. All methods follow the dependency $E arrow q arrow \tau$. Methods are grouped according to the dominant source of environment construction: real or curated environments, LLM-synthesized environments, and programmatic or executable environments. **Resource** links to public GitHub repositories or Hugging Face websites when available.

**Table 2.** Representative works in reverse agentic data generation. Methods are grouped into task-first, trajectory-first, and structure-first generation, followed by adaptive and self-evolving approaches as a cross-cutting extension. **Domain** indicates the primary application setting, while **Resource** links to public repositories when available.

Accuracy: The Prerequisite

Fluency alone does not guarantee correctness; all four ACE factors must be satisfied.

Fluency does not imply accuracy in agentic settings because a sample can be well‑formed yet violate environment consistency, task feasibility, interaction causality, or verifier correctness.

This camp composes inexpensive deterministic tests, learned model judgments, and selective human review to filter out structural, behavioral, and semantic failures. Early rule‑based checks prune obvious errors; model‑based critics evaluate semantic coherence; human auditors intervene on ambiguous or safety‑critical cases.

Methods in this camp restrict generation to syntactically and semantically valid structures before any rollout. Blueprint verification, tool‑graph constraints, and stateful environment generators ensure that tasks, actions, and success conditions are mutually consistent from the outset.

This direction grounds accuracy in actual execution outcomes. Generated actions are run against real tools or simulators, and resulting states, tests, or formal proofs are inspected to confirm that observations follow from actions and that the verifier recognises the intended terminal condition.

Rather than discarding failed samples outright, pipelines use verification feedback to iteratively repair or regenerate components. Early‑stage failures trigger localized reconstruction, while successful seeds seed later generations, creating a self‑improving data creation loop.

The community still disagrees on the optimal trade‑off between verification cost and granularity: cheap rule‑based checks miss subtle semantic errors, while full execution‑based verification is expensive for long‑horizon tasks.

**Figure 4** Four complementary factors of accuracy: environment, task, interaction, and verifier/outcome accuracy.

**Figure 5.** Four major research directions and emerging trends in accuracy assurance for agentic data generation.

Complexity: Informative Learning Signals

Mapping the landscape of complexity methods and their trade‑offs.

Complexity methods converge on shaping task difficulty around the learner’s capability frontier, yet they differ in which factor they manipulate and how they calibrate difficulty.

Complexity is defined as the verified failure probability $C_n(d)=1-\Pr[v(d,\tau)=1\mid d,z]$, which varies with the execution configuration $z$. The ACE objective favours a moving band of tasks that are neither trivially solved nor universally unsolvable, formalised by $p_{z_0}(d)<\rho\le p_{z_A}(d)$ where $p_z(d)=\Pr[v(d,\tau)=1\mid d,z]$. Structural attributes such as horizon, dependency depth, and partial observability serve as controllable explanatory variables rather than absolute difficulty scores.

Agentic complexity decomposes into four jointly interpreted factors: environment, task‑signal, interaction‑realization, and verifier‑conditioned completion. Environment complexity stems from state‑action structure and uncertainties, while task‑signal complexity arises from compositional goals and hidden constraints beyond linguistic obscurity. Interaction‑realization complexity captures the minimal serial and parallel dependencies required, and verifier‑conditioned completion complexity reflects the strictness of success criteria.

Methods that pre‑define a dependency graph or subgoal plan before natural‑language realization, then instantiate tasks that require traversing the specified structure. By fixing depth, width, joins, and conditional edges, these approaches directly control the minimal interaction required to solve the task.

Approaches that manipulate what must be achieved (task‑side) or what information is initially available (information‑side). Withholding information creates clarification and state‑tracking burdens, while adding compositional goals or interacting constraints raises the reasoning load.

Techniques that alter the decision process by changing tool dependencies, shared state, observation protocols, or policy boundaries. These changes affect the transition dynamics and the set of actions that meaningfully influence the task.

Methods that redefine success criteria or provide richer feedback signals. By tightening verifier conditions or introducing step‑level rewards, these approaches increase the decision burden while preserving learnability.

Pipelines that start from valid seeds and progressively modify them—removing cues, strengthening goals, or extending workflows. This preserves a grounded core while allowing controlled escalation of difficulty.

Approaches that estimate verified success under a target model, scaffold, and budget, then retain candidates near a target success band. This aligns generated difficulty with the learner’s current capability.

Mechanisms that both add burden to overly easy tasks and simplify overly hard ones, often by inserting intermediate goals, procedural hints, or auxiliary agents. The execution configuration $z$ is altered while keeping the underlying instance fixed.

Complexity evaluation should pair a structural description of the manipulated factors with a behavioural calibration that reports verified success rates under a specified model, scaffold, and budget. Structural reports list goal constraints, information exposure, dependency topology, state design, horizon, branching, and verifier semantics to support controlled ablations. Behavioural reports stratify success by domain, skill, and structural band, and compare matched instances before and after interventions.

Increasing dependencies or stricter verifier conditions raises both failure risk and verification burden, making long trajectories fragile. Focusing on model‑aware difficulty can narrow diversity, while overly realistic but routine scenarios may add little useful signal. Balancing cheap structural proxies with costly model‑based calibration is essential to avoid over‑ or under‑estimating useful complexity.

**Figure 6** Learner-relative task complexity. Learning utility is maximized within a learnable band between tasks that are too easy and too difficult. This band can also shift with learner capability.

**Figure 7.** Factor-level complexity in agentic data across environments, tasks, interactions, and completion criteria.

Diversity: Breadth of Experience

Surface-level variation alone fails to yield meaningful agentic diversity; true diversity requires behaviorally distinct coverage.

Surface-level variation alone fails to yield meaningful agentic diversity; true diversity requires behaviorally distinct environment‑task‑interaction coverage.

The survey organizes the literature into five major camps, each emphasizing different mechanisms for expanding the three ACE diversity factors.

This camp expands the environment‑specification factor by adding executable APIs, tool‑state effects, and multi‑turn information flows. Pipelines increase coverage of tool families, state regimes, and policy variations, showing that richer tool pools improve out‑of‑distribution performance more efficiently than merely increasing sample counts.

This camp targets interaction‑realization diversity by varying application layouts, operating‑system modalities, and cross‑application workflows. By recombining existing UI components and injecting state changes (e.g., permissions), pipelines expose distinct interaction topologies that are not captured by mere screenshot counts.

This camp expands task‑signal diversity through repositories, build systems, and issue‑type variations. Pipelines generate multilingual code snippets, dependency graphs, and test‑driven tasks, thereby widening the set of executable software behaviors an agent must learn.

This camp enriches environment‑specification diversity by varying scenes, objects, physics, sensors, and partner policies. It also adds interaction‑realization diversity through alternative action sequences and social communication protocols, yielding richer behavioral repertoires.

This camp focuses on generator and provenance diversity by exploring materials, proteins, hypotheses, and formal proofs. The pipelines couple candidate generation with property predictors, simulators, or theorem provers, ensuring that each new candidate is validated before being added to the dataset.

**Figure 8** Four complementary factors of diversity: environment, task, interaction, and generator/provenance.

**Figure 9** Domain-specific instantiations of diversity across diverse kinds of agents. Meaningful diversity should be characterized by domain-specific behavioral units rather than surface-level variation alone.

Discussion and Future Directions

Extending ACE to scaling, data sources, and evolving generation pipelines.

The ACE framework asserts that agentic data must be consistent across environment, task, interaction, and success signals. This refresher reminds us that ACE (Accuracy, Complexity, Diversity) organizes these requirements.

Scaling under the ACE objective changes the meaning of “more data”. Simply adding raw trajectories no longer guarantees progress because each new sample contributes unevenly to accuracy, complexity, and diversity. Recent agentic studies show that targeting diversity can yield better performance than increasing sheer quantity, even with smaller budgets.

From the ACE perspective, the useful scaling variable is “effective support” rather than raw dataset size. Effective support grows by producing more valid experiences, moving probability mass toward the learner’s complexity frontier, or expanding coverage over environments, tasks, and interaction structures. The strongest strategy often balances all three dimensions instead of maximizing any single one.

Because the learner’s capabilities evolve, static quantity scaling quickly loses value. As tasks saturate and the complexity frontier shifts, generation must adapt, using failures, success rates, and coverage gaps to decide what experience to create next. This dynamic scaling aligns with self‑evolving and difficulty‑aware systems.

Real and synthetic data differ in origin. Real data come from deployed environments and capture authentic interactions, but they may contain failures, hidden context, or privacy concerns. Synthetic data are generated by models or simulators, offering explicit states and verifiable supervision, yet their correctness hinges on generator fidelity.

Under ACE, real data excel at grounding and external auditing for accuracy, while synthetic data excel at providing controlled complexity and filling diversity gaps. Neither source uniformly dominates; a hybrid mix leverages the strengths of each depending on which ACE dimension is limiting.

Most current agentic data generation is post‑training, placing a heavy burden on a late, data‑limited stage. Shifting generation earlier—into pre‑training and mid‑training—exposes models to state transitions, tool composition, and long‑horizon information seeking, reducing reliance on expensive expert rollouts.

In this earlier regime, data units need not be full (E, q, $\tau$, v) records; fragments such as local transitions, inverse‑dynamics examples, or distilled policies become useful building blocks. ACE still requires accuracy for local correctness, complexity for informative fragments, and diversity for broad priors.

Self‑evolving agents turn data generation into a continual loop: the system observes capability gaps, generates or discovers relevant experience, and updates the agent. This loop can create targeted tasks from failures, adjust difficulty, extract reusable skills, or replay prior experience.

Within this loop, ACE remains dynamic: accuracy blocks erroneous feedback, complexity calibrates the difficulty of new experience, and diversity prevents the system from focusing only on recent failures. Stable evolution also requires fixed anchors—held‑out tasks or real environments—to avoid drift.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers