Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents
Yihao Hu, Zhihao Wen, Yitong Lin, Siyuan Lu, Pan Wang, Wei Wu, Jiang-Ming Yang, Tao Lin
Trajectory2Gym constructs tool-use training data by prioritizing compositional dependencies and decision-grounded supervision over simple success.
How can we transform raw, successful interaction logs into high-quality training data that teaches agents to respect multi-step dependencies and visible evidence?
Tool-use agents are often trained on successful trajectories, but success certifies an outcome rather than the learnability of the steps taken. This leads to models that fail when tasks require multi-step dependencies or recovery from errors, as they lack exposure to the causal links between observations and actions. Trajectory2Gym reframes data collection as the deliberate construction of learnable interactions: it synthesizes paths that exercise specific cross-step dependencies, filters for actions justified by the agent's visible history, and allocates training support based on structural complexity rather than raw volume. This approach improves performance across five model scales, with the 30B-A3B model reaching 84.17 on the Tau2 benchmark—within 1.2 points of closed-source references—and consistently outperforming existing agent-training checkpoints.
Paper Primer
The core mechanism hinges on three stages: synthesizing interactions to fill gaps in compositional coverage, masking targets to ensure every supervised action is grounded in the agent's visible context, and using parent-normalized weighting to ensure long multi-turn interactions do not dominate the loss. Trajectory2Gym is like a curriculum designer: it identifies which "lessons" (dependencies) are missing from the student's experience, constructs specific scenarios to teach them, and ensures the student is only graded on decisions they had enough information to make.
Trajectory2Gym significantly improves stateful tool-use performance across all tested model scales.
Paired comparisons on Tau1 and Tau2 benchmarks show mean performance gains of 20.04 to 40.92 percentage points compared to public Qwen3 initializations. The 30B-A3B model achieves a 75.47 mean across benchmarks, outperforming all re-evaluated agent-training checkpoints.
Parent-normalized supervised fine-tuning provides the largest isolated gain among the proposed design choices.
A matched ablation on the 8B model shows that treating a semantic interaction as the unit of optimization—rather than individual rendered rows—improves the three-benchmark mean by 8.04 points.
Why is "success" an insufficient metric for training tool-use agents?
Success certifies that an episode ended well, but it does not guarantee that the agent's actions were justified by the information visible at decision time. A model trained on successful trajectories may imitate actions that rely on "privileged" information unavailable to the agent during inference, leading to failure in real-world settings.
How does this method handle failed tool calls during training?
Failed calls and their authentic error messages are kept in the agent's visible context to provide necessary feedback, but they are masked so the model does not receive imitation loss for the failure itself. The subsequent grounded recovery actions are then supervised as targets.
For tool-use agents, data quality is defined by the dependencies realized and the decision-time grounding of targets. Future pipelines should prioritize structural coverage and decision eligibility over simple trajectory volume.
Introduction: The Problem with Successful Trajectories
We expose why successful tool-use trajectories fail to teach reasoning.
Successful tool-use trajectories often hide reasoning gaps. They may omit multi‑step dependencies or lack justification from the agent’s visible history.
Training on raw successful trajectories does not guarantee that the agent learns the underlying reasoning because supervision can miss crucial dependencies and lack decision grounding.
**Figure 1.** Compositional interaction requires grounded decisions. Top: Trajectory2Gym constructs linked interactions (1) and checks each training action against that history (2). Bottom: a Tau2 Retail user requests a larger bottle and a dimmer lamp, then confirms only the lamp. Both rules judge the same two candidate actions. Filtering by execution keeps both because native tool replay accepts both requests. Trajectory2Gym uses catalog evidence and the latest confirmation to keep the lamp exchange and reject the additional bottle exchange. Green checks mark correct selection decisions; the red cross marks erroneous retention. This schematic illustrates supervision selection with paraphrased dialogue and a constructed second candidate.
Successful outcomes do not imply logical reasoning.
Benchmarks and Problem Formulation
Survey of tool‑use benchmarks, data pipelines, and the formal problem statement.
Prior work on tool‑use agents clusters around two themes: benchmark suites that stress different interaction horizons, and data pipelines that synthesize or curate executable trajectories.
Berkeley Function Calling Leaderboard – a standard benchmark measuring tool‑use accuracy across single‑turn and multi‑turn scenarios.
Couples diagnosis‑guided training‑interface adaptation with policy updates to improve credit assignment in multi‑turn interactions.
Leverages next‑turn user reactions for fine‑grained credit assignment in interactive tool‑use episodes.
Failure‑aware method that diagnoses breakdowns and selects recovery interventions during execution.
Deficit‑guided synthesis pipeline that builds native tool‑use environments, exposes policy‑visible target eligibility, and allocates structural support across learner scales.
Derives reusable procedural skills from source code and verifies them as executable tool‑use primitives.
Curates prefix‑grounded, efficient steps using privileged reference patches to accelerate trajectory construction.
Selection method that ranks candidate trajectories by interaction topology to favor diverse decision graphs.
Benchmarks that evaluate agents on multi‑turn, long‑context tool‑use scenarios, typically involving a customer‑service dialogue ($\tau$) that stresses temporal reasoning.
Our formulation requires two properties of $D_s$: compositional coverage of observation–action dependencies, and decision grounding that ties each supervised action to its realized context $c_t$.
Compositional Interaction Synthesis
We synthesize missing interaction paths by generating and executing targeted native traces.
Figure 2 visualizes the pipeline that the paper builds on: (1) compositional interaction synthesis creates native executions for missing paths, (2) grounded supervision turns those executions into student contexts, and (3) structural‑support learning decides how often each certified structure appears during training.
The figure illustrates a framework consisting of two main components: "Grounded interaction construction" and "Structural-support learning." The first part shows a teacher agent interacting with a native environment to generate executed logs, which are then processed through "Grounded supervision" involving collection, filtering, and certification to produce student traces. The second part, "Structural-support learning," utilizes a "Certified trajectory pool" with "Discover," "Recover," and "Complete" stages, followed by "Structural selection" and "Parent-normalized SFT" to train "Tool-use agents."
We turn a sparsely supported interaction path into a concrete native execution by asking a generator to fabricate a request and then letting a teacher agent run it in the real environment.
Generator $G$ samples a request $x$ = “retrieve order #42” and an initial state $s_0$ containing a user ID.
Teacher $\pi_T$ executes $x$ in the native environment, producing a trajectory $\tau$ that first calls the lookup API and then receives a confirmation response.
Canonical path $p(\tau)$ is identified as $(\text{lookup},\text{confirm})$, so $n_{\{\tau\}}(p)$ increments from 0 to 1.
The path is certified because both the lookup action and its confirmation observation appear in $\tau$.
The certified path is added to the support inventory, making it available for future synthesis rounds.
This toy example shows how a missing two‑step interaction becomes a concrete, certified trajectory, increasing its support count from zero to one.
How does targeted synthesis avoid the failure of uniform random selection described in Proposition 3?
Uniform selection would miss a singleton path with probability $1\!-\!B/N$, leaving it unrepresented. Targeted synthesis explicitly conditions the generator on under‑supported paths, guaranteeing that each such path is executed at least once and thus enters the support inventory.
Decision-Grounded Supervision
Method details for grounding supervision and selecting structural support trajectories.
Grounded supervision converts raw execution logs into student contexts ($c_t$) and eligible targets, checking visibility, local execution, and workflow completion sequentially. This pipeline isolates what the agent can actually observe from what the teacher certifies.
Decision Grounding filters a tool call to the subset that the agent can justify using only information visible in its current context, discarding calls that rely on hidden collector knowledge.
Compute $gt = et \times vt = 1 \times 1 = 1$ — the call is eligible.
If the query had been a hidden API key, $vt$ would be 0, yielding $gt = 0$ and the call would be filtered out.
Decision Grounding ensures the training set only contains actions the agent could have justified using its own observations, eliminating spurious supervision from hidden information.
How is Decision Grounding different from ordinary execution validity?
Execution validity ($et$) only verifies that a call did not error; Decision Grounding ($gt$) additionally requires that every argument and the tool itself be discoverable from the agent’s visible context ($vt$). Thus $gt$ filters out calls that are technically successful but rely on hidden information.
Performance and Structural Support
Trajectory2Gym delivers up to 41‑point gains on Tau benchmarks and boosts structural path coverage.
Trajectory2Gym raises the Tau1 and Tau2 mean scores by up to 41 points across all model scales.
Table 1 shows paired gains ranging from +20.04 at 1.7 B to +41 at larger scales.
**Figure 3.** Structural vs. random selection at a fixed 8B budget (14,819 parents, equal target tokens). (a) Canonical paths covered. (b) Score change of structural selection (points). Structural selection covers 942 more paths and improves both Tau benchmarks, with BFCL essentially unchanged.
It selects training interactions that maximally expand the set of covered capability atoms while respecting a per‑task limit, ensuring diverse structural support for decision grounding.
How does Structural‑Support Learning differ from random interaction selection?
Random selection ignores the gain g($\tau$) and the per‑task cap, so it may repeatedly pick interactions that add no new atoms or exceed K, leading to lower path coverage. Structural‑Support explicitly maximizes new atom coverage while respecting K, which yields the observed 942‑path advantage.
Structural selection markedly improves path coverage compared to random sampling.
Ablation Study
We test each design choice by removing it and measuring the performance impact.
Recall that training on raw successful trajectories can hide poor reasoning; we therefore filter logs for decision grounding and diversify structural support.
It redistributes optimization mass across semantic interaction paths, treating each parent interaction as a unit of optimization rather than each rendered row.
How does parent normalization differ from simply weighting examples by frequency?
Weighting by frequency still privileges interactions that appear many times; parent normalization equalizes the contribution of each distinct semantic interaction regardless of how many rendered rows it generates.
Parent normalization yields the largest isolated performance gain.
An ablation that only changes parent normalization raises the three‑benchmark mean by 8.04 points.
**Table 4.** More structural support is not automatically better. Performance does not increase monotonically with support depth (Table 4): the three-benchmark mean is highest at the shallowest setting (72.28 versus 70.55 and 69.44) even as trajectories and target tokens continue to grow. The decline is concentrated in Tau2 (82.50 to 75.00), while BFCL remains comparatively stable. The headline schedule was fixed before these runs, so this is a sensitivity test rather than checkpoint selection; because depth also changes data volume, it does not isolate path support from token count. In short, structural support is an allocation problem rather than a simple data-scaling rule, and we set depth per scale by a predeclared schedule (Appendices F.5 and D).
The ablation results confirm that each proposed component—parent normalization, targeted expansion, grounding, and structural selection—contributes positively, while the sensitivity analysis reveals that simply increasing support depth can hurt certain benchmarks, underscoring the allocation nature of structural support.
Qualitative Analysis
Qualitative examples illustrate how decision grounding and structural support shape agent outcomes.
Trajectory2Gym (8B) attains a native reward of 1 on the Retail workflow, while Simia and MUA‑RL each receive 0.
Simia fails to apply the retrieved address, MUA‑RL edits the item before the address update causing a state‑dependency error, and Trajectory2Gym correctly updates the address while the order is pending, then edits the items, resulting in a pass.
**Figure 4.** Same-task comparison of evidence use and action order. Simia overlooks visible evidence; MUA-RL violates a state-dependent ordering; Trajectory2Gym reuses the evidence and preserves the required dependency (native rewards 0, 0, and 1). Interactions are independently generated and abridged for display; a step-by-step account appears in Appendix G.1.
These behaviors exemplify the paper’s construction: decision grounding admits only targets whose arguments trace to visible evidence, and workflow‑order certification builds and certifies dependent‑state paths.
Appendix G provides six additional paired cases, a recovery scenario (G.2) where a failed lookup is followed by an alternative lookup preserving context, and further travel and connectivity workflows (G.3) that link decisions through observations and user‑mediated state changes.
Interaction Synthesis Details
Details of the interaction corpus, its representation, and how synthesis targets missing structures.
The 1.7B–14B learners draw from a common corpus of 34,772 certified interactions covering 2,515 tasks, while the 30B‑A3B learner uses an expanded set of 35,819 interactions across 3,602 tasks. Selected views contain 11,583 to 19,937 interactions per model, corresponding to roughly 13.0 M to 23.9 M supervised target tokens.
Within the common corpus, 40.9 % of interactions have at most two supervised assistant decisions, yet only 10.0 % feature a grounded decision after an authentic error observation. This skew shows that corpus size alone does not guarantee exposure to recovery or cross‑step dependencies, motivating explicit control over compositional support.
Interactions are represented by the dependencies that connect decisions across time, not by the sequence of native tool names. A shared capability vocabulary K abstracts semantic operations, observations, control‑flow events, and interaction horizons, enabling comparison of structural paths across tasks and environments.
Intended structure describes the target composition a task aims to require, but only realized structure—where the dependency actually occurs during native execution—contributes support to the learner. Thus structural support is defined by the learner’s observed experience, not by the generator’s specification.
Operational encoding records decision kinds (text, tool call, multiple calls, handoff), maps tool access to read/write/irreversible‑write/hand‑off categories, and classifies objects (identity, commerce, travel, finance, connectivity, communication, location, time). Argument roles (identifiers, amounts, temporal values, etc.) are tokenized and sorted into composite classes; values contribute type information only.
Set‑valued coverage adds call and argument counts, decision‑horizon buckets (1, 2, 4, 8, 16, 32, overflow), adjacent transition types, and an error‑recovery indicator. Each supervised assistant decision yields a descriptor that includes its kind, ordered call descriptors, subsequent observations, and whether visible error feedback occurred since the previous decision.
The canonical path is the ordered sequence of these descriptors, serialized deterministically and assigned a stable key. Consequently, different native interactions share a path when their descriptor sequences match, while any change in decision order, observation shape, or recovery context yields a distinct path.
Mapping the retail interaction example: decisions 0–19 are indexed, decision 1 (failed email lookup) is excluded from the path, and the path consists of descriptors from decisions 0, 2,…, 19. Decision 0 lacks error feedback; the subsequent recovery decision carries the after‑error indicator, illustrating how observed errors reshape the descriptor.
Synthesis targets compositional structures that remain weakly represented in the current corpus. Given a target path, task construction generates a user request and initial environment state that require the corresponding dependency for successful completion.
Three task families arise: information‑dependent tasks need a value later obtainable only through earlier evidence; recovery tasks introduce a resolvable obstacle whose cause and solution appear via native feedback; state‑dependent tasks demand later operations respect information or state established earlier.
Execution proceeds in the native environment; observations, state transitions, user responses, errors, and terminal outcomes are consequences of that execution, not annotations inserted by the generator. Only structures that actually occur become support, and decision‑grounded supervision selects which realized decisions become learning targets.
Thus intended structure guides where to generate new interactions, while realized structure determines the experience the learner actually receives.
Decision Grounding Nuances
When does an assistant action qualify as supervised target under decision grounding?
Decision grounding requires that the information needed to choose an action be recoverable from the policy‑visible history V(H < t). Only observations exposed through the native agent interface—such as user messages or tool outputs—count as evidence; collector‑private state and hidden external operations do not.
Execution validity (whether the environment accepts the action) and decision grounding (whether the policy had the required evidence) are independent criteria. A successful trajectory does not guarantee that every intermediate action is a valid imitation target; an action that depends on hidden identifiers remains ungrounded even if the episode succeeds.
Failed actions receive no imitation loss, but if the failure and its error response are visible, they enrich the policy‑visible history for a later recovery decision. Likewise, a repeated call can be supervised only when the intervening state change yields new evidence; mere repetition without new information offers no additional supervision.
Structural Support Extensions
Describes how structural‑support learning selects and weights interactions for diverse training.
Structural‑support learning answers two practical questions: which interaction paths should be kept for a learner, and how much repeated training weight each retained path should receive.
Structural‑support learning separates “does a compositional structure appear?” from “how often it is reinforced” during training.
Having defined how much support each path should receive, the next step is to decide which interactions to keep.
The selector first guarantees broad task coverage, then adds structural diversity, and finally repeats under‑supported paths until the depth target is met.
The allocation stops once all requested depth levels have been processed; remaining deficits indicate either infeasibility due to task caps or lack of eligible candidates.
Theoretical Notes
Formal properties of the learning objective and selection stages.
When a semantic interaction produces several rendered training examples, a naïve row‑averaged loss would overweight interactions that happen to generate more examples. The parent‑normalized objective instead distributes the total optimization mass uniformly across selected interactions, regardless of how many examples they render.
Because the inner weights in Lparent sum to one for every interaction, each selected interaction receives the same aggregate coefficient 1/Ps. Consequently, repartitioning a semantic interaction into a different number of rendered examples does not alter its overall contribution—this is the rendering‑invariance property proved in Appendix E.4.
In the grounding analysis we consider a decision triple (c, z, a): visible context c, collector‑private information z, and assistant action a. A student policy $\pi$$\theta$(a | c) never sees z.
Proposition 1 (Grounding floor) states that for any student policy the expected negative log‑likelihood satisfies E[−log $\pi$$\theta$(A | C)] ≥ H(A | C) = H(A | C,Z) + I(A;Z | C), with equality only when the policy matches the true conditional distribution $p$(· | c). The extra mutual information term I(A;Z | C) quantifies the unavoidable loss due to hidden collector information.
Remark 1 notes that the grounding check removes targets lacking visible support, but this removal does not automatically guarantee a positive I(A;Z | C). Remark 2 clarifies that certification acts as a nested filter: only decisions passing all four checks survive, and the proposition characterizes the log‑loss penalty incurred by the remaining hidden information.
The deficit‑driven selection stage optimizes a capped support function Fd(S)=∑p∈Pmin(nS(p), d), where nS(p) counts how many selected candidates follow canonical path p. Reducing the total deficit $\delta$d(p;S)=d|P|−Fd(S) is equivalent to increasing Fd.
Proposition 2 shows that Fd is monotone submodular and that a greedy algorithm adding any candidate with positive marginal gain ($\Delta$d($\tau$ | S)=1) attains the optimal capped support given a budget B. When the cap is infinite (d=∞) the objective becomes modular (simply |S|), so any budget‑feasible selection is optimal; the cap d=1 counts covered paths.
Remark 3 explains that the full selection pipeline consists of several stages (core, exemplar, atom/path coverage, deficit reduction). The deficit stage corresponds to the d=1 case, and the greedy order is exact for the capped objective; tie‑breaking only affects candidates with equal marginal gain.
Additional Experimental Analysis
Provides the full public‑checkpoint results and deeper analysis of matched controls and structural‑support allocation.
This appendix extends the main results by publishing the full set of sixteen public checkpoints evaluated under the unified benchmark protocol, and by offering additional analyses of the matched controls and the structural‑support allocation rule.
**Table 5.** Complete public-checkpoint evaluation (%). Tau scores are domain macro averages; Mean equally averages Tau1, Tau2, and BFCL. Trajectory2Gym-8B is shown for reference.
Cross‑benchmark behavior (Section F.2) reveals a consistent gap between function‑calling accuracy (BFCL) and stateful task execution (Tau1/2). Qwen3 Simia checkpoints trained on Tau trajectories transfer better to Tau than their AgentBench or OfficeBench counterparts, while LoopTool, EnvFactory, and MidTool achieve strong BFCL scores without comparable Tau2 performance.
Matched component controls (Section F.3) isolate four factors: targeted expansion, decision grounding, structural selection, and parent normalization. Each factor is compared against a stage‑matched baseline, showing that the gains reported in Table 3 are not additive but stem from distinct interventions applied at different stages of the training pipeline.
Structural coverage under a matched budget (Section F.4) demonstrates that the allocation rule increases the number of covered canonical paths from 5 418 to 6 360—a gain of 942 paths—while keeping the total semantic‑interaction budget (14 819) constant. This extra coverage translates into a 2.32‑point improvement in the equal‑weight three‑benchmark mean, driven primarily by gains on Tau1 and Tau2.
Support‑depth sensitivity (Section F.5) varies the structural‑support depth for Qwen3‑8B. Depth 2 yields the highest three‑benchmark mean, while deeper support reduces performance, especially on Tau2, indicating that more repeated exposure to the same paths does not guarantee better results.