TEV: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards

Wenbo Hu, Shraman Pramanick, Krishna Kumar Singh, Nanyun Peng, Kai-Wei Chang, Yong Jae Lee

TEV-GYM provides a latent-world-first synthesis pipeline and dense progress rewards to train long-horizon agents.

How can we train long-horizon agents to solve complex, multi-step tasks when terminal-only rewards provide no signal for intermediate progress?

Long-horizon agents struggle because training data is scarce and rewards are sparse, leaving models unable to learn from complex, multi-step tasks that only provide feedback upon final completion. The authors introduce a latent-world-first synthesis approach: they construct tasks backward from a hidden, correct specification, allowing them to render verifiable intermediate states that serve as a dense progress signal. Training with this recipe, TEV-27B matches frontier performance on computing and math benchmarks and generalizes to longer-horizon tasks, closing two-thirds of the gap to state-of-the-art models.

Paper Primer

The core mechanism, Progress-Potential Reward Shaping (PPRS), converts the latent world's intermediate states into a per-step reward. It functions like a progress bar in a game: as the agent completes stages of the workflow, its potential score rises, providing a continuous gradient that prevents the "starvation" of reinforcement learning groups where no rollout reaches the final goal.

TEV-27B significantly outperforms its base model on long-horizon tasks.

On the ALE-CM benchmark, TEV-27B improves full task solutions (P@1) by 33% over the Qwen3.8-27B base model. It reaches 85% of the frontier performance level, closing over two-thirds of the original gap to Claude Opus 5.

The training recipe generalizes to larger base models and harder benchmarks.

Applying the recipe to GLM-5.3-Flash yields TEV-320B, which improves P@1 on Terminal-Bench 3.0 and 4.0 subsets by 35% and 28% respectively. TEV-320B becomes the strongest open-source model on these subsets, trailing only the top-tier proprietary frontier models.

Why is a "latent-world-first" approach necessary compared to simply using a language model to generate tasks?

Language-model-generated tasks often suffer from "artifact drift," where the task, verifier, and solution describe different problems. By constructing the task backward from a single, hidden latent world, the authors ensure the ground truth is correct by construction and the verifier is mathematically anchored to the task.

What happens to training if the agent fails to complete the task entirely?

In standard terminal-only reward systems, failed rollouts contribute zero gradient, wasting the batch. With PPRS, even failing rollouts receive a reward based on their final progress potential, allowing the model to learn from partial successes and near-misses.

This paper demonstrates that long-horizon agent training is tractable when the environment provides a verifiable, dense progress signal derived from a latent specification, rather than relying on sparse terminal feedback.

Introduction: The Long-Horizon Bottleneck

We expose why terminal-only rewards cripple long-horizon training and preview TEV’s dense reward solution.

Training long‑horizon agents fails because terminal‑only rewards provide only a sparse signal, starving learning. Imagine searching for treasure only at the maze’s exit versus receiving clues at each intersection—sparse feedback makes progress impossible.

When an environment gives reward only at task completion, early training steps receive no gradient signal, so the policy cannot learn to make progress.

Rollout 1 reward = 0, Rollout 2 reward = 0.

GRPO computes the mean reward for the group: (0 + 0)/2 = 0.

Advantage for every timestep in both rollouts = reward − mean = 0 − 0 = 0.

Policy gradient update = $\Sigma$ advantage × ∇log$\pi$ = 0, so no learning occurs.

This toy example shows how terminal‑only rewards can completely starve gradient information when all rollouts fail.

**Figure 1.** We demonstrate TEV’s results on ALE-CM, and generalization to Terminal-Bench (TB) 3.0-CM, 4.0-CM, and 2.1 on the left and a summary of our latent-world-first data synthesis approach on the right. TEV showcases significant performance gains across two model scales and TEV-27B matches GPT-6 Astra and Claude Opus 5 at Pass@5 on ALE-CM.

The failure of terminal‑only rewards in long‑horizon tasks is the core obstacle this work addresses.

Latent-World-First Environment Synthesis

We synthesize environments backward from a latent world, enforcing coverage, difficulty, and correctness.

Long‑horizon agents need dense, verifiable feedback; we therefore generate the full task environment from a clean latent world and then corrupt it in a controlled way.

We first build a perfect “latent world” that contains the complete, correct data, then expose the agent only to a corrupted view while keeping a verifier that can check any output against the latent world.

How does TEV synthesis differ from the usual forward‑generation of environments?

Typical pipelines start from a specification and generate data directly; TEV first creates a complete, correct latent world $W$, then projects a clean version and finally applies a declared defect plan. This backward‑first approach guarantees that every corrupted artifact can be traced back to a unique ground‑truth $W$, which a forward‑only pipeline cannot ensure.

We treat the latent world as the single source of truth and build every component of the environment—task prompt, input files, verifier, and progress contract—by deterministic transformations of that world.

Generate $W$: customers = {C₁, C₂}, products = {P₁, P₂}, transactions = {(C₁,P₁), (C₂,P₂)}.

Render clean CSV files:

Apply defect plan that duplicates the first transaction and introduces a null product ID in the second row.

Resulting corrupted file

The verifier $ver_W$ reads the corrupted file, reconstructs $W$ by undoing the declared defects, and confirms that the original two‑transaction state is recoverable.

Because the corruption is declared and invertible, the agent’s observed data $s_0$ uniquely determines the latent world $W$, so any correct solution can be verified without external solvers.

Why can’t we simply corrupt the data without first rendering a clean version?

If we corrupt raw data directly, we lose a reference to the uncorrupted state, making it impossible to guarantee that the agent’s output can be checked for exact correctness. Rendering a clean version first gives a provably invertible mapping between $W$ and the public files, which the defect plan then perturbs in a traceable way.

**Figure 3.** Latent-world-first synthesis, on one example TEV-GYM environment. This environment's task is “Retail Warehouse ETL Recovery. Goal: build a clean star-schema SQLite warehouse from six months of messy retail data and produce two truthful JSON sidecars.”

Dense Progress Rewards

Dense progress rewards turn sparse terminal signals into frequent, verifiable feedback.

Group‑relative RL (GRPO) samples a batch of $G$ rollouts per task and weights each rollout by a group‑normalized advantage. With a terminal‑only reward, a group in which every rollout fails yields zero advantage for all members, so the entire batch contributes nothing to the gradient. On long‑horizon tasks the success probability $p$ is tiny (≈ 0.05), meaning $(1\!-\!p)^G$ of groups are wasted—over 66 % when $G\!=\!8$.

Instead of rewarding only the final submission, we give the agent a dense signal that measures how much of the latent‑world contract it has already satisfied.

Compute gated scores: $\tilde c_{v_1}=0.8$ (no prereqs), $\tilde c_{v_2}=0.6$ (no prereqs), $\tilde c_{v_3}=\min(0.9,0.8,0.6)=0.6$ because $v_3$ depends on both $v_1$ and $v_2$.

Weight and sum: $\Phi(s)=0.5\cdot0.8 + 0.3\cdot0.6 + 0.2\cdot0.6 = 0.40 + 0.18 + 0.12 = 0.70$.

If the agent later improves $c_{v_2}$ to 0.9 while leaving others unchanged, the new gated scores become $\tilde c_{v_2}=0.9$, $\tilde c_{v_3}=\min(0.9,0.8,0.9)=0.8$, yielding $\Phi(s')=0.5\cdot0.8 + 0.3\cdot0.9 + 0.2\cdot0.8 = 0.40 + 0.27 + 0.16 = 0.83$.

The per‑step shaping reward for this improvement is $r=\Phi(s')-\Phi(s)=0.83-0.70=0.13$.

Gated scoring forces the agent to satisfy prerequisites before gaining credit for downstream nodes; a high raw score on a leaf node cannot compensate for missing ancestors.

How does this dense reward differ from classic potential‑based reward shaping (PBRS)?

Standard PBRS forces the terminal potential to be zero so that the optimal policy is unchanged; here we keep the terminal term $\lambda\Phi(s_T)$ and deliberately alter the objective. Moreover, $\Phi$ is defined solely on the workspace (independent of the policy’s internal actions), guaranteeing that two rollouts reaching the same final workspace receive identical rewards regardless of the path taken.

**Figure 4.** The progress potential along one group ($G=8$) of rollouts. Eight rollouts of the trained 27B model on one warehouse environment, graded live after every turn. Under terminal-only reward (red) every rollout scores 0. Under progressive rewards, by contrast, two submit an incomplete warehouse (circles), five exhaust the 65k-token context (crosses), and one builds a complete warehouse, so the rewards vary within the group.

Table 2 (referenced) shows that agents equipped with dense progress rewards achieve substantially higher scores on the ALE‑CM benchmark than their terminal‑only counterparts, confirming that the additional gradient signal translates into better long‑horizon performance.

Experimental Results

Experimental results demonstrate TEV‑GYM’s advantage over terminal‑only baselines.

Training long‑horizon agents with only terminal rewards yields sparse feedback. TEV‑GYM instead synthesizes dense progress rewards from a latent world.

TEV‑27B achieves a +28% Pass@1 improvement over its base model on the TB 4.0‑CM benchmark.

Figure 2 shows the orange arrow indicating a +28% gain for TEV‑27B relative to Qwen3.8‑27B.

**Figure 2.** All models on the Terminal-Bench 4.0 “CM” subset (TB 4.0-CM), ranked by Pass@1. TEV-320B and TEV-27B are trained with our recipe from GLM-5.3-Flash and Qwen3.8-27B, respectively; each arrow gives the relative gain over its base.

A benchmark suite that grades agents on a variety of computing and math problems, reporting both a mean score (S@k) and the fraction of fully solved tasks (P@k).

How does ALE‑CM differ from standard code‑generation benchmarks?

Standard benchmarks usually report a single pass/fail score per task. ALE‑CM records a graded score (S@k) and a solve fraction (P@k) across multiple rollouts, revealing both quality and reliability of the agent’s solutions.

A collection of long‑horizon tasks where success is measured only at episode termination, making sparse reward signals the norm.

Why can’t we simply add intermediate checkpoints to TB instead of using TEV‑GYM?

Adding handcrafted checkpoints would require domain‑specific engineering for each task, breaking TB’s goal of a unified, scalable benchmark. TEV‑GYM automatically generates dense rewards from a latent world model, preserving the benchmark’s generality.

**Figure 5.** Qualitative example on the ALE-CM task: k8s payment api root cause analysis (Coding & System). From a `kubectl` capture, the Deployment/HPA YAML and a crashing pod's log, the agent writes a JSON root-cause report with *verbatim* evidence strings and a rollback verdict. Green marks a decision that earns a component, red one that forfeits it.

TEV consistently outperforms terminal‑only baselines across long‑horizon benchmarks.

Related Work

We position our approach among prior methods for synthetic environments and reward shaping.

Recent synthetic corpora for coding and terminal agents often mine software repositories for issues and tests, or rely on language models to generate tasks such as bug‑injected code and reconstructed terminal sessions. Scaling executable tool‑use environments has shown success, but only when task difficulty is carefully controlled.

Our environments differ by containing long, multi‑stage workflows across mathematics, science, machine learning, coding, and systems, instantiated by seeded generator code rather than directly authored by a language model, and they provide a dense progress signal for every task. Unlike prior backward‑construction families—denoising, controlled error injection, code mutation, environment rewinding, instruction back‑translation, and procedural puzzle generation—we corrupt a rendering of an entire latent world, making the task, verifier, and progress signal all functions of that single world.

In agentic reinforcement learning, existing methods avoid uninformative rollout groups by filtering identical rewards, selecting tasks near the policy’s current ability, or adding expert prefixes. Other approaches provide denser feedback through learned judges or proxy signals such as test‑pass rates, but these require extra models and capture only terminal outcomes. Our potential $\Phi$ instead measures exact progress and regression against the latent world at every workflow stage, without requiring a critic or language‑model judge.

Data Construction Details

We detail how TEV‑GYM environments are built, verified, and diversified.

This appendix expands the condensed description in §2, detailing how each TEV‑GYM environment is generated, verified, and diversified.

A.1 Task sources and domain taxonomy enumerates the seven computing‑and‑math subdomains of ALE‑CM that TEV‑GYM spans, each populated by at least two generators as required by the coverage policy.

**Table 6.** The seven computing-and-math subdomains of ALE-CM that TEV-GYM spans. Subdomains (9.1–9.7) of the Computing & Math domain of Agents’ Last Exam (Sun et al., 2026), which we refer to as TEV-GYM’s seven domains. Ids and names are taken from the ALE taxonomy; scope summaries follow it, reconstructed from O*NET/SOC 2018 and the released ALE task specifications. “Verified by” names the deterministic signals we build the strict verifier and progress potential around.

A.2 Determinism, reproducibility, and generation cost emphasizes that every stage of §2.1 is a pure, seeded function; no language model participates, making each environment fully reproducible from its generator, seed, and scale tier.

The authoring cost of a generator is amortized, while the marginal cost of producing a single environment—including both reference solvers and the progress potential $\Phi_{W}$—is a median of $30$ s of CPU time (90th percentile $82$ s), far cheaper than hand‑authoring.

A.3 Coverage axes and covering array defines a strength‑2 covering array over 19 axes, guaranteeing that every pair of axis values co‑occurs in at least one admitted environment.

The corpus is extended greedily: at each step the generator, seed, and scale tier that fill the largest number of currently uncovered pairs are added, making “diverse” a verifiable property.

The eight gates act as a checklist that guarantees each environment is well‑specified, non‑gaming, and structurally sound before it ever reaches an agent.

How does “Reward causality” differ from ordinary reward shaping?

Ordinary shaping may add a scalar that still increases when the underlying cause is corrupted; Reward causality requires the reward component to strictly decrease whenever its causal workspace element is altered, making the signal provably tied to the intended behavior.

A.4 Scale ladder describes how each generator exposes a scale axis; for example, the warehouse environment ranges from 30 customers/48 transactions (small) to 480 customers/960 transactions (large), with proportional growth in duplicates and missing values.

Across the three scale tiers the median input tree grows from 17 files/51 KB (unit) to 20 files/168 KB (small) and 28 files/586 KB (full), while the workflow DAG remains six stages.

A.5 Measuring horizon and difficulty uses the critical‑path length of the workflow DAG (Stage 2 of §2.1) as the structural horizon measure, avoiding inflation from agent detours.

Under the training harness, an untrained Qwen 3.8‑27B solves only $3.2\%$ of environments (mean graded score $0.06$), confirming that terminal‑only reward is near‑zero for most of the corpus.

When evaluated with Gemini 3.6 Flash, TEV‑GYM reaches pass@1 $29\%$ and pass@8 $40\%$ on a 92‑environment subsample, while a substantial fraction remains unsolved, highlighting a meaningful difficulty gap.

**Table 8.** Composition of TEV-GYM by capability category. Counts are task environments; horizon columns are per-environment medians over the category (workflow stages, input files the agent sees, injected corruptions); “formats” counts distinct wire formats among the inputs. Base P@1 is the single-rollout pass rate of untrained Qwen3.8-27B under the training harness, from one probe on our environments; frontier P@1 is the fixed evaluator of Appendix A.8 on its 92-environment subsample.

A.8 Comparison protocol measures “agent time” as the wall‑clock interval from the first action to the last step; TEV‑GYM rollouts have a median of $12.4$ min and a mean of $14.8$ min, far longer than prior corpora (median $0.5$–$2.0$ min).

**Table 1.** Comparison of existing corpora with TEV-GYM.

A.7 Dataset statistics report a normalized‑entropy balance of $0.89$ across capability categories and $0.91$ across domains, confirming near‑uniform distribution of environments.

Horizon statistics show a median of six workflow stages, twelve input files, six reward components, and two injected corruptions per environment, with a median of $51$ KB of input data spanning $23$ wire formats.

A.9 Decontamination guarantees structural cleanliness: the six‑seed diversity gate ensures no two environments share a public tree, and lexical overlap analysis finds zero $13$‑gram and zero $8$‑gram hits with any evaluation set.

The table presents an evaluation of data contamination across various benchmarks (ALE-CM, TB 3.0, TB 4.0, TB 2.1) by measuring 13-gram hits, 8-gram hits, maximum token coverage, and nearest-neighbor Jaccard similarity (median/maximum). A second section, "Calibration: same code, same-benchmark tasks," provides a baseline comparison by evaluating tasks against other tasks within the same benchmark.

Worked Training Environments

Shows how dense rewards avoid terminal reward starvation and the impact of removing them.

Section A.10 presents three worked training environments: Example 1 (cloud‑cost optimization), Example 2 (closed‑loop building MPC), and Example 3 (financial‑statement reconstruction).

The ablations compare the full TEV‑GYM pipeline with a version that omits dense progress rewards, revealing a persistent all‑fail fraction of 44 % even when the per‑step success probability is $p=0.05$ and the group size is $G=16$.

**Figure 6.** The dashboard screenshot staged as input/`aws_billing_dashboard`.png in Example 1 (900 × 560 px, drawn by the generator from the latent world). The upper table is per-service context and plays no role in grading; the lower table holds the three dashboard-only findings that appear in no CSV, so the agent must read the image to recover their ids, actions, and savings.

``` input/ question.md the task prompt above instance.json horizon, schedules, tariff, controller weights (excerpt below) experiment_plan.json four runs, identification recipe and split, two policies output_contract.json the nine deliverables, their columns, and their rounding (below) model.idf one-zone EnergyPlus model, materialized from the pinned example weather.epw Golden, CO (NREL) TMY3 weather file, from the same pinned install ``` instance.json declares every quantity the procedure depends on; its seven 672-entry schedules are elided here.

Extended Related Work

Survey of synthetic and verification approaches for coding and terminal agents.

The first wave of coding corpora mined real repositories: SWE‑bench and SWE‑Gym collected GitHub issues with tests, while SWE‑rebench and Skywork‑SWE automated mining. The second wave synthesizes tasks by injecting bugs (SWE‑smith), stripping dependency graphs (SWE‑Flow), or back‑translating commits (R2E‑Gym).

Terminal‑focused corpora scale up similarly: Endless Terminals, SETA, recursive solution extension, and TMAX generate tasks with model‑written tests; CLI‑Gym and Terminal‑Universe reconstruct environments from agent trajectories.

Our approach follows the “latent‑world‑first” synthesis line: instead of corrupting a single artifact, we corrupt an entire world representation, making the world itself the oracle and progress signal. This mirrors CLI‑Gym’s rewind but adds dense intermediate verification across many files.

Verification against final world states appears in AppWorld, $\tau$‑bench, WebArena, and OSWorld, yet those systems rely on hand‑authored tasks. Recent generators such as AgentMercury and Zhang et al. (2026) produce worlds with executable invariants, but they lack a declared corruption that guarantees exact intermediate grounding.

Sparse‑reward methods discard identical‑reward rollouts or schedule prompts near the learner’s success band, limiting difficulty growth. Dense‑signal approaches—learned step rewards, advantage shaping, or proxy quantities like test‑fraction scores—provide richer feedback but often require trained critics or suffer from non‑monotonicity.

```text 2025,balance_sheet,cash,246,us-gaap:CashAndCashEquivalentsAtCarryingValue,companyfacts.json ,#/facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD/1,"SEC Companyfacts JSON, us-gaap:CashAndCashEquivalentsAtCarryingValue, FY 2025, companyfacts.json#/ facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD/1", latest_exact_period_annual_structured_fact 2024,balance_sheet,cash,256,us-gaap:CashAndCashEquivalentsAtCarryingValue,companyfacts.json ,#/facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD/2,"SEC Companyfacts JSON, us-gaap:CashAndCashEquivalentsAtCarryingValue, FY 2024, companyfacts.json#/ facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD/2", latest_exact_period_annual_structured_fact ```

Conclusion

Summarizes contributions, limitations, future directions, and acknowledges support.

We introduced a latent‑world‑first pipeline that builds tasks backward from a known‑correct latent world, enabling dense progress rewards and exact verification. Instantiating this pipeline as TEV‑GYM yields 1,170 verified environments across seven domains, improving both 9B and 27B open models on ALE‑CM and transferring gains to Terminal‑Bench tasks of varying horizons. The study is limited to synthetic terminal environments, and the hardest benchmarks still lag behind leading frontier agents; future work should simplify environment authoring, broaden domain coverage, and combine the data with stronger open models.

We thank Scott Cohen at Adobe Research for mentorship, guidance, and support, and we acknowledge Adobe Research and our UCLA colleagues for helpful discussions and feedback.

Generative AI tools assisted in creating synthetic data, implementing TEV‑GYM generators, and writing evaluation scripts; all deterministic generators were reviewed and qualified before use. Language models appear only as objects of study in experiments, never as agents that produce training rewards or grade tasks.

TEV‑GYM is entirely synthetic, containing no human‑derived data, and all artifacts are generated from public benchmarks under their licenses. While the environments enable more capable coding agents, they also raise dual‑use concerns typical of advanced autonomous systems, which we discuss alongside benchmark overlap measurements.

Every result in this paper is reproducible: each TEV‑GYM environment is a deterministic function of its generator, seed, and scale tier, regenerable in ~30 s of CPU time. Full training and evaluation pipelines, hyperparameters, and dataset specifications are detailed in the appendix, enabling exact replication of all experiments.

Training and Evaluation Details

Training recipes, hyperparameters, and evaluation protocols for TEV‑GYM experiments.

We train all agents with DPPO, using group‑centered advantages and a total‑variation divergence bound of 0.1 on the policy ratio; no KL anchor is applied. Groups whose rewards have zero variance are discarded and refilled by active sampling, guaranteeing eight informative groups per step. The reward follows Eq. (4) with $λ\!=\!0.25$, applied per‑turn with return‑to‑go ($γ\!=\!1$) and a group‑mean baseline, plus a terminal verifier score of 1.0 for a pass.

**Table 11.** Training hyperparameters shared by every RL run in this paper.

The evaluation protocol runs ALE‑CM’s 18 long‑horizon tasks (Math & Science, ML, Coding & System) with five independent rollouts per task, using 64 turns and a per‑turn token budget of 16,384. Metrics reported are mean graded score (S@1), pass‑rate (P@1), and their best‑of‑5 counterparts (S@5, P@5). Frontier models are evaluated in their native harnesses, while all Qwen variants use the Vanillux2 single‑bash container.

**Table.** The table lists 18 tasks categorized into "Math & Science", "ML", and "Coding & System", providing a task identifier and a descriptive title for each.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers