T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li, Haitao Mi, Leowei Liang
T1 uses reinforcement learning with dense execution rewards and routing replay to master long-horizon terminal tasks.
How can we stabilize reinforcement learning for long-horizon terminal agents to outperform frontier models on complex, multi-step tasks?
Large language models struggle to maintain performance over long-horizon terminal tasks, where minor numeric drifts in sparse models cause catastrophic policy divergence during multi-turn reinforcement learning. The authors introduce T1, a 122B Mixture-of-Experts model trained with two stabilization mechanisms: TITO (Token-In-Token-Out) to enforce bit-exact token sequences, and R3 (Rollout Routing Replay) to lock expert selection during training. On Terminal-Bench 2.1, this pipeline raises the base model from 43.8% to 64.0% resolved, outperforming several frontier models while using only 10B active parameters.
Paper Primer
Reinforcement learning on terminal tasks is notoriously unstable because the training and inference systems operate on different hardware kernels and reduction orders. In a sparse Mixture-of-Experts (MoE) model, these tiny numeric discrepancies cause the router to select different experts during training than it did during generation, effectively training a different sub-network than the one that produced the behavior.
T1 solves this by enforcing two fidelity conditions: TITO ensures the trainer consumes the exact token identifiers generated by the sampler, while R3 records the expert routing mask during inference and forces the trainer to reuse that specific mask. This alignment allows the model to learn from dense execution rewards—scored by the absolute number of passing assertions in a sandbox—without the policy collapsing due to bookkeeping errors.
T1 achieves state-of-the-art performance on long-horizon terminal benchmarks using reinforcement learning.
Evaluation on Terminal-Bench 2.1 shows T1 reaching 64.0% resolved, surpassing GPT-5.4 (54.8%) and DeepSeek-V4-Flash (56.9%). 28.5% relative gain over the supervised fine-tuned checkpoint.
TITO and R3 mechanisms effectively eliminate training-inference drift.
Measured log-probability gap between training and inference reduced from 0.021 to 0.013, with 0.0000% drift in the loss region. Near-perfect alignment of policy evaluation across disparate hardware stacks.
Why is a dense reward necessary for these tasks?
Binary pass/fail rewards provide only one bit of signal per trajectory, which is insufficient for tasks requiring hundreds of tool-call turns. Dense rewards based on per-assertion outcomes allow the model to receive credit for partial progress, turning otherwise zero-reward trajectories into useful learning signals.
Does this approach rely on benchmark overfitting?
No; the training corpus is fully out-of-distribution relative to the evaluation benchmarks. The authors use isolated seeds and synthesized tasks disjoint from the test sets to ensure the observed gains reflect genuine capability transfer.
The core assumption is that the verifier's per-assertion report is a reliable proxy for progress. The authors mitigate reward hacking by filtering the training pool for verifiers that are too weak to validate their own goals, ensuring the model is rewarded for actual task completion rather than gaming the assertion count.
For researchers building agentic systems, T1 demonstrates that stable RL at scale requires treating the training-inference mismatch as a first-class engineering constraint, specifically by forcing bit-exact token and routing fidelity.
Introduction to T1
We expose why training terminal agents via RL is hard and preview our T1 solution.
Training terminal agents with reinforcement learning is hampered by two intertwined issues: sparse, binary rewards from execution and a mismatch between the high‑throughput inference system and the rollout process used during training. These gaps cause unstable gradients and impede long‑horizon reasoning.
A terminal agent is a model that interacts with a command‑line environment, issuing sequences of tool calls until a task’s verifier signals success.
First, training‑inference consistency breaks for sparse Mixture‑of‑Experts models because tiny numeric differences flip expert routing at turn boundaries, leading to divergent gradients. Second, dense reward is scarce: a binary pass/fail per trajectory supplies only one bit of signal, making it hard for the critic to guide the actor.
**Figure 1** Performance vs. model size under the same agent harness (Terminal-Bench 2.1 resolved rate). At 122B, *T1* achieves 64.0%, outperforming GPT-5.4 (54.8%), DeepSeek-V4-Flash (56.9%), and Claude Opus 4.6 (63.8%). It closely approaches Claude Opus 4.7 (66.1%), and is the best model in its size band.
T1 demonstrates that a 122 B Mixture‑of‑Experts architecture, combined with TITO alignment, can substantially close the RL training‑inference gap on terminal tasks.
The T1 Training Pipeline
The training pipeline stitches asynchronous rollouts with dense verification to enable stable long‑horizon learning.
Training terminal agents requires aligning high‑throughput inference with sparse‑reward rollouts; our pipeline hides latency by pipelining generation and training steps across separate accelerators.
**Figure 2** The *T1* training pipeline. **Left**: each task is self-contained, comprising metadata, a long-horizon instruction, an environment image with resource limits, a held-out verifier and a reference solution, and its verifier reports each assertion individually, which makes the reward executable rather than modelled. **Middle**: the inference replicas serve the behaviour policy under oversampling; the trajectory assembler normalizes interaction logs into TITO-stitched training samples carrying routing records and the dense reward; the trainer updates the critic and then the actor before synchronizing weights back. **Right**: sandboxes are created, loaded, driven turn by turn, verified and reclaimed. Rollout and training run concurrently on disjoint devices.
Each task bundles a self‑contained environment, a long‑horizon instruction, and a verifier that reports every assertion individually—turning the reward into an executable signal.
How does the verifier turn a binary pass/fail into a dense learning signal?
The verifier emits a separate score for every assertion (e.g., “file compiled”, “test passed”), and the sum of these scores becomes the scalar reward that the critic distributes across the entire trajectory.
Assertion 1 succeeds → score +1.
Assertion 2 fails → score +0.
Assertion 3 succeeds → score +1.
Dense reward = 1 + 0 + 1 = 2.
Even though the overall task succeeded, the dense reward reveals which sub‑steps contributed, enabling the critic to reinforce the successful edits while penalizing the failing unit test.
Inference replicas generate token identifiers in parallel while the trainer consumes a trajectory assembler that normalizes multi‑turn logs, ensuring that updates are anchored to a consistent token stream.
Why does the trainer update the critic before the actor?
Updating the critic first provides a fresh value baseline that the actor can use to compute advantages; this prevents the advantage estimate from being corrupted by the actor’s own parameter change within the same step.
Turn 1 token IDs → “ls” command logged.
Turn 2 token IDs → “gcc” command logged; compilation succeeds.
Turn 3 token IDs → “./`run_tests`” command logged; two of three tests pass.
Assembler aligns the three token streams, masks the tool output, and applies the same offset to each layer.
Critic is updated using the aligned logs; actor is updated afterward.
The same offset across layers guarantees that the policy’s internal representation stays consistent despite the multi‑turn, multi‑tool nature of the task.
Instead of a single binary outcome, the verifier emits a per‑assertion score, producing a dense scalar that scales with task difficulty and gives the critic a consistent target across steps.
Why is a dense per‑assertion reward preferable to a binary pass/fail signal?
A dense reward supplies a gradient at every turn, allowing the critic to assign credit proportionally to each sub‑task; a binary signal would give a single scalar after many steps, making credit assignment noisy and inefficient.
Compile succeeds → +1.
Static analysis fails → +0.
Unit test A succeeds → +1.
Unit test B succeeds → +1.
Dense reward = 1 + 0 + 1 + 1 = 3.
Even though the overall task “passes” the binary check, the dense reward reveals that static analysis was a bottleneck, guiding the learner to improve that specific aspect.
Across three epochs of PPO fine‑tuning, T1 lifts the supervised checkpoint from 49.4 % to 64.0 % on Terminal‑Bench 2.1 and matches state‑of‑the‑art performance on long‑horizon benchmarks, demonstrating that the asynchronous pipeline and dense rewards scale to challenging terminal tasks.
Terminal Dataset Design
Defines the task pools, how they are built, and the weighted scoring that selects the final 15 k tasks.
Reinforcement‑learning agents that must act over long horizons receive only sparse feedback, making learning unstable. A curated terminal dataset with per‑assertion verification supplies the dense signal needed for reliable training.
The dataset grows like a curriculum: each round takes previously accepted tasks, expands them with new steps, and only keeps the extensions that still pass a held‑out verifier.
Seed A: solution $(s_1, s_2)$ → apply rewrite → $(s_1, s_2, s_3)$.
Seed B: solution $(t_1, t_2)$ → apply rewrite → $(t_1, t_2, t_3)$.
Re‑align environment and verifier for each extended solution.
Execute both candidates in fresh sandboxes; both pass their verifiers.
Both extended tasks are admitted to the next‑round pool.
The recursion builds a curriculum where each round’s tasks are strictly harder (one extra executable step) while guaranteeing that the verifier still validates the whole workflow.
Each task is evaluated on eight orthogonal dimensions; a weighted sum produces a single QualityScore, and a per‑dimension minimum guards against catastrophic failures.
Apply weights: $0.20\!\times\!4 = 0.8$, $0.15\!\times\!3 = 0.45$, $0.15\!\times\!5 = 0.75$, $0.10\!\times\!2 = 0.2$, $0.10\!\times\!5 = 0.5$, $0.10\!\times\!3 = 0.3$, $0.10\!\times\!4 = 0.4$, $0.10\!\times\!5 = 0.5$.
Sum the weighted contributions: QualityScore $= 0.8+0.45+0.75+0.2+0.5+0.3+0.4+0.5 = 3.9$.
Check the critical‑minimum across the six critical dimensions (all except value): the lowest weighted score is $0.2$ (clarity), which is above a hypothetical threshold of $0.1$, so the task passes.
The weighted sum ranks tasks, while the per‑dimension floor prevents a task from being accepted solely because it excels in a few dimensions while failing badly in another.
**Figure 3** Category composition of T1-15k. All 15,000 tasks are counted once across 17 merged categories (from 47 raw labels); angle encodes share exactly, while radius is a rank-based power scale chosen to keep small slices visible and is therefore *not* proportional to share. The pool is concentrated in command-line engineering work: the top five categories account for 67.9% of all tasks.
**Figure 4.** The eight audit dimensions and their weights, summing to 1.00. Aggregated by facet: Verifier 45%, Solution 25%, Instruction 20%, and Task Value 10%. Darker bars highlight the highest-weighted dimensions.
How does this multi‑dimensional scoring differ from a single metric like BLEU?
BLEU aggregates n‑gram overlap into one number and cannot detect failures such as verifier‑instruction mismatches or unfair test design. The weighted quality score explicitly evaluates each orthogonal property, applies a minimum guard on critical dimensions, and therefore guarantees that accepted tasks are both verifiable and well‑specified, which is essential for dense RL rewards.
The terminal dataset is built by recursive synthesis and filtered through a multi‑dimensional quality score, ensuring dense verification and a balanced task mix for stable RL training.
Stabilizing MoE RL Training
We expose and fix the token and routing fidelity gaps that cause training‑inference divergence.
Rollout and training run on distinct back‑ends: the inference side is optimized for throughput, while the training side computes exact gradients. Because the two systems do not share tensors, the policy updates see a different token stream than the one that generated the data.
The mismatch arises from two independent fidelity gaps: (1) token fidelity – the trainer must see exactly the same token identifiers the sampler produced, and (2) routing fidelity – the MoE routing decisions must be identical between sampler and trainer.
**Figure 6.** Train–inference log-probability gap $|\Delta \log p|$ (Equation 15). (Left) Per-step series over the production dense-reward run, with an exponential moving average at $\alpha=0.25$ in bold. (Right) Mean gap on the same 122B stack with and without the two mechanisms.
Token fidelity check: compare $T^{t}_2 = 45$ with $T^{r}_2 = 46$ → mismatch.
Compute log‑probabilities under the two policies: $\log p_{\text{sampler}} = -0.30$, $\log p_{\text{trainer}} = -0.45$.
Gap $|\Delta \log p| = | -0.30 - (-0.45) | = 0.15$ for this position.
Aggregate over the two positions (first token matches, second mismatches) → average gap $0.075$.
Even a single tokenization discrepancy inflates the log‑probability gap, illustrating why exact token fidelity is essential for stable RL updates.
How does token fidelity differ from simply checking that token IDs match?
Token fidelity is stricter: it requires the entire tokenization pipeline—including whitespace handling, special‑token insertion, and template re‑encoding—to produce identical identifiers. A naïve ID check would miss subtle shifts caused by the chat template or normalization, which still break the RL ratio.
TITO Stitching Mechanism
Align token streams and routing masks to eliminate training‑inference drift.
RL training of terminal agents suffers from a drift between the high‑throughput inference stream and the training rollout, because encoding is canonical while decoding is many‑to‑one and tool‑call boundaries introduce mismatched whitespace.
TITO forces each turn’s token‑id stream to be a bit‑exact prefix of the next turn’s prompt, eliminating the re‑encoding drift that otherwise accumulates across turns.
Form $T_1 = $`p_1`$ \parallel $a_1$ = [
Check strict prefix: $T_1 \preceq p_2$ holds because $p_2$ begins with $[
Since the condition holds, the assembler appends the remaining context (none), leaving $T = (T_1, T_2)$ unchanged.
Mask $m$ marks the three sampled positions as 1; all other positions (e.g., padding) are 0.
Log‑probabilities $q$ are computed only for the three masked positions, preserving exact alignment.
When the strict case succeeds, no additional transformation is needed, guaranteeing that the trainer sees exactly the tokens the sampler produced.
How does TITO differ from simply feeding the sampled token ids into the model?
Plain feeding would still pass the ids through the encoder’s canonical tokenizer, which can re‑encode them differently (e.g., whitespace changes). TITO bypasses that step by treating the raw ids as the prompt prefix, guaranteeing bit‑exact prefix equality across turns.
**Figure 5** TITO stitching, drawn for the RETOKENIZED case (Equation 8), the hardest of the four. The re-tokenized copy $\tilde{a}_i$ never enters the training stream, whereas the sampled $a_i$ does; observations and glue enter masked, with $m=0$ and $q=0$.
R3 records the routing mask produced during inference and replays it unchanged during training, so the same experts receive gradient updates that they actually contributed to the sampled trajectory.
Store $R_1[1,1,:] = [3,5]$ and $R_1[1,2,:] = [1,4]$ as 4‑byte integers.
During training, compute live logits $s_1^1(\theta_t)$ and $s_2^1(\theta_t)$.
Apply the replayed mask: for layer 1, normalize $\exp(s_1^1)$ over experts 3 and 5 only.
For layer 2, normalize over experts 1 and 4 only.
Aggregate the expert outputs using the replayed weights, yielding the same expert mixture that generated the rollout.
Replay guarantees that the gradient updates the exact experts that contributed to the sampled trajectory, removing the discontinuity that would otherwise arise from a different top‑$k$ selection.
Why not simply recompute the top‑$k$ selection during training instead of replaying the recorded mask?
Recomputing would select a different set of experts because the router’s logits change as training progresses. The recorded mask fixes the selection to the experts that actually produced the rollout, aligning gradients with the sampled trajectory and avoiding the “expert‑mismatch” problem.
R3 replay pseudocode – reuse recorded routing mask during training.
Training‑stability of T1 is measured by the mask‑weighted mean of the absolute log‑probability gap $|d_j|$, confirming that both token and routing fidelity are preserved after applying TITO and R3.
Critic‑side stability relies on a separate full‑size critic that shares the actor’s device allocation; a warm‑up phase trains the critic alone for one epoch before any policy update, yielding explained variance between 0.71 and 0.86 from the first update.
Dense Verification Reward Design
We introduce a dense verification reward that grades partial task progress via per‑assertion counts.
Sparse binary rewards left most trajectories at zero, even when agents satisfied many intermediate requirements, making learning on hard tasks impossible.
Instead of a single “solved / not solved” signal, the reward counts how many verification assertions the agent passes, turning every trajectory into a graded learning signal.
Why not use a pass‑ratio ($P/T$) instead of the absolute count $P/S$?
Because tasks have varying numbers of assertions; a ratio would give the same 0.5 score to a hard task with ten passes out of twenty and an easy task with two passes out of four, even though the former required substantially more work. The absolute count preserves that effort difference.
Compute $r = P/S = 5/20 = 0.25$.
The agent receives a reward of 0.25 for this trajectory.
If later the same task yields $P=7$, the reward becomes $7/20 = 0.35$, a $0.10$ increase reflecting two additional verified steps.
The reward grows linearly with each new passed assertion, so the agent can see incremental progress even when the task remains unsolved.
Algorithm 2: Dense reward assignment for one rollout step.
Credit is assigned by the critic alone: the scalar $r$ is placed on the final response token, and GAE with $\gamma=\lambda=1$ propagates it backward over the entire interaction horizon.
Reward hacking is mitigated on two fronts. First, tasks that could be gamed are filtered out by a semantic audit with DeepSeek‑V4‑Pro, rejecting any task with hidden requirements, test leakage, shortcut solutions, or weak verifiers. Second, the fixed denominator $S$ prevents a single extreme sample from reshaping the batch‑wise reward scale, eliminating a common manipulation avenue.
**Figure 7** Critic explained variance, defined in Equation 19, with and without Critic Warm-Up. Blue (*T1*): the production dense-reward run on T1-15k, whose critic comes from Critic Warm-Up over TMax-15k. Red: the cold-started critic of the TMax-15k campaign. Faint lines are per-step values and bold lines an exponential moving average at $\alpha=0.25$, while the dashed rule marks EV=0. The cold start starts at EV= $-33.6$ and is negative for 30 of 58 logged steps, whereas the Critic Warm-Up run never goes negative and plateaus between 0.71 and 0.86.
**Figure 8** Mean rollout reward under the test-count reward over the production dense-reward run, reported before any normalization, since under PPO the reward stage returns raw values as Algorithm 2 states. Faint line: per-step mean; bold line: exponential moving average at $\alpha=0.25$. Reward rises from 0.250 to roughly 0.345 in the first 50 steps and then holds a band of 0.34 to 0.36 for the remaining 60.
Benchmark Performance
We report the final performance of T1 on our held‑out benchmarks and compare it to contemporary models.
Training terminal agents via RL suffers from sparse rewards and a mismatch between inference and rollout; T1 bridges this gap by aligning token streams with TITO and using dense verification rewards. Terminal‑Bench 2.1 is a held‑out suite of 89 terminal tasks, reported as the fraction resolved.
T1 achieves 64.0 % resolved rate on Terminal‑Bench 2.1, surpassing all comparison models evaluated under the same harness.
Table 1 and Figure 9 show T1’s 64.0 % score, ahead of the next best model Claude Opus 4.6 at 63.8 %.
**Figure 9** Terminal-Bench 2.1 standing of *T1* against contemporary frontier and open-weight models. Blue bars trace our own pipeline: the Qwen3.5-122B-A10B base model (43.8), the RST-38k SFT checkpoint we initialize from (49.4), and *T1* after RL (64.0); the dashed arrows mark the two stages, a gain of 5.6 percentage points from SFT and a further 14.6 percentage points from reinforcement learning. Grey bars are the comparison models. With 10B active parameters *T1* ranks fourth overall and ahead of Claude Opus 4.6, and is the only model in the leading group that reaches that band from a sub-50 starting point.
**Figure 10** Performance on Long-Horizon Terminal Bench. The base model, RST-SFT checkpoint, and T1 score 18.9, 23.6, and 27.9, respectively. RL adds 4.3 average-reward points over SFT, and T1 matches Gemini-3.1-Pro among the models shown.
**Figure 11** Terminal-Bench Hard resolved rate: *T1* outperforms the RST-SFT checkpoint by 9.7 points and the base model by 18.0 points.
Domain Performance Breakdown
Rebalancing devices cuts step time by 2.6×, enabling scalable training.
Rebalancing inference and training devices yields a 2.6× reduction in step time.
Moving from one to three inference replicas cut generation from 58.9 min to 19.6 min, reducing the overall step from ~60 min to ~23 min.
Figure 15 reveals that T1’s advantage is not uniform: it resolves every task in data‑processing and machine‑learning, yet GPT‑5.6 Sol still leads on data‑science, scientific‑computing, and mathematics, while file‑operation tasks remain hard for T1.
All three checkpoints achieve 100 % on Easy tasks, so the gains concentrate on Medium and Hard groups. T1 reaches 78 % on Medium (≈20 pp above SFT) and 33 % on Hard (≈3 pp above SFT), indicating that reinforcement learning mainly improves medium‑difficulty reasoning.
T1’s policy uses 94.4 turns on average—about 3× more than the SFT checkpoint (31.5) and 2.3× more than the base model (41.1). The longer interaction correlates with modest Hard‑task gains, suggesting that extra turns are only beneficial when paired with effective diagnosis and stopping.
Failure analysis shows T1 spends 164–473 turns on six unsolved tasks, whereas GPT‑5.6 Sol solves comparable tasks in 6–40 turns before timing out. Resource differences (input limit 56 k vs 120 k tokens, output limit 8 k vs 32 k tokens, and disabled explicit thinking) further complicate direct attribution.
Memory, not arithmetic throughput, limits placing two 122 B networks on the same device set.
Proposition 7.1 states that the optimizer footprint of the expert parameters is invariant to the sharding plan $\Pi$; reducing expert sharding merely swaps resident weights for optimizer state without changing total memory.
Splitting the recurrent scan across ranks preserves exact sequential semantics while dividing activation memory.
Liveness under hour‑scale steps is ensured by bounding every control‑plane operation with a deadline while allowing bulk transfers to run unbounded. A two‑stage probing scheme (600 s grace, then 30 s checks with 120 s deadlines) distinguishes genuine hangs from slow but healthy tasks.
Environment concurrency is limited by per‑node caps (35 at batch 256, up to 94 at batch 512) and a token‑bucket admission rate of 6 s⁻¹ against a 600 min⁻¹ quota. Oversampling 560 trials for a batch of 512 and cancelling the surplus bounds wall‑clock time but introduces a selection bias toward harder task families.
The publication barrier synchronizes inference and training updates: parameters are published once per step after generation quiesces, making the staleness exactly one update. With disjoint pipelines, step time is $\displaystyle T_{\text{step}}(n)=\max\!\bigl(\frac{R}{n},\frac{C}{G-n}\bigr)$, and the optimal split $n^{\ast}$ minimizes this maximum.
**Figure 12** Held-out Terminal-Bench 2.1 resolved rate over the production dense-reward run. Filled markers are the evaluated checkpoints (every 10 rollout steps), each annotated with its score; the starred point is the peak. The dashed rules are the two fixed anchors, namely the base model at 43.8% and the SFT checkpoint the run is initialized from at 49.4%. The very first evaluated checkpoint already clears SFT by 6.8 percentage points, no evaluated checkpoint ever falls back to the initialization, and the peak at step 110 reaches 64.0%, which stands 14.6 percentage points above SFT and 20.2 percentage points above base.