Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, Leowei Liang

A benchmark of 46 long-horizon terminal tasks that uses dense subtask grading to expose agent failure patterns.

How do current AI agents perform on complex, long-horizon terminal tasks that require sustained reasoning and dense reward feedback?

Existing agent benchmarks focus on short, one-shot tasks, leaving a gap in evaluating complex, multi-stage workflows that require sustained planning and iterative debugging. Long-Horizon-Terminal-Bench (LHTB) addresses this by decomposing 46 realistic terminal tasks into graded subtasks, providing dense partial-credit signals instead of binary pass/fail outcomes. Even with this relaxed grading, frontier models struggle, with the strongest model achieving only a 28.3% success rate and the average model pass rate sitting at just 6.4%.

Paper Primer

Real-world workflows like reproducing research experiments or auditing scientific pipelines require agents to maintain state over hundreds of steps and hours of execution. Current benchmarks fail to capture this because they grade only the final outcome, masking the difference between an agent that fails immediately and one that makes significant progress before stalling.

LHTB forces agents to work entirely within containerized terminal environments, requiring them to read files, run scripts, and analyze diverse data formats. The core move is subtask-based grading: the benchmark decomposes each workflow into semantically meaningful intermediate goals, assigning partial credit based on objective environment state checks.

Current frontier models lack the reliability required for long-horizon execution.

Across 17 models, the mean pass rate at a 0.95 reward threshold is 6.4%, with the strongest model (Grok 4.5) resolving only 28.3% of tasks. The benchmark is significantly more demanding than prior standards, requiring an average of 239 episodes and 88.9 minutes of execution per task.

Dense grading reveals "false finish" failure modes that binary metrics hide.

Models frequently exit early with high partial rewards (R ≥ 0.75) despite failing to satisfy the hidden verifier, indicating a systematic weakness in self-verification. Near-misses (0.75 ≤ R < 0.95) occur nearly twice as often as full successes.

Why is dense, subtask-level grading necessary for this benchmark?

Binary pass/fail grading collapses diverse failure modes into a single bucket, making it impossible to distinguish between models that make no progress and those that nearly complete the task but fail at the final step.

What does this benchmark reveal about the relationship between cost and performance?

Higher inference spending does not guarantee better long-horizon performance; several high-cost models perform significantly worse than more efficient models on the Pareto frontier.

The Challenge of Long-Horizon Agents

We expose why short‑task benchmarks miss critical long‑horizon failures and introduce a new benchmark to capture them.

Current terminal benchmarks finish within minutes and score agents only on the final state, so intermediate work is invisible and reward signals are extremely sparse.

Real‑world workflows—reproducing research, installing software stacks, debugging compilers, or steering multi‑stage ML pipelines—require hundreds of steps, long‑term plan updates, and continuous verification, which short benchmarks cannot stress.

LHTB is a collection of 46 challenging terminal tasks that span nine domains and are broken into many graded subtasks, so agents receive credit continuously as they make progress.

When we evaluated 17 frontier models on LHTB, agents consumed on average 9.8 M tokens, ran 239 episodes, and spent 88.9 minutes per task—orders of magnitude larger than prior terminal benchmarks.

Even the strongest model, Grok 4.5, succeeded on only 28.3 % of tasks at a 0.95 reward threshold, while the mean pass rate across all models was just 6.4 % (3.2 % for perfect reward), highlighting a substantial gap in long‑horizon capability.

The shift from short‑task benchmarks to long‑horizon terminal environments reveals a large, previously hidden performance gap.

Designing the LHTB Benchmark

Design of the Long‑Horizon‑Terminal‑Bench benchmark, including task format, dense reward grading, and dataset construction.

This section details how the Long‑Horizon‑Terminal‑Bench (LHTB) is built, what signals it provides, and how the benchmark varies across tasks.

Each task is a self‑contained terminal environment that an agent interacts with solely via command‑line operations.

Instead of a binary pass/fail, the benchmark assigns a normalized score for each meaningful subtask, then aggregates them into a single dense reward.

Subtask 1 succeeds, so $r_1=1$.

Subtask 2 produces a plot with 0.8 similarity to the reference, yielding $r_2=0.8$.

Compute $R = (1\cdot1 + 1\cdot0.8)/(1+1) = 0.9$.

This example shows how partial progress on the second subtask still contributes substantially to the overall reward, encouraging agents to keep improving even when the final goal is not yet reached.

Specify a terminal‑only task format (Docker image, instruction, oracle).

Decompose the overall goal into a small set of semantically meaningful subtasks.

Design deterministic graders for each subtask, choosing binary or continuous scoring as appropriate.

Assign equal or task‑specific weights $w_k$ and define the aggregation formula for $R$.

Collect a diverse set of real‑world workflows, adapt them into the terminal format, and inject hidden stress cases.

Calibrate difficulty by running a strong baseline (e.g., Deepseek‑V4‑Pro) under a fixed time budget and adjusting task design until the baseline achieves a target success rate.

Finalize the benchmark with 46 tasks spanning 21 high‑level categories.

**Figure 1** The overall structure of long-horizon terminal task with dense reward grading. Partial rewards show how far models are into completing the task.

**Figure 2. Task distribution across the LHTB categories.** The 46 tasks span nine long-horizon domains, ranging from software and reverse engineering to scientific computing, earth and climate science, and multimodal analysis.

**Table 2.** Task list for Long-Horizon-Terminal-Bench. Each row shows a task ID, its paper-level category, a one-sentence description, and a coarse difficulty label derived from average task reward in Figure 3 (Easy if mean reward ≥ 0.5, otherwise Hard).

Performance and Failure Analysis

Top models improve pass rates while keeping costs modest.

Grok 4.5 resolves 28.3 % of Long‑Horizon‑Terminal‑Bench tasks at the $R\ge0.95$ threshold, outpacing the next best models by a wide margin.

Pass@1 for Grok 4.5 is 28.3 % (13/46) versus 15.2 % for GPT‑5.6‑sol and GPT‑5.5.

The evaluation protocol, dense‑reward grading, and episode budget were held constant across models; only GPT‑5.3 used a different harness (Codex) for compatibility.

A lightweight harness that standardizes episode management, token accounting, and cost estimation for long‑horizon agents.

How does the Harbor framework differ from simply calling an LLM API?

Beyond raw generation, Harbor orchestrates a full episode: it tracks state, enforces a time budget, records token usage, and translates those counts into a monetary cost, which a plain API call would not provide.

The Pareto curve that shows which models achieve the best pass rates for a given monetary cost.

Why plot cost on a log scale instead of linear?

Model costs range from a few dollars to tens of dollars per task; a log axis compresses this span, making the trade‑off shape visible and preventing the cheap models from collapsing into a single point.

A systematic breakdown of why agents do not finish tasks, distinguishing timeouts, early exits, and false finishes.

What distinguishes a false finish from a timeout?

A false finish occurs when the agent stops early while already achieving a high reward (e.g., $R=0.92$), whereas a timeout forces termination after the budget runs out, often with near‑zero reward.

**Figure 3.** LHTB leaderboard. Pass@1 at reward thresholds 0.9, 0.95, and 1.0, together with mean normalized reward, for each model under a shared agent setup, shown in a single model order sorted by pass@1 at $R \geq 0.95$. Pass rate is the fraction of the 46 tasks whose overall reward $R$ meets a threshold $\tau$, with $R \geq 1.0$ corresponding to full completion, while mean $R$ captures partial progress across all tasks. Grok 4.5 achieves the strongest overall performance, resolving 28.3% of tasks at $R \geq 0.95$. Tightening the threshold to $R \geq 1.0$ collapses most models to zero pass rate, showing that partial-credit evaluation captures progress that a binary threshold misses.

**Figure 4.** Distribution of final rewards over all $17 \times 46$ model-task runs. Only 50 runs (6.4%, green) pass the $R \geq 0.95$ threshold, while 241 runs (30.8%, gray) make no meaningful progress ($R < 0.05$). The remaining 491 runs (62.8%, blue) achieve partial reward but would all be counted as failures under binary pass/fail evaluation. This partial progress is often substantial rather than trivial: 223 runs (28.5%) reach $R \geq 0.5$, and the mass in $[0.85, 0.95)$ (68 runs) is still larger than the number of full passes. Binary grading collapses a wide range of partially successful runs into a single failure bucket, whereas dense subtask-level rewards preserve the signal needed to distinguish model progress at this difficulty level.

**Figure 5** Cost-reward frontier on LHTB. Estimated total cost (log scale) against pass rate at $R \geq 0.95$; the dashed line traces the Pareto frontier, and models on the frontier are shown in bold. Grok 4.5 achieves the highest pass rate among the reported models while remaining on the Pareto frontier, and Hy3 stays at the low-cost end of that frontier.

**Figure 6. Composition of unresolved runs per model.** Each bar decomposes a model's unresolved tasks ($R < 0.95$, out of 46) into timeouts (the agent is still working when the 90-minute budget expires), early exits (the agent terminates on its own with no harness error), and harness errors (non-timeout exceptions such as API- or verifier-side failures). The label inside each timeout segment shows the number of timed-out runs and their mean reward; the label inside each early-exit segment shows the number of early exits and their mean reward. Timeouts dominate for every model (79% of all unresolved runs), showing that many failures reflect incomplete progress rather than immediate breakdowns, while the smaller early-exit segments highlight cases where agents stop too soon despite not having fully solved the task.

Contextualizing Agent Benchmarks

Existing benchmarks are too short; LHTB adds long‑horizon tasks with dense rewards to expose partial progress.

Recent agent benchmarks have moved from static QA toward interactive, execution‑grounded tasks. In software engineering, SWE‑Bench evaluates repository‑level issue resolution, while LiveCodeBench tests code generation, execution, self‑repair, and test‑output prediction.

Further benchmarks such as SWT‑Bench, Terminal‑Bench, FrontierSWE, SWE‑EVO, SWE‑Marathon, and Edge‑Bench target increasingly complex, multi‑stage, or ultra‑long‑horizon workflows.

Measuring long‑horizon autonomy has shifted to analyzing task duration, as in METR’s time‑horizon analysis, RE‑Bench, and HCAST, which compare agents against human performance on extended tasks.

Beyond outcome‑only grading, process reward models and rubric‑based approaches provide dense intermediate supervision, a principle adopted by LHTB for subtask‑level deterministic grading.

Agent harnesses such as SWE‑agent, OpenHands, and the Terminus family mediate model interaction with environments, while open‑source agents like Codex and OpenClaw add prompting, tool orchestration, and action‑selection policies.

Benchmark Task Taxonomy

Task taxonomy and difficulty labeling for the Long‑Horizon‑Terminal‑Bench.

The taxonomy groups tasks into nine categories and assigns an Easy/Hard label based on mean reward.

The table lists various tasks with their corresponding IDs, categories, descriptions, and difficulty levels.

Extended Task Descriptions

Appendix lists diverse reproducibility and audit tasks across domains.

This appendix enumerates the concrete reproducibility, audit, and repair tasks that were used to evaluate the benchmark suite, spanning climate, simulation, security, and interactive domains.

This table lists various tasks with their corresponding IDs, categories, descriptions, and difficulty levels.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers