Longhorizon-Harness: Horizon Agents for Real-World Tasks

Ziyu Ma, Hailang Huang, Shun Zou, Yong Wang, Shidong Yang, Yiming Hu, Fei Wei, XiangXiang Chu

LongHorizon-Harness decouples task-state management from execution to prevent compounding errors in long-horizon agent tasks.

How can we improve long-horizon agent reliability by decoupling task-state management from environment interaction?

Agents performing long-horizon tasks often fail because they maintain their task state and execution history in a single, growing context, leading to compounding errors and goal drift. LongHorizon-Harness reformulates execution as a Manage-Execute-Audit (MEA) loop, where a manager maintains an explicit task state, an executor performs subtasks in fresh contexts, and an auditor independently verifies environment changes. This approach significantly improves performance across hybrid GUI-CLI, desktop, and command-line benchmarks, nearly doubling success rates on complex tasks like WeaveBench.

Paper Primer

Existing agent harnesses suffer from "context rot" and state loss because they conflate the execution trajectory with the task state. When an agent performs a multi-step task, it often relies on its own potentially incorrect self-assessments, which then propagate into future decisions and steer the agent away from the original goal.

LongHorizon-Harness introduces a Manage-Execute-Audit (MEA) loop: the manager defines a bounded subtask contract, the executor performs it in a fresh, isolated context, and the auditor independently inspects the environment to verify progress. Only verified facts are incorporated into the persistent task state, ensuring that subsequent rounds build on confirmed outcomes rather than unverified execution claims.

LongHorizon-Harness substantially improves task completion rates across diverse long-horizon benchmarks.

On WeaveBench, the PassRate for Qwen 3.7-Plus increased from 51.8% to 80.7%; on OSWorld 2.0, binary completion rose from 2.8% to 8.3%. Nearly 1.5x to 3x improvement in success metrics depending on the benchmark and model.

The framework provides consistent gains across different backbone models and interaction domains.

Claude Opus 4.7 performance on an OSWorld 2.0 subset improved from 20.0% to 34.3%, demonstrating that the harness complements rather than replaces model capability. Consistent performance lift across both GUI-heavy and pure CLI environments.

Why does the framework use a "fresh-context" executor for each subtask?

Discarding the raw interaction trajectory after each round prevents the agent from being misled by its own accumulated errors or irrelevant history, forcing it to rely on the verified task state maintained by the manager.

Does this approach add significant computational overhead?

The manager and auditor roles account for a small fraction of total tokens (typically under 40% combined), and the framework can actually reduce total token consumption on some tasks by avoiding the repeated, failed execution cycles common in baseline agents.

Introduction and Motivation

We expose why long‑horizon agents fail and introduce the Manage‑Execute‑Audit loop to fix it.

Current long‑horizon LLM agents keep all execution history, task state, and progress checks in a single, ever‑growing context. As the context expands, the agent loses track of what it has already done and can mis‑judge its own progress, leading to error accumulation and goal drift.

When an agent mixes the record of what it has done with the representation of what still needs to be done, it cannot reliably know which facts are fresh and which are stale.

The authors restate the problem as a task‑state management issue and propose LongHorizon‑Harness, which isolates the task state from execution. Their Manage‑Execute‑Audit (MEA) loop lets a manager read the current state, a fresh‑context executor carry out a single subtask, and a read‑only auditor verify the environment change before the state is updated.

**Figure 1** Left: LongHorizon-Harness improves long-horizon execution across benchmarks and backbones. With the same backbone and execution backend, it lifts WeaveBench PassRate from 51.8% to 80.7%, Terminal-Bench 2.1 from 69.7% to 77.2%, and OSWorld 2.0 binary completion by 3.0×, and the gains transfer from Qwen 3.7-Plus to Claude Opus 4.7. Right: Audited state transitions. Instead of one continuously growing session that judges its own progress, a manager re-plans the next subtask from audited facts, a fresh-context executor performs it, and a read-only auditor certifies what actually changed in the environment. Audit reports are the only cross-round memory, and interchangeable backends (e.g., Claude Code, Codex) serve each role.

Empirically, the framework improves Qwen 3.7‑Plus on WeaveBench from 51.8 % to 80.7 %, on Terminal‑Bench 2.1 from 69.7 % to 77.2 %, and on OSWorld 2.0 from 2.8 % to 8.3 %. Similar gains appear with Claude Opus 4.7, confirming that the benefits transfer across models and harness backends.

The paper’s contributions are threefold: (i) reframing long‑horizon execution as explicit task‑state management, (ii) designing the Manage‑Execute‑Audit loop with interchangeable manager, executor, and auditor components, and (iii) demonstrating consistent performance lifts across three diverse long‑horizon benchmarks.

The core failure is conflating state tracking with execution; separating them via an audited task state restores reliable long‑horizon performance.

The Manage-Execute-Audit Loop

Methodology details the Manage‑Execute‑Audit loop that structures long‑horizon tasks.

The LongHorizon‑Harness framework breaks a long‑horizon task into a series of self‑contained rounds. Each round keeps only an explicit task state and audit evidence, discarding the raw execution trace. This design prevents state‑tracking from being tangled with environment interaction.

**Figure 2.** Overview of LongHorizon-Harness. LongHorizon-Harness processes a long-horizon task $\mathcal{T}$ through repeated Manage-Execute-Audit rounds, shown by the dashed box and numbered flow 1–3. The manager reads task state $S_i$ and constructs subtask contract $c_i$, which specifies the goal, acceptance criteria, boundary constraints, and relevant prior evidence (1). The manager may instead request user information or authorization through the ask route. The selected GUI or CLI executor performs the subtask in a fresh, budget-bounded context and modifies the environment (2). The auditor independently inspects the resulting environment through read-only tools and produces audit report $v_i$ (3). The manager uses $v_i$ to update the task state before the next round.

The loop isolates planning, action, and verification so that each round can be reasoned about independently, avoiding the “state‑interaction conflation” that plagues monolithic agents.

How does this loop differ from a simple “plan‑then‑act” pipeline?

In a plain plan‑then‑act system the planner directly observes the environment and trusts its own execution output. Here the manager never sees the environment; it must rely on an independent auditor’s read‑only evidence before updating the persistent state.

Round 1 Manage: manager emits contract $c_1$ = “create file $f$”.

Round 1 Execute: executor creates $f$, producing environment $e_1$ and report $o_1$.

Round 1 Audit: auditor reads the filesystem, confirms $f$ exists, and emits $v_1$ marking the requirement as completed.

Round 2 Manage: manager updates $S_2$ (requirement now completed) and emits contract $c_2$ = “open file $f$”.

Round 2 Execute: executor opens $f$, producing $e_2$ and $o_2$.

Round 2 Audit: auditor verifies the file is open and emits $v_2$; $S_3$ now satisfies the overall task.

The loop guarantees that each subtask’s success is independently verified, so a failure in round 1 would never be silently propagated to later rounds.

Task state is a structured ledger that records what the overall task still needs, what has been produced, and what evidence backs each entry.

Why can’t the manager simply trust the executor’s report $o_i$?

Because $o_i$ may be fabricated or incomplete; the auditor’s read‑only inspection provides an objective ground truth that the manager can rely on to update $S_{i+1}$ safely.

Manage emits $c_1$ = “run `pip install foo`”.

Execute runs the command, producing $o_1$ that claims success.

Audit runs `pip show foo` in read‑only mode, sees version 1.3, and emits $v_1$ marking the requirement completed.

If the executor had failed silently, the auditor would report “missing package” and the requirement would stay pending.

The auditor’s independent check catches execution failures that the executor might misreport.

Receive the original task $T$, current state $S_i$, and accumulated audit history $V_i$.

Apply the latest audit $v_i$ to $S_i$, updating record statuses (completed, blocked, etc.).

Compare the updated $S_{i+1}$ against $T$ to decide the next control signal $q_{i+1}$ (execute, done, blocked, ask).

If $q_{i+1}= \text{execute}$, select an unresolved requirement that can be advanced and package its context into a bounded contract $c_{i+1}$.

Return the tuple $(S_{i+1}, q_{i+1}, c_{i+1})$ to the harness.

Receive task $T$, state $S_i$, and contract $c_i$ together with any audit reports referenced by $c_i$.

Launch a bounded episode with a fresh sandbox that contains only the inputs supplied for this round.

Perform the actions required by $c_i$ (GUI clicks, CLI commands, file edits) until the episode budget expires or the contract’s goal is reached.

Return the new environment snapshot $e_i$ and an execution report $o_i$ summarizing actions, artifacts, and any issues.

Receive $T$, $S_i$, $c_i$, the executor’s report $o_i$, and the fresh environment $e_i$.

Using read‑only tools, locate the artifacts, files, or UI elements referenced in $c_i$.

Check the contract’s acceptance criteria and boundary constraints against $e_i$.

Emit audit report $v_i$ indicating completion status (complete/incomplete/blocked) and integrity status (clean/suspect/violation), together with any newly verified facts.

Experimental Results

LongHorizon‑Harness lifts task success far beyond the baseline.

LongHorizon‑Harness raises WeaveBench PassRate from 51.8 % to 80.7 %, a +28.9 % gain over the Claude Code baseline.

Table 1 shows the baseline (Claude Code + Qwen 3.7‑Plus) at 51.8 % PassRate and the LH‑Harness version at 80.7 %.

Claude Code is the execution backend that runs the low‑level actions (clicks, commands, API calls) required by a task.

OSWorld 2.0 is a desktop‑workflow benchmark that strings together GUI and CLI actions into long‑horizon tasks.

WeaveBench evaluates agents on tasks that require coordinated GUI and CLI interactions within a single workflow.

Terminal‑Bench 2.1 measures agents on pure command‑line tasks that require precise sequencing and verification.

Detailed Benchmark Results

Key quantitative gains of LongHorizon‑Harness across all benchmarks.

The Manage‑Execute‑Audit loop isolates environment state from transient interaction, letting the agent verify progress against an explicit task state.

LongHorizon‑Harness raises the OSWorld 2.0 Opus 4.7 partial score by +11.0 percentage points over the baseline.

Table 5 shows the baseline at 55.83 % and LH‑Harness at 66.86 %.

**Table 5.** Detailed per-task results on the OSWorld 2.0 Opus 4.7 subset. Ours denotes LongHorizon-Harness.

**Table.** Performance comparison between Baseline and LH-Harness across various tasks.

**Table 7.** Fine-grained results for WeaveBench Games tasks. Task names omit the shared `GAM_task_*` prefix for compactness.

**Table 8.** OSWorld 2.0 results by capability tag. Tags are overlapping. “Zero” and “Full” count tasks in the tag group where LH-Harness obtains score 0 or 1, respectively.

**Table 10.** Terminal-Bench 2.1 results by difficulty.

**Table 11.** Terminal-Bench 2.1 tag-level results. Tags may overlap across tasks.

Qualitative Case Studies

Qualitative case studies show how the Manage‑Execute‑Audit loop rescues long‑horizon tasks.

The paper’s core claim is that long‑horizon failures stem from conflating state tracking with interaction; the Manage‑Execute‑Audit loop isolates a clear task state and audits each step.

This appendix expands the qualitative analysis with concrete case studies across WeaveBench and Terminal‑Bench, each illustrating how the loop turns partial progress into auditable state propositions.

**Figure 7.** Recovering from a stalled interaction. The case is `WEB_task_16`, a WebRTC simulcast-layer audit. Top: The Claude Code baseline recognizes that Wireshark’s “Decode As” dialog is unresponsive, but continues retrying the same interaction for more than 400 steps and obtains a score of 0.59. Bottom: LongHorizon-Harness records the unresolved evidence gaps in the task state. Subsequent execution rounds collect the missing chart- and packet-level evidence, resulting in a score of 0.92.

LongHorizon‑Harness improves the WEB _task 16 score from 0.59 to 0.92.

Figure 7 shows the baseline’s endless retries versus the harness’s evidence‑driven recovery.

**Figure 8. Auditing apparent completion.** The case is `DOC_task_2`, which requires heading-style normalization. Top: The Claude Code baseline edits the document XML directly and terminates with a visually plausible result. Because the required LibreOffice workflow is not followed, the result receives a score of 0.00. Bottom: LongHorizon-Harness applies the heading styles through the prescribed GUI workflow and rechecks the document. The auditor then parses the document XML to confirm the final style of all 15 headings, obtaining a score of 0.89.

LongHorizon‑Harness raises the DOC _task 2 score from 0.00 to 0.89.

Figure 8 illustrates the baseline’s visual‑only edit versus the harness’s style‑level verification.

**Figure 9. Preserving pre-repair evidence.** In `DOC_task_4`, the agent must repair a Calc VLOOKUP formula while preserving before-and-after evidence. *Top*: The Claude Code baseline records initial formula errors but edits the spreadsheet before completing the required pre-repair evidence, leaving the final file and evidence sequence inconsistent and scoring 0.45. *Bottom*: LongHorizon-Harness keeps the missing pre-repair evidence as a pending requirement, records it before modification, and audits the full nine-screenshot sequence, scoring 0.87.

LongHorizon‑Harness lifts the DOC _task 4 score from 0.45 to 0.87.

Figure 9 highlights the missing pre‑repair evidence in the baseline.

**Figure 10.** Continuing from verified progress. `WEB_task_10` is a Lighthouse performance-optimization task. *Top*: The Claude Code baseline completes the core optimization but gets stuck during DevTools evidence collection, fails to produce all required deliverables, and scores 0.53. *Bottom*: LongHorizon-Harness preserves the verified optimization state, delegates the remaining evidence requirements to later rounds, and audits the final screenshot and metadata, scoring 0.85.

LongHorizon‑Harness improves the WEB _task 10 score from 0.53 to 0.85.

Figure 10 shows the baseline’s dead‑end versus the harness’s audited continuation.

**Figure 11** Desktop workflow case. LongHorizon-Harness creates the rich note, recovers from failed GUI interactions by updating the task state, and verifies the reproduced rendering bug through screenshot evidence.

**Figure 12.** Document-processing case. The harness verifies the underlying ODT structure instead of relying on visual formatting, ensuring that all headings are semantically normalized.

**Figure 13.** Game-analysis case. GUI replay identifies the illegal PGN transition, while file-level auditing checks that the final reports are consistent with the board evidence.

**Figure 14.** Web-diagnostics case. LongHorizon-Harness combines browser charts, tooltip evidence, and Wireshark inspection into a single audited evidence chain.

**Figure 15.** Data-analysis case. The harness rejects mislabeled evidence, captures the correct Airflow views, and audits whether each screenshot supports the required diagnosis.

**Figure 16.** DevOps case. The harness links management-UI symptoms to queue and binding evidence, then verifies the repaired routing state across GUI and CLI views.

**Figure 17** Spatial/CAD case. The harness combines DXF-level inspection, LibreCAD layer repair, print-preview evidence, and screenshot-integrity checks.

**Figure 18.** Design case. LongHorizon-Harness grounds visual quality comparison in real GIMP state, including histograms and side-by-side rendered outputs.

**Figure 19.** Terminal-Bench build case. LongHorizon-Harness turns a multi-condition build task into an explicit acceptance contract and verifies the final SQLite installation with gcov evidence.

**Figure 20.** Terminal-Bench reverse-engineering case. The harness preserves exploratory evidence as audited facts and verifies the final independent C implementation against behavior, compilation, size, and independence constraints.

Across all domains the pattern is identical: local actions become useful only after they are recorded as auditable state propositions, enabling the manager to issue the next bounded subtask without losing verified progress.

These case studies substantiate the paper’s central premise: performance limits arise not merely from model capability but from how the harness represents, verifies, and carries forward task state.

Experimental Setup Details

We detail the LongHorizon‑Harness framework and its experimental evaluation.

LongHorizon‑Harness wraps agent execution in a Manage‑Execute‑Audit loop that separates state tracking from environment interaction.

WeaveBench comprises 114 long‑horizon computer‑use tasks spanning eight domains (Desktop, Document, Games, Web, Data Analysis, DevOps, Spatial/3D, Design). Each task runs in an isolated Ubuntu desktop VM, using Qwen 3.7‑Plus as the model and Claude Code 2.1.76 as the executor. We allow up to 25 Manage‑Execute‑Audit rounds per task, with a 1800‑second timeout for the executor and 300‑second timeouts for the manager and verifier.

OSWorld 2.0 contains 108 professional desktop workflow tasks, each averaging 1.6 hours of human effort. Experiments use the official Docker‑based VM (1920×1080 resolution) and a hybrid tool pool that combines GUI actions with CLI commands. The same Qwen 3.7‑Plus model runs under Claude Code 2.1.176, and we report Binary Accuracy (strict success) and Partial Accuracy (average fine‑grained score).

Terminal‑Bench 2.1 evaluates agents on realistic command‑line software‑engineering tasks via the Harbor framework. Both the baseline and LongHorizon‑Harness settings use Claude Code 2.1.211 with temperature = 1.0, `top_p` = 0.95, `top_k` = 20, and a 5‑hour per‑task timeout. Each task is run three times; we report the average score across the trials.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers