StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, Kai Wang

StateM improves agent reliability by externalizing execution control into a versioned, state-machine runbook.

How can we maintain agent performance over long-horizon tasks by replacing implicit model-based planning with an explicit, state-machine-based execution harness?

Long-horizon agents often fail to complete tasks because they lose track of mutable state, skip necessary checks, or deviate from plans as their interaction history grows. StateM introduces an agent-native runtime that organizes work into explicit, versioned states with defined entry/exit contracts and transition checks, keeping the agent’s reasoning loop intact while enforcing procedural compliance. This harness scaling approach allows a fixed model to reach 95.3% accuracy on Terminal-Bench 2.1, effectively closing a model-generation-sized performance gap without changing model weights.

Paper Primer

StateM functions as a YAML-configured state machine where each state acts as a context-and-contract boundary. When an agent enters a state, the runtime refreshes phase-local instructions and durable progress, while exit hooks enforce verification before the agent can transition to the next phase.

The system is agent-native: the agent operates the control layer through the same CLI action space it uses for the task, allowing it to inspect its current state, propose runbook changes, and recover from interruptions using the persistent execution record.

Harness scaling achieves performance gains comparable to model-generation upgrades.

GPT-5.6 Sol xhigh with StateM reaches 95.3% raw accuracy on Terminal-Bench 2.1, compared to the 84.9% reference score.

Control profiles transfer across model generations without modification.

A runbook developed with GPT-5.5 raised GPT-5.6 Luna from 76.7% to 85.4% accuracy without any target-model-specific tuning.

Why is this approach better than simply giving the agent a better prompt or a longer context window?

Longer context often leads to control-signal dilution, where the agent loses track of its obligations in a growing trace. StateM externalizes the state, ensuring that phase-relevant instructions and exit conditions are always explicit and enforceable, rather than relying on the model to reconstruct them from history.

Does StateM replace the agent's reasoning capabilities?

No. The agent remains the primary executive with broad autonomy within each state. StateM provides the "guardrails" and "checkpoints" that ensure the agent follows necessary procedures and verifies its work before moving forward.

Reliability in long-horizon agents is an execution-harness problem, not just a model-scaling problem; procedural knowledge can be accumulated and enforced externally through versioned runbooks.

The Long-Horizon Agent Problem

We expose why capable models still fail on long tasks and frame the need for explicit control.

Long‑horizon agents often collapse even when their underlying models can solve each individual step. Implicit model‑based planning dilutes the control signal and leaves mutable state ambiguous, causing agents to lose track of progress, skip required checks, or terminate prematurely. StateM replaces this implicit planning with an explicit state‑machine harness that enforces execution protocols and preserves durable state across long traces.

Agents can execute each local operation correctly yet still fail to complete the overall task because the surrounding control layer loses or misinterprets state.

A benchmark that measures the ability of agents to complete long‑horizon tasks under realistic execution constraints.

Long‑horizon agents fail not because the model lacks capability, but because the surrounding control layer dilutes signals and obscures mutable state.

StateM: Agent-Native Control

StateM Harness: a YAML‑driven state‑machine runtime that makes phase transitions explicit while preserving agent autonomy.

Long‑horizon agents lose a clear execution signal as traces grow, and the model’s internal state becomes ambiguous. StateM tackles this by externalizing control into an explicit, versioned state‑machine that the agent drives via a simple CLI.

StateM replaces the implicit, model‑driven plan with a concrete runbook that defines coarse phases, validates each transition, and records durable progress—so the agent keeps its freedom inside a phase while the system guarantees that moving between phases is checked and logged.

Agent issues

EXECUTE’s `in_hook` loads a placeholder script and presents the prompt “Implement the function”.

Agent runs the implementation, then calls

All tests pass, so StateM records the transition, executes VERIFY’s `out_hook` to archive the artifact, and enters the terminal state.

The example shows how a single CLI command triggers a multi‑step protocol that guarantees entry, validation, and persistence without the model having to remember the entire history.

How does StateM differ from a traditional workflow orchestrator that also defines phases?

Traditional orchestrators usually run outside the model and require the agent to follow a fixed script, losing autonomy inside each phase. StateM keeps the agent in the loop: within a phase the model can invoke tools, edit files, and iterate freely, while only the phase boundaries are enforced by explicit checks and hooks.

Verify that the requested edge from the current state to

Evaluate the current state’s

Run the current state’s persistence or

Evaluate any edge‑level guards or transfer hooks.

Commit the target state and append the transition event to history only if all pre‑commit steps succeed.

Create the target‑state entry and execute its

CLI‑driven transition example

Harness Scaling in Practice

Harness scaling shows how a well‑designed execution harness can boost a fixed model’s performance.

Harness scaling treats agent capability as a joint property of the model weights and the execution system that preserves state, reactivates experience, and checks progress.

It is the systematic boost you obtain when a well‑engineered harness (runtime, runbook, and control profile) extracts more reliable behavior from the same underlying model.

Run the model once: success with probability 0.60, failure with 0.40.

If failure occurs, the harness triggers a retry; the second attempt succeeds with probability 0.60.

Overall success probability = $0.60 + 0.40 \times 0.60 = 0.75$.

The harness converts a single‑shot 60 % chance into a 75 % chance by explicitly handling failure, illustrating how state‑preserving control can amplify raw model ability.

How is harness scaling different from simply scaling up the model?

Model scaling improves the raw predictive power of the network, but the same model may still fail to complete long‑horizon tasks because the execution logic cannot preserve or repair state. Harness scaling adds a separate layer that manages state, retries, and checks, so even a fixed model can achieve higher task success rates.

**Figure 3.** A model-generation-sized harness gain on Terminal-Bench 2.1 GPT-5.5 xhigh with StateM records 92.1%, compared with the 83.1% GPT-5.5 reference. GPT-5.6 Sol xhigh with the frozen StateM profile records 95.28% raw accuracy, compared with the 84.9% Sol xhigh reference. Reference configurations and our runs are visually distinguished; 95.28% is shown as 95.3% at one-decimal precision

Harness scaling is a property of both the model and the execution system, enabling large performance gains without changing model weights.

Cost Frontiers and Model Transfer

StateM shifts the cost–accuracy frontier, achieving high accuracy at a fraction of the original cost.

StateM moves the cost–accuracy frontier dramatically: DeepSeek‑V4‑Flash with StateM achieves 88.76 % accuracy while costing only \$15.20, which is 2.65 % of the \$574.68 cost of the public GPT‑5.6 Sol max submission.

DeepSeek‑V4‑Flash + StateM: 88.76 % accuracy, \$15.20 cost; GPT-5.6 Sol max: 83.37 % raw accuracy, \$574.68 cost.

Cost‑Frontier Transfer means a reusable runbook can lift a cheaper model up the accuracy‑cost curve without extra training.

**Figure 4** Frozen cross-generation transfer with zero target-model runbook changes The profile developed with GPT-5.5 transfers unchanged to GPT-5.6 Sol xhigh, where the reference difference grows from +9.0 to +10.4 points. The evaluated profile is frozen before any GPT-5.6 outcomes are observed

**Figure 5** StateM moves both the quality and cost frontiers on Terminal-Bench 2.1 The orange points are matched score–cost pairs from our runs: GPT-5.6 Sol xhigh + StateM at 95.28% raw and \$1,062.95 submission-reported model cost, and DeepSeek-V4-Flash + StateM at 88.76% descriptive accuracy and \$15.20 realized final-evaluation expenditure. The horizontal 88.76% reference denotes the separately reported GPT-5.6 Sol max score. The public \$574.68 GPT-5.6 Sol max submission is plotted at its matched 83.37% raw score. The upper-left direction is better

**Table 1.** Frozen one-shot BusinessBench results with Codex + GPT-5.6 Luna. The first four rows summarize the six treated families; family rows report the corresponding Round-1 aggregates. Attendance is excluded because no StateM workflow is applied. †The 10.04-point subgroup delta is computed from unrounded source values; the displayed values differ by 10.03 after independent rounding. The 80.56 RefactorBench Round-1 baseline and the later 80.56 StateM development score in Table 2 belong to different runs and splits.

**Table 2.** Post-evaluation diagnostic validation after selective profile refinement. These results use aggregate evaluation feedback and are separated from the untouched one-shot evidence in Table 1. The first three rows are refined StateM aggregate scores; the remaining rows are matched reruns or split-level diagnostics.

Failure Boundaries and Intervention

StateM’s controls close failure gaps, turning zero‑success tasks into full success.

Long‑horizon runs stumble over three failure modes: control‑signal dilution, mutable‑state ambiguity, and the resulting epistemic, compliance, and memory gaps.

When a long sequence mixes high‑level “plan” tokens with many low‑level “execution” tokens, the attention weight on the plan thins out, so the agent loses the guiding signal.

Isn’t the weakening of plan attention just a consequence of longer sequences?

No—length alone isn’t the issue; the soft‑max normalization spreads weight across all tokens. Even a short run can suffer if many low‑level tokens appear before the plan, so the problem is the *distribution* of attention, not raw length.

When a process revisits or branches, multiple versions of a variable coexist, and the agent cannot tell which version is the current live state.

Doesn’t ambiguity simply mean the model is uncertain, which it can handle with probabilities?

Probabilistic uncertainty still assumes a single underlying value. Ambiguity here means *multiple* concrete values exist simultaneously, and the model must choose the correct one—not just assign a confidence score.

**Figure 6** Attention flaw for mixed plan and execution tokens. Plan tokens are at higher abstraction level, which is much fewer but critical. They are diluted by execution tokens in a long run.

Table 3 quantifies the impact of StateM’s controls on Terminal‑Bench 2.1. Every task that previously scored 0/5 or low success with the baseline Codex CLI reaches 5/5 with StateM‑Codex, often via a single added control (e.g., “service/deploy state” or “transition gate”).

**Table 3.** Representative Terminal-Bench 2.1 task-level improvements for GPT-5.5 Each result is the number of successful trials out of five for one task. The listed controls were proposed and implemented by GPT-5.5 Codex under the human-specified high-level design and golden rules. They are selected from visible task semantics and workspace evidence, without using task identifiers, task names, hashes, hidden tests, verifier implementations, or answer artifacts as activation keys

**Figure b.** Task with loops and branches: repeated iterations append many interleaved versions of the same item, making the current state ambiguous.

The figure illustrates a state reconstruction process where specific elements in a sequence are linked to a "Reconstruct state" block. Elements labeled A1, B1, A2, and B2 are highlighted within a row of empty boxes. Solid lines connect A2 and B2 to the "Reconstruct state" block, while dashed lines connect A1 and B1 to the same block, indicating a distinction between active and inactive or historical dependencies.

Beyond the table, the runbook (Section 4.7) acts as a selective procedural memory: each “practice”—a state‑local check, constraint, or activation condition—captures a reusable lesson without bloating the model’s reasoning.

BusinessBench experiments confirm that remembering only the essential controls (e.g., the “transition gate” for primers) yields better generalization than indiscriminately storing every failure trace.

Related Work

We situate StateM among prior work on planning, orchestration, aging, and harness adaptation.

StateM sits at the intersection of four research strands: long‑horizon planning, stateful orchestration, long‑lived memory, and automated harness adaptation.

Planning is the classic response to long‑horizon complexity, but agents frequently drift from their plans, skip required steps, or stop before meeting the original completion criteria.

When plans are expressed only in natural language they act as advisory notes; without a runtime that monitors progress and validates conditions, such plans cannot guarantee task fulfillment.

Stronger orchestration is not a panacea: breaking a procedure into many prompted nodes can fragment reasoning and introduce routing errors, and recent work shows that feeding the full procedure in context can outperform heavily orchestrated variants.

StateM addresses this tension by providing tighter execution control while keeping the primary agent intact, avoiding the need to decompose the task into a cascade of narrowly scoped model calls.

Finite‑state workflows such as StateFlow and durable graph systems like LangGraph already offer explicit states, transitions, checkpointing, and human‑in‑the‑loop recovery.

The key distinction is where the control resides: conventional controller‑led graphs place the workflow definition outside the agent, whereas StateM embeds the control layer inside a general‑purpose CLI agent, allowing the agent to query its own state and request transitions without leaving its tool environment.

CLI agents (e.g., Codex, Claude Code) preserve broad autonomy and expose useful control fragments—planning prompts, TODO lists, rule files, memory files, and hooks—but these fragments are typically soft, natural‑language artifacts rather than authoritative execution states.

StateM unifies these fragments into a shared runbook that records the current state, valid transitions, state‑local instructions, checks, hooks, recovery rules, and execution history, enabling both the agent and a human supervisor to edit the same control artifact.

AgingBench treats deployed agents as time‑evolving systems and identifies four longitudinal failure mechanisms (compression, interference, revision, maintenance aging), providing a diagnostic lens for long‑term reliability.

Its revision‑aging findings—showing that mutable or derived state can fail even with ample text memory—support StateM’s motivation for explicit state representations.

AgingBench interventions (typed‑state overlays, threshold‑triggered controllers) operate at a higher level of lifespan diagnostics, whereas StateM focuses on intra‑run procedural state and transition enforcement.

Recent harness‑adaptation work (Agentic Harness Engineering, Life‑Harness, Self‑Harness, Better Harnesses) demonstrates that runtime harnesses can be optimized from trajectory feedback, but these studies do not claim novelty in the harness itself.

StateM’s contribution is the explicit state‑machine runbook that serves three roles: a runtime control surface, an audit log, and the search space for failure‑driven improvement.

Unlike prompt‑only optimization, external workflow graphs, or longitudinal benchmarks, StateM directly modifies the execution substrate that the agent operates on.

Figure 1’s quadrant positions StateM between soft‑plan autonomy and developer‑authored workflow graphs, targeting the intermediate regime where tasks are open‑ended yet long enough to require enforceable state transitions.

The quadrant does not assert universal superiority; stable, repetitive workflows may favor fixed graphs, while exploratory tasks may need only minimal control.

System Results and Transfer Regimes

Key system-level scores and failure‑mode taxonomy for Terminal‑Bench 2.1.

StateM attains the highest reported score on Terminal‑Bench 2.1, reaching 95.28 % on the GPT‑5.6 Sol profile.

Table 4 shows a 95.28 % raw score for the GPT‑5.6 Sol + StateM (high) entry, surpassing all other listed configurations.

Other configurations achieve scores ranging from 83.1 % to 92.1 %, with varying coverage levels. DeepSeek‑V4‑Flash, even when adapted with StateM, reaches up to 88.76 % under an extended‑timeout condition.

**Figure.** Two prerequisites converge to one result

**Complex task (loop + branch)**

**Table 4.** Terminal-Bench 2.1 system-level results and transfer regimes. Reference rows reproduce the stated comparator result; StateM rows report our runs. The descriptive DeepSeek aggregate replaces only the five gpt2-codegolf trials with the disclosed extended-timeout evaluation.

Failure Modes and Caveats

This section catalogs failure modes and the StateM controls that bound them.

We now enumerate the concrete limits of long‑horizon agents and the StateM mechanisms that enforce safe execution.

This table outlines various abstract failure modes in agentic systems, their corresponding "Generalized StateM control" strategies, "Enforcement and evidence" mechanisms, and "Main applications."

These limits illustrate how StateM’s explicit harness mitigates the ambiguities that cripple implicit planning in long‑horizon agents.

Discussion and Conclusion

We discuss runtime design for harness scaling, summarize experimental findings, and outline future directions.

A harness‑scaling runtime must be cheap to modify, otherwise rapid failure‑driven iteration becomes burdensome. Conventional state‑machine or graph‑based runtimes give explicit orchestration but push much reasoning into manually built nodes and handlers. Agent‑native systems like Codex and Claude Code keep reasoning inside the model but lack persistent, auditable transition semantics, which StateM supplies by exposing states, hooks, and transition conditions to both agents and humans.

Beyond single‑agent control, separating agents by role can provide distinct context views and independent decision boundaries. Role‑specific read/write permissions restrict which artifacts, state, or actions each agent may modify, supporting review, verification, and recovery from self‑reinforcing decisions. We leave these coordination and permission designs to future work, focusing here on single‑agent flows.

StateM enables harness scaling as a capability axis complementary to model scaling, preserving a unified reasoning loop while externalizing mutable execution state. On Terminal‑Bench 2.1, GPT‑5.6 Sol‑high with a frozen runbook from GPT‑5.5 achieves 95.3 % raw accuracy (424/445 successful trials), whereas an adapted DeepSeek‑V4 Flash reaches 88.09 % (full benchmark) and 89.09 % (common core) with evidence costs of about \$15 and a total API spend of \$52.22. Transfer experiments show procedural control is hierarchical: practices transfer unchanged between GPT‑5.5 and GPT‑5.6 but not to DeepSeek, while the failure‑driven development methodology remains reusable across providers.

Our analysis identifies three failure modes for capable models: missing relevant knowledge, unretained lessons, and incomplete active procedures. StateM addresses these gaps through state‑local context, versioned procedural practices, and checked transitions, and BusinessBench shows that frozen aggregate generalization is modest while the biggest gains appear where control is sparse and matched to explicit execution boundaries. Thus, the central scaling problem is not how many rules a harness remembers, but which lessons should persist, where they should intervene, and whether they survive changes in model or task, while model scaling expands what an agent can do.

Runbook Implementation Details

Appendix provides a full runbook example and details the modest compute budget used.

This appendix shares a concrete runbook named “coding‑agent‑loop” that encodes the disciplined sequence of prompts and checks used by the agent.

Runbook definition for the coding‑agent‑loop.

The experiment was run on a personal Codex pro plan using a MacBook Pro 2025 (M4 chip) for development, staying under a \$200 budget with actual spend below \$125; formal submissions employed an AWS m7i.4xlarge instance.

On the personal machine, GPT‑5.5 xhigh achieved roughly a 91 % pass rate on Terminal‑Bench 2.1, demonstrating that high‑quality long‑horizon execution is feasible with modest hardware.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers