Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

Haoyang Yan, Min-le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai, Shuyue Hu

Harness-of-Harness (HoH) enables autonomous software development by organizing coding agents into iterative, evidence-guided planning-coding-testing loops.

How can autonomous coding agents improve software quality over long-running development cycles without human intervention?

Autonomous coding agents often struggle to build complex software from scratch because they lose track of requirements, introduce regressions, or get stuck in repetitive repair cycles over long development trajectories. HoH solves this by wrapping existing coding harnesses in a three-role loop—Planner, Developer, and Tester—that persists both the evolving software artifact and structured execution evidence across iterations. On standard benchmarks, HoH consistently outperforms standalone agents, achieving average relative gains of over 50% and successfully developing a complete, playable game over 70 autonomous iterations.

Paper Primer

HoH organizes development into a continuous cycle where each iteration produces a bounded, verifiable software increment. The core move is the separation of concerns: the Planner selects the next objective based on accumulated evidence, the Developer implements it within the existing codebase, and the Tester independently verifies the result against the specification.

HoH significantly improves software quality across diverse benchmarks compared to standalone agent harnesses.

Across GameCraft-Bench, FrontierSWE, and ProgramBench, HoH@3 consistently outperformed Vanilla baselines, with average relative gains of 52.25%. Maximum gain of 82.86% observed after three iterations.

The framework maintains continuity through two persistent states: the artifact state (the code and resources) and the evidence state (a record of verified behaviors, unresolved failures, and regression risks). This allows the system to "warm-start" each iteration, building on prior progress rather than reconstructing the project from scratch.

Why does HoH require a separate QA Tester role instead of letting the Developer verify its own code?

The Developer has direct knowledge of its own implementation, which can bias its completion claims. Independent QA ensures that acceptance is based on observable evidence from a frozen, read-only candidate, preventing the implementation and verification processes from collapsing into a single, potentially flawed decision.

Does HoH require a new, specialized agent architecture to function?

No. HoH is a framework that operates on existing coding-agent harnesses. It constrains the verifiable outputs and the workflow sequence (Plan-Develop-Test) but leaves the agent's internal reasoning, tool use, and implementation strategy entirely autonomous.

Autonomous Software Development

We define autonomous software development and expose its scaling challenges.

Autonomous coding agents struggle to scale because they lack a mechanism that preserves verified functionality and integrates feedback across many development iterations. As development trajectories grow, agents can lose track of earlier requirements, introduce local fixes that break other parts, and waste effort on redundant verification. Harness-of-Harness (HoH) addresses this by enforcing a cyclical, state‑preserving loop of planning, coding, and testing.

It is the task of turning high‑level requirements into a complete, runnable program without any human guidance.

**Figure 2 | Two different modes of software development.** In human-in-the-loop development, coding agents generate code under continuous human oversight, guidance, review, and intervention. In autonomous software development, agents independently transform high-level requirements into complete, functional, and deployable software systems without human guidance or intervention.

The key shift is moving from one‑shot code generation to an iterative development process that continuously refines and verifies software.

The HoH Framework

Harness-of-Harness orchestrates planning, coding, and testing in a cyclical loop that preserves state and evidence.

When a coding agent iterates over many development cycles, earlier requirements, design decisions, and verified behavior can be lost, causing regressions and wasted effort.

HoH turns a fixed coding‑agent system into a perpetual three‑stage loop—planning, coding, testing—where each stage hands off both the updated artifact and the evidence of what was verified.

Loop 1: Planner reads the empty artifact and creates D₁ = “implement add(x, y) and add a unit test for add(1,2) → 3”.

Developer writes add(x, y) = x + y, producing A₁ containing the new function and the test file.

QA Tester runs the test; it passes, so E₁ records “add(1,2) verified”.

Loop 2: Planner sees that the next requirement is “handle negative numbers”; it emits D₂ = “extend add to support negative inputs and add test add(-1, -2) → -3”.

Developer updates add to handle negatives (no code change needed), producing A₂ (same code, new test).

QA Tester runs both tests; both pass, so E₂ records both verifications.

Even though the code changed only once, the preserved evidence E₁ → E₂ lets the planner know which behavior is already safe, preventing redundant re‑testing of already‑verified cases.

The Planner looks at the specification and the accumulated evidence to decide the next bounded objective, without touching the code.

Why can’t the Project Planner modify the artifact directly instead of delegating to the Developer?

Because the Planner’s role is to maintain a high‑level, project‑wide view; allowing it to edit code would blur the separation of concerns and make it hard to trace which agent introduced which change, undermining reproducibility.

The Developer receives the development document Dₜ and the current artifact Aₜ₋₁, then produces a new artifact Aₜ that implements the requested changes.

What stops the Developer from fixing failures that the QA Tester later discovers?

The Developer works with a read‑only view of the evidence Eₜ₋₁; it cannot see the new failures that arise after its changes until the QA stage, preserving an independent assessment.

The QA Tester evaluates the frozen candidate Aₜ, records what succeeded and what failed, and returns the evidence Eₜ for the next planning step.

Why is read‑only access essential for the QA Tester’s assessment?

Read‑only access guarantees that the evidence Eₜ reflects the exact behavior of the candidate produced by the Developer, preventing the tester from silently fixing bugs and thus contaminating the evaluation.

Read prior artifact Aₜ₋₁ and evidence Eₜ₋₁.

Project Planner generates development document Dₜ.

Developer updates Aₜ₋₁ → Aₜ according to Dₜ.

QA Tester evaluates Aₜ read‑only, producing evidence Eₜ.

Loop repeats with (Aₜ, Eₜ) as the new seed.

Algorithm 1: Harness-of-Harness loop

**Figure 3 | Harness-of-Harness overview.** HoH repeatedly invokes a *Project Planner*, *Developer*, and *QA Tester* around an evolving software artifact. The deterministic Runtime freezes each role's inputs, enforces its permissions, binds evidence to the tested candidate, and records the resulting project state. The model, base harness, role definitions, and runtime policy remain fixed within a run; the development document, software artifact, and execution evidence evolve across iterations.

Benchmark Evaluation

HoH consistently outperforms Vanilla baselines across benchmarks, with detailed results and ablations.

HoH@3 achieves a mean GameCraft-Bench Overall score of 71.52, surpassing the Vanilla baseline by 22.78 points.

Table 1 reports 71.52 for HoH@3 versus 48.74 for Vanilla.

GameCraft-Bench evaluates agents on building complete, playable Godot games from natural‑language specs across 45 sampled tasks.

FrontierSWE measures software‑engineering task performance, reporting mean reward and a dominance win‑rate against random baselines.

**Table 1.** Main results on GameCraft-Bench, FrontierSWE, and ProgramBench. Each harness is evaluated under Vanilla and HoH@1–3. Within each harness and metric, bold values mark the best setting; rankings use unrounded values, and exact ties share the same formatting. Small green values shown only for HoH@3 give absolute gains over Vanilla; Dominance gains are percentage points. Bold italic labels distinguish aggregate metrics from task categories. Strat., Sim., Adv., Impl., and Perf. denote Strategy, Simulation, Adventure, Implementation, and Performance, respectively. FrontierSWE reports category scores and Dominance. For ProgramBench, Pass Rate$^\dagger$ denotes the benchmark's Avg. Test Pass Rate.

**Figure 4.** Vanilla and HoH@3 scores across the four GameCraft-Bench rubric categories. Panels (a)–(c) show results for Codex with GPT-5.5 (high), OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3, respectively. Bars report mean category scores over 45 tasks for Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation; error bars indicate 95% bootstrap confidence intervals.

**Figure 5 | FrontierSWE Dominance over 10 loops.** Shading shows ±1 SE; the star marks the best checkpoint and the dashed line denotes Vanilla baseline.

**Figure 6.** Qualitative comparison of final game artifacts produced by Vanilla and HoH@3 using Codex with GPT-5.5 (high). Columns show three GameCraft-Bench tasks from distinct game families: Momentum Lab (momentum-based platformer), Kitchen Rush (restaurant-management simulation), and Ant Empire (idle colony-management game). Rows show gameplay frames from Vanilla (top) and HoH@3 (bottom). Numbered dashed boxes identify the regions discussed in the annotations; red crosses and green checks denote limitations and implemented functionality, respectively.

**Table 2.** Comparison of HoH and repeated Vanilla development after 1, 2, and 3 development passes on GameCraft-Bench. Results use Codex with GPT-5.5 (high). Score denotes the mean GameCraft-Bench Overall score over 45 tasks, and tokens are mean cumulative coding-harness tokens per task.

**Table 3.** GameCraft-Bench ablation study with Codex and GPT-5.5 (high). Parentheses show score differences from Full HoH@3; tokens are mean cumulative totals per task.

HoH consistently outperforms Vanilla baselines across diverse benchmarks.

Multi-Day Case Study

Long‑term autonomous development of a FPS game demonstrates HoH’s sustained progress.

HoH sustained incremental progress over 70 autonomous loops, closing 65 of 81 recorded issues (≈80% closure).

By Loop 70 the system had closed 65 issues, leaving 16 unresolved, with 17 issues reopened after earlier closure.

**Figure 1.** Across successive iterations by Harness-of-Harness, the resulting First-Person-Shooter game features a coherent storyline, implemented combat, weapon and enemy-interaction systems, player guidance, heads-up display and menu systems, cinematic animation, and polished visual and audio presentation, yielding human-playable experience. The game, development traces, and gameplay videos are available on GitHub.

The case study confirms that HoH can orchestrate long‑horizon software development, preserving functionality and integrating feedback across many iterations, and the authors envision extending it to broader domains.

Execution Protocol Details

Details the executable protocol, role prompts, tools, and evidence handling for Harness‑of‑Harness.

The supplementary methods expand the high‑level HoH description into concrete, executable interfaces for each role and the surrounding tooling.

One HoH iteration composed of three role‑specific harness calls.

| Role | Inputs | Invocation contract | Materialized output | | :--- | :--- | :--- | :--- | | Project Planner | $S$ and $\mathcal{E}_{t-1}$ | Select bounded priorities from public requirements and preceding evidence; identify verified functionality to preserve; specify observable acceptance requirements; do not modify production code | Development document $D_t$ | | Developer | $S$, $D_t$, and the workspace containing $A_{t-1}$ | Address prioritized targets with native coding tools; preserve verified functionality; keep the project buildable and runnable; write changes into the existing workspace | Updated artifact $A_t$ and execution records | | QA Tester | $S$, $D_t$, $A_t$, and public execution records | Derive checkable claims; execute and inspect the artifact; associate findings with observable records; distinguish supported functionality from unresolved or insufficiently evidenced requirements | Evidence bundle $\mathcal{E}_t$ |

Role‑specific responsibilities are as follows: the Planner selects bounded priorities from public requirements and prior evidence, preserves verified functionality, and defines observable acceptance criteria without touching production code; the Developer addresses those priorities using native coding tools, keeps the project buildable, and writes changes into the shared workspace; the QA Tester derives checkable claims, inspects the artifact, and records evidence that distinguishes supported functionality from gaps.

Project Planner prompt template (iteration {{`loop_index`}}).

Developer prompt template (iteration {{`loop_index`}}).

QA Tester prompt template (iteration {{`loop_index`}}).

The table outlines five core capabilities: Project operations, Build and execution, Game interaction, Visual inspection, and Task-specific tools. Each capability is mapped to its representative operations and the corresponding retained records.

Evidence collection follows a formal pipeline: claims $C_t = \text{Claims}(S, D_t)$ are generated from the public specification and development document; each claim $c_i$ is observed in the artifact $A_t$ producing records $r_i = \text{Observe}(A_t, c_i)$; the QA Tester assesses each pair to assign a status $s_i = \text{Assess}(c_i, r_i)$. The evidence bundle $E_t = \{(c_i, r_i, s_i)\}_{c_i\in C_t}$ is then partitioned into verified evidence $E^{\text{ver}}_t$ (where $s_i$ = verified) and gaps $E^{\text{gap}}_t$ (where $s_i$ = gap).

GameCraft Task Scores

HoH@3 lifts the overall GameCraft‑Bench score by over two dozen points.

HoH@3 improves the overall GameCraft‑Bench score by +24.61 points over the Vanilla Baseline.

Overall scores across families show a HoH@3 total of 72.22 versus 47.61 for Vanilla (Codex + GPT‑5.5 high).

Across individual tasks, HoH@3 yields dramatic gains on Racing ($\Delta$ +38.16 on Void Patrol) and Void Harvest ($\Delta$ +38.61), while it hurts performance on Shooter ($\Delta$ ‑13.68 on Ivory Beats) and Roguelike ($\Delta$ ‑10.30 on Thunder Valkyrie). Many other tasks show modest improvements, e.g., Rhythm (+16.59) and Sports (+19.87).

**Table 12.** FrontiersWE tasks excluded from the evaluated subset.

Detailed GameCraft Performance

HoH@3 lifts GameCraft‑Bench scores by a large margin.

HoH@3 improves the average GameCraft‑Bench score by +24.73 points over the Vanilla Baseline.

Average of the $\Delta$ column in Table 16 (Pi + MiniMax‑M3) shows a consistent positive gain across 50 tasks.

The table describes four variants of a system, comparing them across three columns: "Development document", "Coding start", and "Evidence for next plan". The variants are "Full HoH", "w/o Plan Update", "w/o Evidence Feedback", and "w/o Warm-Start".

**Table 14.** Complete GameCraft-Bench task scores for Codex + GPT-5.5 (high). Scores use the benchmark's 0–100 scale; $\Delta$ denotes HoH@3 minus Vanilla.

FrontierSWE Performance

HoH@3 outperforms Vanilla on mean reward across all FrontierSWE model configurations.

HoH@3 reduces the mean reward by $0.45$ points on the Codex + GPT‑5.5 configuration compared to the Vanilla baseline.

Table 17 shows a Vanilla mean of $0.99$ versus a HoH@3 mean of $0.54$.

Across all three model families, HoH@3 consistently beats the Vanilla baseline, with gains ranging from $0.15$ (OpenCode + DeepSeek‑V4‑Pro) to $0.55$ (Pi + MiniMax‑M3). Figure 8 visualizes these category‑level improvements, confirming that the cyclical development loop benefits both implementation and research tasks.

**Figure 8.** FrontierSWE mean rewards by official category and harness-model configuration.

Budget-Controlled Comparison

HoH achieves 71.52 average score, a 21.94‑point gain over Vanilla.

HoH reaches an average score of 71.52, outperforming the Vanilla baseline by 21.94 points.

Table 20 reports average scores of 49.58 for Vanilla, 58.24 for three‑pass Vanilla Continuation, and 71.52 for HoH across 45 GameCraft‑Bench tasks.

HoH consumes 8.41 M tokens per task—3.22 M more than Vanilla—but yields a 21.94‑point score gain, corresponding to 3.77 score points per additional million tokens, versus 2.32 for Vanilla Continuation.

**Figure 9.** Score and cumulative token trajectories in the budget-controlled GameCraft-Bench comparison using Codex with GPT-5.5 (high). HoH includes planning, coding, and testing at each pass.

The table presents performance metrics across various game families and tasks, comparing a "Vanilla" baseline against three "HoH" (Hierarchy of Hierarchies) configurations (HoH@1, HoH@2, HoH@3), with a final column showing the delta ($\Delta$) between the best HoH result and the Vanilla baseline.

**Table 1.** Performance comparison across different game families and tasks, comparing the "Vanilla" baseline against "HoH@1", "HoH@2", and "HoH@3" variants, with the final column showing the delta between "Vanilla" and "HoH@3".

Ablation Study

Ablations reveal each HoH component’s contribution to performance and token efficiency.

We run three HoH variants with $T = 3$ on all 45 GameCraft‑Bench tasks and report the results in Tables 22–24.

Removing plan updates reduces the mean score by 8.13 points.

Mean score falls from 71.52 (full HoH) to 63.39 without plan updates (Table 22).

Removing evidence feedback reduces the mean score by 6.29 points.

Mean score falls from 71.52 (full HoH) to 65.23 without evidence feedback (Table 22).

Removing warm‑start reduces the mean score by 7.85 points.

Mean score falls from 71.52 (full HoH) to 63.67 without warm‑start (Table 22).

**Figure 7** | GameCraft-Bench Overall scores by reporting group and harness-model configuration. Each group contains nine tasks.

Table 24 reports the total tokens and wall‑clock time for each harness‑model configuration on FrontierSWE, illustrating the trade‑off between performance gains and resource consumption.

Qualitative Artifact Analysis

HoH@3 delivers the strongest Overall scores across all GameCraft‑Bench families.

Across all 15 GameCraft‑Bench families, Harness-of-Harness (HoH) consistently beats the Vanilla Baseline, with the third iteration (HoH@3) achieving the highest Overall scores.

Table 25 shows HoH@3 outperforms Vanilla on every game, with the largest gap of +38.5 points.

**Figure 12.** Action: Platformer, Shooter, and Roguelike.

**Table 20.** Task-level scores for the budget-controlled comparison on GameCraft-Bench using Codex + GPT-5.5 (high).

Protocol and Data Records

Details the experimental setup, data recording, and benchmark configurations for the Harness-of-Harness study.

The protocol records two parallel channels each iteration: the artifact channel delivers the updated project $A_t$ to the next Developer, while the evidence channel supplies $E_t$ to the next Planner. Within iteration $t$, $D_t$ is the shared specification, and the subsequent Planner builds a new document from $S$ and $E_t$ rather than persisting $D_t$ as a third state channel.

Each iteration produces a human‑readable tester report and a structured record that feeds the next planning step. Listing 1 (not shown) normalizes these records into verified and gap subsets, preserving the claim, execution evidence, and status for auditability.

**Table 6.** Harness-model configurations used in the experiments.

The main experiments run three HoH iterations ($T=3$). Vanilla performs a single coding pass, while the budget‑controlled study evaluates one, two, or three sequential Vanilla passes. HoH reports only the artifact after the prescribed budget; intermediate artifacts are logged for analysis but never selected by benchmark scores.

The table outlines the experimental setup, including the experiment type, benchmark used, development structure, and the number of runs per task-condition.

Two distinct execution environments are used. GameCraft‑Bench runs in a local subprocess workspace, whereas FrontierSWE launches an official task container inside a Docker‑in‑Docker daemon, enforcing explicit CPU, memory, storage, and GPU limits per task.

**Table 1.** Comparison of execution environments between GameCraft-Bench and FrontierSWE.

Benchmark sampling follows a fixed‑size subset: GameCraft‑Bench contributes 45 tasks across 15 families, while FrontierSWE contributes 15 tasks across 3 categories. Each family supplies three tasks, enabling balanced reporting groups.

**Table 9 |** Composition of the evaluated benchmark subsets. Counts refer to the tasks used for every harness-model configuration in the main experiments.

For GameCraft‑Bench, the 45‑task subset is stratified by seed 20260707, ensuring three tasks per family without reference to model scores. Families are further grouped into five coarse categories (Action, Timing, Strategy, Simulation, Adventure) for aggregate analysis.

**Table 10.** GameCraft-Bench reporting groups, benchmark families, and sampled tasks. Each family contributes three tasks.

Benchmark Task Taxonomy

Details of FrontierSWE task taxonomy, baseline protocols, ablations, metrics, and reproducibility artifacts.

FrontierSWE comprises 15 tasks organized into three thematic families—Horror, Open World, and Visual Novel—each further classified by construction scope: Implementation, Performance, or Research. Tables 11–13 enumerate the full task set, the excluded tasks, and the information channels retained by the ablation variants.

Tasks that require building a horror‑themed game environment.

Tasks that involve large‑scale, open‑world game construction.

Tasks that require scripting narrative‑driven visual‑novel experiences.

Tasks that require building a complete, end‑to‑end deliverable from scratch.

Tasks that focus on optimizing an existing system without rebuilding it.

Tasks that explore novel research directions rather than delivering a concrete software artifact.

Baseline protocols use the “Vanilla” harness‑model configuration without the Harness‑of‑Harness (HoH) protocol, performing a single development pass. “Vanilla Continuation” repeats the same harness‑model configuration for two additional passes, using a fixed continuation prompt but omitting planning and evidence feedback.

Ablation variants keep the three‑iteration budget of full HoH while disabling one cross‑iteration input at a time: w/o Plan Update fixes the development document after the first iteration; w/o Evidence Feedback withholds QA evidence from the next planner; w/o Warm‑Start always starts from the initial artifact instead of the previous one.

Metrics include the GameCraft‑Bench overall score (a weighted sum of four rubric dimensions) and the FrontierSWE official reward. Token usage is measured in millions of tokens per condition, and the efficiency gain $\eta$(c) compares quality improvement to additional token consumption relative to Vanilla.

Player‑Experience evaluation uses the source‑blinded Fusepoint playtest with the full ten‑construct Player Experience Inventory (PXI). Scores are averaged per construct; no global PXI total is reported.

Reproducibility artifacts comprise the open‑source HoH implementation, role‑prompt templates, and adapters, while omitting private credentials, provider secrets, and benchmark‑hidden evaluator contents.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers