HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, Wenxuan Zhang

HarnessDev evaluates whether LLMs can build and evolve the execution infrastructure they run inside.

Can LLMs autonomously design and iteratively improve the execution infrastructure (harness) required to solve complex software engineering tasks?

Agent performance depends heavily on the "harness"—the software loop, tool use, and state management that turns model outputs into actions. Current benchmarks treat this harness as a fixed experimental setting, ignoring the model's ability to design its own execution substrate. HarnessDev shifts the evaluation unit from task outputs to the harness itself. Models start from a minimal, non-functional seed and must build a complete, reusable execution system (Creation) and iteratively improve it using downstream feedback (Evolution). Generated harnesses currently trail human-engineered systems in code and research tasks, though they match or exceed them in writing and machine-learning experimentation. Evolution produces local performance gains, but these are often unstable and fail to generalize across different runtime models.

Paper Primer

The benchmark isolates harness development by separating the creator model (which writes the code) from the executor model (which runs the tasks). The core move is to freeze the generated harness as a persistent artifact, then evaluate it on held-out tasks to measure both capability and execution-token efficiency.

Generated harnesses show high variability in quality and efficiency, with no direct correlation between execution cost and task success.

Across MLE-bench, token usage varies by approximately nineteen-fold between different model-built harnesses while achieving similar performance. Top-performing models like Opus 4.8 achieve a 67.8 average score, remaining below the 86.2 human-engineered reference.

Evolution is treated as a local search process where models use feedback from a fixed set of tasks to refine their harness. While models can successfully diagnose and patch specific failure modes—such as adding completion checks to prevent premature termination—these improvements often overfit to the specific executor model used during development.

Why is harness engineering fundamentally different from standard code editing?

When a model edits its own harness, it modifies the execution substrate through which it observes and plans, meaning a single change alters its behavior across all future tasks rather than just a local, verifiable output.

What does it mean for a harness to be "frozen" in this benchmark?

The harness is treated as a static, runnable artifact that is evaluated on held-out tasks after development concludes, ensuring that performance gains reflect durable capability rather than temporary, task-specific overfitting.

Introduction to HarnessDev

We expose the missing ability of agents to build and evolve their own execution harnesses.

Agent performance now hinges on the surrounding execution infrastructure, yet current benchmarks treat that infrastructure as a fixed backdrop. This leaves a blind spot: can models themselves construct and continuously improve the harness that powers them?

HarnessDev measures a model’s ability to create and evolve a runnable execution harness instead of relying on a pre‑built one.

How does HarnessDev differ from traditional agent benchmarks?

Traditional benchmarks fix the execution harness and only measure downstream task performance, whereas HarnessDev treats the harness itself as the target of evaluation, requiring the model to construct a runnable system and then assess its capability and efficiency on unseen tasks.

**Figure 1.** HarnessDev covers two stages of harness development. In Creation, a creator builds a complete harness from a weak but runnable seed and a small number of development cases. In Evolution, it continues to improve its own persistent harness using downstream execution feedback. Both stages evaluate runnable infrastructure that persists across tasks rather than a one-off task output.

The key shift is moving from a static execution environment to a model‑built harness that can be created and continuously improved.

Benchmark Design and Scope

We outline deployment challenges and present HarnessDev, which tests a model’s ability to build and evolve execution harnesses.

Most agent benchmarks assume the execution scaffold is already in place, hiding the engineering effort required for real‑world deployment. In industry this work is performed by forward‑deployed engineers (FDEs), who adapt general‑purpose models to customer‑specific data formats, workflows, and compliance constraints. The success of an FDE is measured by sustained usage and reliability, not by a single benchmark score.

From the model’s side, three pieces of structure are typically assumed: (1) the target is vague, requiring translation of business intent into concrete objectives; (2) the feedback signal is absent or unreliable, so self‑evaluation mechanisms must be built; and (3) the execution system itself does not exist in a usable form and must be constructed, adapted, and maintained.

This paper concentrates on the third piece—the execution scaffold—by introducing HarnessDev, which asks language models to build and evolve that scaffold from a minimal seed.

SWE‑bench Pro is a suite of software‑engineering tasks that measures how well a model can create, modify, and run execution harnesses for real‑world workloads.

In HarnessDev, the submitted artifact is a frozen, runnable harness that is reused across downstream tasks. A creator LLM (LC) works inside a development environment (D) to produce the harness (H); later an executor LLM (LE) runs inside H on a downstream task (x), and an evaluator (J) scores the output (y).

RQ1 (Creation) asks whether a model can build an effective harness from a weak but runnable seed. RQ2 (Evolution) asks whether a model can improve an existing harness while preserving previously working behavior.

The Seed Harness Environment

Defines the weak seed harness and the creation process for building functional harnesses.

The core challenge is to isolate the ability to construct an execution harness from the ability to solve the downstream tasks themselves. By providing a minimal “seed” that can run but does nothing useful, we force the creator model to add the missing control logic.

The Seed Harness is a runnable compatibility layer that parses inputs and exposes passive primitives, but it contains no decision‑making or task‑solving logic.

Step 1:

Step 2: The harness writes the returned string to the audit log via

Step 3: No further actions are taken; the output artifact is just the log entry.

The seed can move data but cannot transform it or decide to invoke additional tools, so the resulting artifact is always a direct echo of the input.

How does this seed differ from a full autonomous agent?

Unlike an agent, the seed has no internal loop that can iteratively plan, decompose, or retry; it merely forwards data through a fixed set of passive primitives, so any adaptation must be added externally by the creator.

Harness Creation asks the creator model to augment the Seed Harness with control logic so that it can solve real tasks, using only a handful of public development cases.

Step 2: The added loop iterates over the lines, reverses each (“olleh”, “dlrow”).

Step 3:

By inserting a control loop, the harness now transforms input data, turning a zero‑score seed into a functional solution.

Why does the creator never see the hidden evaluation set during development?

The hidden set is kept private to prevent the creator from overfitting to the test distribution; only the public development cases are visible, so the final frozen harness must generalize to unseen tasks.

Evolution (RQ2) builds on the frozen harness $H_0$ from Creation. The creator receives results on a fixed 100‑task SWE‑Pro feedback set and all 89 Terminal‑Bench tasks, then iteratively refines $H$ under a budget of ten full‑evaluation pairs.

Between two charged evaluation pairs the creator may run at most two diagnostic probes covering the first five tasks of each benchmark. Probes are never counted toward the official score; only complete pairs of 100‑task SWE‑Pro and 89‑task Terminal‑Bench evaluations determine progression.

**Figure 2.** The weak Seed Harness and its development environment. The seed fixes only the input and audit envelopes and exposes passive, unorchestrated primitives; unmodified, it performs no task work. Development feedback is public; hidden tasks, answers, and official scores remain inaccessible.

Table 2 summarizes the downstream evaluation coverage for Creation: the SWE‑bench Pro public split (731 tasks), Terminal‑Bench (89 tasks), MLE‑bench (75 tasks), EQ‑Bench3 (46 tasks), and BrowseComp (1,266 tasks), together spanning four domains and 2,207 unique instances.

Harness Creation Performance

Evaluates how well creators build functional harnesses from minimal seeds.

Human‑engineered harnesses still outperform the best creator by 17.4 points.

Human reference scores 86.2 vs. best model 68.8.

**Figure 3.** From control layer to scorable artifacts. The creator must implement the six control modules (E/T/C/S/L/V, colored as in Figure 1). A finished harness reports through unified auditable outputs and delivers a domain-specific, scorer-readable final artifact.

**Table 3. RQ1 / Harness Creation under Self-Eval.** Unless noted otherwise, each creator independently builds and evaluates three harnesses and the score is avg@3. Each benchmark uses its native metric, tok. is the mean execution tokens per harness in millions, and Avg. is the unweighted mean over SWE-Pro, Terminal-Bench, EQ-Bench3, and BrowseComp. MLE-bench covers 33 physical cells and 2,475 results. The human row is a system-level reference; * marks external results that we did not re-run in this experiment, and — marks unavailable entries.

Implementation and Architecture Analysis

Assess how each creator’s harness construction impacts overall performance.

We ask whether removing any component of the creator’s harness construction hurts performance, reporting the resulting drop in the average score.

**Figure 5** Harness architecture evidence across fields and stages. The six columns are execution loop, tool policy, context management, state and memory, lifecycle and recovery, and result verification. Each field is first normalized by the number of independent harnesses actually available and then aggregated over RQ1 Code/Data/Writing/Search and RQ2 Main; Seed has no RQ2 trajectory and is therefore built only from the four RQ1 fields. Full weight requires a mechanism to enter the main path or to trigger during a formal run, while declared code, configuration, or transient state receives only partial weight. Color shows mechanism evidence density, not score, significance, or causal effect.

**Table 4.** RQ1 / Harness Creation under a fixed Gemini executor. Every creator harness is executed by Gemini 3.1 Pro. The Gemini row reuses the SELF-EVAL control and is not counted as a new physical cell. Every score entry is avg@3. Code cells marked ‡ contain one collapsed R3 replica; dropping it gives post-hoc clean sensitivity means of 49.1 for Opus on SWE-Pro and 43.8/57.3 for DeepSeek on SWE-Pro/Terminal-Bench. GPT-5.5's EQ-Bench3 cell is avg@3 (46.5); excluding its zero-valued first harness the mean is 69.7. All other definitions follow Table 3.

**Table 5. Edit size of the frozen RQ1 Code artifacts.** File counts and total net LOC are summed over each creator's three independently created artifacts; the median and range are per artifact. The statistics cover only harness/, which is the creator's responsibility, and exclude the runtime substrate injected by the runner. Performance is SELF-EVAL avg@3.

Evaluation Methodology

We expose the Self‑Eval vs Unified‑Eval trick and why it matters before evolution.

Benchmarks normally hand a ready‑made execution harness, yet real‑world autonomy demands that agents construct and evolve their own infrastructure. This mismatch creates a pain point: without a self‑contained harness, an agent cannot be deployed beyond the benchmark sandbox.

Self‑Eval lets a creator model grade its own harness output, whereas Unified‑Eval uses a separate, fixed evaluator to score the harness.

Compute the average for Task A: (0.6 + 0.8 + 0.7) / 3 = 0.70.

Compute the average for Task B: (0.5 + 0.6 + 0.55) / 3 ≈ 0.55.

Self‑Eval reports the higher of the two averages, 0.70, as the harness’s score.

Unified‑Eval runs the same three attempts through a fixed evaluator that yields (0.55, 0.58, 0.60) for Task A and (0.48, 0.52, 0.50) for Task B.

Unified‑Eval averages these to 0.543, which is lower than the Self‑Eval score, revealing a bias.

Self‑Eval can overstate performance when the creator’s own inference loop is tuned to its own outputs, while Unified‑Eval exposes that gap.

The six creators—Opus, GPT‑5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro—adopt distinct implementation strategies, ranging from rewriting the execution stack to extending the seed with modular toolkits.

Table 5 shows that edit size (net lines of code) varies widely; Gemini adds only 1,006 LOC yet achieves the highest Terminal‑Bench score (68.8), indicating that focused changes matter more than sheer code volume.

Across the 18 Code harnesses, 108 component instances appear, but only 72 ever trigger in real runs; the remaining 36 are dead code, most of which involve state‑handling mechanisms that never execute.

Self‑test count correlates weakly with downstream performance (Spearman 0.13–0.26), whereas revision‑call frequency shows a strong correlation (0.57, p ≤ .0005), suggesting that targeted revisions are a better predictor of success.

When transferred to the Gemini executor, Qwen and DeepSeek harnesses improve (e.g., Qwen gains 17.6 points on BrowseComp), while Opus suffers a steep drop (Self‑Eval SWE‑Pro falls from 69.3 to 33.0) and its duplicate‑query rate jumps from 10.1 % to 88.2 %.

How does Self‑Eval differ from Unified‑Eval in practice?

Self‑Eval reuses the creator model to score its own harness, so the evaluation loop is identical to generation and can mask deficiencies. Unified‑Eval invokes a separate, fixed evaluator, breaking that loop and revealing whether the harness truly generalizes across executors.

Harness Evolution Results

Evolution yields modest held‑out gains, highlighted by a +4.44‑point improvement.

Recall that HarnessDev tests whether a creator can build a functional harness from a minimal seed, reflecting real‑world autonomy where agents must evolve their own infrastructure.

Harness evolution yields modest but measurable held‑out gains, with the largest observed improvement of +4.44 points.

Table 6 shows Opus 4.8 attaining a +4.44‑point held‑out increase after evolution.

RQ2 asks whether a creator can improve its RQ1 Code harness using downstream execution feedback. The feedback set consists of tasks repeatedly evaluated during evolution, while the held‑out set contains tasks evaluated only after evolution, never shown to the creator. We report the average of SWE‑Pro‑100 and Terminal‑Bench‑89 percentages as $\overline{P}_t = \tfrac12(P_{\text{SWE100}}^t + P_{\text{Term89}}^t)$ for each frozen version $t$.

**Figure 6** Harness portability under a fixed Gemini executor. Filled markers are SELF-EVAL, hollow markers are the fixed Gemini result for the same creator harness, labels are UNIFIED-EVAL minus SELF-EVAL, and dashed lines are external human-engineered system references. All panels use the avg@3 values of Table 3 and Table 4; the clean sensitivity means for the two collapsed code cells are given in the caption of Table 4.

**Figure 7.** Feedback-set evolution trajectories on SWE-Pro and Terminal-Bench. The top row is self runtime and the bottom row is the fixed Gemini runtime; the left column is SWE-Pro-100 and the right column is Terminal-Bench-89 $H_0$ is the frozen RQ1 harness, $H_i$ are later frozen commits that completed a formal evaluation on both benchmarks, and stars mark the version declared by the creator. The Gemini control is reused in both settings, and each creator-runtime cell has a single trajectory.

**Figure 8.** SWE-Pro-100 feedback trajectories and post-freeze held-out-630 performance. Each column isolates one creator model and overlays its visible 100-task feedback trajectory with its post-freeze 630-task held-out trajectory. The top row is self runtime and the bottom row is fixed Gemini. Stars mark the final versions declared from 100-task feedback. Faint vertical segments connect matched harness versions. The Gemini control is repeated only for visual comparison.

Discussion and Conclusion

We synthesize findings, acknowledge limits, and outline the broader impact of HarnessDev.

Across the nine RQ2 lineages, most models manage to close the full loop—reading results, editing the harness, re‑evaluating, and selecting a new version—yet the diagnostic step remains fragile, inspecting only a tiny fraction of feedback tasks.

**Table 7. Code edit size of the nine RQ2 lineages.** "Official switches" counts only adjacent official versions; the cumulative diff compares $H_0$ with the version declared by the creator for each lineage. The edit focus describes behavior observed in that single trajectory and is not a general property of the corresponding creator model.

Opus stands out by tracing a success‑rate gap (99 % reported success vs. 48 % passing) to premature completion and inserting a completion gate, illustrating how concrete feedback can drive a useful fix.

Stability analysis of the 64 official switches reveals a noisy landscape: eight regress on both benchmarks, sixteen regress on a single benchmark, and only two switches show clear positive gains beyond the noise band.

Code reachability statistics show that of the 169 new functions or classes introduced, 113 are reachable from the entry point, 31 exist only in dead‑code paths, and 25 have no callers, underscoring that added code is not automatically active.

Feedback scores and held‑out scores align only 34 times out of 64 comparable switches (53.1 %), and merely two of the nine declared versions turn out to be optimal on the held‑out split, indicating that local feedback is an unreliable guide for final selection.

**Figure 9. Cost vs. performance (RQ1).** Downstream score against execution tokens (log scale) under SELF-EVAL (filled circles) and UNIFIED-EVAL (open squares; fixed executor Gemini 3.1 Pro); dashed lines mark the human-engineered reference where measured. All points are the avg@3 entries of Table 3 and Table 4; Gemini is the fixed-executor control, so its two markers coincide and appear as a dot inside a square. Similar quality can differ by close to an order of magnitude in cost—on MLE-bench, GPT-5.5 reaches a medal rate of 19.1 with 29.3M tokens while DeepSeek V4 reaches 19.6 with 208.4M. We report this performance-token trade-off directly rather than collapsing it into a single cost-adjusted score (Section 3.5).

**Figure 10** RQ2 executor-token cost across frozen harness versions. The top row uses self runtime and the bottom row uses the fixed Gemini runtime; columns show SWE-Pro-100, Terminal-Bench 2.1, and the post-freeze SWE-Pro held-out-630 split. Each point sums the observed task-agent/runtime `total_tokens` for one frozen harness and benchmark leg. Creator, judge, and probe tokens are excluded.

Limitations include incomplete benchmark coverage, a single trajectory per creator‑runtime cell, and reliance on a fixed development environment $D$, which prevents assessing whether an evolved harness can serve as a new development platform.

In conclusion, HarnessDev reframes agent evaluation from fixed‑system task solving to the ability to construct and evolve execution harnesses, exposing a new measurable dimension of intelligence beyond model weights.

The ethics statement notes that all harnesses are built from public benchmarks and open‑source code, with explicit constraints preventing insecure tool use; audit artifacts are released for reproducibility.

Contributions: core contributors are Yuhao Wu, Jingyuan Zhang, and Jiajun Shi; additional contributors and corresponding authors are listed in the full author list.

Appendix: Candidate Systems

Provides candidate harness listings, experimental setup details, and the full harness interface specification.

The benchmark draws candidate systems from four functional categories. Here, a Creator is the LLM that generates the harness code, and a Harness is the executable environment the agent uses. The categories are code agents (Claude Code, OpenCode, OpenHands, SWE‑agent, mini‑SWE‑agent), notebook/data‑analysis agents (DataAgent, DB‑GPT), writing agents (AutoResearchClaw, webnovel‑writer), and research/retrieval agents (Alibaba‑NLP/DeepResearch, dzhng/deep‑research, modelscope/ms‑agent, gpt‑researcher).

**Table 8.** Creator-LLM and development-environment configuration. *default* means that the provider does not expose or we do not override the value; — means not applicable. Max output is measured in tokens. The Opus 4.8 value is the measured endpoint cap.

Downstream runs for data‑analysis benchmarks (MLE‑bench) execute in isolated containers on a single NVIDIA A800‑SXM4‑80GB GPU, 14 vCPUs, and 227 GiB RAM, with a wall‑clock limit of 36 000 s split into 34 200 s for the generated harness and 1 800 s for the fixed grader, and a hard 500‑step cap.

Table 9 records the human‑engineered harness‑executor pairs used as references for each downstream benchmark, e.g., the SWE‑Pro benchmark pairs the public coding‑agent setup with Claude Code 2.1.177, while Terminal‑Bench 2.1 and BrowseComp both use the GPT‑5.6 Sol executor.

**Table 9.** Human-engineered reference systems. Each row records the harness-executor pair associated with the public system-level reference used in Table 3.

On MLE‑bench, GPT‑5.5 achieves a median rate of 19.1 while consuming 29.3 M tokens, whereas DeepSeek V4 reaches 19.6 but requires 208.4 M tokens, illustrating an order‑of‑magnitude cost difference for comparable quality.

RQ2 score trajectories are complemented by executor‑side usage statistics for each frozen harness version, keeping task‑agent runtime separate from the token consumption of the creator, judge, and diagnostic probes.

The harness interface defines six modules—execution, tools, context, state, lifecycle, and evaluation—each exposing a small set of JSON‑serializable functions such as run, register, call, save, beforeAction, and evaluate.

The seed harness skeleton provides a minimal runnable foundation, with a directory layout including workspace, dev, runner, harness, `entry_module`, `seed_runner`, `audit_contract`, and primitives, but contains no task‑solving policy.

Appendix: Implementation Details

Evaluation settings, model roles, and the full harness specification for the meta‑evaluation.

This appendix formalizes the evaluation regimes, clarifies the distinct model roles, and spells out the complete harness specification that all downstream experiments rely on.

Self‑Eval measures whether a creator can build a harness that works for its own language model, while Unified‑Eval tests the same harness under a fixed external executor to isolate harness quality.

The three roles separate responsibilities: the Creator (LC) designs the harness, the Developer (D) supplies file‑level operations, and the Executor (LE) runs downstream tasks after the harness is frozen.

A harness is the execution system surrounding the runtime LLM; it must implement six functional blocks.

The image displays a dashboard-style summary of a software engineering evaluation process. It includes four metric cards at the top: - **51.0 / 67.416** (H0 · SWE / Terminal) - **T2 · 56.0 / 74.157** (final selection) - **+5.87 pp** (pair gain) - **7** (complete evaluation loops) Below these cards is a task description: **01 Rewrite the architecture before complete H0 was available**, followed by metadata: **H0 → T1 | 0b33583 · core 5038dd0 | edit first | 4 files | +426 / -48**.

The image displays a dashboard-style summary of software engineering evaluation metrics. The top row contains four cards: - "68.0 / 74.157" labeled "H0 · SWE / Terminal" - "T3 · 74.0 / 74.157" labeled "final selection" - "+3.00 pp" labeled "pair gain" - "3" labeled "complete evaluation loops" Below this, a header reads: "**01 Build failure categories from 189 cases before a structural repair**", followed by metadata: "H0 → T1 | 9d6f78e | diagnosis-first | 8 files | +304 / -18".

**Figure.** Downstream score as % of the human-engineered reference (dashed = reference; labels = raw score)

**Figure.** Complete self-test ledger. Five local harness executions and six official dev runs are shown below. Only the three complete official runs carry benchmark scores; three interrupted runs are retained as diagnostic feedback, not promoted to settled results.

The image displays four distinct metric cards, each containing a numerical value and a descriptive label: 1. **20**: seed reads before first edit 2. **10**: harness modules written upfront 3. **3**: valid official self-tests 4. **+1252 / -596**: seed-to-final harness diff

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers