Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Jiayu Yao, Jiafeng Guo, Xueqi Cheng
Doc2Harness evaluates LLM harness engineering by tasking models to build infrastructure from specifications and testing it deterministically.
How effectively do current LLM-driven coding agents perform when tasked with building the complex "harnesses" required to manage long-horizon software development tasks?
Coding agents rely on harnesses to manage tool execution and session state, but existing benchmarks measure agent performance on fixed infrastructure rather than the quality of the harness itself. Doc2Harness tasks a model with constructing a complete harness from a specification document and an interface-only skeleton, then drives the resulting candidate as a black box using a scripted mock model to ensure deterministic evaluation. Even the strongest model, GPT-6 Astra, fails nearly half of the test cases, revealing that harness engineering remains a significant bottleneck for agent capability.
Paper Primer
The benchmark operates in two stages: a construction phase where the model builds the harness inside a fixed environment, and an evaluation phase where the candidate harness is executed in isolated containers. The core mechanism is a scripted mock model that drives the harness through preset sessions, allowing the system to observe four channels—requests, disk files, exit codes, and command output—to verify behavior against the specification.
The benchmark effectively discriminates between model capabilities in harness construction.
Evaluation of nineteen frontier models shows a wide performance spread. The strongest model (GPT-6 Astra) achieves a 53.05% pass rate, while the weakest reaches 27.44%, a 26-point gap.
To prevent "vacuous passes"—where a test case passes simply because a feature was never triggered—the authors implement positive controls. These controls verify that the observation preconditions are met (e.g., confirming a file is readable before asserting that an out-of-bounds read was correctly blocked), ensuring that a pass reflects actual implementation correctness.
Why is this benchmark needed if we already have software engineering benchmarks like SWE-bench?
Existing benchmarks judge whether a model can resolve a specific issue in a mature codebase, whereas Doc2Harness evaluates the ability to build the infrastructure (the harness) itself. Harnesses are updated frequently and lack the stable, mature repositories found in other benchmarks, requiring models to generalize from specifications rather than relying on memorized code.
What is the primary architectural factor influencing success in this benchmark?
The authors observe that fragmentation is costly: candidate harnesses with more modules tend to score lower, while those that consolidate cross-cutting functionality into fewer, more cohesive modules perform better.
Doc2Harness shifts the focus from agent performance to infrastructure quality, providing a deterministic metric for how well models can engineer the very systems that govern their own behavior.
Introduction: The Harness Bottleneck
We expose the missing evaluation of harness engineering and introduce Doc2Harness to measure it.
LLM‑driven agents are increasingly deployed on long‑horizon tasks, but their success hinges on the surrounding harness that mediates tool use, state, and error recovery. Current benchmarks freeze this harness, leaving the engineering of the harness itself unmeasured.
A Candidate Harness is the concrete implementation a model builds to expose interfaces, manage tool execution, and maintain session state, ready to be evaluated as a black‑box component.
Doc2Harness implements a two‑stage protocol: Stage 1 asks a build model, running inside a fixed Builder Harness, to synthesize a Candidate Harness from a merged development document and a contract‑only skeleton. Stage 2 then executes each Candidate Harness in its own container, feeding it a scripted mock model and checking four channels—model requests, on‑disk files, exit code, and command output—while positive controls guard against vacuous passes.
The field must shift from evaluating only model outputs to assessing the quality of the surrounding harness.
The Doc2Harness Framework
Defines the documents, skeleton, and harness components that feed the benchmark.
Task Inputs specify the textual specification, the contract skeleton, and the fixed builder harness that together define what the benchmark evaluates.
The benchmark measures how well a language model can synthesize a functional software harness from a specification document.
How does Doc2Harness differ from a standard code‑generation benchmark?
Standard benchmarks score raw generation quality (e.g., BLEU), while Doc2Harness scores the end‑to‑end functional correctness of a complete harness built from a specification, isolating the engineering step.
The Builder Harness is a fixed, open‑source coding‑agent framework that the model uses as a scaffold to assemble the Candidate Harness.
Why not replace the Builder Harness with a model‑specific tool like Claude Code?
Model‑specific tools are co‑developed with their own models, giving those models an unfair advantage; a shared open‑source harness ensures a level playing field.
The Development Document is a three‑step specification that enumerates features, turns them into concrete clauses, and resolves ambiguities.
During disambiguation the system selects the canonical name “/compact” and maps both original commands to it.
Any clause that referenced “/summarize” is rewritten to “/compact”, eliminating the semantic clash.
The resulting Development Document contains a single clause “/compact” with a unified definition, ensuring consistency across the merged feature set.
This merging prevents duplicate semantics from inflating the clause count and guarantees that the candidate harness implements a single, unambiguous command.
The Contract Skeleton is a stripped‑down mini‑swe‑agent that supplies the immutable interfaces required by the benchmark.
Copy the Contract Skeleton and the Development Document into a new isolated container.
The Builder (powered by OpenCode) reads the Development Document and, using the skeleton’s interfaces, assembles a Candidate Harness.
When the Builder believes the harness is complete, it issues a Submission signal.
The Contract check validates compilation, presence of required commands, absence of placeholders, and existence of the three contract interfaces.
If the check passes, the Candidate Harness is frozen; otherwise the Builder revises the code and resubmits (quota limited).
During evaluation, a deterministic Mock model runs each test case inside its own container, optionally gating with Positive Controls.
Pass/fail outcomes are recorded; the overall pass rate serves as the performance metric for the build model.
This flowchart illustrates the development and verification process for a software contract. It begins with a "Development Document" and "Contract Skeleton" as inputs, which feed into a "Builder" component containing a "Build Model" and "Builder Harness (OpenCode)". The output of the builder is processed by a "Submission Harness" and then subjected to a "Contract Check" (represented by a diamond shape). If the check passes, the process proceeds to a "Candidate Harness"; if it fails, the process loops back to the "Builder Harness" for revision.
Synthesizing Test Cases
Describes how test cases are built, varied, and measured to probe harness implementations.
Test case synthesis creates concrete scenarios that are tied to documentation clauses and that can tell apart different harness implementations.
It is a systematic way to generate, filter, and mutate test cases so that each case is anchored to the specification and can reveal a concrete difference between a candidate harness and a reference implementation.
How does this test‑case synthesis differ from ordinary fuzz testing?
Fuzz testing generates random inputs without regard to the specification, whereas test‑case synthesis deliberately ties each scenario to a documented clause, clusters and filters real defect reports, and varies only one controlled variable at a time to ensure that the resulting case isolates a specific implementation difference.
Tool variation: replace
Input‑form variation: change the symlink target from a directory to a regular file (
Erroneous‑parameter variation: set the parameter
Result‑site variation: capture the error message on standard error instead of standard output.
Each variation isolates a single factor; only the tool‑change variant still triggers the rejection, confirming that the candidate harness’s bug lies in its handling of the specific tool rather than in generic path validation.
**Figure 2.** Overview of Doc2Harness. Top (Stage 1): the builder, formed from the build model under evaluation and a fixed builder harness (OpenCode), takes the development document and the contract skeleton as input and emits a submission; the contract check verifies form rather than behavior, freezing the deliverable as the candidate harness on success and returning it to the builder on failure. Bottom (Stage 2): the frozen candidate harness and a mock model form the driven harness, which runs the scenario of each test case; effects are read from four channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output, and the positive control gates the status before the assertion decides PASS or FAIL.
Validating with Positive Controls
Positive controls ensure test passes reflect true capability rather than vacuous behavior.
A positive control is a companion test that confirms the effect a test case checks is actually observable, preventing vacuous passes.
Fixture: external file “/tmp/secret.txt” = “SECRET”; workspace symlink “/workspace/link.txt” → “/tmp/secret.txt”.
Run candidate harness with the read tool on “link.txt”; the model receives no “SECRET” content, so the test passes.
Run the positive control: invoke the bash tool on the same symlink, then read the file directly; the control asserts that “SECRET” is indeed readable.
The positive control shows that the read tool is capable of reaching the external file; therefore the original test’s pass indicates that the harness correctly blocked the out‑of‑bounds read rather than being non‑functional.
Channel control checks whether the tool’s result ever reaches the observed channel. If the implementation never places file content onto that channel, the absence of forbidden content is meaningless, so we read a normal file with the tool and verify its content appears in the model’s requests.
Same‑form control rules out failures that would happen even without the constraint. By accessing the same symlink through the bash tool—outside the file‑tool scope—we confirm that the implementation can perform the access when unconstrained.
Reverse control rules out blanket refusal of the input type. We replace the symlink with one that points inside the workspace and verify that its content still reaches the model, proving the implementation does not reject all symlinks outright.
How does a positive control differ from a regular test case that simply asserts a condition?
A regular test case only checks that a condition holds; a positive control deliberately changes the input or access path to demonstrate that the condition is observable, thereby confirming that a pass is not due to a non‑functional tool or missing output.
Benchmark Statistics and Coverage
Key changes in the evaluation pipeline and test suite composition.
The evaluation now follows a four‑stage pipeline (Collection → Ranking → Clustering → Filtering).
Figure 4 visualizes the four stages and their sequential flow.
Both stages run on the same AMD EPYC 7K83 server (48 logical processors, 128 GB RAM). The suite is split into nine disjoint categories; 130 of the 328 cases are gated by at least one positive control (114 by one, 15 by two, 1 by three).
The figure illustrates a four-stage pipeline process: Collection, Ranking, Clustering, and Filtering. Each stage is enclosed in a dashed box and connected by arrows indicating a sequential flow. The "Collection" stage involves pulling issues from public repositories. The "Ranking" stage ranks these issues by relevance to a development document. The "Clustering" stage groups records reporting the same phenomenon into a single candidate. Finally, the "Filtering" stage drops candidates that overlap or resist deterministic assertion.
**Figure 4.** File operations and permissions have the highest positive-control coverage, whereas the interactive terminal has the lowest. Artifact checks is marked NA, because the contract check already settles in full what its test cases examine, namely that the deliverable is complete.
The distribution of test cases across system operations reveals where harnesses are most exercised and where validation gaps remain.
Main Benchmark Results
Category-level performance reveals a strong‑weak gap across models.
The benchmark measures how well LLM‑driven agents construct harnesses from documentation, isolating the impact of harness quality on overall performance.
The difference between the best‑performing model and the worst‑performing model within a category, expressed as a percentage point gap.
GPT‑6 Astra achieves the highest overall pass rate at 53.05 % while the weakest model, Hy3, reaches only 27.44 %.
Table 2 shows the full set of 19 models; the gap spans roughly twenty‑six percentage points.
On ordinary scenarios Opus 5 leads with 80.68 % versus GPT‑6 Astra’s 72.46 %; on boundary scenarios GPT‑6 Astra leads with 48.97 % versus Opus 5’s 37.02 %.
Both numbers are taken from the detailed breakdown in Table 2.
**Figure 1.** Pass rate of the nineteen models on the Doc2Harness benchmark, ordered from the strongest model on the left to the weakest on the right. Each model is scored on three independently built candidate harnesses, and each bar gives their mean.
**Table 3.** The categories differ sharply in discriminative power.
Performance Data Analysis
We highlight the performance shift, quantify its size, and note the fixed settings.
The top Candidate Harness reached a score of 93.3, far exceeding the baseline performance.
Raw scores range from 3.3 to 93.3; the highest value appears in the final block of the section.
All other experimental variables—model architecture, tokenization, and inference budget—were kept identical throughout the evaluation.
Test Case Difficulty Analysis
Analyzes test‑case composition and model performance across evaluation categories.
Table 3 breaks down each evaluation category into three test‑case groups: floor cases (no candidate passes), ceiling cases (all candidates pass), and discriminating cases (determine the ranking). This partition reveals where the benchmark is most informative.
**Table 3.** Strong-weak gap and test-case composition by evaluation category. The last three columns split each category's test cases into floor test cases, which no candidate harness passes, ceiling test cases, which every candidate harness passes, and the remaining discriminating test cases that determine the ranking.
Permissions stands out with the largest strong‑weak gap, while file operations show a modest gap despite a comparable pass rate. The interactive terminal category is almost saturated (≈80 % pass), leaving little discriminative power.
Fifty‑six test cases are failed by every candidate harness, providing no discrimination; they cluster in direct drive, session state, permissions, and headless categories. Model rankings shift dramatically across categories—e.g., Grok 4.7 is first on file operations but eleventh on session state.
All fifteen pairwise correlations among the six main categories are positive (mean +0.521), and overall pass rate correlates strongly (0.69–0.94) with each category, indicating a largely one‑dimensional capability.
The KR‑20 coefficient across the fifty‑seven candidate harnesses is 0.913, confirming high internal consistency of the test suite.
Table 4 reports architecture statistics for each model’s candidate harnesses, including pass rate, code lines, module count, and the proportion of code held by the largest module.
Floor cases are those that no candidate harness can solve; ceiling cases are those that every candidate harness solves—only the middle, discriminating cases actually separate models.
Why do floor and ceiling test cases matter if they don’t affect the ranking?
They set the bounds of difficulty. A large number of floor cases indicates many unattainable requirements, while many ceiling cases suggest the benchmark contains easy checks that inflate overall pass rates without providing discriminative signal.
Structural Analysis of Generated Harnesses
Analyzes code size and modularity of candidate harnesses and their effect on pass rate.
This analysis quantifies how the size and modular structure of candidate harnesses relate to their success rates.
Candidate harnesses with more modules achieve lower pass rates, even after accounting for total code size.
Partial correlation between module count and pass rate is –0.423 when controlling for code lines.
More lines per module associate with higher pass rates, holding code size constant.
Partial correlation between lines per module and pass rate is +0.504.
**Table 1.** Model performance and structural metrics.
**Figure 5.** Code lines against pass rate. Left, one point per model at its mean over the three scored builds, with a least-squares fit and its 95% confidence band. Right, each model's three scored builds, spanned by a segment with the mean marked, one row per model, ordered by pass rate.
Related Benchmarks
We position Doc2Harness among existing harness benchmarks and highlight its unique evaluation design.
Repository‑level generation benchmarks ask models to reconstruct existing libraries and evaluate them with supplied unit tests. Prior suites include Commit0, NL2Repo‑Bench, RepoZero, ProgramBench, and SWE, each measuring reconstruction quality under different protocols.
Doc2Harness differs in three respects. First, the target deliverable is a novel harness synthesized from a development document rather than a reconstruction of an existing implementation. Second, a mock model follows a preset script, making the evaluation deterministic. Third, the candidate harness is executed end‑to‑end as a black box, with effects read from four channels and adjudicated clause by clause, using positive controls to verify preconditions.
The protocol measures a single build, allowing up to three submissions per build and not supporting an iterative build–evaluate–repair loop. It tests contract form rather than runtime behavior, so the pass rate of an early submitter is a lower bound. Correlated failures arise because many test cases share a single‑variable expansion, and distance checks prune candidates that are too close to seed cases.
Doc2Harness is the first benchmark that deterministically assesses how well a model develops a harness. Experiments show that even the strongest model fails nearly half of the test cases, with failures concentrating on whether the same functionality is split across several modules. The authors hope the benchmark will become the standard for measuring this construction ability.