Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao, Sicheng Zhou
TraceTrial-Bench decouples AI agent performance from harness reliability to diagnose systemic failures.
How can we reliably evaluate AI agent harnesses—the systems managing model-tool interactions—rather than just the underlying LLM's performance?
AI agents are often evaluated solely on final task outcomes, which masks whether a failure stems from the underlying model's reasoning or the harness's inability to manage context, tools, or execution state. The authors introduce TraceTrial-Bench, a two-layer evaluation framework that uses trajectory tracing and multi-source evidence to separately score agent task performance and harness reliability. Across 3,420 runs, the benchmark reveals that 31.1% of successful agent tasks still contain at least one harness-level failure, demonstrating that outcome-only metrics are insufficient for diagnosing system reliability.
Paper Primer
The benchmark treats the agent harness as a distinct system under test, organizing reliability into seven dimensions: Context, Execution, Governance, Lifecycle, Observability, Tools, and Verification. By reconstructing native execution records into a shared Trace Intermediate Representation (Trace IR), the system allows for cross-harness comparisons that preserve native behavior while standardizing evidence collection.
The core mechanism is a two-layer evaluation protocol: Agent-layer scoring measures task deliverables, while Harness-layer scoring uses trajectory and probe evidence to verify native obligations. A Failure Attribution module then links these layers using predefined mechanism chains, allowing researchers to distinguish between independent failures and those where a harness defect propagates to an agent-side error.
Outcome-based evaluation significantly underestimates harness-level defects.
Comparison of task success versus harness-criterion satisfaction across 3,420 runs. 31.1% of runs achieved task success despite failing at least one harness criterion.
Harness reliability varies significantly across current implementations.
Macro-averaged harness scores across four evaluated harnesses.
Why is it necessary to evaluate the harness separately from the model?
Because a model can produce a correct final artifact despite underlying harness failures, such as lost context or violated interaction requirements, which remain hidden if only the final output is checked.
Does this benchmark replace existing agent benchmarks?
No; it is a complementary framework that adds a harness-reliability layer to existing task-completion metrics, providing an auditable foundation for diagnosing why agents fail.
Researchers and developers should shift from outcome-only evaluation to layered diagnostics, as harness-level failures are common even in successful agent runs and represent a critical, often overlooked, source of system instability.
Introduction
Introducing the experimental gap: benchmarks conflate model and harness failures.
Current agent benchmarks report only the final answer, so they cannot tell whether a failure stems from the model’s reasoning or from the harness that routes tool results, manages state, or enforces safety. This conflation hides systematic reliability gaps across harnesses, as illustrated by Figure 1.
The harness is the surrounding system that feeds inputs to the LLM, invokes tools, captures their outputs, and maintains state; it determines what the model can see and act upon.
**Figure.** Performance and cost analysis of various models and harnesses. (a) Bar chart showing the overall score (%) for different models. (b) Bar chart showing the overall score (%) for different harnesses. The bottom scatter plot displays the overall score (%) versus the mean cost per run (USD, log scale) for various models and harnesses, with a dashed line indicating the Pareto frontier.
The figure displays two side-by-side panels comparing failure reporting mechanisms. The left panel, labeled "Explicit failure" for "OpenClaw," shows an initial result with exit code 23, a post-tool observer reporting an exit code of 23 and error as true, and an agent report confirming the failure. The right panel, labeled "Text-only failure" for "DeepSeek Harness," shows an initial result with exit code 23, but the post-tool observer incorrectly reports the exit code as null and error as false, despite the agent report correctly identifying the failure.
**Task prompt** **Objective** Run the museum storage-room sensor calibration exactly once with bash /workspace/tools/`ok_then_fail`.sh. Record the returned process outcome separately from the status text emitted by the check. **Rules** Do not inspect the fixture or retry, poll, wrap or delegate the command. Use only its initial native tool result; do not infer an exit code from the file name, later messages or other sources. **Deliverable** Write output/exit-observation.json with command, `status_ok_visible`, `result_complete_visible`, `exit_code_visible`, `failure_reported` and `observation_basis`. Use null if no exit code is explicit; finish with `E4B_DONE`.
The key shift is moving from model‑centric to system‑centric evaluation of AI agents.
Agent Harness Fundamentals
Defines the agent‑harness abstraction and outlines evidence boundaries for evaluation.
The background formalizes the agent as the sum of a model and its harness (see Eq. 1) and then examines how evidence is collected across the interaction pipeline.
Different points in the execution pipeline expose distinct slices of what actually happened; each slice constitutes its own evidence source.
Claims are stratified by how much evidence they require and where that evidence originates, ranging from simple satisfaction checks to full propagation validation.
TraceTrial-Bench Overview
TraceTrial-Bench evaluates agent‑harness interactions by measuring agent outputs, harness behavior, and failure propagation.
To assess how well an agent harness fulfills its contract, TraceTrial‑Bench treats the harness as a first‑class system rather than a transparent wrapper around the model.
It runs a model inside its native harness, records everything the harness does, and then scores the model and the harness separately before checking whether a harness failure explains any observed model failure.
How does TraceTrial‑Bench differ from a conventional end‑to‑end benchmark that only reports a single success/failure score?
Conventional benchmarks collapse model and harness behavior into one aggregate metric, hiding whether a failure stems from the model’s decision or from the harness’s tool handling. TraceTrial‑Bench keeps the Agent and Harness verdicts separate, then explicitly checks a pre‑declared propagation rule to see if a harness defect can explain an observed model failure.
The figure illustrates the framework of "TraceTrial-Bench" for evidence-based evaluation of AI agents. It is divided into three main sections: "Task outcomes can mislead," "Native model-harness interaction," and "Evidence-based evaluation." The left section highlights that task outcomes can be misleading, showing scenarios where an agent passes despite a harness shortfall, or fails despite faithful execution. The center section depicts the interaction between a model (which selects actions) and a native harness (which provides context, tools, and control), both of which interact with external tools and the environment. The right section outlines the "TraceTrial-Bench" process, which uses native traces, workspace artifacts, and environment state to pass through an "Apparatus admission gate," leading to failure attribution for both the agent and the harness. A key statistic is noted: "31.1% of runs pass Agent criteria but fail ≥1 Harness criterion."
**Figure 2.** TraceTrial-Bench overview. Mechanism-oriented tasks exercise native harnesses; trajectory tracing and multi-source evidence support separate Agent and Harness scores and evidence-based failure attribution. The figure's evidence-extraction stage corresponds to the MEA module.
Run the model inside its native harness on a selected task.
Capture the full event sequence $\tau$ and workspace delta $\Delta W$.
Pass the captured data through the apparatus admission gate; reject runs with apparatus failures.
Score the Agent layer using task‑specific artifact checks (producing $V_A$).
Score the Harness layer using trajectory tracing and probe evidence (producing $V_H$).
Apply deterministic, criterion‑specific evidence selectors to build evidence bundles.
Activate any candidate mechanism links $L_{act}$ whose pre‑declared conditions are satisfied by the pair of failed criteria.
Emit the run record $R_{h,m,t,r}= (\tau, \Delta W, G, V_H, V_A, L_{act})$ for downstream analysis.
Capture $\tau$: the model writes $f$ at step 5; the harness logs a permission error at step 6.
Admission gate verifies container lifecycle and archive integrity – both succeed, so the run proceeds.
Agent scoring checks for $f$ – passes, yielding $V_A = \text{pass}$.
Harness scoring checks permission preservation – fails, yielding $V_H = \text{fail}$.
Pre‑declared propagation rule links “Permission preservation” to “Missing artifact $f$”; both criteria fail, so $L_{act}$ activates the candidate link.
This concrete trace shows how a harness‑side permission error can be automatically linked to an agent‑side artifact failure, enabling evidence‑based attribution without manual inspection.
Harness Reliability Taxonomy
Defines the taxonomy, task suite, harness selection, and measurement setup for the benchmark.
This section details how the benchmark is built: first the reliability taxonomy, then the concrete task suite, followed by harness selection and the isolated measurement apparatus.
The taxonomy groups reliability obligations of a harness into seven orthogonal dimensions, each capturing a distinct kind of failure or guarantee.
Turn 1: Agent receives prompt “store token X” and writes X to its private workspace.
Runner forces a session reset, clearing any in‑memory state.
Turn 2: Agent is asked “what is the token?”; the correct answer is “unknown” because the session was reset.
Evidence collector checks the output log for the token string; absence confirms isolation.
This concrete run shows how the measurement apparatus isolates the harness’s state, allowing the taxonomy to attribute a failure to a breach of the Context dimension.
How does this taxonomy differ from typical benchmark categories?
Typical benchmarks group tasks by application scenario (e.g., translation or reasoning). The Harness Reliability Taxonomy groups by the harness obligation being exercised, so failures are attributed to the harness layer rather than the model’s task performance.
Select a taxonomy sub‑dimension (e.g., Execution).
Identify the native harness mechanism that implements the sub‑dimension.
Define the property under test (e.g., idempotent retry).
Specify the model‑visible consequence (e.g., log entry indicating retry).
Anchor the task with an external truth source (oracle or deterministic state).
Fix the prompt, initial workspace, allowed tools, and resource budgets.
Verify deterministic state or artifact checks before execution.
Four commercial harnesses (Codex, Claude Code, OpenClaw, DeepSeek) are evaluated via source‑specific adapters that freeze the task revision and translate native records without altering prompt construction or tool transport.
**Figure 3.** TraceTrial-Bench system boundary and implemented pipeline. The native harness is the SUT. Dashed arrows denote evidence acquisition; solid arrows denote task execution or subsequent analysis. The evaluator consumes normalized evidence only after run-identity and apparatus checks.
Runs occur in task‑scoped containers with a frozen workspace. The runner records private baselines, provisions deterministic tools, and enforces resource limits before invoking the measurement pipeline.
**Figure 1.** The TraceTrial-Bench hierarchy, consisting of 40 leaf nodes organized into 7 components.
Trajectory Tracing and Evidence Acquisition
Describes how raw harness traces are collected, normalized, and packaged into evidence packets for evaluation.
Trajectory Tracing (TT) captures every native harness event, normalizes it to a common schema, and feeds the result into Multi‑source Evidence Acquisition (MEA) so that downstream judges see a uniform view of tool interactions and model outputs.
TT is the systematic collection of raw harness logs and their conversion into a single, time‑ordered JSON stream that all evaluation components can read.
Instantiate the appropriate harness adapter and stream the native JSON events.
For each incoming event, construct a canonical envelope and copy the original payload into
Append the canonical event to a growing JSON array ordered by a contiguous
When the session ends, write a redacted copy of the array to the run archive and record its cryptographic digest.
Define a selector in the criterion’s score sheet (e.g., tool calls, state changes) and query the archive for matching events.
Package the matched events into an evidence packet: short strings are returned verbatim; longer strings are truncated with keyword‑centered windows.
Attach provenance tags (e.g., digest, provenance‑type) so judges can distinguish archival evidence from generated evidence.
Adapter creates envelope: {`schema_version`:1, seq:0, timestamp:12345, `run_id`:"runA", `event_type`:"`tool_call`"}.
Original payload is attached as
Event is appended to the JSON array; after the run finishes the array is redacted and its SHA‑256 digest is recorded.
This concrete trace shows how TT preserves both a uniform view for evaluation and a full audit trail for verification.
The table outlines four contract elements: Task and pressure, Agent score sheet, Harness score sheet, and Oracle and provenance. Each element is defined by what is "Fixed before execution" and its "Evaluation role."
**Table 1.** Event families and their representative types and normalized content.
**Figure 5.** Cross-boundary reconstruction of a tool action. The model-facing result, native record, external execution state, and normalized event are related but not interchangeable. Solid arrows show execution; dashed arrows terminate in separately sourced, identity-aligned evidence.
**Figure 1.** The architecture of the evidence collection and judgment system.
Two-Layer Evaluation
Two-Layer Evaluation separates Agent and Harness judgments into independent score sheets.
The evaluation framework enforces a strict admission phase, then runs two parallel score sheets—one for the Agent layer and one for the Harness layer. Deterministic criteria are checked first; semantic rubrics are evaluated only after deterministic outcomes are fixed. This separation lets the system attribute failures to the correct layer without conflating them.
It runs two independent verdict tables—Agent and Harness—so that a failure in one does not mask the other.
How does Two-Layer Evaluation differ from a single‑layer scoring approach?
In a single‑layer scheme the same verdict table would mix deterministic checks with semantic judgments, making it impossible to tell whether a failure originated in the Agent’s logic or in the Harness infrastructure. Two‑Layer Evaluation keeps the tables separate, so a Harness malfunction (state 3) never overwrites an Agent pass, and vice‑versa.
Verify admission: ensure result, configuration, task, model, and harness identity agree; check that lifecycle, cleanup, trace, and archive manifest are complete.
Validate deterministic criteria: compare each declared identifier against the fixed oracle output; record states 0 or 1.
Evaluate Agent semantic rubrics: use changed artifacts as evidence packets; assign states 0 or 1 for each Agent criterion.
Evaluate Harness semantic rubrics: consume trajectory and probe evidence; assign states 0, 1, 2, or 3 according to the rubric rules.
Compute layer scores: $p_H/n_H$ for Harness, $p_A/n_A$ for Agent.
Compute combined continuous score $S = 50 + \frac{p_A}{n_A}\times50$ and store all verdicts, evidence packets, and Judge prompts for downstream analysis.
Admission succeeds because all provenance records are present.
Deterministic checks yield states [1,1,1,1] for the four Agent criteria.
Agent semantic rubrics produce states [1,1,1,0] (one failure).
Harness semantic rubrics produce states [1,1,1,1,2] (the last criterion is not exercised).
Compute $p_A=3$, $p_H=4$; layer scores are $p_A/n_A = 3/4 = 0.75$ and $p_H/n_H = 4/5 = 0.80$.
Combined score $S = 50 + 0.75 \times 50 = 87.5$.
This concrete trace shows how a non‑exercised Harness criterion (state 2) does not penalize the final Harness score, while an Agent failure directly reduces the combined score.
**Figure 7.** Admission and layered-evaluation order. Agent results are completed before Harness semantic judging, and the two layers retain distinct evidence packets and verdict vectors.
Experimental Setup
Key performance variations across Harnesses are quantified while keeping models and tasks fixed.
Harness macro rates span a 12.2 percentage‑point range across the four native Harnesses, while Agent task‑success varies over only 8.2 points.
Table 6 shows Claude Code at 75.9 % Harness macro and 44.0 % Agent success, versus Codex at 59.8 % and 35.8 % respectively.
All 95 tasks and nine model groups were held constant across the four Harnesses, ensuring that any observed differences stem from the Harness implementations themselves.
Harness-Model Interaction Results
We compare how each harness influences model performance and resource usage across the benchmark.
Claude Code harness yields the highest Agent Pass rate among the evaluated harnesses.
Claude Code reaches 44.0 % Agent Pass, while the runner‑up DeepSeek Harness attains 43.2 %.
All harnesses were evaluated on the same nine models, each on 95 tasks, with identical token budgets and cost estimates; only judge costs are omitted.
**Table 6.** Aggregate results (%). All harnesses use 855 matched model-task cells. Overall averages unrounded Agent Score and Harness Macro equally, matching the first-page plots. Agent Pass requires all Agent criteria to pass, while Clean requires all criteria in both layers to pass. Score and Macro average per-run fractions, whereas Micro pools Harness items. Full model-level results appear in Table 7.
Detailed Performance Metrics
OpenClaw boosts Harness macro rate while revealing large performance variability across harnesses.
Harness performance varies widely across harnesses within the same model.
The within‑model Harness range spans 10.0–16.2 percentage points (Table 7).
All four harnesses are evaluated on the same 95 tasks per model, using nine models per task. This constant task‑and‑model setup isolates the effect of the harness configuration.
Reliability by Taxonomy Dimension
Reliability varies across harness dimensions, revealing opposite agent‑harness gaps.
Execution shows the largest positive agent‑harness gap (13.3 pp), while Tools shows a negative gap (‑13.6 pp), indicating that dimension‑wise performance cannot be captured by a uniform offset.
Figure 8(a) reports agent scores of 78.0 % (Execution) vs harness 64.7 % and agent 53.2 % (Tools) vs harness 66.8 %.
These gaps imply that harness selection must be tailored to the reliability profile of the target task rather than relying on a single, task‑agnostic offset.
**Figure 8.** Reliability and candidate links on the 3,420-run grid. (a) Mean per-run layer scores by dimension. (b) Four criterion-verdict states over 855 runs per harness. (c) Runs with a candidate link among joint-failure runs, with linked/joint-failure counts and the pooled 49.5% rate.
Failure Attribution Analysis
Breaks down run failures between agent and harness and evaluates candidate‑link attribution.
It tells you whether a run’s failure comes from the agent’s decision logic or from the harness that mediates tool use.
How does Failure Attribution differ from simply reporting an overall error rate?
Overall error rates collapse all failures into one number, hiding whether the agent or the harness caused the problem. Failure Attribution separates the failures into distinct layers and also records whether a stored candidate link was triggered, so you can pinpoint whether improvements should target the model’s reasoning or the surrounding infrastructure.
**Table 10.** Failure states and candidate-link activation. Results across Harnesses and primary dimensions on the full 3,420-run grid.
Limitations and Related Work
We outline current limits, future work, and situate our approach among prior harness research.
Limitations: the evaluation runs a single execution per task‑model‑harness triple on a targeted task suite that does not uniformly cover all mechanisms. Consequently, the results give limited insight into how the harness will behave on real‑world workloads or under repeated runs.
Future Directions: a trace‑guided improvement loop could automatically identify mechanism failures, revise the harness, and re‑evaluate on held‑out tasks with fixed models and budgets. Controlled interventions and version tracking would reveal whether each change fixes the target failure without introducing regressions.
Related Works – Agent Harnesses: prior work tackles context handling, tool interfaces, continuity, policy enforcement, and automated optimization (e.g., ACE, LongHorizon‑Harness, HARBOR, HarnessFix). TraceTrial‑Bench adds mechanism‑level criteria and evidence while preserving native harness execution, enabling direct comparison of task gains against harness reliability.
Related Works – Agent Evaluation and Failure Analysis: existing benchmarks such as Terminal‑Bench, Harness‑Bench, and REFLECT focus on task outcomes or execution fidelity. Our two‑layer evaluation separates agent success from harness failures, providing an evidence‑based attribution framework that highlights when agents succeed despite harness shortcomings.
Benchmark Taxonomy Details
Defines the harness reliability taxonomy and details the benchmark task suite.
This appendix enumerates the taxonomy that structures the benchmark and describes the concrete task suite that exercises each harness responsibility.
Manages the information presented to the model and preserves task‑relevant content across reductions, turns, sessions, and agent boundaries.
Executes requested actions and returns process state, output, and failure information without altering their operational meaning.
Applies permissions, safety policies, and environmental restrictions consistently while preserving a clear relationship between declared and enforced behavior.
Coordinates state and control flow across turns, sessions, delegated work, recovery operations, and resource boundaries.
Exposes sufficient and accurate information for the model or operator to identify failures, distinguish their sources, and reconstruct important execution events.
Exposes external capabilities through stable model‑facing interfaces and faithfully transports tool definitions, arguments, results, and interaction requests.
Supports evidence‑based confirmation of execution progress and completion, and distinguishes verified results from unsupported success claims.
**Table 11.** The seven components of the TraceTrial-Bench harness reliability taxonomy.
**Table 12.** Construction groups represented in the final 95-task release.
**Table 13.** Normalized task-card fields for the released benchmark tasks.
**Table 14.** Main oracle sources used by TraceTrial-Bench.
This table outlines various "Oracle sources" used in evaluation, providing examples for each and describing their specific "Evaluation role."
**Table 16.** Primary component coverage in lean95.manifest.
**Table 15.** Runner-level admission and hard-gate policy, distinct from the final-score aggregation used in the present analysis.
**Table 18.** Task metadata, oracle, probe, and validation coverage.
Harness Configurations
Details the harness configurations, capability contracts, and adapter canonicalization for cross‑harness evaluation.
Section B.1 enumerates the four harnesses used in the benchmark, their package versions, and the execution paths they expose to the model.
**Table 18.** Artifact or declaration coverage.
Section B.2 introduces the capability contracts that expose what evidence each harness can emit during a run.
Section B.3 describes how adapters launch the selected harness, expose a common tool surface, and normalize native records into a unified Trace IR.
**Table.** Native evidence paths. OpenClaw combines Gateway results with session JSONL and native parent-child relationships. Codex combines JSONL execution events with thread-resume and session records. Claude Code combines stream-JSON with session archives and native tool-call identifiers. DeepSeek Harness uses persistent JSON-RPC notifications across turns. Task-specific contracts enable observation of compaction, permissions, and asynchronous child return where the runtime exposes them.