AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

AutoSaddler automates LLM agent harness optimization by treating harness updates as an offline learning problem.

How can we automatically optimize LLM agent harnesses to improve robustness on long-horizon tasks by using execution traces to generate durable patches?

LLM agents often fail on long-horizon tasks because small, local errors compound over time, and manually tuning the external "harness" (prompts, tools, and control logic) to fix these is slow and difficult to scale. AutoSaddler automates this by treating the harness as code, iteratively diagnosing execution failures, and applying targeted patches that are verified for generalization. On benchmarks like GAIA2 and SWE-Bench Pro, this approach improves agent performance by up to 10 percentage points over base harnesses, consistently outperforming manual and automated baselines.

Paper Primer

AutoSaddler formulates harness optimization as an offline learning process, mirroring mini-batch training. It uses a "Diagnosis-Patch-Verification" loop where an agent analyzes failed execution traces to generate structured patches, which are then validated on a development set to ensure they generalize beyond the training mini-batch.

The system maintains an "EvoDAG" (Evolutionary Directed Acyclic Graph) that stores the history of harness updates, lessons learned, and performance signals. This memory allows the framework to compose successful components from previous iterations, enabling the agent to escape local optima through history-aware evolution.

AutoSaddler significantly improves long-horizon agent performance across diverse benchmarks.

Pass@1 success rates on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0. Gains of 9.0, 9.6, and 10.0 percentage points over base harnesses, respectively.

Structured intervention and generalization-aware selection are critical for performance.

Ablation studies show that removing generalization-aware selection causes the largest performance drop (e.g., 62.0% to 50.6% on GAIA2) due to increased regression rates on unseen tasks. Generalization-aware selection prevents "over-scoped" patches that disrupt agent behavior on unrelated scenarios.

Why is "in-depth diagnosis" necessary compared to simple reflection?

Long-horizon failures are complex; shallow reflection often misses the root cause. In-depth diagnosis actively explores both execution traces and the harness codebase, leading to more accepted patches and better identification of systemic issues.

How does AutoSaddler avoid overfitting to specific training tasks?

It uses a generalization-aware selection mechanism that validates patches on a held-out development set and uses an EvoDAG to filter out patches that cause regressions on unseen scenarios, ensuring updates are durable rather than trajectory-specific.

AutoSaddler demonstrates that harness optimization can be treated as a principled, iterative learning problem, shifting the burden of agent reliability from manual tuning to automated, trace-grounded symbolic optimization.

Introduction

Why LLM agents stumble on long‑horizon tasks and how AutoSaddler aims to fix it.

LLM agents often collapse on long‑horizon tasks: a tiny slip early on can snowball into a full failure, and fixing the harness that wraps the model is both labor‑intensive and expensive.

LLM agents fail on long‑horizon tasks because each local mistake propagates, making the overall behavior brittle; AutoSaddler treats the harness as a symbolic program to be learned from execution traces, enabling durable fixes.

The compounding nature of local failures makes long‑horizon agent tasks especially fragile.

Related Work

We map prior efforts onto AutoSaddler’s three pillars: prompt, harness, and trace repair.

Related work spans five strands that intersect with AutoSaddler: prompt‑level tuning, self‑evolving agents, harness‑level pipelines, trace‑level diagnosis, and LLM‑augmented evolutionary search.

GEPA is a structured representation that records how a patch modifies the agent’s external harness, enabling systematic reuse and analysis.

Meta‑Harness is a coding‑agent framework that builds an end‑to‑end pipeline for automatically improving the external layers surrounding an LLM.

AutoSaddler unifies these strands by using GEPA‑driven patches within a Meta‑Harness pipeline, guided by trace‑level diagnostics and LLM‑based evolutionary selection.

Agentic Tasks and Traces

Formalizes harness optimization as a trace‑driven, budget‑constrained learning problem.

Agentic tasks span multiple reasoning steps, tool invocations, and environment interactions, so small local mistakes can snowball into complete failure. The core trick is to treat the surrounding harness as a set of symbolic parameters and evaluate them with stochastic execution traces under a strict rollout budget.

Instead of modifying the LLM itself, we view the harness as a fixed vector of symbols (prompt, tool, middleware) and use sampled execution traces to assess and improve them within a limited rollout budget.

How does this differ from standard reinforcement‑learning policy optimization?

Here we are not learning a policy that maps states to actions; we are directly searching over a static set of harness parameters using trace‑level performance estimates. The optimization is a black‑box search over $\theta$ rather than a gradient‑based policy update.

For task x₁ under $\theta^{(1)}$ we run two rollouts: first yields success ($\mu$ = 1), second yields failure ($\mu$ = 0); average $\mu$ = 0.5.

For task x₁ under $\theta^{(2)}$ both rollouts succeed, giving average $\mu$ = 1.0.

For task x₂ under $\theta^{(1)}$ both rollouts fail (average $\mu$ = 0), while $\theta^{(2)}$ yields one success and one failure (average $\mu$ = 0.5).

Empirical estimate Ĵ_D($\theta$) = (0.5 + 1.0 + 0 + 0.5) / 4 = 0.5 for $\theta^{(1)}$ and (1.0 + 0.5) / 2 = 0.75 for $\theta^{(2)}$.

AutoSaddler selects $\theta$̂_AS = $\theta^{(2)}$ because it maximizes the empirical score within the budget.

Even with a tiny budget, the trace‑level evaluation can distinguish candidates that differ only in a single middleware tweak, highlighting the value of stochastic traces for fine‑grained diagnostics.

The AutoSaddler Framework

AutoSaddler iteratively optimizes LLM agent harnesses via diagnosis, patching, reflection, and evolution.

AutoSaddler treats harness optimization as an offline learning problem, iterating over mini‑batches of tasks to progressively improve a LLM agent’s harness.

Evaluate current harness $H_n$ (parameters $\theta_n$) on a mini‑batch $B_n \subset D_{\text{train}}$.

Enter Diagnosis–Patch Session: analyze failed traces, generate a structured patch $\Delta\theta_n$, and form $H'_n = H_n + \Delta\theta_n$.

Verify $H'_n$ on the same mini‑batch; accept the patch if $J_{B_n}(H'_n) > J_{B_n}(H_n)$.

Optionally evaluate $H'_n$ on $D_{\text{dev}}$ to gauge generalization.

Reflection Session records fixed, regressed, still‑failing, and still‑passing cases and stores lessons in EvoDAG.

Evolution Session queries EvoDAG to compose a new candidate harness $H_{n+1}$ for the next iteration.

Optimizing a harness means adjusting the code, prompts, and tool configurations that govern how an LLM agent interacts with its environment, so that long‑horizon tasks succeed more reliably.

Diagnosis identifies the missing tool invocation as the root cause.

Patch $\Delta\theta$ adds a new tool argument to T (Capability Patch) and inserts a prompt rule “use tool X when needed” (Steering Patch).

Apply $\Delta\theta$ to obtain $H'$, yielding updated $\theta'$ with the new rule and argument.

Verification on $B$ shows the task now succeeds, so the patch is accepted.

The example shows how a single iteration can combine a Capability Patch (code change) with a Steering Patch (prompt edit) to fix a concrete failure.

How does Harness Optimization differ from standard prompt tuning?

Prompt tuning only edits the textual prompt, while Harness Optimization also allows structural changes to tool code and middleware logic (Capability Patches), giving a richer set of interventions.

EvoDAG is a directed acyclic graph that records every harness version and the patches that produced it, enabling the system to recombine successful components from any past iteration.

Node $v_2$ inherits $v_1$’s parameters and applies the Capability Patch to add tool T.

Node $v_3$ inherits $v_2$’s parameters and applies the Steering Patch to adjust the prompt.

The Evolution Session can now create a new node $v_4$ by merging $\Delta\theta_{1\to2}$ and $\Delta\theta_{2\to3}$, yielding a harness that has both the new tool and the refined prompt.

EvoDAG lets the system reuse successful patches from different lineages, escaping local optima that a purely sequential update would get stuck in.

Why is a DAG used instead of a simple list of past harnesses?

A DAG captures the dependency structure between patches; it shows which patches can be safely combined and which would conflict, something a flat list cannot represent.

Capability Patches modify executable code or agent configuration, while Steering Patches only tweak textual prompts or tool descriptions without changing underlying logic.

When should I use a Capability Patch versus a Steering Patch?

Use a Capability Patch when the failure stems from missing functionality or a bug in the code; use a Steering Patch when the functionality exists but the agent’s prompting or tool usage is suboptimal.

**Figure 2.** Overview of AutoSaddler. The iterative optimization loop: the current harness is tested on a mini-batch, diagnosed and patched across harness components, verified for improvement, and then reflected upon to extract lessons into the EvoDAG, which guides the evolution of the next harness.

Experimental Setup

AutoSaddler’s best‑in‑class Pass@1 performance across all benchmarks.

AutoSaddler outperforms the strongest baseline by a large margin on the hardest SBP task.

On the Ansible task (96 tasks) AutoSaddler reaches 58.0 % Pass@1 versus 50.0 % for GEPA.

We evaluate on three benchmarks—GAIA2, SWE‑Bench Pro (SBP), and Terminal‑Bench 2.0 (TB2)—using the default ReAct, SWE‑agent, and Terminus 2 harnesses respectively. Baselines include GEPA and Meta‑Harness, adapted where necessary.

**Table 3.** Test-set Pass@1 results on SWE-Bench Pro and Terminal-Bench 2.0, reported as mean ± standard deviation over three runs. Parentheses indicate the number of tasks; bold denotes the best result per column.

Ablation studies reveal that removing the in‑depth diagnosis step drops performance to 56.7 %, while omitting structured intervention or generalization‑aware selection reduces it further, confirming each component’s contribution.

Main Results

AutoSaddler delivers up to 10 pp gains and far higher efficiency than prior harnesses.

LLM agents stumble on long‑horizon tasks because early mistakes snowball; AutoSaddler treats harness optimization as a symbolic learning problem, using execution traces to diagnose and patch components durably.

AutoSaddler outperforms the strongest baseline on each benchmark, delivering up to a 10 pp gain.

On Terminal‑Bench 2.0, AutoSaddler raises Pass@1 from 40.0 % to 50.0 % ( +10.0 pp ) versus the best automated baseline at 43.3 %.

**Figure 1.** Comparison of optimization performance and efficiency on GAIA2. (a) AutoSaddler reaches 72.3% dev accuracy with ~1,000 total task executions, whereas GEPA and Meta-Harness saturate at 64.6% and 61.5%, respectively, despite consuming ~2,800 task executions. (b) When measured by the number of execution rollouts leveraged for optimization, AutoSaddler achieves its best performance after consuming only 147 traces, ~10x fewer than Meta-Harness (1,400 traces).

**Figure 3.** Patch type distribution and acceptance: AutoSaddler vs. w/o Structured Intervention. Without structural intervention, patches collapse onto Steering (91.5%), while AutoSaddler produces a balanced mix spanning Prompt, Tool, and Middleware edits.

**Figure 4.** Performance comparison on the dev-set across iterations. AutoSaddler maintains a lower regression rate while achieving comparable fix rates to the w/o Generalization-Aware Selection ablation, yielding a consistently positive net gain that the ablation fails to sustain.

**Figure 5.** Comparison of optimization performance and efficiency on Terminal-Bench 2.0. (a) AutoSaddler reaches 73.7% dev accuracy with only 31 task executions, whereas Meta-Harness requires 98 executions to plateau at 63.2% and GEPA reaches 57.9% at 90 executions. (b) When measured by the number of execution traces leveraged for optimization, AutoSaddler achieves its best performance after leveraging only 12 traces, over 8× fewer than Meta-Harness (98 traces), while surpassing it by 10.5 percentage points.

**Figure 6.** Evolutionary search trajectory of AutoSaddler as the EvoDAG. Four phases (foundation, rapid improvement, selective merging, and consolidation) lead to a global dev-set peak of 72.3% at Iter27, escaping the Iter20 regression and local optima.

**Figure 9.** Case study on a cab-booking task with two candidate rides on the same day. Shallow diagnosis overlooks the second matching ride and proposes an off-target fix, while AutoSaddler inspects the full ride history and identifies the missed candidate.

Training Loop Details

Mapping AutoSaddler’s loop onto the classic mini‑batch training pipeline.

AutoSaddler preserves the outer skeleton of standard mini‑batch training while swapping the differentiable gradient step for a trace‑grounded symbolic optimization stage. Table 4 makes the correspondence explicit, showing how each traditional step maps to an AutoSaddler stage.

**Table 4.** Mapping between conventional mini-batch training and the AutoSaddler loop in Figure 2. AutoSaddler follows the outer structure of mini-batch learning but replaces differentiable gradient computation with trace-grounded symbolic optimization.

Sample a mini‑batch $B_n \subset D_{\text{train}}$.

Execute the current harness $H_n$ on $B_n$, collecting outcomes and traces $\tau_n$.

Diagnose failures in $\tau_n$ and infer root‑cause hypotheses.

Generate a targeted patch $\Delta\theta_n$, yielding candidate harness $H'$.

Re‑evaluate $H'$ on the same mini‑batch.

If same‑batch performance improves, update EvoDAG $G_n$ with the patch and evolve the next harness $H_{n+1}$; otherwise discard the patch.

Continue until the rollout budget $K$ is exhausted.

A capability patch rewrites executable components of the harness—adding new tools, changing tool arguments, or altering the agent‑loop logic—so the agent can perform actions it previously could not.

How does a capability patch differ from simply increasing the learning‑rate in gradient‑based training?

Increasing the learning‑rate scales an existing numeric update; a capability patch rewrites the code that generates the update, potentially adding new actions or changing the agent’s execution flow, which cannot be captured by any scalar step size.

A steering patch tweaks textual descriptors—prompts, tool descriptions, or hook reminders—without altering the underlying executable code, thereby nudging the agent’s decision‑making within the existing capability set.

Why might a steering patch sometimes cause a large behavioral shift despite being a “small” edit?

Because the agent’s policy can be highly sensitive to prompt phrasing; a minor lexical change may flip the attention distribution, leading to a disproportionate effect on downstream actions.

Is EvoDAG comparable to a conventional optimizer like Adam?

No—Adam updates numeric parameters directly using gradient statistics; EvoDAG updates a symbolic knowledge base of textual patches and their empirical effects, which the search algorithm queries to propose new candidates.

The two‑phase schedule splits the search into an exploration phase (favoring capability patches) and a refinement phase (favoring steering patches). The transition point $k$ is computed as $k = E \cdot \frac{|D_{\text{train}}|}{B}$, where $E$ is the number of capability‑phase epochs and $B$ the mini‑batch size.

AutoSaddler deliberately swaps the order of optimizer update and validation compared to standard training: dev‑set validation precedes the EvoDAG update, ensuring that only generalizable patches are stored.

AutoSaddler’s loop replaces numeric gradients with a diagnosis‑patch‑verification cycle, using EvoDAG as a memory of verified textual updates.

Optimizer Cost Analysis

This section details the optimizer and task‑agent costs, and analyzes the impact of in‑depth diagnosis.

Optimizer‑side cost is summarized in Table 13, which reports generated, rejected, and accepted patches together with wall‑clock time, monetary cost, LLM calls, and token usage per generated patch for GEPA, Meta‑Harness, and AutoSaddler.

**Table 13.** Optimizer-side cost on GAIA2. Runtime, monetary cost, LLM calls, and token usage are averaged per generated patch.

**Table 14.** Average task-agent evaluation cost per GAIA2 rollout.

**Table 15.** Average tool calls and file accesses per optimization step for AutoSaddler and its ablated variant without in-depth diagnosis.

The search trajectory visualization (described in the text) shows how reflection‑guided rebasing and selective merging shape AutoSaddler’s optimization over 50 iterations.

**Figure 7.** Case study on a calendar task with ambiguous cancellations, where AutoSaddler's in-depth diagnosis accurately identifies the true root cause via deep debugging, unlike shallow diagnosis.

**Table 16.** Representative patches discovered by AutoSaddler on GAIA2, producing the final harness $\hat{\theta}_{\text{AS}}$ from the base harness. C = Capability, S = Steering. Inserted text is highlighted in teal.

Overall, the modest optimizer‑side overhead of AutoSaddler is offset by its selective use of expensive task‑agent evaluations, yielding higher performance with fewer rollouts.

Case Studies

Component ablations reveal how each patch contributes to AutoSaddler’s performance.

We evaluate the impact of each individual patch by removing it from the final harness and measuring the resulting drop in performance.

**Figure 8.** Case study on a file-management task that requires moving PNG files into a sims3 folder under benchmarks. Shallow diagnosis misattributes the failure to a relative-path error, while AutoSaddler verifies the filesystem and locates the true cause: an unverified parent directory.

AutoSaddler accepts more patches than the w/o In‑depth Diagnosis ablation.

Figure 11 shows AutoSaddler reaching 20 accepted patches versus 15 for the ablation.

**Figure 10.** Cumulative accepted patches. AutoSaddler accumulates 20 accepted patches, whereas the w/o In-depth Diagnosis ablation reaches 15.

Capability and Steering patches together yield a 24.6 pp accuracy gain.

Figure 12 reports a cumulative 24.6 pp improvement after the full optimization trajectory.

**Figure 11.** Representative patches discovered over the optimization trajectory. During the Exploration phase (Iterations 1–25), the search is dominated by Capability patches, including New Tool Addition, Implementation Fix, and Agent Loop Logic Change, which address structural limitations in the harness (e.g., Iterations 11 and 13). During the Refinement phase (Iterations 26–50), the search shifts toward Steering patches that refine prompt rules, tool descriptions, and PreToolUse hooks (e.g., Iteration 27). Together, these two phases produce a cumulative 24.6 pp improvement.

**Figure (c).** From Lessons to Principles: The Temporal Maturation of Reflection

**Table.** Summary of component modifications and patches, detailing the transition from the Base Harness to the Final Harness ($\hat{\theta}_{AS}$).

EvoDAG Implementation Details

Additional experimental details, robustness analyses, and EvoDAG CLI information.

The EvoDAG DAG can be built trivially, but exposing it to agents efficiently requires a dedicated interface. We therefore provide the evo-dag CLI, which offers on‑demand summaries such as patch history, lessons learned, scenario registry, and code diffs, avoiding the need to serialize the entire DAG into a prompt.

GEPA expects a single monolithic system prompt, while the default GAIA2 ReAct agent splits its prompt into multiple variables (core principles, tool‑calling JSON, ARE instructions). To make GEPA compatible we concatenate these sections using ===== `VAR_NAME` ===== delimiters, preserve them during reflection, and later split the evolved prompt back into the original variables for the CA‑SDK.

Meta‑Harness originally assumes a single‑file Python harness. GAIA2 agents, however, may spread their harness across an entire repository. We therefore extend Meta‑Harness to operate at the repository level, applying the same patching logic to every file in the repo.

For GAIA2 we run the ReAct agent via are‑run, record full trajectories, and use Llama‑3.3‑70B‑Instruct as the judge model to label task success according to the GAIA2 protocol [10].

SWE‑Bench Pro follows its official pipeline: each generated patch is applied inside a Docker image seeded with the target repository at the issue commit, and success requires all fail‑to‑pass tests to pass without introducing regressions on pass‑to‑pass tests.

Terminal‑Bench 2.0 executes each task in an isolated Harbor container and judges success via a task‑specific test script that inspects the final terminal and system state.

We construct train, development, and test splits per benchmark (Table 6). GAIA2 splits by Universe (persona), SWE‑Bench Pro splits by repository (programming language), and Terminal‑Bench 2.0 uses a uniform random split because it lacks a natural grouping axis.

Optimization budgets differ by method: AutoSaddler and GEPA receive 2 epochs on GAIA2 and SWE‑Bench Pro (4 epochs on Terminal‑Bench 2.0), while Meta‑Harness is allocated more epochs (20 on GAIA2, 15 on Terminal‑Bench 2.0, 8 on SWE‑Bench Pro) to match total task executions.

Robustness to stochastic optimization is evaluated by running two independent optimizations per method on GAIA2 and measuring Pass@1 on Universe 22 over three test executions (Table 7). AutoSaddler’s second run attains 58.6 % ± 0.5, confirming consistent gains.

To test sensitivity to training‑distribution shift we re‑optimize AutoSaddler on a different training universe (Universe 24) and evaluate on Universe 22 (Table 8). The resulting 57.4 % ± 2.1 performance remains well above the default agent.

Cross‑model transferability is assessed by deploying harnesses optimized with Claude Opus 4.6 on a weaker Claude Haiku 4.5 backbone. AutoSaddler improves Pass@1 by +5.6 pp over the default agent across all GAIA2 universes (Table 9).

Fine‑grained ablations isolate the contributions of structured intervention and generalization‑aware selection. Removing only Phase Scheduling drops Pass@1 to 54.8 % (−5.9 pp), while removing only Development‑Set Filtering drops it to 50.0 % (−10.7 pp) (Table 10).

Patch durability by harness component (Prompt, Tool, Middleware) shows similar fix rates (57–59 %) but higher regression for Tool patches (19 %) (Table 11).

Re‑classifying patches by the Capability–Steering taxonomy reveals that Steering Patches have comparable fix rates but substantially higher regression (17 % vs. 8 % for Capability Patches) (Table 12).

On Terminal‑Bench 2.0, AutoSaddler reaches 73.7 % dev accuracy after only 31 task executions and 12 leveraged traces, outperforming Meta‑Harness (63.2 %) and GEPA (57.9 %) (Figure 5).

diagnosis $\longrightarrow$ patch-as-intervention $\longrightarrow$ same-batch verification

Finally, we profile end‑to‑end optimization cost by separating optimizer‑side LLM overhead (candidate generation, diagnosis, reflection) from task‑agent evaluation cost, providing a fuller picture of computational expense on GAIA2.

EvoDAG CLI Reference

Procedural checklist for the final analysis and selection phase.

This appendix enumerates the exact workflow that must be followed before finishing the session, ensuring that every scenario in the mini‑batch receives a structured reflection and that the base codebase is correctly prepared for the next iteration.

Core commands for the analysis phase

When classifying a regression or a fix, compare the before/after traces, locate the divergence point, and check whether the diff shown by evo-dag show edge touches the exercised code path; only then label the change as a true causal effect.

**Table 17.** Representative patches discovered by AutoSaddler on Terminal-Bench 2.0, producing the final harness $\hat{\theta}_{AS}$ from the base harness. C = Capability, S = Steering. Inserted text is highlighted in teal.

This table compares the "Base Harness" and the "Final Harness ($\hat{\theta}_{AS}$)" across two components: `_get_completion_confirmation_message` (labeled S) and `run()` (labeled C).

All reflections must be recorded with the evo-dag update-reflection command before the session ends; omitting any entry corrupts the lesson pool that drives future iterations.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers