ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization
Sungho Park, Wonjoong Kim, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Victor Rühle
ActiveSaddler adapts training curricula for agent harness optimization by dynamically prioritizing failure patterns.
How can we optimize LLM agent harnesses more efficiently by dynamically choosing between exploring new scenarios and repairing known failure patterns?
Automated harness optimization improves agent performance by iteratively patching prompts and control logic, but these methods typically use a fixed schedule of training scenarios that fails to adapt as the harness evolves. ActiveSaddler treats curriculum construction as a non-stationary bandit: it dynamically groups failures into reusable "patterns," prioritizes them by estimated learning potential, and balances revisiting known weaknesses with exploring unseen scenarios. This adaptive approach consistently outperforms fixed-order baselines, improving test Pass@1 by 4.4 and 7.5 percentage points on GAIA2 and Terminal-Bench 2.0, respectively.
Paper Primer
Harness optimization is constrained by a limited rollout budget, making the choice of training scenarios critical. Existing methods often rely on static mini-batches, which leads to wasted compute on already-repaired failures or premature abandonment of unresolved weaknesses.
ActiveSaddler is a curriculum-learning framework that co-evolves the training schedule with the harness. It functions like a triage system: it identifies recurring failure patterns, assigns them a priority score based on their potential for improvement, and decides whether to "pull" (revisit a known weakness) or "draw" (explore a new scenario) at each iteration.
ActiveSaddler significantly improves final harness performance under a fixed rollout budget.
Comparison against fixed-order baselines on GAIA2 and Terminal-Bench 2.0 benchmarks. +4.4 pp and +7.5 pp improvement in Pass@1, respectively.
Adaptive curriculum selection is more cost-efficient than fixed-order optimization.
End-to-end cost-accuracy trade-off analysis on GAIA2 and TB2. Achieves target accuracy with significantly fewer task-agent rollouts (e.g., $298 vs \$1,360 on GAIA2).
Why is a "failure-pattern" arm more effective than simply grouping by scenario or category?
Category-level arms are too coarse, allowing successes on some tasks to mask unresolved weaknesses in others, while scenario-level arms are too fragmented, separating recurring weaknesses across individual instances and preventing efficient budget allocation.
How does the system decide when to stop optimizing a known failure and start exploring new ones?
The Exploration Controller uses an LLM to weigh the remaining learning value of known arms against the potential for discovering new weaknesses in the unseen scenario pool, shifting from exploration to exploitation as the arm pool matures.
ActiveSaddler demonstrates that the training curriculum is a primary optimization dimension; researchers should prioritize adaptive scenario selection alongside harness-update mechanisms to maximize performance under limited compute.
Introduction
We expose the inefficiency of fixed-scenario harness optimization and motivate adaptive curricula.
Large Language Model agents rely on an external harness that defines prompts, tool usage, and execution control. Current harness‑optimization methods focus on how to update this harness but keep the training scenarios static, which limits their ability to target the most informative failures as the harness improves.
An Agent Harness is the surrounding system that determines how an LLM‑based agent is prompted, which external tools it may call, and how its execution is monitored and corrected.
ActiveSaddler addresses this gap by treating scenario selection as an automated curriculum‑learning problem: a non‑stationary bandit dynamically decides whether to discover new failure‑pattern arms or to refine existing ones, continuously adapting the curriculum as the harness evolves.
The core inefficiency of current harness optimization is its reliance on a fixed scenario order that cannot respond to the evolving needs of the harness.
Problem Formulation
Defines the agent harness and the curriculum policy that drives its optimization.
Optimizing an agent harness requires a concrete way to evaluate its behavior on tasks and a mechanism to choose which tasks to present at each iteration.
How does an “agent harness” differ from simply fine‑tuning a language model?
Fine‑tuning changes the model’s internal weights, while a harness keeps the model fixed and instead adjusts the surrounding prompts, tool calls, and control flow. This separation lets the same model be reused across many tasks with only lightweight harness updates.
The curriculum policy $\pi$ decides which training scenarios to present next, using everything the optimizer has learned so far.
Compute the current performance on each task: $\mu_A=0.9$, $\mu_B=0.85$, $\mu_C=0.3$, $\mu_D=0.25$.
The policy $\pi$ ranks tasks by difficulty and selects the two hardest: $X_{t}=\{C,D\}$.
Execute $H_{t}$ on $C$ and $D$, obtaining new traces $E_{t}$.
Pass $E_{t}$ to optimizer $O$, which updates $\theta$ to improve performance on $C$ and $D$.
Next iteration the policy re‑evaluates and may now include $B$ if its difficulty has risen relative to $C$.
The curriculum focuses computation on the current failure modes, ensuring that each update targets the most informative examples rather than wasting effort on already‑solved tasks.
In what way does this curriculum differ from standard “easy‑to‑hard” curricula?
Standard curricula often follow a fixed schedule (e.g., increasing difficulty monotonically). Here $\pi$ is fully data‑driven: it continuously re‑samples based on the actual performance of the current harness, allowing it to revisit previously easy tasks if they become hard again.
Related Work
We position ActiveSaddler among prior automatic harness optimization and curriculum learning research.
Automatic Harness Optimization (AHO) expands automatic tuning from prompts to the full agent harness—tools, memory, skills, and runtime control logic. Prior work splits into search‑based approaches that explore configurations via evolutionary or Bayesian search, and learning‑based approaches that treat the harness as a learnable object updated from rollout feedback without touching model weights. GEPA combines reflective updates from mini‑batch feedback with evolutionary Pareto search over validation performance.
Automated Curriculum Learning studies how to schedule training data or tasks; early methods prioritized examples by competence or learning progress, while later techniques formalized the choice with non‑stationary bandits, teacher‑student selection, and progress‑driven environment sampling. ActiveSaddler inherits the principle of valuing training scenarios but faces the added difficulty that the curriculum target—the harness weakness—is only revealed after execution.
Adaptive Task Selection for Agent Optimization adapts task allocation at different stages: CoEvolve generates new training tasks for model‑weight updates, while Task‑CoEvolve and HarnessLens select validation or verification tasks for harness modifications. In contrast, ActiveSaddler continuously adapts the training scenarios that generate evidence for harness updates, balancing revisits of known weaknesses with exploration of unseen scenarios.
The ActiveSaddler Framework
ActiveSaddler iteratively balances exploring new failures and refining known arms to optimize the harness under a limited budget.
Optimizing a harness under a limited rollout budget forces the curriculum to allocate each optimization step wisely. We cast this allocation as a multi‑armed bandit where each arm corresponds to a concrete failure‑pattern target. The resulting loop repeatedly decides whether to explore unseen scenarios or to pull an existing arm for further refinement.
An arm groups together all scenarios that share the same underlying weakness in the harness, so improving the arm repairs many failures at once.
All three failures are abstracted to the same symptom “index‑error”.
The symptom matches no existing arm, so a new arm a₁ is instantiated.
Arm a₁ stores the scenario set $S$(a₁) = {s₁, s₂, s₃}.
When a₁ is pulled, the next batch Xₜ is drawn from S(a₁), e.g., {s₁, s₃}.
After the batch is optimized, any remaining failures are re‑examined for further pattern updates.
Grouping failures into an arm lets a single patch eliminate all three errors, dramatically reducing the number of required optimization steps.
How does a Failure‑Pattern Arm differ from a standard multi‑armed bandit arm?
Standard bandit arms are fixed actions with static rewards; a Failure‑Pattern Arm is a dynamic collection of training scenarios whose relevance changes as the harness evolves, and its reward is the reduction of a shared failure pattern.
The controller decides whether the next step should “draw” new, unseen scenarios or “pull” an existing arm, balancing discovery of new weaknesses against exploitation of known ones.
Why does the Exploration Controller make a hard binary choice instead of a soft probability over many actions?
Because the curriculum’s primary tension is between two qualitatively different regimes—discovering new failure patterns versus refining known ones—so a binary decision cleanly separates the two and avoids diluting the signal with marginally useful intermediate actions.
Curriculum learning treats the sequence of optimization targets as a curriculum, gradually presenting harder or more informative tasks as the harness improves.
How does this curriculum differ from classic curriculum learning that pre‑orders tasks?
Instead of a fixed ordering, the curriculum here is data‑driven: the policy continuously re‑evaluates which arms or unseen scenarios promise the greatest learning progress, allowing the order to change as the harness repairs or regresses.
Observe current harness $H_t$, arm set $A_t$, unseen pool $U_t$, and history $I_{<t}$.
The Exploration Controller outputs a decision $d_t \in \{\text{DRAW},\text{PULL}\}$.
If $d_t$ = DRAW, sample a batch $X_t \subseteq U_t$; otherwise, the Arm Prioritizer samples an arm $a_t \sim q_t$ and sets $X_t \subseteq S_t(a_t)$.
Execute $X_t$ under $H_t$ to obtain execution records $E_t$.
Pass $E_t$ to the harness optimizer $O$ to produce updated harness $H_{t+1}$.
Failure‑Pattern Extractor processes failed records in $E_t$, updates the arm set to $A_{t+1}$, and removes used scenarios from $U_t$.
Increment $t$ and repeat.
**Figure 2.** Overview of ActiveSaddler. At each iteration, the curriculum chooses between exploring unseen scenarios and revisiting an instantiated arm, optimizes the harness on the selected scenario batch, and updates its state based on the resulting execution and optimization outcomes.
**Figure 3.** Effect of adaptive arm scoring. Adaptive scoring dynamically reallocates optimization priority and yields more effective repair of observed failures.
**Figure 4.** Adaptive exploration over the optimization trajectory. The policy chooses DRAW when discovery is more valuable and PULL when promising repair opportunities remain.
Experimental Results
ActiveSaddler sets new performance records on GAIA2 and Terminal‑Bench 2.0, with large efficiency gains.
GAIA2 is a simulated smartphone environment containing ten distinct “Universes” that test a general‑purpose assistant’s ability to handle varied user contexts.
TB2 is a terminal‑based benchmark comprising 89 realistic tasks spanning system administration, machine learning, and cybersecurity, used to evaluate agent harnesses in command‑line scenarios.
ActiveSaddler achieves the highest Pass@1 on GAIA2, reaching 59.8 %.
Table 1 shows ActiveSaddler outperforming all baselines on the GAIA2 split.
Beyond raw accuracy, Figure 1 demonstrates that ActiveSaddler reaches comparable development performance with far fewer task‑agent rollouts, yielding a markedly better cost–accuracy trade‑off on both benchmarks.
Cost and Failure Analysis
We quantify how each component’s removal impacts optimization cost and efficiency.
ActiveSaddler treats optimization as a curriculum‑learning problem: an exploration controller decides whether to discover new failure modes (Pull) or refine existing arms (Draw) to maximise repair efficiency.
We first separate optimizer‑side computation from task‑agent rollouts, then combine them to see the full end‑to‑end cost.
Task‑agent rollout cost is roughly the same for all methods, so the total cost difference stems mainly from how many rollouts each method allocates.
**Table 8.** Average task-agent evaluation cost per rollout.
ActiveSaddler reaches 58.5 % development accuracy on GAIA2 for only \$298 of cumulative cost.
End‑to‑end cost‑accuracy curve (Figure 5) shows ActiveSaddler achieving 58.5 % at \$298, whereas AutoSaddler needs \$1,360.
On Terminal‑Bench 2.0, ActiveSaddler attains 78.9 % development accuracy for just \$128 of cumulative cost.
Figure 5 (bottom row) reports ActiveSaddler’s 78.9 % at \$128, while the next best method reaches the same accuracy only after \$220.
**Figure 5.** End-to-end optimization efficiency. Best-so-far development accuracy against cumulative monetary cost, LLM calls, input tokens, and output tokens on GAIA2 (top) and Terminal-Bench 2.0 (bottom). Cumulative usage includes both optimizer-side and task-agent computation.
Case studies (Figures 6 & 7) illustrate how failure‑pattern arms enable iterative refinement when a repair initially fails.
**Figure 6.** Case study on `tnxtee` (GAIA2), where a failure-pattern arm instantiated from the initial failure enables an unsuccessful repair to be retargeted and refined in a subsequent iteration.
**Figure 7.** Case study on 4bytkx (GAIA2), where retargeting the same failure-pattern arm corrects an executor hook that was initially inserted too late in the execution flow.
Category arms are too coarse—successes on unrelated scenarios can mask an unresolved weakness—while scenario arms are too fragmented, making it hard to revisit a specific weakness under a limited budget. Failure‑pattern arms strike a balance, keeping weaknesses explicit yet compact.
**Figure 8.** Failure hit rate by arm definition. Fraction of optimization iterations whose sampled mini-batch contains at least one scenario with an unresolved failure. Budgets are 1,400 rollouts for GAIA2 and 490 for TB2.
**Table 9.** Failure patterns induced by ActiveSaddler on GAIA2
**Table 9.** Failure-pattern arms discovered during the GAIA2 and TerminalBench 2.0 optimization runs.
Arm‑scoring dynamics are illustrated in Figures 9 & 10: a repaired arm’s priority drops sharply after a successful patch, while a recurring weakness retains high priority until sufficient evidence accumulates.
**Figure 9.** Case study of a resolved weakness (P3 of GAIA2). After an accepted patch at Iteration 5, the arm’s priority drops sharply. A later pull finds all supporting scenarios successful and produces no further patch, after which the arm remains at low priority.
**Figure 10.** Case study of a recurring weakness (P8 of GAIA2). Early pulls produce no patch because the associated scenarios succeed when re-executed, yet the arm remains highly prioritized. The same failure pattern later appears in additional scenarios, providing new evidence that the weakness remains active; a subsequent pull produces an accepted patch. Cross markers indicate scenarios newly associated with P8.
Exploration decisions depend on unresolved weaknesses: ActiveSaddler pulls when many arms remain unresolved (ratio 2.0× on GAIA2, 6.9× on TB2), whereas the fixed schedule shows little dependence.
**Figure 13.** Dependence of exploration decisions on unresolved weaknesses. Ratio of the average number of unresolved arms at PULL to DRAW decisions. Values above one indicate more unresolved weaknesses at PULL, while values near one indicate little difference between the two decision types.
Finally, the curriculum prompts (Figures 14‑16) make the Failure‑Pattern Extractor, Arm Prioritizer, and Exploration Controller reproducible and interpretable.
Implementation Details
Additional experimental details, data splits, and algorithmic specifications.
The Pattern Registry records each failure‑pattern arm as a symptom‑level label together with its (harness, trace, scenario) tuples, root‑cause evidence, activity observations, and learning‑progress scores. Because the registry grows across iterations, serializing it into the prompt is infeasible, so we expose a pattern command‑line interface (CLI) that mediates all agent interactions.
The CLI offers on‑demand scoped views of the arm population—canonical pattern tables, per‑pattern evidence, score rankings, and full pull histories—and is the sole channel for registering new arms, tagging tuples, and recording scores or decisions. Sessions receive a capability manifest derived from the sampling strategy; any command outside the manifest is rejected.
The table presents the benchmark splits for GAIA2 and Terminal-Bench 2.0, detailing the split axis, split type (Train, Dev, Test), task groups, and the number of tasks associated with each.
We set the shared rollout budget to $K = 10(|D_{\text{train}}|+|D_{\text{dev}}|)$, yielding 1,400 rollouts on GAIA2 and 490 on Terminal‑Bench 2.0. All methods limit each optimization iteration to three scenarios, matching the mini‑batch size of AutoSaddler, and we fix the arm‑selection temperature at $\tau_{\text{sel}} = 0.15$.
Algorithm 1 details the ActiveSaddler curriculum: the Exploration Controller decides between exploring unseen scenarios or optimizing an existing arm; the Arm Prioritizer scores arms and samples the next target; the fixed harness optimizer $O$ updates the harness; and the Failure‑Pattern Extractor normalizes new patterns.
The regression‑rediscovery rule re‑enables previously successful scenarios that did not instantiate an arm, allowing them to be explored again once all arms have been visited.
Algorithm 2 shows how the same curriculum is instantiated on AutoSaddler: after executing $H_t$ on $X_t$, failures are diagnosed, a structured patch is derived, and the patched harness $H'_t$ is re‑executed before updating EvoDAG.
**Table.** To assess robustness to optimization stochasticity, we conduct two independent GAIA2 optimization runs for AutoSaddler, its category- and scenario-level difficulty-based fixed curricula, and ActiveSaddler. Each resulting harness is evaluated on three held-out universes (Universes 21, 22, and 27; 300 scenarios in total) with three repeated test-time executions. Both runs of ActiveSaddler outperform all fixed-order baselines on each held-out universe. Across the three universes, ActiveSaddler achieves 59.8% and 61.6% Pass@1 in Runs 1 and 2, exceeding the fixed-order baselines by 3.9 to 4.4 and 5.0 to 5.7 percentage points, respectively. These consistent gains across independent optimization runs and held-out universes indicate that the improvement of ActiveSaddler is not specific to a single optimization trajectory.
**Table 5.** Comparison with alternative curriculum strategies. Pass@1 on three held-out GAIA2 universes when replacing either the Arm Prioritizer or Exploration Controller in ActiveSaddler with an alternative strategy while keeping the remaining curriculum components unchanged. Performance on each universe is reported as mean ± standard deviation over three test-time executions.
EMA scoring updates each arm’s priority via an exponential moving average: $\phi_t(a) = (1-\eta)\,\phi_{t-1}(a) + \eta\,\text{active}_t(a)$ with $\eta=0.1$, where $\text{active}_t(a)$ is the fraction of scenarios still exhibiting the failure after patching.
The UCB‑AIR controller decides to explore unseen scenarios when the cumulative number of scenario executions $N_t$ exceeds $\alpha|A_t|$ (with $\alpha=0.6$); otherwise it selects an existing arm.
The table presents the performance of different methods on the GAIA2 Universe benchmark, measured by Pass@1 accuracy across three specific subsets (21, 22, and 27) and their average.
**Table 6.** Results for GAIA2 and Terminal-Bench 2.0.
Finally, the average task‑agent evaluation cost per rollout (last two tables) shows that ActiveSaddler’s overhead is comparable to AutoSaddler while delivering consistent performance improvements across benchmarks.
Deployment Considerations
Guidelines for extracting failure patterns, scoring arms, and responsibly deploying ActiveSaddler.
The extraction stage requires two mandatory skills: symptom‑extract to generate raw symptom candidates from each failed scenario, and symptom‑normalize to reconcile those candidates against existing patterns, ensuring atomicity and avoiding anchoring bias.
Arm scoring uses history‑analysis to retrieve the prepared harness’s evolution history, then progress‑scoring to assign a learning‑progress score $\phi$∈[0,1] based on severity, fixability, breadth, and side‑effect risk, which feeds a softmax sampler for the next iteration.
Deployment considerations stress that ActiveSaddler is a research prototype evaluated only on public benchmarks; safety, security, privacy, and governance controls must be added before any production use, with explicit human oversight and rollback mechanisms.
**Figure 11.** Evolution of pull-probability mass across failure-pattern arms. The ten arms with the highest average pull probability over the run are shown individually, while the remaining 25 arms are aggregated into Other arms. The upper panel shows the number of instantiated arms, which grows from 6 to 35 as new failure patterns are discovered. Pull probabilities are shown only at the 34 iterations where arm selection is performed.
**Figure 12.** Pull-probability trajectories for all failure-pattern arms. Rows show all 35 discovered arms ordered by instantiation time, and columns correspond to the 34 iterations in which arm selection is performed. Cell intensity denotes pull probability, orange markers indicate the arm sampled at each iteration, and gray cells denote iterations before an arm entered the pool.
```bash pattern rate -pattern-id <id> \ -severity <s> -fixability <f> -breadth <b> -side-effect <e> \ -rationale "..." ```
## Mandatory Skills You MUST read and follow the SKILL.md for each skill listed below. Do NOT skip or summarize the skill -- execute the full procedure described in it. This skill is installed at `.claude/skills/<name>/SKILL.md` in the current worktree. | Order | Skill | When | Why | |-----|-----|----|---| | 1 | `history-analysis` | Before the decision (Step 1) | Understand the Prepared harness and evolution history | ## Goal Decide whether this iteration should exploit a known failure-pattern arm (PULL) or explore unseen scenarios that may reveal a new failure type (DRAW). Record exactly one decision with `pattern decide`. This is an exploration-exploitation trade-off in **arm space**: - **PULL** spends this iteration on an already-discovered pattern. arm-scoring step then scores every known pattern so the sampler can choose an arm to repair. - **DRAW** probes scenarios that have never been executed. This may reveal a failure pattern not yet represented by the known arms; Arm scoring is skipped.