RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhang, Yoonho Lee, Chengsong Huang, Han Yu, Zhongying CuiZhu, Yifei Ming, Huaxiu Yao, Burak Gokturk, Tomas Pfister, Chen-Yu Lee
Regularized Recursive Self-Improvement (RRSI) constrains agent harness evolution to prevent overfitting and bloat.
How can we evolve LLM agent harnesses to improve performance without overfitting to the specific tasks used during the evolution process?
Automated agent harness evolution often overfits to the specific tasks used for training, leading to performance gains that vanish on held-out benchmarks. RRSI regularizes this recursive process by constraining how candidate edits are proposed and selected, favoring reusable mechanisms over benchmark-specific noise. Across eight benchmarks, RRSI improves performance on held-out tasks by up to 4.7 points while reducing the policy token cost compared to unregularized evolution.
Paper Primer
RRSI treats harness evolution as an adaptive search over an open space of prompts, tools, and control flows. It applies three regularization principles: annealed edit budgets to limit update capacity, evidence-aware credit assignment to track hypothesis utility, and conservative selection criteria to filter out noise and leakage.
The core mechanism is a dual-constraint loop: the proposer uses an annealed budget to bundle fewer edits in later rounds, while the selector acts as a gatekeeper: "Screen for leakage, justify cost with gain, and prune unproductive components." This ensures that only robust, efficient mechanisms persist in the harness.
RRSI improves generalization on out-of-distribution benchmarks.
Evaluation across eight benchmarks spanning coding, agentic workspace, and engineering design. Up to 4.7 points gain on out-of-distribution tasks, outperforming prior baselines by up to 22.9%.
Why does unregularized harness evolution fail to generalize?
It tends to encode benchmark-specific patterns, chase evaluation noise, and accumulate unnecessary complexity that improves scores on the training set without creating reusable agent mechanisms.
Does RRSI require a specific backbone model?
No; experiments with Claude Opus 4.8 and Gemini 3.5 Flash show that the learned harness mechanisms transfer across different policy families and even to smaller models not used during the evolution process.
For researchers building self-improving agents, RRSI demonstrates that controlling the dynamics of the evolution loop—rather than just the edit space—is essential for achieving transferable, efficient system improvements.
The Challenge of Harness Evolution
We expose how harness evolution overfits and introduce RRSI to close the generalization gap.
LLM agents rely on a harness—prompts, control flow, tooling, memory, and context management—that amplifies a frozen backbone model. Recent work automates harness edits by letting the system propose and select changes based on task feedback, effectively creating a form of recursive self‑improvement at the agent‑system level.
Iterative harness evolution tends to overfit to the tasks it is trained on, producing a sizable gap between performance on the evolve set and on unseen, out‑of‑distribution benchmarks.
**Figure 1 |** Evolution overfits on the split it is scored on whereas RRSI generalizes the improvements. (a) Gains on the evolve split against gains out of distribution for the agentic workspace benchmark. Prior methods retain little of their evolve-set gain and several end below $H_0$, the initial harness. (b-d) Out-of-distribution held-out score for $H_0$, average of the four baseline methods and our RRSI on SWE-bench Verified, the mean of JobBench, GDPval and APEX-Agents, and Frontier-Eng.
The key challenge is the gap between evolve‑set performance and out‑of‑distribution generalization.
Defining Agents and Harnesses
Define the agent harness and its evaluation metrics.
This section formalizes the notion of an agent harness and introduces the metrics used to evaluate its performance and token cost.
The harness $H$ is everything that surrounds a fixed policy $\pi$: prompts, control flow, tool interfaces, memory/skill files, and context management that together shape what the policy sees at each step.
Harness evolution treats $H$ as the optimization variable while keeping $\pi$ fixed, iteratively proposing and selecting candidate harnesses based on empirical estimates $\hat{S}$ and $\hat{C}$.
At each round the current harness $H_t$ is run on an evolve set $D_{\text{evolve}}$, feedback $F_t$ is collected, and a proposal process $P_0$ generates candidate harnesses which are evaluated and the best becomes $H_{t+1}$.
The RRSI Framework
RRSI regularizes recursive harness evolution to keep improvements generalizable.
Iterative harness evolution quickly overfits to the finite evolve set, turning every discovered tweak into a brittle shortcut that fails on new tasks.
RRSI keeps the harness edit space open but shapes the search trajectory with three proposal‑side and four selection‑side constraints, so each round makes a small, attributable change that survives only if it is robust and efficient.
Round 0: $b_0 = 3$ (full budget, three independent edits allowed).
Round 5: $b_5 = 2$ (mid‑schedule, two edits).
Round 10: $b_{10} = 1$ (final round, only a single edit).
The budget drops smoothly because $\cos(\pi t/T)$ moves from $+1$ to $-1$.
Annealed sparsity forces early rounds to discover coarse mechanisms while later rounds isolate the exact edit responsible for any gain, making credit assignment tractable.
Algorithm 1 – RRSI proposal side (high‑level pseudocode)
Algorithm 2 – RRSI selection side (high‑level pseudocode)
**Figure 2 | Overview of RRSI.** RRSI regularizes the search trajectory, not restricting the potential harness edit space: proposal-side constraints control how search capacity is used, while selection-side constraints control which measured improvements are allowed to become a permanent state.
**Algorithm 1** RRSI, proposal side. **Require:** $H_t$, history $\mathcal{L}_t$, round $t$ of $T$ **Require:** $b_{\min}$, $b_{\max}$, stall window $w$, noise band $\delta$ 1: $\mathcal{F}_t \leftarrow \text{ANALYZE}(H_t, \mathcal{D}_{\text{evolve}})$ 2: $b_t \leftarrow \lceil b_{\min} + (b_{\max} - b_{\min})\frac{1}{2}(1 + \cos \frac{\pi t}{T}) \rceil$ $\quad \triangleright L_0$-style edit-cardinality control 3: $\sigma_t \leftarrow \mathbb{1}[\hat{S}_t - \hat{S}_{t-w} \leq \delta]$ 4: $\mathcal{T}_t \leftarrow \{\ell_i : (t_i, \ell_i, \dots) \in \mathcal{L}_t\}$ 5: $\mathcal{U}_t \leftarrow \mathcal{K} \setminus \mathcal{T}_t$ 6: $\mathcal{E}_t \leftarrow (\sigma_t, \mathcal{U}_t, m_{\text{draft}})$ 7: $\mathcal{B}_t \leftarrow \{\ell \in \mathcal{T}_t : g_t(\ell) \leq 0\}$ $\quad \triangleright \text{Lasso}/L_1$-style pruning targets 8: $\mathcal{H}_t \sim P_{\text{reg}}(\cdot \mid H_t, \mathcal{F}_t, \mathcal{L}_t, b_t, \mathcal{E}_t, \mathcal{B}_t)$ 9: tag each atomic edit with component and hypothesis metadata 10: **return** candidates that pass the pre-evaluation screen
**Algorithm 2** RRSI, selection side. **Require:** screened $\mathcal{H}_t, (\hat{S}_t, \hat{C}_t), S^\star, \delta, k$ **Require:** $\beta_0, \beta_1, w_s, w_c, w_n$ 1: $\mathcal{A}_t \leftarrow \emptyset$ 2: **for** $H' \in \mathcal{H}_t$ **in parallel do** 3: $\quad \hat{S}', \hat{C}' \leftarrow \text{EVALUATE}(H', \mathcal{D}_{\text{evolve}}, k)$ 4: $\quad \Delta S \leftarrow \hat{S}' - \hat{S}_t; \Delta C \leftarrow (\hat{C}' - \hat{C}_t)/\hat{C}_t$ 5: $\quad \nu \leftarrow \nu_t(H')$ $\quad \triangleright$ new structural component types 6: $\quad$ **if** $\Delta S > \delta$ **then** 7: $\quad \quad c \leftarrow [\Delta C \leq \beta_0 + \beta_1 \Delta S]$ $\quad \triangleright$ gain-dependent cost rule, Eq. (7) 8: $\quad$ **else** 9: $\quad \quad c \leftarrow [w_s \Delta S - w_c \Delta C + w_n \nu > 0]$ $\quad \triangleright$ within-band rule, Eq. (17) 10: $\quad$ **end if** 11: $\quad g \leftarrow \text{DOMAINGUARD}(H_t, H')$ 12: $\quad$ **if** $\hat{S}' \geq S^\star - \delta$ **and** $c$ **and** $g$ **then** 13: $\quad \quad \mathcal{A}_t \leftarrow \mathcal{A}_t \cup \{H'\}$ 14: $\quad$ **end if** 15: **end for** 16: $H_{t+1} \leftarrow \arg \max_{H' \in \mathcal{A}_t} \hat{S}'$, or $H_t$ if $\mathcal{A}_t = \emptyset$ 17: $S^\star \leftarrow \max(S^\star, \hat{S}_{t+1})$ 18: record each measured edit with $a = 1$ iff its candidate is $H_{t+1} \neq H_t$ 19: **return** $H_{t+1}$
How does RRSI’s annealed update sparsity differ from a naïve $L_0$ penalty applied to a static model?
In a static $L_0$ penalty the set of parameters is fixed and the penalty merely discourages many non‑zero entries. RRSI’s sparsity budget $b_t$ is dynamic: it caps the number of *independent edits* per round and shrinks over time, forcing early broad exploration and later fine‑grained attribution. Moreover, the edit pool itself changes each round, so the constraint shapes the *trajectory* through hypothesis space rather than a single parameter vector.
Empirical Results and Generalization
RRSI’s performance, component impact, and token cost across three domains.
We evaluate RRSI on eight benchmarks spanning coding, agentic workspace, and engineering design, and we ask how it compares to prior harness‑evolution methods, whether gains transfer to unseen splits, which components matter, and what the token cost is.
RRSI outperforms all baselines on the out‑of‑distribution average by +3.9 points.
Table 2 shows RRSI achieving 43.6 versus the next‑best baseline 39.7.
**Figure 3 | Main results in all three domains.**
**Figure 4.** Cost of the final harness of each arm, measured on the evolve split of the agentic workspace instance. OOD Avg. is the mean over JobBench, GDPval and APEX-Agents. The shaded region in (a) is everything RRSI dominates: more policy tokens per trial for a lower out-of-distribution average.
Related Work
We survey how harness design and its evolution shape agent capabilities and generalization.
Agent harnesses—beyond the backbone model—shape what an agent can achieve, with prompt structure, tool interfaces, context compaction, and recovery logic all influencing task completion.
Recent work automates harness evolution by letting LLM proposers rewrite harnesses and retain edits that raise benchmark scores, evolving components such as skills, memory, or preference signals, yet this process inherits the same generalization risks as self‑improving agents that search over their own code.
Because the search is driven solely by suite scores, reported gains often fail to transfer to new suites; concurrent approaches address this by adding explicit generalization objectives or diversity‑preserving archives, while our method instead regularizes the search dynamics—assigning credit across the full evolution history, filtering task‑specific logic before scoring, and accepting edits against a noise‑adjusted baseline—to retain mechanisms rather than overfit to the evolution suite.
Baseline Methods and Harness Changes
We detail the baseline harness methods and the round‑level RRSI implementation.
We summarize the five baseline harness‑evolution methods and then describe the round‑level formulation and bookkeeping used in our experiments.
A meta‑level optimizer treats the harness code itself as the variable, proposing and evaluating new harness programs in an outer loop.
Algorithm 1 – RRSI proposal side (pseudocode)
Algorithm 2 – RRSI selection side (pseudocode)
This table summarizes various harness changes across different domains and rounds, detailing the outcome and the underlying principle illustrated by each change.
``` context_mgmt, client_tool, skill, memory, subagent ```
Evaluation Methodology
Evaluation details across eight benchmarks and their scoring protocols.
Each environment is run and scored under identical conditions: the same evaluation window, tool suite, judge model, and number of trials for both the harness and its baseline.
Terminal‑Bench 2.1 packages each of the 89 tasks as a container image with hidden unit tests; the agent interacts via a real shell, and a task counts as solved only when its test suite passes after the agent stops.
SWE‑bench Verified presents real GitHub issues paired with repository snapshots; the agent must produce a patch that flips failing tests to passing while preserving already‑passing tests, and the reported resolve rate is the fraction of instances satisfying both conditions.
Harvey LAB supplies source documents (Word, Excel, PDF) and a strict rubric of 20–100 criteria per task; scoring is the fraction of criteria passed, with a fixed 120‑task evolve split and a 40‑task held‑out split that remain constant throughout the experiment.
JobBench draws tasks from professional workflows, hides reference material, and exposes a filesystem, code execution, and web‑search tools; results are graded by the benchmark’s weighted rubric using the average of Gemini‑3.5‑Flash and Claude Opus 4.8 judges.
GDPval pits each harness‑generated deliverable against a human expert version; three judges (Qwen3.6‑35B‑A3B, Claude Sonnet 4.6, Gemini‑3.1 Pro) vote, and the win‑rate is the proportion of tasks where the harness wins the majority of the 204 comparisons.
APEX‑Agents evaluates 480 sandboxed tasks across three domains; a task counts as a success under pass@1 only when the single sampled rollout satisfies the full rubric, and failures due to infrastructure crashes are counted as failures.
EngDesign uses a license‑free subset of 61 engineering design tasks, each graded by a frozen deterministic simulator; no held‑out split is used because the suite is too small to allocate separate evaluation tasks.
Frontier‑Eng comprises 47 real‑world engineering optimization problems; each task awards gold, silver, or bronze credit (1, 0.67, 0.33) based on frozen v1 thresholds, and the overall Medal Score is the mean credit across the 38 tasks that could be built in our sandbox.
Hyperparameter Settings
Appendix C details RRSI hyperparameters and illustrative evolution decisions.
RRSI introduces a modest set of hyperparameters that govern update sparsity, exploration, pruning, and the cost‑performance trade‑off. These values are chosen solely within the evolve environment; no held‑out or out‑of‑distribution benchmarks influence them.
Table 5 lists the full hyperparameter matrix: edit‑budget bounds (`b_min`, `b_max`), exploration controls (w, `m_draft`), pruning window `n_prune`, and the base‑cost $\beta$₀ (fixed at 0.10) together with the instance‑specific $\beta$₁ values.
The qualitative case study (Table 6) inspects concrete decisions made during RRSI evolution, showing how the cost‑aware selection rule filters candidates beyond raw score improvements.
These examples illustrate three key patterns: (1) a measurable improvement is discarded if its cost‑increase exceeds the $\beta$₁ budget; (2) lower inference cost cannot rescue a candidate whose performance falls below the noise‑adjusted floor; (3) tiny, task‑agnostic control‑flow corrections are kept because they add negligible complexity.