Self-Evolving Coding Agents via Harness-Policy Optimization
Zhepei Wei, Wenlin Yao, Baolin Peng, Danqing Wang, Qianhui Wu, Hao Cheng, Yu Meng, Jianfeng Gao
Self-HPO recursively improves coding agents by alternating harness search and policy training.
How can we recursively improve both the coding agent's policy and its environment-interaction harness to solve complex software engineering tasks?
Coding agents rely on a harness—the scaffold controlling tools and feedback—to elicit capabilities from a policy model. Because a harness tuned for one model often fails to address the weaknesses of an updated one, agents frequently hit a performance ceiling where neither component can improve the other.
Paper Primer
Self-HPO functions as a self-improving loop: the policy acts as a harness proposer, and the harness acts as a targeted training environment. The core move is the evolving search set: instead of optimizing against a static task list, the system identifies tasks that the current policy failed to resolve and uses them to force the harness to adapt to the policy's current frontier.
Joint optimization of harness and policy yields gains exceeding the sum of individual component improvements.
Cross-evaluation shows that combining the optimized harness (H2) and policy (P2) achieves a 69.5% resolve rate, an 8.5-point gain over the base agent that surpasses the combined individual gains of 6.9 points.
Why is this approach better than simply fine-tuning the model on successful trajectories?
Fine-tuning (SFT) distills existing knowledge but does not adapt the agent's interaction environment to its specific weaknesses. Self-HPO's harness optimization intervenes in the agent's runtime logic—such as adding empty-patch guards or submission checks—which allows the agent to express capabilities it already possesses but previously failed to execute correctly.
Does this method require an external, more powerful model to propose harness changes?
No. While using an external model like Opus 4.8 provides a performance boost, the framework is designed to be self-improving; the current policy checkpoint can serve as its own harness proposer, still outperforming single-component baselines.
Researchers should treat the harness not as a fixed interface, but as a dynamic component that must co-evolve with the policy to maintain effective feedback loops.
Introduction and Motivation
Self-HPO tackles the static‑harness bottleneck by co‑optimizing harness and policy.
Current coding agents are hamstrung by a static harness: the external scaffold that decides what the model sees, how actions are executed, and when episodes end never adapts to the policy’s evolving weaknesses. Consequently, a policy can only learn from trajectories that the fixed harness elicits, and a harness tuned for one policy checkpoint may miss the failure modes of an updated policy.
Agents fail because their interaction scaffold is frozen; letting the policy itself propose and refine the harness lets each component target the other's current blind spots.
**Figure 1.** Self-HPO recursively improves the harness and policy using the agent itself. Left: In each H-step, the current policy proposes an improved harness, and each P-step then updates the policy through reinforcement learning with the optimized harness. Right: Average resolve rate over SWE-Bench Verified, SWE-Bench Multilingual, and SWE-Bench Pro along the H- and P-steps. Policy-only and Harness-only denote the RLVR (Peng et al., 2026) and Self-Harness (Zhang et al., 2026a) baselines.
**Figure 6.** Best-so-far performance during harness evolution on SWE-Bench Verified. Circles show individual candidate scores. Candidates 1–6 are shared, while 7–12 belong to the separate Self-Harness and Self-HPO branches. Vertical increases and stars mark Self-HPO’s two P-steps. Dashed horizontal lines show Base and final RLVR performance.
The harness is as critical as the policy model for agent success.
The Self-HPO Framework
Self-HPO iteratively co‑optimizes harnesses and policies to close failure loops.
The core of Self‑HPO is an alternating loop that refines a harness and a policy in lock‑step. Each iteration first fixes the policy to improve the harness (H‑step), then fixes the harness to improve the policy (P‑step). This loop drives the agent toward solving ever‑harder software‑engineering tasks.
The loop treats the harness and the policy as co‑evolving partners: the policy exposes failures, the harness repairs them, and the repaired harness gives the policy a richer playground to learn from.
H‑step iteration 1: propose $H_0^{1}$, evaluate on $S_0$, obtain score 0.6 (solves $x_2$ as well).
H‑step iteration 2: propose $H_0^{2}$, evaluate, obtain score 0.55 (no improvement).
Select best candidate $H_1 = H_0^{1}$ (score 0.6).
P‑step: with $H_1$ fixed, run RL; the policy now solves both tasks, yielding reward 1 for each rollout.
Hard‑task identification: no tasks remain unsolved, so $S_1 = \emptyset$ and the loop terminates.
The toy example shows how a single harness improvement can turn a previously hard task into an easy one, causing the subsequent policy update to achieve perfect solve rates.
How does this loop differ from ordinary RL where only the policy is updated?
Ordinary RL keeps the environment (here the harness) static, so the policy can only learn to work within the same constraints. Self‑HPO actively changes the environment between policy updates, expanding the set of behaviours the policy can express and thereby exposing new learning signals.
During H‑step the policy is frozen; the harness is treated as a mutable “prompt” that the policy itself critiques and rewrites to fix observed failures.
Why not simply fine‑tune the policy instead of modifying the harness?
Fine‑tuning the policy cannot introduce new primitives (e.g., additional tool calls or prompt structures) that the current harness lacks. Some failures stem from missing context or tooling, which only a harness change can resolve.
With the harness fixed, the policy learns to exploit the newly available capabilities, using reinforcement learning guided by binary success signals.
What would happen if the harness were changed during the P‑step?
Changing the harness mid‑update would break the on‑policy assumption of GRPO/BAR, causing the advantage estimates to become inconsistent and destabilising the policy gradient update.
**Figure 2.** **Overview of Self-HPO.** Top (H-step): The frozen policy analyzes the agent's failures on the search set and proposes new harness candidates. Bottom (P-step): With the updated harness, the policy evolves through reinforcement learning. The updated policy becomes the next harness proposer, while tasks never resolved during this P-step form the next search set.
**Harness Proposer System Prompt** You are the harness optimizer for a fixed policy {$P_t$}. The search set contains {N} hard training tasks that the policy never solved during the preceding P-step. The parent harness is {`H_best`}, the current best harness. The starting harness of this search solves {s}% of the search set under the frozen sampling configuration. Your goal is to design ONE new harness mechanism, of your own design, that is added on top of the parent harness and solves more search-set tasks than the parent. CONSTRAINTS - Keep every method of the parent harness verbatim and add exactly one new mechanism. - The policy, sampling configuration (temperature 0.95, top-p 0.95), step limit, maximum tokens, and verifier are frozen. - Never inject reference patches or task-specific facts into the agent's context; the mechanism must compute its own signal at runtime. EXPERIENCE (read it before designing; treat it as evidence, not a mandate) - Candidates evaluated so far in this search: scores, code, and trajectories. - Notes left by earlier proposals in this search. Diagnostics computed from the starting harness's runs on the search set: - Coverage gap: the source files required by the reference fix versus the files the agent touched. - Precision gap: failures in which every required file was edited but the fix was incorrect. - Submission loss: failures in which a real source edit appears in the trajectory but the final submitted patch is empty or malformed. - Reference-fix archetypes (multi-file, signature change, new import, guard, error handling, ...) versus what the agent produced. Decide for yourself which failure is the largest and most reachable by a harness, or find another one. WORKFLOW 0. Review the experience: the earlier notes and every candidate evaluated so far. For the most recent one, read the tasks it made worse and better than the starting harness (trajectories and reference fixes), and work out why. 1. Read several failing trajectories and compare a few with the reference fix to form your own hypothesis about the dominant failure cause. 2. Design a mechanism with a detectable trigger in the agent's own state and a concrete action: for example, intercept a submission and return feedback so that the agent gets another turn to act on it, replace the submitted patch, or act at any step. 3. Implement it in a copy of the parent harness. All repository access must go through the sandbox environment. 4. Write a one-line purpose (diagnosed cause -> mechanism), append what you learned this round to the notes, and stop. The search then checks that the candidate keeps every parent method and fires on real sandbox tasks, and returns any failure to you to fix.
**(3) Contract grounding: `GROUND_TEXT`** [GROUND YOUR DIFF IN THE REPO'S EXISTING TESTS -- one-time check; your submission is NOT finalized yet] The most common way this task is failed is submitting a patch that APPLIES cleanly but implements the WRONG contract -- a guessed function/option name, a signature the codebase never uses, a rename or new spelling that no caller expects, or an error/return shape different from what the code around it uses. Re-reading the PR will NOT catch this, because your reading of the PR is what produced the wrong contract. The codebase itself is the authority. The repo ships EXISTING tests that already exercise the code you touched and the API next to it -- they show the EXACT names, signatures, argument spellings, and error/return types the code is expected to have. Use them as ground truth now. 1. Print your changes and identify the concrete symbols you added or changed: `git --no-pager diff` List every NEW or RENAMED symbol/option/message string in your diff. 2. For each such symbol, search the repo's tests and existing call sites for how that symbol -- and the nearest EXISTING sibling API -- is actually used: `grep -rn --include='*.py' -e '<symbol>' -e '<sibling_api>' . | head -40` Then OPEN and READ the specific test files that reference the area you changed (e.g. tests/ or test_*.py near your file) and note the exact spelling, call signature, and expected result/exception the codebase uses. 3. Confront your diff against what the tests/call sites actually expect. If you introduced a name, option spelling, signature, or error shape that NO existing test or caller uses -- while an existing convention does -- change your diff to MATCH the codebase's convention. Do not invent a new API; conform to the one already there. Also confirm you did not MISS a behavior the tests imply is required. 4. If you changed anything, regenerate the patch: `git diff -- <the source files you modified> > patch.txt` and re-inspect patch.txt so it contains ONLY your intended source changes. 5. When your diff conforms to the codebase's real contract, re-issue the EXACT submit command: `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && cat patch.txt` This check happens only once -- after you re-submit, it goes through.
```bash cd /testbed && python3 -m pytest tests/test_ext_autodoc*.py tests/test_util_typing.py \ tests/test_ext_inheritance_diagram.py -x 2>&1 | tail -20 ```
**Algorithm 1** Self Harness-Policy Optimization (Self-HPO) **Require:** Seed harness $H_0$, base policy $P_0$, training data $\mathcal{D}$, harness search budget $K$ 1: $t \leftarrow 0$, randomly initialize harness search set $\mathcal{S}_t \subset \mathcal{D}$ 2: **while** not converged **do** /* H-step: search harness with the current policy as proposer */ 3: Score $v_0$, trajectories $\mathcal{T}_0 \leftarrow \text{Eval}(H_t, P_t, \mathcal{S}_t)$, experience $\mathcal{E} \leftarrow \{(H_t, \mathcal{T}_0)\}$, $v_{\text{best}} \leftarrow v_0$, $H_{t+1} \leftarrow H_t$ 4: **for** $k = 1, \dots, K$ **do** 5: $H_t^k \leftarrow \text{ProposeHarness}(H_{t+1}, P_t, \mathcal{E})$ 6: $v_k, \mathcal{T}_k \leftarrow \text{Eval}(H_t^k, P_t, \mathcal{S}_t)$, $\mathcal{E} \leftarrow \mathcal{E} \cup \{(H_t^k, \mathcal{T}_k)\}$ 7: **if** $v_k > v_{\text{best}}$ **then** 8: $H_{t+1} \leftarrow H_t^k, v_{\text{best}} \leftarrow v_k$ /* P-step: train the policy under the updated harness */ 9: $P_{t+1}, \mathcal{T}_{\text{RL}} \leftarrow \text{RL}(H_{t+1}, P_t, \mathcal{D})$ 10: $\mathcal{S}_{t+1} \leftarrow \text{SelectHardTasks}(\mathcal{T}_{\text{RL}})$ 11: $t \leftarrow t + 1$ 12: **return** $H_t, P_t$
Empirical Results
Self‑HPO outperforms static‑harness baselines across SWE‑Bench benchmarks.
Self‑HPO achieves the highest resolve rate of 69.5 % on SWE‑Bench Verified, surpassing the next best baseline by at least 2 percentage points.
Table 1 shows Self‑HPO (ours) reaching 69.5 % while RLVR and Self‑Harness attain 67.5 % and 65.2 % respectively.
SWE‑Bench is a collection of real‑world coding tasks where each task is a GitHub issue and success is measured by whether the generated patch passes an automated verifier.
Why does improving the resolve rate on Verified matter for the other splits?
Verified tasks are the most rigorously checked; a higher resolve rate indicates that the agent has learned robust code‑generation patterns that transfer to the less‑constrained Multi and Pro splits.
mini‑SWE‑agent is a lightweight coding agent that uses the Qwen 3.5‑35B‑A3B model as its policy and a simple harness to interact with the repository.
How does mini‑SWE‑agent differ from a full‑scale SWE agent?
mini‑SWE‑agent strips away auxiliary components (e.g., large retrieval modules) and focuses on the core policy‑harness interaction, allowing clearer attribution of performance gains to harness optimisation.
**Figure 3.** Optimization trajectories across harness and policy updates. Resolve rates (%) on SWE-Bench Verified, SWE-Bench Multilingual, and SWE-Bench Pro are shown on the common Base-$H_1$-$P_1$-$H_2$-$P_2$ schedule. Self-HPO alternates H- and P-steps, whereas Self-Harness and RLVR update only the harness and policy, respectively. Markers with dashed outlines indicate steps at which the corresponding method performs no update.
**Figure 4.** **Pass@k performance on SWE-Bench Verified and RL training signal.** (a) pass@k curve of Base and Self-HPO; (b) pass@1 by task difficulty, categorized by Base success rate p: Hard (p = 0), Medium (0 < p ≤ 0.5), Easy (0.5 < p < 1), and Resolved (p = 1). (c) Fraction of zero-success rollout groups for RLVR and Self-HPO’s final P-step.
**Table 1.** Performance comparison of different optimization methods across SWE-bench benchmarks.
**Table 2.** Ablation study. Resolve rates reported on SWE-Bench Verified.
**Table 3.** Cross-evaluating Self-HPO policies and harnesses on SWE-Bench Verified. Rows vary the harness and columns vary the policy.
The table presents a comparison of various models evaluated on the mini-SWE-agent harness, reporting Total Parameters, Active Parameters, and Resolve Rate. The table is divided into two sections: "Frontier models" and "Qwen3.5-35B-A3B" variants, with the latter showing the impact of different training stages culminating in "Self-HPO (ours)".
Self‑HPO consistently outperforms static‑harness baselines, confirming the benefit of joint harness‑policy optimisation.
Harness Evolution Analysis
Ablation shows how each harness check prevents failure and enables successful patches.
We compare two runs of the final Self‑HPO policy P₂ on the SWE‑Bench task sphinx‑doc__sphinx‑9658: one under the seed harness H₀ and one under the evolved harness H₂.
The harness can block or correct agent actions that would otherwise produce an invalid or empty submission.
How do these failure‑mode checks differ from ordinary error handling in software agents?
Ordinary agents typically raise an exception when something goes wrong; here the harness actively inspects the agent’s output (budget, patch content, intent match) and intervenes before the submission is accepted, turning a silent failure into a corrective feedback loop.
Under H₀ the agent’s final summary is accepted as the submission, leaving the bug unresolved; under H₂ the harness intercepts the empty patch, forces the policy to emit a proper diff, and the task is solved.
**Figure 5.** Seed versus optimized harness (HPO, second H-step). Left: the default agent loop, with the four failure modes it lets through (X). Right: the optimized harness subclasses the seed agent and overrides only `step()`, and each added check (✓) addresses the matching failure mode. Shared functions and classes use the same color on both sides. See Figure 8 for a detailed diff.
**Figure 8.** Harness changes from HPO’s second H-step. Shown as a diff against the seed agent’s `step()` (+ added, - removed). Helper calls are condensed into short names. The message texts are listed in Figure 9.
**Figure 11.** Success case: policy $P_2$ with the evolved harness $H_2$. The harness holds the first submission, and the policy then submits a valid patch that resolves sphinx-doc__sphinx-9658.
Related Work and Conclusion
Survey of prior harness and policy work, then recap Self‑HPO’s contributions.
Harnesses shape how LLM agents interact with tools and environments. Prior work optimizes prompts, programs, contexts, and entire workflows, and agents reuse feedback through refinement, reflection, skills, and memory. These approaches keep the task‑solving policy static while the harness adapts.
Policy learning literature focuses on iterative training via filtered reasoning, refreshed preferences, self‑play curricula, negative teachers, or evolving environments. Software‑engineering agents further benefit from executable environments, supervision from software evolution, and self‑play code repair. All these methods improve the policy under a fixed harness.
Joint harness–policy evolution research explores prompts + weights, executable harnesses + policies, and policies with evolving skills or external state. Self‑HPO stands out on two axes: (1) the P‑step policy becomes the next H‑step harness proposer, unlike WHALE’s frozen proposer or Evo²’s separate proposer; (2) the harness search set is drawn from tasks unsolved during RL, whereas competing methods use a fixed set. Appendix C details further comparisons.
**Table 5.** Conceptual comparison of Self-HPO and related methods. Coding indicates whether the method is evaluated on coding tasks. Policy update indicates whether the task-solving policy is frozen or updated through training such as supervised fine-tuning (SFT) or reinforcement learning (RL). Harness proposer is the model that modifies the harness, which can be either frozen or updated across rounds. Harness search set denotes the task set used to collect diagnostic trajectories for harness optimization. In Self-HPO, the evolving policy itself proposes harness revisions, and hard tasks identified during training form the next harness search set, keeping harness optimization targeted to the policy’s evolving failure modes.
We introduced Self‑HPO, a recursive framework that alternates harness and policy optimization for coding agents. The evolving policy proposes its own harness revisions, learns through RL under the selected harness, and supplies hard tasks for the next harness search. Across three coding benchmarks Self‑HPO raises the average resolve rate from 57.8 % to 64.8 % and improves reliability (pass@1) and capability (pass@k) over the base agent.
Implementation Details
Implementation specifics of the agent, harness search, and policy training.
We build the agent on the Qwen3.5-35B-A3B model (Qwen Team, 2026) and serve it with SGLang (Zheng et al., 2024). RL and SFT training run on a single node equipped with eight B200 GPUs, orchestrated by the slime framework.
The seed harness follows the mini‑SWE‑agent interface, allowing the model to execute bash commands inside a task‑specific repository environment.
Optimization data start with an initial search set $S_0$ of 300 tasks randomly drawn from the training corpus $D$. For later H‑steps we reuse task IDs that were identified as hard—i.e., never resolved in any rollout of the preceding P‑step—and generate fresh failure evidence with the updated policy.
Self‑HPO treats the current policy checkpoint as both task solver and harness proposer. The proposer inspects the experience of evaluated harnesses, including failed trajectories, and revises the best harness. When Opus 4.8 is used, it replaces the proposer while preserving the alternating H‑step–P‑step loop. Each H‑step evaluates $K=6$ candidate harnesses, each run once per search task; the chosen harness remains fixed during the following policy update.
Balanced Adaptive Rollout (BAR) generates eight rollouts per task, selects four to form a reward‑balanced group whose success fraction lies between 0.25 and 0.75, and repeats until 32 groups satisfy the filter, yielding 128 training trajectories per batch. Each batch (global size 128) is optimized for one epoch using Adam with a constant learning rate $10^{-6}$, weight decay $0.01$, entropy coefficient $10^{-4}$, and no KL penalty. Rollouts run at temperature $0.95$ and top‑$p$ $0.95$; RLVR follows the same configuration under the seed harness with 50 updates per P‑step.
Evaluation and Harness Details
Evaluation setup and observed performance trends for Self‑HPO on SWE‑Bench
All evaluations run with a hard step limit of 250 and sampling parameters temperature = 0.95 and top‑p = 0.95. When a task attempt fails due to sandbox or API errors it is automatically retried, and resolve rates are computed over the full set of attempts for each benchmark.
The SFT baseline in Table 1 fine‑tunes the base policy on Opus 4.8 trajectories, discarding any that did not resolve. Training runs for two epochs with batch size 64 and a constant learning rate of 5 × 10⁻⁶ after a 10 % warm‑up phase. In Table 2 the same SFT procedure replaces the RL component in the P‑step of the HPO loop.
After the shared first H‑step reaches 64.3 %, a harness‑only search pushes the best score to 65.2 %. Self‑HPO then updates the policy to 67.8 % before resuming search; the second H‑step climbs to 68.6 % and finally 69.5 % after the second policy update. The gap between the two branches demonstrates the advantage of interleaving policy updates with continued harness search.
Table 4 reports resolve rates on the SWE‑Bench Verified leaderboard for several frontier models and for Qwen3.5‑35B‑A3B using the methods of Table 1. Self‑HPO’s results are highlighted in bold; parameter counts follow the values listed in the developers’ model cards, with “–” indicating missing data.