Scaling Harness Intelligence via Just-in-Time Harness Evolution

Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun Li, Yue Liao, Heng Chang, Xiaobin Hu, Qibing Ren, Wangchunshu Zhou, Shuicheng Yan

JIT-Agent synthesizes task-specific agent harnesses on the fly, outperforming fixed runtimes while reducing inference costs.

Can we improve agent performance by dynamically generating a task-specific "harness" (memory, planning, action, and tool protocols) at inference time, rather than using a static, one-size-fits-all agent scaffold?

Agent performance is currently bottlenecked by "Ahead-of-Time" (AOT) harnesses—fixed scaffolds that struggle to adapt to the diverse requirements of search, coding, and planning tasks. JIT-Agent treats the harness as a composable, machine-generatable artifact, using a meta-agent to synthesize a task-specific four-module scaffold (memory, planning, action, and capability orchestration) at inference time. This approach consistently improves performance across model families, often surpassing frontier models while simultaneously reducing token consumption and API costs by up to 54%.

Paper Primer

The core mechanism hinges on a four-tuple protocol $(M, P, A, F)$ that factorizes any agent runtime into memory, planning, action, and capability-orchestration modules. JIT-Agent is trained via a three-stage pipeline: Stage I learns task-conditioned customization, Stage II uses failed execution traces to learn reliable repair, and Stage III employs Evolutionary Group-Decoupled Policy Optimization (Evo-GDPO) to distill performance signals from an expanding archive of prior harness configurations.

JIT-Agent significantly elevates the performance of existing foundation models.

On the DeepSeek-V4-Flash backbone, JIT-Agent improved performance on DeepSearchQA by +9.1 points and OdysseyBench by +4.3 points compared to the vanilla model. Large absolute gains, including +20.2 points on DeepPlanning-Travel for GLM-5.2.

JIT-generated harnesses are more cost-efficient than mature, fixed-harness runtimes.

Across controlled evaluations, JIT-Agent reduced per-case API costs by 14.9% to 54.1% compared to fixed alternatives like Claude Code and NanoBot. Average cost reduction of 36.0% across six controlled backbone-benchmark settings.

Why is a "Just-in-Time" approach superior to optimizing a single, durable harness?

Different tasks have heterogeneous demands—such as parallel evidence exploration for search versus serial ReAct loops for terminal tasks—that a single AOT harness cannot satisfy without cumbersome, instance-agnostic search.

Does this method require retraining the underlying foundation model?

No; JIT-Agent acts as a meta-agent that generates an executable harness for an arbitrary, frozen off-the-shelf LLM, making it orthogonal to model scaling.

The Case for Dynamic Harnesses

We expose why static harnesses limit agents and how JIT‑Agent generates task‑specific harnesses.

Agent capability depends not only on the foundation model but also on the surrounding harness, which governs memory management, planning strategy, action execution, and tool orchestration. Existing harnesses are hand‑crafted per task, making them brittle, task‑specific, and fundamentally unscalable.

Prior work optimizes a single ahead‑of‑time (AOT) harness under the assumption that the same scaffold will serve many future tasks. This assumption breaks down when tasks demand divergent memory layouts, planning horizons, or tool sets, forcing costly redesigns for each new domain.

The harness is the operational wrapper that decides what history is kept, how intent is formed, which tools are exposed, and when verification or recovery occurs.

**Figure 1.** Leaderboard across four representative agent benchmarks. JIT-Agent-generated harnesses consistently improve the underlying backbone agents across deep research, daily work, planning, and workspace tasks, demonstrating that just-in-time harness synthesis provides gains beyond model scaling alone.

Agent performance is bottlenecked by static operational protocols.

JIT-Agent Architecture

JIT-Agent builds a task‑specific harness at inference time, enabling adaptive, reliable, and evolvable agents.

Static harnesses lock a model into a fixed memory, planning, and action protocol, which limits performance on tasks that demand different state organizations or tool use.

JIT-Agent is a compact meta‑agent that stitches together a custom harness at inference time, much like a tailor assembles a suit from the exact fabric pieces a client needs.

How is JIT‑Agent different from simply prompting a language model to write a harness?

Prompting asks the model to produce free‑form code, which often violates syntax or tool contracts. JIT‑Agent, by contrast, selects concrete module templates from HarnessFactory and fills them with task‑conditioned parameters, guaranteeing a syntactically valid, executable harness before any execution attempt.

JIT‑Agent reads the task spec and maps it to $M=2$ (store two recent queries), $P=1$ (single‑step plan), $A=$“retrieve”, $C=$“summarize”.

It selects the memory template `mem_small.py`, the planning template `plan_onehop.py`, the action template `act_retrieve.py`, and the capability template `cap_summarize.py` from HarnessFactory.

Each template is instantiated with the concrete parameters (e.g., buffer size = 2) producing four short Python modules.

The modules are concatenated into a single script `harness_search_faq.py` and handed to the LLM for execution.

After the first run, the trace shows a missing import error; Stage II repair rewrites the import line, updates the archive, and the next execution succeeds.

This walk‑through shows how a high‑level task description is turned into a concrete, executable harness by selecting and parameterizing a handful of predefined modules.

**Figure 2.** Overview of JIT-Agent. Given a task, JIT-Agent composes a problem-specific agent harness by instantiating (rather than simply combining) four modules: memory, planning, action, and capability. Different task structures therefore induce distinct executable protocols and state organizations, as illustrated by the specialized harnesses for deep research, product generation, and autonomous research.

JIT‑Agent harness synthesis – high‑level pseudocode.

The three‑stage training recipe first teaches JIT‑Agent to customize modules from teacher‑generated examples (Stage I), then to repair compilation or runtime failures (Stage II), and finally to propose ever‑better harnesses using Evo‑GDPO (Stage III).

Harness Engineering Landscape

Survey of static harness designs and their optimization trade‑offs.

Related work frames the problem space of agent runtimes. Across recent systems, the harness is recognized as the decisive layer that couples a foundation model to its environment.

A static harness is a fixed collection of modules that together decide what the model sees, remembers, plans, and how it invokes tools.

Harness engineering spans code‑generation runtimes (Claude Code, Codex, Hermes, OpenClaw) and memory‑centric systems (Mem0, EverMemOS, MemEvolve). These implementations differ in language, execution model, and the degree of tool integration they expose.

Once modularized, harness construction becomes an optimization problem over executable artifacts rather than a one‑off engineering decision. Recent work expands the search space from prompts to full workflow graphs, allowing feedback loops that edit both local instructions and global control flow.

The dominant paradigm has been ahead‑of‑time (AOT) synthesis, where a durable harness is built from a distribution of past experiences. By contrast, JIT‑Agent generates a task‑conditioned harness at inference time and can refine it on‑the‑fly, shifting the optimization burden from repeated artifact search to learned generation.

Modular Harness Design Space

Modular harnesses let JIT‑Agent replace a static wrapper with a task‑specific pipeline.

Static harnesses lock agents into a one‑size‑fits‑all runtime, capping performance on diverse tasks. By exposing a design space of interchangeable modules, JIT‑Agent can synthesize a harness that matches each task’s needs.

Think of the harness as a car: the memory tank stores past trips, the planner GPS decides the route, the capability gearbox selects which tools to engage, and the action accelerator drives the next move.

Step 1: $M$ sees empty history $\psi_{<1}=\emptyset$ and state $s_1$, returns view $v_1$ = “no prior info”.

$P$ maps $v_1$ to directive $d_1$ = “use next tool”.

$F$ filters $C_{\tau}$ with $d_1$, yielding $C_1=\{\text{calc}\}$.

$A$ executes $\text{calc}$, emits $e_1$ = result $42$, and updates state to $s_2$.

Step 2 repeats: $M$ now sees $\psi_{<2}$ containing $(s_1,e_1, o_1)$, returns view $v_2$ = “last result = 42”.

$P$ again emits “use next tool”, $F$ now selects $\text{search}$, $A$ calls it, emits $e_2$ = “found URL”, and terminates with $e_2\in Y$.

The four‑module factorization lets the harness reuse the same memory and planner across steps while swapping only the capability slot to pick a different tool.

How does this four‑module harness differ from a monolithic harness that mixes memory, planning, and action in one block?

In a monolithic design the three concerns share internal state, making it hard to replace just one component (e.g., swapping a planner). The modular factorization isolates each concern, so JIT‑Agent can keep $M$ and $F$ fixed while trying many $P$ implementations, dramatically expanding the searchable design space.

Training Stage I: Customization

Stage I learns task‑conditioned harnesses by imitating a frozen teacher and preferring efficient, high‑reward generations.

Static harnesses cannot adapt to the idiosyncrasies of a new task, so the generated harness may satisfy the protocol yet still waste latency or miss reward opportunities.

Instead of hand‑crafting a one‑size‑fits‑all harness, the model learns to synthesize a task‑conditioned harness that already obeys the four‑module protocol and is nudged toward higher reward, lower latency, and lower cost.

How does this preference learning differ from standard RL‑HF where a reward model ranks completions?

Here the ranking is based on three concrete execution metrics (reward, latency, cost) evaluated on the actual harness execution, not on a learned proxy. Moreover, the loss directly compares the model’s own probability ratios to a frozen reference checkpoint, whereas RL‑HF typically updates a policy against a fixed reward model.

Compute gaps: $r^{+}-r^{-}=3$, $\ell^{-}-\ell^{+}=0$, $\kappa^{-}-\kappa^{+}=0$.

Apply clipping: $[\ell^{-}-\ell^{+}]_{+}=0$, $[\kappa^{-}-\kappa^{+}]_{+}=0$.

Aggregate valuation: $\Delta_{\text{val}} = 1\cdot3 + 0.5\cdot0 + 0.5\cdot0 = 3$.

Assume the current model assigns $\log p_{\theta}(h^{+}\mid c_{\tau})=-1.2$, $\log p_{\theta}(h^{-}\mid c_{\tau})=-2.0$, and the reference checkpoint gives $-1.5$ and $-2.5$ respectively.

Compute log‑ratio difference: $\log\frac{p_{\theta}(h^{+})}{p_{\theta}(h^{-})}=0.8$, $\log\frac{p_{\text{ref}}(h^{+})}{p_{\text{ref}}(h^{-})}=0.8$.

With $\beta_{\text{pref}}=1$, the sigmoid argument becomes $0.8-0.8=0$, so $\sigma(0)=0.5$; the preference loss contribution is $- \Delta_{\text{val}}\log 0.5 = -3\cdot(-0.693)=2.08$.

The preference term only activates when the model’s ranking deviates from the reference; otherwise the loss is zero, preventing unnecessary pressure to change already‑good behavior.

Sample a task $\tau$ and retrieve three reference scaffolds $E_{\tau}$ from the seed bank.

Feed $(\tau,\Phi,C_{\tau},E_{\tau})$ to the frozen teacher $q_{\phi}$ to generate candidate harnesses $h_{\text{each}}$.

Run each candidate through the protocol validator $\text{Valid}_{\Phi}$ and the frozen executor $\pi_{\psi}$; discard any that fail.

Compute the supervised generation loss $L^{\text{gen}}_{\text{I}}$ on the tokens of all accepted candidates.

Form preference pairs $(h^{+},h^{-})$ by comparing reward, latency, and cost; evaluate $\Delta_{\text{val}}$ for each pair.

Apply the preference loss $L^{\text{pref}}_{\text{I}}$ using the log‑ratio formulation against the reference checkpoint.

Combine the two losses with weight $\lambda_{\text{pref}}$ and update $\theta$.

**Figure 3.** Training pipeline of JIT-Agent. Stage I learns task-conditioned harness customization. Stage II converts failed harnesses and execution diagnostics into bounded repair trajectories. Stage III performs online evolution by comparing candidate harnesses against the incumbent bank, optimizing decoupled reward, latency, and cost advantages, and retaining frontier-improving designs.

Training Stage II: Repair

Stage II repairs failed harnesses by iteratively applying teacher‑proposed patches.

When a generated harness almost works but crashes, the system treats the crash report as a clue and applies a tiny, teacher‑crafted patch to fix it, repeating up to two times.

Diagnostic $g(0)$ reports “NameError: name ‘np’ is not defined”.

Teacher selects $\Delta^{1}=$ “add import numpy as np” from $P$.

Apply inserts the import at the top of $h(0)$, producing $h(1)$.

Re‑validation succeeds, so $K^{\star}=1$ and the trajectory is kept.

This single‑line edit fixes the crash without altering any downstream logic, exemplifying the high‑leverage nature of patches in $P$.

How does this repair loop differ from ordinary fine‑tuning of the harness?

Fine‑tuning adjusts model weights over many epochs, whereas the repair loop keeps the harness code fixed and applies at most two discrete edits suggested by a teacher. The goal is immediate executability, not gradual performance improvement.

Collect failing harnesses $h(0)$ and their diagnostic reports $g(0)$ from Stage I.

For each failure, let the teacher propose a patch $\Delta^{k+1}$ from $P$ and apply it to obtain $h(k+1)$.

Validate $h(k+1)$; if executable within two steps, record the trajectory $R_{K^{\star}}$.

Assemble dataset $D_{II}$ containing task description, history $\{(h(j),g(j))\}$, and the teacher patches.

Train JIT‑Agent by minimizing $L_{II}(\theta)$, encouraging it to predict the next patch given the observed history.

Training Stage III: Evolution

Stage III optimizes JIT-Agent to iteratively synthesize harnesses that outperform prior designs in reward and efficiency.

Stage III treats harness design as an active optimization problem. Instead of merely imitating past designs, JIT-Agent learns to evolve its own harnesses at test time, aiming to surpass the current performance frontier in both reward and operational efficiency.

Evo-GDPO trains the agent to propose a group of candidate harnesses and select the best one by comparing them against a "bank" of high-quality prior designs, rewarding the model for both higher rewards and lower resource costs.

Sample a task $\tau$ and retrieve a reference set $E_{\tau,n}$ from the harness bank $B_n$.

Generate $G$ candidate harnesses using the current policy snapshot $\theta_{\text{old}}$.

Execute candidates and measure reward $r_i$, latency $\bar\ell_i$, and cost $\bar\kappa_i$.

Normalize advantage signals across the group to stabilize optimization.

Update the policy $\theta$ using the aggregated advantage and retain candidates that strictly improve the frontier.

Inference follows two paths: static inference uses parallel generation to boost diversity without extra rollouts, while streaming inference persists successful harnesses in the bank to transfer experience across a sequence of tasks.

Experimental Evaluation

JIT‑Agent delivers large performance gains across nine benchmarks.

Recall that a static harness caps an agent’s capability; JIT‑Agent creates a task‑specific harness at inference time.

Replacing the default scaffold with a JIT‑generated harness raises the average benchmark score by 8.8 points on DeepSeek‑V4‑Flash.

Table 3 shows the DeepSeek‑V4‑Flash average moving from 66.7 to 75.5.

Across both backbones the JIT‑generated harness improves every paired benchmark, with the biggest jumps on planning tasks (+24.8 on DeepPlanning‑Shopping, +20.2 on DeepPlanning‑Travel). Compared to frontier models, JIT‑Agent + GLM‑5.2 ranks first on seven of nine columns, and Table 4 shows it also reduces token usage and API cost.

**Table 3.** Main results across nine agentic benchmarks. Rows prefixed by JIT-Agent use the same backbone as their vanilla counterpart but replace its default scaffold with a JIT-generated harness. Scores are reported on a 0–100 scale; higher is better. Top-1, Top-2, and Top-3 results in each column are highlighted in soft green, yellow-green, and yellow, respectively; Top-1 is also bolded. Missing evaluations are denoted by “—”.

**Table.** Performance, token usage, and cost comparison across different backbones and harnesses on DeepSearchQA, xBench-DS, and AgentIF benchmarks.

**Figure 5.** JIT-generated harnesses consistently improve paired backbones over ReAct. Rows group three model families and two variants within each family; columns correspond to DeepSearchQA, AgentIF-Oneday, DeepPlanning-Shopping, and OfficeBench. Each panel compares the same backbone under a fixed ReAct harness (dashed) and a JIT-generated harness (solid), with callouts reporting the absolute score gain. DeepSearchQA uses a 100-example subset; the other three benchmarks use 50-example subsets.

Performance and Cost Analysis

Extended analyses quantify JIT‑Agent’s performance, efficiency, and generalization across models and tasks.

JIT‑Agent improves task performance by an average of 8.7 points over the strongest fixed harness across all benchmarks.

PinchBench shows JIT‑Agent + DeepSeek‑V4‑Flash leads DeepPlanning‑Shopping at 83.9 and beats the DeepSeek‑V4‑Pro baseline on every reported benchmark, yielding an average advantage of 8.7 points.

Under controlled backbones, JIT‑Agent outperforms the best fixed harness by up to 7.0 points.

On Qwen3.6‑Flash × xBench‑DS it scores 70.0 versus 63.0 for the strongest fixed harness, a 7.0‑point gain.

Across 24 backbone‑benchmark pairs, the JIT‑generated harness beats the ReAct baseline by 7.6 points on average.

DeepSeek‑V4 family gains 10.2 points, Qwen 3.6 gains 4.0 points, and Mimo 2.5 gains 8.6 points; DeepSearchQA shows the largest single gain of 15.2 points.

**Figure 4.** Cost–performance trade-offs on DeepSearchQA and AgentIF. Marker color identifies the harness, while filled and open circles distinguish DeepSeek-V4-Flash and Qwen3.6-Flash. The horizontal axis is API cost per case in USD and the vertical axis is task performance. The dark-green step line traces the global Pareto frontier, and the pale yellow-green and blue-green regions contain pairings dominated by at least one Pareto-optimal point.

**Figure 6.** Streaming test-time harness evolution across task streams. Cumulative accuracy (top), per-task API cost (middle), and per-task tool calls (bottom) on DeepPlanning-Shopping, DeepPlanning-Travel, and OfficeBench. Dashed curves represent Static JIT, where task-specific harness generations are independent, while solid curves represent Streaming JIT, which continuously incorporates execution feedback as new tasks arrive. Shaded regions show local variation around the streaming trajectories. Streaming JIT finishes with higher cumulative accuracy on all three benchmarks, while API-cost and tool-use trajectories remain task-dependent and of broadly similar scale.

Qualitative Harness Inspection

JIT‑Agent synthesizes task‑specific harnesses that boost performance and cost efficiency.

Quantitative analysis confirms that the harnesses generated by JIT‑Agent consistently accelerate execution and lower compute cost across all evaluated tasks.

**Figure 7.** Palimpsest: graph-planned artifact execution. GraphPlanPlanning converts the contact-processing request into a DAG; GraphPlanAction executes it with bounded width and depth, while GraphPlanMemory stores reusable artifacts and reasoning state.

**Figure 8.** Trapdoor: bounded research behind a delegated tool call. A synthesized delegate capability is intercepted by OrchestratorLoop, which runs a bounded subagent and writes extracted facts into FactGraphMemory.

The contrast between Palimpsest and Trapdoor illustrates the central qualitative finding: the same generator maps one task to graph execution with an artifact store and another to recursive orchestration with a fact store, all while respecting a shared four‑module interface.

JIT‑Agent, trained through customization learning, repair supervision, and Evo‑GDPO, turns harness construction from a static engineering effort into a learned capability, delivering consistent performance gains across daily‑work, planning, and deep‑research domains.

Future work envisions harness intelligence as a third scaling axis alongside model size and compute, ultimately leading to co‑design of foundation models and their operational scaffolds; production systems may retain a stable core while allowing selective, just‑in‑time redesign of harness components when tasks demand.

Additional Harness Visualizations

Appendix showcases eight extra JIT‑Agent harness visualizations and a hierarchical architecture diagram.

This appendix extends the two representative cases from Section 6.7 with eight additional harness visualizations, each generated under the four‑module protocol and tailored to distinct task structures.

**Figure A.1.** Origami: hierarchical work with folded context. ROMAPlanning creates isolated subtasks; HierarchicalMemory retains their trajectories and artifacts while `fold_thought` replaces only the active working context before aggregation.

Collectively, these cases demonstrate how JIT‑Agent selects a state representation and control regime suited to each request, instantiating planning, action, memory, and capability orchestration around that choice.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers