Training Agents to Evolve with Their Harness

TaoLive AIGC LLM Team, Yuhan Sun, Wenhao Lin, Yongdong Luo, Yibo Hu, Meiguang Jin, Junfeng Ma, Weihang Pan, Jiaxin Zhao, Zulong Chen

Harness-Aware Training (HAT) enables compact agents to adapt to evolving business rules without retraining.

How can we train digital-avatar agents to remain robust and effective when their business-logic "harness" (prompts, tools, and skills) evolves frequently?

Live-streaming digital avatars require frequent updates to their business logic—such as new product rules or marketing hooks—but retraining the underlying model for every change is too slow, and fixed-model agents overfit to specific configurations. The authors propose Harness-Aware Training (HAT), which exposes the model to diverse, augmented harness configurations during training so it learns to interpret the current environment rather than memorizing static identifiers. This approach maintains high performance across evolving harness states while preserving general instruction-following capabilities, achieving a 94.8 score on live-stream QA and meeting real-time latency constraints.

Paper Primer

The core mechanism, Harness-State Augmentation (HSA), acts like a translator that constantly shuffles the labels and structure of the agent's environment. By renaming skills, reordering prompt instructions, and injecting synthetic tool definitions during training, the model is forced to rely on the functional meaning of the harness rather than memorizing fixed strings or templates.

The training pipeline follows a three-stage progression: HSA-SFT for supervised learning on diverse environments, General On-Policy Distillation (General OPD) to recover general-purpose reasoning, and Agentic RL with HSA to optimize tool-calling and robustness in a simulated live-streaming environment.

HAT prevents the generalization degradation typically caused by domain-specific fine-tuning.

While standard Fixed-Harness SFT causes a 7.7-point drop on the IFEval benchmark, HAT maintains a strong score of 83.5, effectively preserving general instruction-following ability. +2.0 points over the base model on IFEval, compared to a -7.7 point drop for standard SFT.

HAT achieves superior robustness to harness variations compared to fixed-configuration training.

On the Harness-Variant QA benchmark, HAT achieves an average score of 94.6, whereas the base model scores 75.4 and Fixed-Harness SFT scores 88.2. 19.2-point improvement over the base model on variant-heavy tasks.

Why is "surface-form overfitting" a critical failure mode for these agents?

When a model is trained on a single, static harness, it learns to associate specific skill names or prompt templates with actions. When the business logic evolves—renaming a tool or reordering a prompt—the model fails because it cannot interpret the new configuration, treating it as an entirely different task.

Why not just use a larger, zero-shot capable model that can adapt to any harness?

Large models possess the necessary generalization but fail the strict real-time latency requirements of live-streaming commerce. The authors report that a strong zero-shot model (DeepSeek-V4-flash) has a median latency exceeding 11 seconds, whereas the HAT-trained compact model achieves a P50 latency of 3.4 seconds.

HAT demonstrates that by treating the execution environment as a variable distribution rather than a static constant, developers can build compact, low-latency agents that evolve alongside business requirements without the overhead of continuous retraining.

The method relies on the assumption that the "change envelope" of future harness updates can be effectively simulated during training; it may not generalize to radical architectural shifts in the harness that fall outside the augmented distribution.

The Challenge of Evolving Harnesses

Live‑stream avatars need fast, adaptable, accurate responses, but fixed‑harness training makes them brittle.

Live‑stream e‑commerce avatars must satisfy three strict requirements: ultra‑low latency to keep viewers engaged, rapid adaptation to ever‑changing campaign rules, and high‑accuracy, effective replies that avoid hallucination.

To enable fast updates, the system adopts an evolvable Harness that isolates Skills, Hooks, the system prompt pipeline, and the tool registry so they can be edited without retraining the policy model. However, compact models trained on a single, fixed Harness overfit to the concrete skill and tool names, and when the Harness evolves they lose performance, as evidenced by a 7.7‑point IFEval drop after Fixed‑Harness SFT.

Harness‑Aware Training (HAT) mitigates this brittleness by exposing the model to a distribution of Harness states during training, achieving 94.8 % on Live‑Stream QA (base 80.3 %) and 94.6 % on Harness‑Variant QA (base 75.4 %). The deployed agent meets real‑time constraints with 3.4 s P50 latency and 8.1 s P95 latency, and an online A/B test reports a 4.33 % GMV uplift and a 0.91 % increase in item‑page views over the ReAct baseline.

**Figure 2.** Harness Evolution with a fixed DeepSeek-V4-flash on dev-set (n = 482). Optimizing the agent without training model. Evolution 2 is the selected development checkpoint Harness, while later long-tail edits introduce regressions.

The brittleness of fixed‑policy agents in dynamic e‑commerce environments drives the need for Harness‑Aware Training.

System Architecture

Separating the policy model from a mutable Harness lets operators update business logic without retraining.

Operators must frequently change prompts, tools, or business rules, but retraining a large language model for each change is costly and slow.

The runtime isolates a static policy model from a rapidly mutating Harness state, so any business‑logic update touches only the Harness components.

Load and normalize the viewer request “What is the price of product #7?” together with the live‑room context.

Assemble the prompt by concatenating the base instruction, the active FAQ Skill, the Pricing Skill, and the `price_lookup` Tool signature.

Run the policy loop: the model queries the assembled prompt, decides to call `price_lookup`, receives the price, and drafts a reply.

Apply the Hook check, which verifies that the reply matches the required JSON schema; it passes.

Dispatch the final reply to the TTS module and avatar generator for streaming.

This concrete flow shows that only the Harness components (Skills, Prompt, Hook, Tool) change across deployments, while the underlying policy model stays untouched.

How does this differ from the usual fine‑tuning approach where the model itself is updated?

Fine‑tuning rewrites the model’s weights for every new business rule, requiring expensive GPU cycles and risking catastrophic forgetting. The Harness runtime instead swaps out lightweight modules (Skills, Hooks, etc.) that are orders of magnitude smaller and can be hot‑reloaded without touching the model.

Instead of retraining, the system iteratively refines the Harness state through a human‑in‑the‑loop cycle while keeping the policy model frozen.

Diagnose clusters several failed discount queries and suggests adding a Discount FAQ Skill.

Confirm: The developer approves the addition.

Edit Harness: The new Skill module is version‑controlled and injected into the System Prompt Pipeline.

Evaluate: On the dev set, Accuracy climbs from 92.13 % to 92.55 % and Effectiveness rises from 84.16 % to 92.75 %.

Regression Check: No metric drops, so the edit is promoted.

The loop demonstrates that substantial performance gains can be realized solely by tweaking Harness components, confirming the core premise of Harness‑Aware Training.

Why not fully automate the loop instead of keeping a developer in the middle?

Full automation would rely on imperfect failure diagnostics and could introduce regressions that a human reviewer would catch; the current design balances speed (quick Harness edits) with safety (human confirmation) to maintain production reliability.

**Figure 1.** Architecture of the live-streaming digital-avatar system and its Harness Agent runtime. (a) Viewer requests and live-room context are processed by the interaction agent. The resulting text response is converted to speech, rendered by the avatar generator, and broadcast to the live room. (b) The expanded runtime operates under an evolvable Harness state $h = (\mathcal{S}, \mathcal{T}, \mathcal{P}, \mathcal{K})$, comprising active Skills, the tool registry, the assembled system prompt, and lifecycle Hooks, respectively. In each agent-loop round, Hooks mediate input processing, model inference, tool execution, and stopping, while the policy may load Skills, invoke externally mapped tools, or propose a final response. A successful stop check emits the response; a failed check returns control to context assembly for the next round.

The task interface supports both single‑comment and multi‑comment modes, with an intent mix dominated by product Q&A (~46 %). Runtime interfaces listed in Table 11 expose retrieval, commerce, and flow‑control operations, while Table 12 enumerates the versioned Skill modules that the Harness can load on demand.

Harness-State Augmentation

Introduce diverse harness states to prevent overfitting and improve robustness.

When a harness version changes, a model that has memorized the previous configuration becomes stale, leading to brittle behavior in production.

HSA generates diverse, task‑preserving variants of the harness so the model cannot rely on fixed identifiers or exact wording, forcing it to learn the underlying functionality.

Original skill list: {SkillA, SkillB}. After adding synthetic skill: {SkillA, SkillB, SkillC}.

Rename SkillA → “$S_A$”; SkillB rules masked → only half remain.

ToolX renamed to “$T_X$” while its API remains unchanged.

Prompt blocks reordered: Block2 now precedes Block1.

The resulting harness state $h' = (S'={S_A, SkillB, SkillC}, T'={T_X}, P'=reordered, K$ unchanged$)$.

Even with these superficial changes, the functional behavior of the skills and tool stays the same, but any model that relied on exact names or rule order would be confused.

**Figure 3.** Overview of harness-state augmentation and judging. *Left:* Examples of augmenting tools, skills, hooks, and prompts. The augmented harness is shared across training stages: a teacher model generates candidate trajectories for SFT, while the policy model produces rollouts for RL. *Right:* Representative RL trajectories. From top to bottom, Trajectory 1 invokes Tool A, which is unavailable after augmentation, and is penalized by *Tool Rationality*; Trajectory 2 selects a nonexistent Skill $\alpha$ and is penalized by *Skill Selection*; Trajectory 3 violates the modified prompt and is penalized by *Accuracy*; and Trajectory 4 correctly adapts to the augmented harness and is rewarded. During SFT, the same judges guide rejection sampling. Together, augmentation and judging teach the model to understand the current harness rather than overfit to a fixed configuration.

Simulated Environment

We build a live‑streaming simulator that lets the model train on its own on‑policy failures.

Offline teacher demonstrations only show the ideal path, leaving the model blind to the errors it will encounter when deployed.

The simulator is a flight‑simulator for the agent: it reproduces the full Harness control flow and deliberately injects emergencies so the model can practice recovery without crashing a real service.

Round 1 – Skill routing chooses “ProductInfo” (skill ID = 1).

Tool call “`price_lookup`(item=42)” returns a timeout error (failure flag = 1).

Hook “`timeout_handler`” activates, resetting the tool call flag and incrementing the retry counter.

Round 2 – The same tool call is re‑executed and returns price = \$19.99.

Hook validates the response format, allowing the model to generate the final user reply.

The simulator forces the model to experience error‑recovery loops, teaching it to anticipate and handle failures rather than assuming every tool call succeeds.

How does this simulator differ from simply augmenting offline data with random noise?

Random noise changes surface tokens but leaves the control‑flow structure untouched. The simulator, by contrast, reproduces the full multi‑step Harness execution—including skill routing, hook triggers, and tool failures—so the model learns the consequences of its own actions, not just altered inputs.

Harness-Aware Training (HAT)

We introduce Harness‑Aware Training, a three‑stage method that robustly adapts agents to evolving harnesses.

Standard fine‑tuning locks a model to the exact prompt‑tool configuration (“harness”) it sees during training. When the production harness changes—new tools, altered prompts, or different formatting—the policy overfits and fails, making the agent brittle.

Instead of training on a single static harness, we expose the model to many task‑preserving harness variants so the policy learns to ignore superficial harness details and focus on the underlying intent.

How does HAT differ from ordinary fine‑tuning on a single prompt set?

Ordinary fine‑tuning treats the prompt‑tool configuration as immutable, so the model memorizes exact strings. HAT deliberately samples many “what‑if” harnesses, forcing the policy to base its decisions on the underlying intent rather than surface wording.

Having built a harness‑aware policy, we now train it with on‑policy reinforcement learning that respects the augmented harness distribution. This is the HSA‑RL stage, which uses Group Relative Policy Optimization (GRPO) and Group‑reward Decomposed Policy Optimization (GDPO) to handle multiple reward dimensions per state group.

During each rollout we first flip a coin to decide whether to use the production harness or an augmented version, then generate a full interaction trajectory; the policy is rewarded for both correct tool usage and concise reasoning.

Sample two trajectories: $\tau_1$ under $h_{\text{orig}}$, $\tau_2$ under $h_{\text{aug}}$.

Split each trajectory into tool segment $y^{\text{tool}}$ and reply segment $y^{\text{reply}}$.

Compute rewards: $\tau_1$ gets accuracy = 1, effectiveness = 0.9, tool rationality = 1, skill selection = 1; $\tau_2$ gets accuracy = 0.8, effectiveness = 0.7, tool rationality = 0.6, skill selection = 0.9.

Normalize each reward within its group (e.g., tool group mean = 0.8, std = 0.2) to obtain $\hat{A}^{(d)}$.

Compute CoT lengths: $L_{\text{tool}}^{(1)}=80$, $L_{\text{tool}}^{(2)}=150$; apply the piecewise $r_{\text{CoT}}$ function to get penalties 0 and 0.25 respectively.

Form final advantages $\hat{A}^{\star}_{\text{tool},1}=0.5$, $\hat{A}^{\star}_{\text{tool},2}=0.1$ after subtracting $\lambda$‑weighted CoT penalties.

Discard $\tau_2$ if its final advantage were zero; in this example it remains positive, so both trajectories contribute to the gradient.

Even a small change in harness ordering can shift tool‑rationality scores, showing why training on both original and augmented harnesses is essential for robustness.

Why can’t we simply apply standard PPO on the original harness without the group‑wise decomposition?

Standard PPO treats the whole trajectory as a single reward, which would mix tool‑related and reply‑related signals and obscure which part of the policy is responsible for failures. By separating tool and reply groups and normalizing per‑dimension, GDPO provides clearer credit assignment and prevents any single reward dimension from dominating the update.

**Figure 4.** Overview of Harness-Aware Training (HAT). Harness-State Augmentation (HSA) transforms the original Harness into diverse, task-preserving states. A strong teacher generates candidate trajectories under these states, which are filtered by a rejection-sampling judge to provide diverse, high-quality supervision for HSA-SFT. Starting from the base model, HAT then follows three stages: HSA-SFT on the filtered trajectories; General OPD, using the base model as the teacher and the HSA-SFT checkpoint as the student to mitigate general-capability degradation; and HSA-RL in changing HSA environments. The resulting policy is the final HAT-trained model.

**Require:** Policy $\pi_\theta$; group size $G$; state groups $S = \{\text{tool, reply}\}$; reward dims $\mathcal{D}_{\text{tool}}$ and $\mathcal{D}_{\text{reply}}$; CoT weight $\lambda$; clip range $\varepsilon$ 1: **for** each training iteration **do** 2: $\quad$ Sample a mini-batch of queries and augmented Harness states $(x, h)$ 3: $\quad$ $\pi_{\theta_{\text{old}}} \leftarrow \pi_\theta$; initialize $\mathcal{L}_{\text{tool}} \leftarrow 0, \mathcal{L}_{\text{reply}} \leftarrow 0$ 4: $\quad$ **for** each $(x, h)$ in the mini-batch **do** 5: $\quad$ $\quad$ **for** $i = 1, \dots, G$ **do** 6: $\quad$ $\quad$ $\quad$ Roll out trajectory $\tau_i$ under $\pi_{\theta_{\text{old}}}(\cdot \mid x, h)$ 7: $\quad$ $\quad$ $\quad$ Split $\tau_i$ into tool segment $y_i^{\text{tool}}$ and reply segment $y_i^{\text{reply}}$ 8: $\quad$ $\quad$ $\quad$ Create masks $m_i^{\text{tool}}$ and $m_i^{\text{reply}}$ over generated action tokens; mask prompts and observations 9: $\quad$ $\quad$ $\quad$ Score $\{r_i^{(d)}\}_{d \in \mathcal{D}_{\text{tool}}}$ from tool behavior and $\{r_i^{(d)}\}_{d \in \mathcal{D}_{\text{reply}}}$ from final-reply quality 10: $\quad$ $\quad$ $\quad$ Extract CoT lengths $L_i^{\text{tool}}$ and $L_i^{\text{reply}}$ and compute $q_i^s \leftarrow q(L_i^s)$ for $s \in S$ 11: $\quad$ $\quad$ **end for** 12: $\quad$ $\quad$ **for** each state group $s \in \{\text{tool, reply}\}$ **do** 13: $\quad$ $\quad$ $\quad$ **for** each reward dimension $d \in \mathcal{D}_s$ **do** 14: $\quad$ $\quad$ $\quad$ $\quad$ $\hat{A}_i^{(d)} \leftarrow (r_i^{(d)} - \mu_s^{(d)}) / \sigma_s^{(d)}$ for $i = 1, \dots, G$ 15: $\quad$ $\quad$ $\quad$ **end for** 16: $\quad$ $\quad$ $\quad$ $\hat{A}_i^{(\text{CoT}, s)} \leftarrow (q_i^s - \mu_s^{(\text{CoT})}) / \sigma_s^{(\text{CoT})}$ 17: $\quad$ $\quad$ $\quad$ $\hat{A}_i^{s, \star} \leftarrow \sum_{d \in \mathcal{D}_s} \hat{A}_i^{(d)} - \lambda \hat{A}_i^{(\text{CoT}, s)}$; normalize to $\hat{A}_i^{s, \star}$ 18: $\quad$ $\quad$ $\quad$ **for** each sequence $i = 1, \dots, G$ with $\hat{A}_i^{s, \star} \neq 0$ **do** 19: $\quad$ $\quad$ $\quad$ $\quad$ $\rho_{i,t}^s(\theta) \leftarrow \frac{\pi_\theta(y_{i,t}^s \mid y_{i,<t}^s, x, h)}{\pi_{\theta_{\text{old}}}(y_{i,t}^s \mid y_{i,<t}^s, x, h)}$ 20: $\quad$ $\quad$ $\quad$ $\quad$ $\ell_{i,t}^s \leftarrow -\min(\rho_{i,t}^s \hat{A}_i^{s, \star}, \text{clip}(\rho_{i,t}^s, 1-\varepsilon, 1+\varepsilon) \hat{A}_i^{s, \star})$ 21: $\quad$ $\quad$ $\quad$ $\quad$ $\mathcal{L}_s \leftarrow \mathcal{L}_s + \frac{\sum_t m_{i,t}^s \ell_{i,t}^s}{\sum_t m_{i,t}^s}$ 22: $\quad$ $\quad$ $\quad$ **end for** 23: $\quad$ $\quad$ **end for** 24: $\quad$ **end for** 25: $\quad$ Update $\theta$ by minimizing the mean of $\mathcal{L}_{\text{tool}}$ and $\mathcal{L}_{\text{reply}}$ 26: **end for**

Evaluation Protocol

Comprehensive evaluation across six test sets quantifies both accuracy and deployment latency.

We evaluate the model across six test sets, covering real, augmented, synthetic, and deployment scenarios.

The evaluation suite comprises six distinct sets, providing comprehensive coverage of both offline and online performance.

Table 2 lists the six sets with their sizes, sources, and evidentiary roles.

The table lists various datasets used for evaluation, categorized by their set name, size, source, and evidentiary role.

Main Results

HAT delivers superior average scores and stability across harness variations compared to baselines.

Recall that HAT decouples the agent’s policy from specific harness states, enabling robust performance as the business logic evolves.

HAT attains average scores of 94.8 on T1 and 94.6 on T2, surpassing the best top‑model scores of 93.0 and 93.5 respectively.

Table 3

Training Stage Analysis

How each training stage impacts task performance and generalization.

This section isolates the contribution of each training stage by incrementally adding components to a shared initialization and measuring their effect on task‑specific and generalization metrics.

**Table.** Evaluation of model robustness across Tool Robustness and Prompt Robustness metrics.

Fixed‑Harness SFT yields strong gains on T1, T2, and Tool Robustness but harms Prompt Robustness and IFE, indicating over‑fitting to a single harness. HSA‑SFT recovers those losses and adds modest gains, showing that diversity during SFT preserves generalization.

General OPD can be stacked on either pipeline: after Fixed‑Harness SFT it restores Prompt Robustness and boosts IFE metrics; after HSA‑SFT it further lifts T1/T2 while keeping Prompt roughly stable. The two techniques are complementary.

RL consistently raises task‑specific scores in both pipelines. Crucially, without HSA the RL step erodes Prompt Robustness, whereas HSA‑RL maintains and even slightly improves it, confirming HSA’s role in protecting generalization during RL.

**Figure 5.** Checkpoint trajectories across four training configurations formed by crossing HSA at the SFT and RL stages evaluated on Accuracy, Effectiveness, Skill Selection, and Tool Rationality. In these trajectories, HSA-SFT is associated with higher reply-quality rewards, while HSA-RL is associated with stronger agentic-behavior rewards, especially Skill Selection and Tool Rationality; combining HSA-SFT and HSA-RL gives the most balanced late-stage reward profile.

**Figure 6.** Single-run CoT-length diagnostics for the selected HSA-RL trajectory. Faint lines show recorded step values and bold lines show 25-step trailing means. Panel (a) separates tool-call and final-reply CoT lengths and marks the no-penalty ($L \leq 100$), linear ($100 < L < 200$), and saturated ($L \geq 200$) regions. Panel (b) shows the logged score $q(L) \in [0, 1]$ and the equivalent raw reward contribution $-0.1q(L)$. The figure is a training diagnostic, not a causal ablation of latency or quality.

Deployment Performance

Deployment performance shows latency and throughput gains from MTP and HAT across concurrency levels.

MTP reduces 95th‑percentile latency to 8.1 s at concurrency 1, a ~30 % improvement over the baseline.

Table 7 reports Wall P95 = 8.114 s (MTP On) versus 11.210 s (MTP Off).

MTP lets the model predict several future tokens in a single decoder step, cutting the number of calls needed for low‑latency serving.

The table presents performance metrics across various harness configurations, categorized into "Non-augmented" and "Augmented" groups. Columns include $\mathcal{T}_1$, $\mathcal{T}_2$, Tool Robustness ($\mathcal{T}_3$), Prompt Robustness ($\mathcal{T}_3$), IFE-P ($\mathcal{T}_4$), and IFE-I ($\mathcal{T}_4$). Rows represent different training methods such as Base, +SFT, +General OPD, +RL, +HSA-SFT, and +HSA-RL (HAT). Values are accompanied by superscript indicators of performance change relative to the previous configuration.

**Table 5.** Controlled low-concurrency deployment replay. Latencies are seconds; Decode is the arithmetic mean of client-observed completion tokens/s over model calls. It is length-sensitive and is not an intrinsic cross-checkpoint kernel-speed measure. TTFT is the first call’s client-observed P95 and includes network and queueing. Each row contains 100 measured Agent requests after 10 warm-up cases. API rows are operational references, not same-hardware model comparisons.

**Table 7.** MTP trade-off across concurrency. Arrows show Off → On. Decode TPS is client-observed mean per call; Wall and TTFT are P95 seconds. C=1, 2, and 8 are single runs. C=4 values are means of three run-level statistics, not pooled percentiles.

**Table.** Performance comparison of Qwen3.6-35B-A3B and HAT (Ours) across different concurrency levels ($C$). The table shows metrics for Decode TPS, Speedup, Wall P95 (s), TTFT P95 (s), and $\le 15s$ attainment. $C=4$ run ranges (Off; On). Qwen3.6 Base—Decode: 102.13–107.33; 111.11–116.49. Wall P95: 12.931–13.756; 14.957–18.054. TTFT P95: 0.563–0.758; 0.942–1.226. Attainment: 96–98%; 89–95%. Ours—Decode: 97.40–102.55; 121.61–130.18. Wall P95: 10.269–11.606; 10.532–14.205. TTFT P95: 0.587–0.784; 1.305–1.413. Attainment: 99–100%; 96–100%.

**Table 9.** Attribution of the 35 Harness-preferred examples. Categories are assigned from annotator rationales after the blind decision.

Human Blind Test

Human blind test shows Harness beats ReAct with a 97.2% win rate.

Harness outperforms the ReAct baseline on the human blind‑test set, winning 35 of 36 clear‑preference cases.

Table 8 shows a 97.2% non‑tie win rate for Harness.

**Figure 8.** Qualitative prompt-edit cases under the same Harness revision. The edit prohibits disclosure-process language when the available evidence does not support a definite conclusion. In both examples, the Fixed-Harness SFT checkpoint continues to expose internal uncertainty after the edit, whereas HAT follows the revised instruction and gives a direct response or routes the viewer to product details or customer service. Red text marks the disclosure phrases targeted by the edit. These examples illustrate compliance behavior and are not additional quantitative evidence.

Online A/B Test

Online A/B test shows Harness improves GMV and page views over ReAct in production.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers