ClawGym II: Exploring Black-Box RL on Agent Harness

Huatong Song, Fei Bai, Ming Yang, Renyuan Li, Jia Deng, Jujie He, Zhange Zhang, Daixuan Cheng, Yan Xing, Qi Yun, Xuxing Chen, Danyang Li, Feng Chang, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Mingjie Tang, Ji-Rong Wen

A unified black-box RL framework for optimizing general agents through opaque, complex execution harnesses.

How can we optimize general agents using reinforcement learning without needing access to the internal state or gradients of the complex deployment harnesses they run in?

Modern agent harnesses coordinate complex, long-horizon tasks, but their internal control flows are opaque, making standard reinforcement learning (RL) difficult to apply. The authors decouple policy optimization from harness execution by using a serving proxy to capture model calls, which are then reconstructed into prefix trees for training. This allows stable optimization using PPO or GRPO without modifying the underlying harness. On ClawGym-Bench, this framework improves Pass@1 by up to 14.81 points, demonstrating stable performance across heterogeneous harnesses and task types.

Paper Primer

General agents rely on harnesses like OpenClaw or Claude Code to manage tools and context, but these systems are "black boxes" that hide internal logic. This makes it hard to generate clean training trajectories, as the harness may perform retries, sub-agent delegation, or context compaction that fragments the interaction history.

The framework treats the harness as an opaque rollout engine: it intercepts model calls at the boundary, organizes them into a prefix tree to recover the true interaction structure, and filters out auxiliary noise like retry-induced "dead leaves." It then applies PPO or GRPO to this tree, using token-level importance sampling to correct for the mismatch between the inference-time policy and the training-time policy.

Black-box RL significantly boosts agent performance on long-horizon tasks.

Pass@1 improvements on ClawGym-Bench using Qwen3-30A3B. +9.98 points (OpenClaw) and +14.81 points (Claude Code).

The framework supports mix-harness training, allowing a single model to learn from heterogeneous execution systems simultaneously.

Joint optimization of task-harness pairs in a single batch. Performance matches or exceeds models trained on individual harnesses alone.

Why is a "prefix tree" necessary for training?

Black-box harnesses produce fragmented, redundant, and forked model calls. The prefix tree reconstructs the shared interaction history, ensuring the model learns from the actual task-solving path rather than treating every fragmented call as an independent, isolated event.

Does this approach require modifying the harness code?

No. The framework is designed to be harness-agnostic; it interacts only with the model-serving boundary, allowing it to work with any complex harness as long as the model's inputs and outputs can be intercepted.

Motivation and Problem Framing

We expose why opaque harnesses block RL and introduce a black‑box framework to overcome it.

Modern agent harnesses coordinate large language models with environments, yet their internal logic is opaque, preventing direct reinforcement‑learning optimization.

When a harness hides its control flow and tool interactions, the learning algorithm cannot observe the true state‑action sequence, making policy improvement unreliable.

We therefore ask: how can we perform stable, scalable black‑box reinforcement learning through such harnesses?

The shift from white‑box simulation to black‑box deployment‑harness training unlocks scalable, stable optimization of general agents.

Framework Overview

We validate a unified black‑box RL pipeline that scales across heterogeneous harnesses while remaining stable.

To enable stable, large‑scale concurrent rollouts we provision each task‑specific environment inside a temporary sandbox that is created on demand and destroyed after the rollout, guaranteeing isolation while supporting massive parallelism.

Training remains stable for 200–400 optimization steps, confirming that the decoupled policy optimization and sandboxed execution do not introduce divergence or collapse.

Mix‑harness training—jointly optimizing over rollouts from both OpenClaw and Claude Code—produces a single model that matches or exceeds the performance of models trained on either harness alone, demonstrating seamless integration of heterogeneous environments.

Our contributions are threefold: (1) a unified, extensible black‑box RL framework that works with any opaque harness; (2) a scalable infrastructure plus tree‑structured PPO/GRPO optimization that preserves training‑inference consistency; (3) thorough validation on OpenClaw, Claude Code, and extended tasks (JobBench, OfficeQA), showing consistent gains and stable optimization.

Task and Environment Definitions

Defines the task setup, the harness abstraction, and the RL objectives used to train agents.

A general‑agent task pairs a user instruction with an isolated, stateful workspace. The agent repeatedly observes the workspace, issues tool actions (e.g., web search, file edit, shell command), and gradually transforms the state toward the goal.

Task success is measured either by deterministic rule‑based checks (unit tests, exact matches) or by a rubric‑based evaluator such as an LLM‑as‑a‑Judge that scores the final artifact against task‑specific criteria.

The harness is a thin runtime layer that receives a model action, runs the requested tools, updates the workspace, and returns a fresh observation—so the model never touches the workspace directly.

Initial workspace $W_t$ contains $f_1 = \text{""}$, $f_2 = \text{"data"}$; context $c_t$ is empty.

Harness $H$ runs the append tool, producing $f_1 = \text{"Hello"}$ while leaving $f_2$ unchanged.

Updated workspace $W_{t+1}$ reflects the changed $f_1$; observation $o_{t+1}$ is the string “Hello”.

Context $c_{t+1}$ records that the append tool succeeded, enabling the model to skip redundant checks later.

The harness isolates the model from file‑system details, so the policy only learns to issue high‑level tool commands; low‑level I/O errors are handled inside $H$.

How does the Agent Harness differ from the classic ReAct loop?

ReAct lets the model read and write the workspace directly, requiring the model to manage tool failures and state consistency. The Agent Harness instead centralizes those responsibilities: the model emits a single abstract action, the harness performs the concrete tool calls, handles retries, and returns a clean observation, which removes a large source of instability from the learning problem.

Training proceeds with two RL formulations. PPO augments the policy with a critic $V_\phi$ and applies a binary mask $M_t$ to ignore invalid tokens. GRPO removes the critic entirely, estimating advantages from a group of sampled rollouts.

The Black-Box RL Mechanism

We describe a black-box RL framework that isolates harnesses, builds prefix trees, and trains policies across heterogeneous harnesses.

Training agents through opaque harnesses creates two bottlenecks: scaling rollouts without interfering with each other, and recovering a usable training signal from the harness’s hidden control flow.

The framework treats the harness as an isolated black box, captures model calls at the serving boundary, and reconstructs them into a shared prefix‑tree representation that the policy optimizer can consume.

How does this differ from traditional white‑box RL where the environment is instrumented?

Instead of modifying the harness to emit internal states, we leave the harness untouched and only observe the model’s request‑response pairs at the serving proxy, eliminating the need for any harness‑specific instrumentation.

Model calls are grouped by shared history, forming a tree where each node represents a common prefix of the interaction.

Start with a root node containing “Prompt”.

Attach call $(x_2, y_2)$ as a child because its input shares the longest prefix “Prompt” with the root.

Attach call $(x_3, y_3)$ as a child of the node for $(x_2, y_2)$ since its input extends the prefix “Prompt + ToolResult”.

The resulting tree has a single root‑to‑leaf path of length three, representing the full multi‑turn trajectory.

Even though the three calls appear sequentially, the tree captures their hierarchical dependence, preventing duplicate training on the same prefix.

Why not simply concatenate the calls into a flat sequence?

Flattening would lose the branching information needed to distinguish alternative continuations (e.g., retries or sub‑agents), and would cause the optimizer to over‑count shared prefixes, biasing gradient updates.

After building the prefix tree we prune leaves that do not correspond to meaningful task‑solving behavior.

What threshold determines “excessive” branching?

The implementation uses a fixed leaf‑count limit (e.g., 20 leaves); rollouts exceeding this limit are dropped, a heuristic that empirically balances data richness against noise.

All retained trajectories from a rollout share the same terminal reward, so we assign that reward to every token node while counting shared prefixes only once.

Compute the raw ratio: $\exp(-0.25 - (-0.2)) = \exp(-0.05) \approx 0.951$.

Apply the truncation $\bar{c}=1.0$: $w = \min(0.951, 1.0) = 0.951$.

Scale the token’s loss by $0.951$, slightly reducing its influence because the training policy is marginally less confident than the rollout policy.

Even tiny mismatches between rollout and training probabilities can bias gradients; the importance‑sampling factor corrects this without incurring high variance.

Why does PPO ignore cross‑branch dependencies?

Modeling dependencies across branches would require a complex credit‑assignment scheme over the tree; the paper adopts a pragmatic simplification that treats each branch independently, accepting higher variance as a trade‑off.

By feeding rollouts from multiple heterogeneous harnesses into a single policy, the model learns to operate under diverse interaction protocols without over‑fitting to any single harness.

Why not group rollouts only by task, ignoring the harness?

Different harnesses can produce vastly different interaction patterns and reward scales; mixing them in the same group would distort the normalized advantage and bias the policy toward the dominant harness.

**Figure 1.** Overview of our black-box RL framework for optimizing general agents through harnesses.

With these components—sandboxed execution, prefix‑tree reconstruction, and mix‑harness training—the framework enables scalable, stable reinforcement learning even when the underlying harnesses remain completely opaque.

Performance and Training Dynamics

We report black‑box RL performance, training dynamics, and mix‑harness results across two harnesses.

The black‑box RL premise treats the deployment harness as an opaque environment, letting the policy be optimized without direct access to internal harness logic.

Black‑Box RL with the 30A3B backbone improves Pass@1 on ClawGym‑Bench by up to 14.81 points over the SFT baseline.

Table 1 reports a 14.81‑point gain for the Claude Code harness (ClawII‑CC‑30A3B vs. ClawGym‑30A3B).

Training dynamics are stable for both PPO and GRPO across the two harnesses. PPO shows smoother policy‑entropy curves, while GRPO exhibits larger variance and a late‑stage entropy decline under OpenClaw. Under Claude Code both algorithms maintain higher entropy before stabilizing.

**Figure 2.** Training dynamics of PPO and GRPO with OpenClaw as the rollout harness.

**Figure 3.** Training dynamics of PPO and GRPO with Claude Code as the rollout harness.

Mix‑harness training—jointly optimizing with OpenClaw and Claude Code rollouts—produces rewards comparable to single‑harness baselines and matches or slightly exceeds their downstream evaluation scores, indicating no instability from heterogeneous signals.

**Figure 4.** Comparison between single-harness and mix-harness RL training.

Generality and Task Diversity

Unified black‑box RL boosts performance on harder benchmarks and matches white‑box gains.

Unified black‑box RL raises OfficeQA‑Full evaluation from 8.53 to 21.54.

Results on the OfficeQA benchmark show a 13.01‑point gain after training with the black‑box framework.

JobBench‑Easy improves from 20.46 to 27.20, and both benchmarks exhibit clear, stable reward curves throughout optimization.

Cold‑start initialization yields higher initial rewards and smoother entropy dynamics than direct base‑model training, leading to consistently stronger downstream performance.

White‑Box AgentLoop RL achieves an average score of 59.90, a 18.21‑point gain over the base model.

Table 2 shows WhiteBox‑30A3B surpasses the Qwen3‑30A3B initialization by 18.21 points across six task categories.

White‑Box outperforms the black‑box‑trained ClawII‑OC‑30A3B by 8.53 points, and transfers to the OpenClaw harness with a score of 50.33 (+5.22 over the original model), though it remains below the dedicated black‑box baseline of 62.62.

**Figure 5.** Training dynamics with Claude Code on JobBench-style tasks.

**Figure 6.** Training dynamics with Claude Code on OfficeQA-style tasks.

Ablations and White-Box Baselines

We evaluate how each design choice impacts performance.

Recall that the paper’s core idea is to treat a deployment harness as an opaque black‑box, separating policy learning from harness execution.

A white‑box loop runs the policy inside the harness, exposing internal states so the optimizer can directly observe and back‑propagate through them.

How does a white‑box AgentLoop differ from simply logging actions during rollout?

Logging only records the final action‑reward pairs, whereas a white‑box loop also exposes intermediate hidden states and timing information, letting the optimizer adjust the policy based on the full internal trajectory rather than just the end result.

Cold‑start initialization seeds the policy with a supervised‑fine‑tuned (SFT) model before black‑box RL begins, giving the agent a head start on the task.

Why not just train the RL policy from scratch instead of using a cold‑start?

Training from scratch often leads to high variance and slow reward improvement; the cold‑start provides a stable baseline that guides early updates, preventing the optimizer from wandering in low‑reward regions of the policy space.

**Figure 7.** Effect of cold-start initialization on OpenClaw black-box RL.

**Table 2.** Effectiveness under the white-box agentloop.

**Figure 8.** Training dynamics of white-box AgentLoop RL with GRPO and PPO. The curves report training reward, policy entropy, and evaluation score over the course of training.

Removing cold‑start initialization leads to noticeably lower rewards and more erratic entropy, confirming that the warm‑start is essential for stable and fast learning.

When the white‑box AgentLoop is omitted, performance drops across all metrics, demonstrating that exposing internal harness states materially improves the RL signal.

Replacing GRPO with PPO reduces final reward and evaluation scores, though PPO preserves higher entropy, highlighting a trade‑off between performance and exploration.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers