EnvHarness: Awakening Static Worlds for Agent Learning

Chengsong Huang, Zifeng Wang, Rujun Han, Jun Yan, Yanfei Chen, Zoey CuiZhu, Ke Jiang, Peng Xia, Han Yu, Yufan Zhuang, Yifei Ming, Jiaqi Pan, Bhavana Dalvi Mishra, Jiaxin Huang, Burak Gokturk, Tomas Pfister, Chen-Yu Lee

EnvHarness wraps static benchmarks in programmable layers to dynamically tailor training signals to agent weaknesses.

How can we automatically transform static, fixed-difficulty environments into adaptive training grounds that target an agent's specific weaknesses?

Agents learn from static environments that are often too easy or too hard, failing to provide the targeted feedback needed to address specific behavioral flaws. EnvHarness is a programmable wrapper that intercepts the standard environment interface to inject custom logic—such as modifying initial states, filtering actions, or chaining tasks—without altering the underlying environment code. This approach consistently outperforms static environments, achieving up to a 9.0-point improvement on benchmarks and reducing execution steps by 9.8% through targeted, policy-conditioned customization.

Paper Primer

The core mechanism, EnvRigger, treats the agent as a black box to automate this customization. It observes policy trajectories to diagnose systemic weaknesses, synthesizes code-based components to wrap the environment, and iteratively validates these components through fresh rollouts until they provide an effective learning signal.

EnvHarness provides a superior optimization signal compared to static environments.

Across five benchmarks, agents trained with EnvHarness-customized environments outperformed those trained on original environments, with a 9.0-point gain on ALFWorld and 6.5-point gain in reinforcement learning settings.

Targeted environment customization improves agent efficiency.

On SWE-bench Verified, EnvHarness reduced the average steps per episode from 53.6 to 49.6 by diagnosing and repairing wasteful behaviors like repetitive action loops.

Why is this approach more effective than simply generating more training data?

Standard generation often produces redundant practice that fails to address specific policy weaknesses. EnvHarness uses a co-evolutionary loop that conditions environment difficulty on the agent's current performance, ensuring the training signal remains challenging as the agent improves.

Does this framework require building new environments for every domain?

No. EnvHarness operates through a universal interface that wraps existing benchmarks. Once a lightweight "bridge" is implemented for a domain, the core customization logic and components apply across all benchmarks without further modification.

EnvHarness reframes environment construction as a wrapping problem rather than an authoring one, allowing researchers to reuse trusted, human-built verifiers while gaining the flexibility of dynamic, agent-specific curricula.

Researchers can now treat static benchmarks as programmable, adaptive training platforms, enabling continuous co-evolution between agent capabilities and environment difficulty without the overhead of building new simulators.

The Problem of Static Environments

Static environments stall agent learning; EnvHarness reshapes them dynamically.

Current training environments are hand‑crafted and immutable: they present the same challenges regardless of an agent’s progress, providing no focused feedback on emerging weaknesses and eventually exhausting their instructional value.

Static worlds cannot evolve with the learner, so training stalls once the agent masters the fixed tasks.

The inefficiency of static training environments is a primary bottleneck for continual agent improvement.

The EnvHarness Paradigm

EnvHarness dynamically reshapes a frozen environment to expose agent weaknesses during training.

Static environments are immutable, yet agents need varied challenges. EnvHarness injects a programmable layer that reshapes the environment on the fly, targeting the agent’s current weaknesses.

EnvHarness is a thin, external wrapper that sits around a frozen environment, intercepting its reset, step, and observation calls and altering them dynamically to present tailored challenges.

Reset: EnvHarness Stage forces the start state to $s_1$ instead of the default $s_0$.

Agent takes $a_0$: Contract maps $a_0$ to a no‑op, so the base environment stays in $s_1$ and returns observation $1$.

Agent takes $a_1$: Chain detects $a_1$ and forces a transition to $s_0$, yielding observation $0$.

Even though the underlying world never changes, the wrapper can redirect actions and observations, creating a dynamic training signal without touching the original code.

**Figure 2.** While an agent harness transforms a frozen LLM into a capable agent via plug-in components (e.g., skills, memory, tools) without altering model weights, EnvHarness applies this same principle to the other side of the interaction. It customizes a frozen environment with plug-in components while leaving original environment unchanged.

**Figure 3.** Overview of EnvHarness components wrapping the standard environment interface. The underlying base environment (native state transitions and original task verifier) remains completely frozen. From left to right: the base environment, followed by three EnvHarness components—Stage, Contract, and Chain. Highlighted arrows and headers indicate overridden interface methods, with code blocks showing how each wrapper modifies state initialization, transition dynamics, or observation handling without altering the base environment.

The “harness” layer operates as an external, non‑invasive wrapper, leaving the original environment untouched while steering its behavior.

How does EnvHarness differ from directly editing the environment’s source code?

EnvHarness wraps the frozen environment instead of modifying its internals; the base code stays unchanged, guaranteeing reproducibility and allowing the same environment to be reused with different harness configurations.

EnvHarness provides a drop‑in, non‑invasive way to turn any static environment into a dynamic curriculum generator.

Generating Environments with EnvRigger

EnvRigger builds tailored EnvHarness components through a four‑stage loop to expose policy weaknesses.

Static environments give agents a fixed playground, making it hard to surface systematic flaws. EnvRigger turns this limitation into a lever by automatically reshaping the environment to target the policy’s weak spots.

EnvRigger iteratively shapes the environment by observing the policy, diagnosing its gaps, writing targeted wrappers, and validating them.

How does EnvRigger differ from a standard curriculum‑generation method?

Curriculum generation typically changes the task distribution before the policy sees any data. EnvRigger, by contrast, watches the policy’s actual rollouts, diagnoses concrete failure modes, and then writes environment components that directly target those modes, closing the loop after each diagnostic step.

Observe: run $\pi$ for two episodes, both ending with the door opened; the trajectories contain only $a_1$.

Diagnose: the lack of $a_2$ indicates the policy relies on a shortcut that bypasses key acquisition.

Write: generate a Contract component $w_1$ that disables $a_1$ unless the key has been collected.

Validate: wrap $E$ with $w_1$, run two fresh episodes; the policy now fails to open the door in 40 % of steps.

Refine (implicit): the failure signal forces $\pi$ to learn the key‑pickup behavior, raising the success rate to 70 % after further training.

The example shows how a single diagnosed shortcut can be turned into a learning signal by blocking the shortcut and forcing the policy to acquire the missing skill.

The Designer Loop repeatedly writes candidate EnvHarness components and validates them, refining until a component that yields a meaningful training signal is found.

Why isn’t the Designer Loop just another training epoch for the policy?

Because the loop operates on the environment, not on the policy parameters. It creates or modifies EnvHarness components that change the agent’s observation‑action space, providing a new source of learning pressure independent of gradient updates.

Generate a candidate component $w$ from the latest diagnosis.

Wrap the base environment $E$ with $w$ to obtain $E'$.

Run fresh rollouts of $\pi$ on $E'$ and compute the success rate.

If the success rate is within the target range, accept $w$ and add it to the EnvHarness.

Otherwise, adjust $w$ (e.g., tighten constraints) and repeat from step 1.

**Figure 4.** **EnvRigger** generating **EnvHarness** components for a target policy based on given task. The *execution loop* on the left runs the policy against the current environment, which is a frozen base environment wrapped by the active **EnvHarness** containing accepted components $w_1, \dots, w_k$, while the resulting rollout trajectories feed the **EnvRigger** loop on the right. The **EnvRigger** operates systematically through four distinct stages: Observe, Diagnose, Write, and Validate, where the last two steps form a write-and-validate loop that generates a candidate component, evaluates it on fresh rollouts, and revises it upon failure.

Experimental Results

EnvHarness boosts agent success rates across benchmarks and cuts execution steps.

EnvHarness yields up to a 9.0‑point boost on ALFWorld success rate compared to static environments.

Table 2 reports EnvHarness Envs at 66.2 ± 0.3 versus Original Envs at 63.3 ± 1.2, and the authors note up to 9.0‑point gains in out‑of‑distribution settings.

It extracts reusable behaviors (skills) from agent trajectories in customized environments and equips the policy with these skills to accelerate task mastery.

How does skill‑based learning differ from standard behavior cloning?

Behavior cloning copies the policy’s actions verbatim from static demonstrations, while skill‑based learning first reshapes the environment to provoke new behaviors, then extracts those novel actions as reusable skills.

**Figure 1 | Overall performance.** *Left:* Agents learning from EnvHarness environments consistently outperform those learning from the original environments across software engineering and office automation benchmarks, including SWE-bench Verified, OfficeQA, and SpreadsheetBench. *Right:* On SWE-bench Verified, under an identical environment budget, EnvHarness keeps improving as environments scale, while real and generated environments flatten out.

EnvHarness consistently improves agent success rates across diverse benchmarks.

Performance and Scaling Analysis

EnvHarness reshapes static environments to target agent weaknesses, enabling more effective training signals.

EnvHarness wraps static environments in a programmable harness that reshapes the interface to target an agent's specific weaknesses, turning a fixed world into a dynamic training signal. We now examine its impact on reinforcement learning, scaling efficiency, transferability, the Chain component, and user‑defined constraints.

**Table 4.** Reinforcement learning on ALFWorld and WebShop, comparing policies trained on the original environments and on ENVHARNESS environments. ALFWorld is scored by success rate on in-distribution and held-out instance types; WebShop reports environment score and success rate.

We next isolate the contribution of the Chain component, which concatenates two base environments into a single extended episode.

The Combined Skills row achieves the highest success rate (54.30 %) while the Original Envs row yields the lowest average steps (41.96), demonstrating that chaining and stage/contract skills complement each other.

**Figure 5.** Environment scaling on SWE-bench Verified. All three sources supply the same number of environments and feed the same extraction and retrieval protocol.

We also evaluate EnvHarness across four distinct LLM backbones to assess transferability.

**Figure 6 |** Cross-model results on SWE-bench Verified. Each group represents one policy model, ordered from weakest to strongest. Bars show success rates with no skills, with skills from unmodified environments, and with skills from EnvHarness environments. Percentages represent relative gains over original environments.

Finally, EnvRigger can incorporate explicit user‑defined targets, such as a required success rate or a natural‑language description of a weakness, enabling bespoke environment generation.

Related Work

We situate EnvHarness within prior work on environment scaling and self‑evolving agents.

Environment scaling has become a research direction that expands the pool of training environments, ranging from LLM‑driven simulators to full‑world models and programmatic synthesis of executable tasks.

Beyond generating more tasks, recent work also adapts the environment’s presentation to the learner via curriculum generation, corrective feedback, and reward shaping.

Self‑evolving agents improve autonomously by modifying their own components—prompts, skill libraries, memory buffers, or even model weights—while the surrounding world remains static.

In contrast, EnvHarness reshapes the environment itself in response to diagnosed policy weaknesses, using a single interface that works across benchmarks without altering tasks or verifiers.

The ActionableEnv Interface

Defines the ActionableEnv contract and component decorators that reshape static benchmarks for training.

EnvHarness is built around a single design commitment: every benchmark, and every transformation of a benchmark, presents the same interface. This guarantees that a policy or orchestrator sees no difference between a raw environment and one wrapped in an arbitrary stack of EnvHarness components.

A uniform abstract class that every environment must implement, so policies can interact with raw benchmarks and wrapped versions interchangeably.

Reset with seed $42$ → observation

Step with action

Observe returns the same

Separating observe() from reset() allows components to inject deterministic preprocessing between episode start and the first policy decision.

Why is observe() a separate call from reset()?

Because a component may need to replay a sequence of actions after the environment is reset but before the policy sees the first observation; observe() lets the outer layer read the mutated state without incurring another reset cost.

Replays a predefined action list after reset to produce a reachable, deterministic initial state.

How does Setups differ from manually setting the environment’s internal state?

Setups never touches the hidden runtime; it only uses the public step() API, ensuring the resulting state is reachable by the agent and portable across any Bridge implementation.

Three pure‑function hooks that can filter actions, modify transitions, and transform observations on each step.

In what way does Rules differ from a typical environment wrapper that simply modifies observations?

Rules provides three distinct hooks—action filtering, transition modification, and observation filtering—so it can intervene at every point of the step loop, whereas a simple wrapper usually only changes the observation after the step.

Composes two ActionableEnvs into a single episode, handling handoff, lazy reset, and combined success evaluation.

Why does Link avoid importing concrete Bridge implementations?

Link relies exclusively on the ActionableEnv interface, so it can compose any two registered environments without needing to know their underlying runtimes, preserving modularity and extensibility.

Together, ActionableEnv, Bridges, and the EnvHarness component stack give the framework a programmable, composable interface that turns static benchmarks into dynamic training curricula.

Extended Empirical Results

Additional quantitative analysis of EnvHarness’s impact on skill extraction and token usage.

EnvHarness improves generalization on ALFWorld leave‑one‑out tasks, raising average success rate by 3.1 points.

Table 10 shows EnvHarness SR 66.8 % versus Original 63.7 % ($\Delta$ +3.1 %).

Across four policy models, EnvHarness‑generated skills increase success rates while keeping episode length comparable, confirming that the gains are not merely due to longer runs.

Token consumption analysis (Table 11) shows that EnvHarness spends orders of magnitude more rollout tokens than baseline methods, yet design tokens remain a small fraction of the total budget.

Metric targeting (Table 12) demonstrates that EnvHarness pushes the in‑band success‑rate fraction from 6 % to 80 % and the in‑band average‑steps fraction from 18 % to 53 %.

**Table 13.** Nine specified weaknesses, the component the designer generated, and the skill distilled from the trajectories collected in the reshaped environment.

```python class _Rules(Rules): def filter_action(self, action, env_state): if action.name == "bash": command = action.kwargs.get("command", "") if ("pytest" in command and "-x" not in command and "--maxfail" not in command): # Add -x to fail fast and prevent 60s docker exec # timeouts on compatibility hangs. action.kwargs["command"] = command.replace( "pytest", "pytest -x") ```

**Table 1.** Performance comparison across different skill sources and models, measuring Success Rate (SR) and Average Steps (AS).

**Table 10.** Leave-one-out generalization on ALFWorld. Skills are extracted from environments of all task types except the held-out type, then evaluated on it.

**Table 11.** Estimated token consumption. Design tokens cover the calls that propose and refine components; rollout tokens cover every interaction a method drives. These rollouts are not the same kind of work across rows: ENVHARNESS and VeriEnv execute them against the real environment, whereas GenEnv's are LLM-simulated.

**Table 12.** Objective metric targeting on ALFWorld. Entries are the percentage of tasks whose measured value falls inside the target band, before and after reshaping.

Applying pytest -x, patch -p1, and targeted test‑file execution are concrete skills the policy learns to avoid timeouts and resource kills in noisy environments.

When the standard pytest CLI is broken, the policy falls back to python -c “import pytest; pytest.main(…)”, and uses pytest -k filters to run only relevant test cases.

Using the active conda environment’s absolute binary path ensures that python and pytest resolve to the correct interpreter, avoiding version‑mismatch errors.

Tracing exception propagation with grep -rn and locating reference implementations with grep -rn help the policy navigate codebases without in‑place sed edits.

Managing shared step budgets across chained tasks forces the agent to allocate enough steps for the second task after solving the first, as demonstrated in the “Skills from Chain Environments” example.

Re‑orienting the environment after a task handoff (e.g., checking conda env list and which python) prevents command‑not‑found errors when switching between repositories.

Teaching against a specified weakness in ALFWorld leads to the distilled skill “Pre‑Interaction State Verification”, ensuring objects are accessible before manipulation.

In WebArena, the weakness “no scrolling” yields the skill “Incremental Viewport Expansion”, which scrolls to the bottom and re‑inspects the DOM before extraction.

For SWE‑bench Verified, the policy learns “Context‑Aware Code Modification” to read target functions and fixtures before editing, and “Safe File Modification via Python Scripting” to replace fragile sed ‑i operations.

Limitations and Future Work

We outline current limits of EnvHarness and sketch avenues for extending it.

The design loop that builds each EnvHarness environment is iterative: a designer agent proposes a harness, executes it, and revises it until validation passes. Weaker designers need more iterations, and each iteration rolls out the full environment, so assembling a high‑quality pool can consume substantial time and inference compute.

EnvHarness assumes a reset/step interface like a Gym environment, where a Stage must place the environment into a chosen initial state and a Chain must return it to a known state between subtasks. This requirement excludes non‑resettable backends such as live services, real user accounts, or physical robots whose surroundings cannot be restored.

Chains currently compose subtasks only by concatenation and verify the result through the verifiers of each part. This serial composition gives a composite verifier as the conjunction of individual verdicts, but it provides no notion of semantic relatedness and cannot express branching, shared state, or interleaved workflows.

Future work will expand the harness component set beyond the initial Stage, Contract, and Chain. Adding components that inject stochasticity, partial observability, auxiliary feedback channels, or multi‑agent interactions will broaden the space of reshaped environments while preserving the same reset/step contract.

Extending EnvHarness beyond text‑only environments to visual, GUI‑driven, or embodied domains will test the robustness of the wrapping abstraction when observations are no longer symbolic. Such extensions will require new components capable of specifying and verifying non‑textual states.

To support richer control flow, Chains will need mechanisms for branching and semantic composition. This entails defining compatibility measures between subtasks and constructing verifiers that operate over the composed objective rather than merely aggregating individual verdicts.

EnvRigger Prompting Details

Appendix A details the EnvRigger prompt, component name mappings, and related design notes.

This appendix enumerates the auxiliary material supporting the main paper. It begins with the EnvRigger Prompt (Section A), then outlines differences from related co‑evolution frameworks (Section B), and details the interface protocol and design patterns (Section C) with its three sub‑parts. Subsequent sections provide concrete implementation examples, experiment details, analysis breakdowns, additional analysis, limitations, and future directions.

**Table 6.** Component names in the paper and in the release.

The system prompt for the EnvHarness designer agent instructs the “Environment Designer” to reshape the benchmark environment so the policy agent receives appropriate training signals, emitting a Candidate with two independent levers. This prompt forms the basis of the EnvRigger Prompt used throughout the appendix.

Implementation Rules and Hooks

Comparison of EnvHarness hooks and related adaptive environment frameworks.

This appendix details the programmable hooks of EnvHarness and positions them against three prominent adaptive environment paradigms.

EnvHarness per‑step hooks – default pass‑through, overridden as needed.

`in_env_actions` is a list of tool calls that the framework replays through env.step() before the policy begins, providing an initial trajectory (S0) without writing code.

The S0 seed and the three hooks can be used independently or together to shape the training task.

PITFALL – a mutation that makes success impossible yields SR = 0, which is as uninformative as a trivial mutation that yields SR = 1.

BASELINE – examine K unmutated rollouts to see if the policy can solve the task, how many steps it uses, and which environment parts it relies on.

REFINE – after K rollouts of a candidate mutation, accept, refine, or reject based on aggregate success rate and failure distribution, adjusting magnitude rather than discarding working hooks.

Chain Operator Implementation

Implementation snippets illustrate three programmable modes of the Chain (Link) operator.

The Link operator concatenates environments in sequence by default, handing off to the next environment as soon as the current one terminates.

To route the agent based on whether the first task succeeded, the BranchOnOutcome subclass overrides `modify_transition` and selects a harder or easier environment accordingly.

For an immediate switch triggered by a specific action, SwitchOnAction overrides `modify_transition` to jump to a new environment without waiting for the current task to finish.

Interleaving Environment Details

Details on interleaved environments, benchmark splits, baselines, and EnvRigger settings.

Interleaving environments let the transition check execute after each interaction, so the agent can flip back and forth between two worlds continuously.

Alternate class swaps environments on every step.

The table outlines the training and evaluation task distributions for five benchmarks: ALFWorld, WebArena, SWE-bench, OfficeQA, and SpreadsheetBench. It specifies the number of tasks used for training and the corresponding number of tasks or instances used for evaluation for each benchmark.

For SpreadsheetBench, Pass@1 aggregates over base tasks and Mean Score averages over all instances; on ALFWorld, In‑Dist and OOD refer to the benchmark’s own seen and unseen evaluation splits, which we do not construct ourselves.

Baseline generators—GenEnv (Guo et al., 2025), VeriEnv (Chae et al., 2026), and SWE‑smith (Yang et al., 2026a)—each produce the same number of environments as EnvHarness but differ in how they synthesize tasks.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers