Environment Evolution for Terminal Agents

Zhiyuan Fan, Tinghao Yu, Yuanjun Cai, Jiang Zhou, Jiangtao Guan, Jincheng Liu, Yun Yang, Dingxin Hu, Zhuo Han, Xing Wu, Feng Zhang, Lilin Wang

Environment evolution incrementally synthesizes increasingly difficult terminal tasks off-policy to provide continuous learning signals for RL agents.

How can we automatically scale the difficulty of terminal-based coding environments to keep pace with increasingly capable AI agents?

Terminal agents often exhaust the learning potential of static environments, while existing co-evolution methods rely on on-policy rollouts that limit generalization and fail as model performance saturates. The authors propose environment evolution: a loop-engineered harness that incrementally modifies environments along three difficulty axes—scenario novelty, skill rarity, and execution length—without requiring a rollout model. This off-policy approach consistently generates harder tasks, improving performance on Terminal-Bench 2.1 by up to 18.0 percentage points compared to co-evolution and ensemble baselines.

Paper Primer

The method treats environment difficulty as a model-agnostic property derived from the multi-turn learning objective. It uses a multi-agent harness to perform incremental edits: the Proposer updates the execution sequence of scenarios and skills, and the Modifier applies these changes to the environment, which are then validated by a suite of automated verifiers.

Environment evolution is like a curriculum designer: it reads the current task's "recipe" (the sequence of scenarios and skills), tweaks one step to make it more demanding, and verifies that the new task remains solvable before adding it to the training lineage.

Environment evolution provides more effective, longer-lasting learning signals than agent-environment co-evolution.

RL training on Qwen3.6-27B and Qwen3.6-35B-A3B using evolved environments achieved peak accuracies of 71.5% and 64.9% on Terminal-Bench 2.1. Improvements of 14.4 and 18.0 percentage points over the respective baselines.

To prevent the agent from stalling on tasks it cannot yet solve, the Evolution-Lineage (EL) Scheduler manages the training curriculum. It only exposes the next generation of environments once the agent achieves a pass-rate threshold on the current generation, ensuring the training signal remains informative.

Why is an off-policy approach superior to existing on-policy co-evolution?

On-policy methods are constrained by the rollout model's specific weaknesses and the initial environment distribution, which limits generalization. By operating off-policy, environment evolution constructs lineages independently of the target policy, allowing for continuous difficulty scaling even as the model becomes stronger.

What are the three specific directions used to increase environment difficulty?

The authors identify scenario novelty (how common a scenario is), skill rarity (how common a skill is within a scenario), and execution length (the number of solver turns required) as the primary levers for increasing difficulty.

By decoupling environment synthesis from model rollouts, researchers can generate a scalable, verified curriculum of increasingly difficult tasks that sustain agent improvement long after static datasets are exhausted.

Scaling Terminal Environments

We expose why static environments stall frontier agents and introduce environment evolution to keep difficulty rising.

Frontier agents quickly outgrow static, hand‑crafted environments, leaving them with weak learning signals. Existing co‑evolution approaches tie difficulty to on‑policy rollouts, which curtails generalization as models improve. To keep training productive, we need a mechanism that raises environment challenge continuously without depending on the current policy.

Environment evolution mutates an existing environment along a chosen difficulty axis, producing a lineage of progressively harder tasks without consulting the target agent.

**Figure 1.** Comparison of paradigms for environment scaling. (a) Environment ensemble obtains a new environment by composing several difficult primitive environments. (b) Agent–environment co-evolution rolls out the target agent on primitive environments to populate a weakness bank, which guides the construction of new environments. (c) Environment evolution, our proposed paradigm, evolves the environment itself: an evolver uses a derived difficulty signal to produce successive environments along the lineage.

Automated, off‑policy environment scaling is essential to keep frontier agents learning continuously.

Measuring Environment Difficulty

We define a model‑agnostic difficulty metric that quantifies environment challenge independent of any agent.

Estimating difficulty on‑policy ties the metric to a particular model’s pass rate, which conflates environment challenge with model weakness. We therefore need a metric that reflects the environment itself, regardless of which agent is used.

Instead of measuring how well a specific model solves a task, we ask “how hard is the task itself?” by looking at how unlikely the required scenario‑skill pairs are under a broad, model‑agnostic reference distribution.

How does this off‑policy difficulty differ from the on‑policy difficulty that uses a model’s own pass rate?

On‑policy difficulty measures success probability of a specific model, so it mixes environment hardness with that model’s capabilities. Off‑policy difficulty replaces the model‑dependent probabilities with a universal reference $T$, thus capturing only how intrinsically challenging the scenario‑skill pair is, independent of any particular agent.

Compute per‑step difficulties: $d_T(z_1\mid g)= -\log 0.5 - \log 0.2 \approx 0.693 + 1.609 = 2.302$.

Similarly, $d_T(z_2\mid g)= -\log 0.1 - \log 0.05 \approx 2.303 + 2.996 = 5.299$.

Sum to obtain $D_T(\xi)=2.302 + 5.299 = 7.601$.

If a model $\theta$ assigns $p_\theta(\sigma_0\mid g)=0.8$, $p_\theta(\kappa_1\mid\sigma_0,g)=0.6$, $p_\theta(\sigma_1\mid g)=0.4$, $p_\theta(\kappa_2\mid\sigma_1,g)=0.3$, then $d_\theta(z_1\mid g)= -\log 0.8 - \log 0.6 \approx 0.223 + 0.511 = 0.734$ and $d_\theta(z_2\mid g)= -\log 0.4 - \log 0.3 \approx 0.916 + 1.204 = 2.120$.

Excess difficulties: $\delta_\theta(z_1\mid g)=\bigl[0.734 - 2.302\bigr]^+=0$, $\delta_\theta(z_2\mid g)=\bigl[2.120 - 5.299\bigr]^+=0$.

Thus $\Delta_\theta(\xi)=0$, indicating the model is not weaker than the reference on this trajectory.

Even when a model appears to solve a task easily, the off‑policy metric reveals whether the required scenario‑skill pairs are intrinsically rare; a zero weakness score means the model’s performance is fully explained by the environment’s baseline difficulty.

The Multi-Agent Evolution Harness

We describe a two‑loop multi‑agent harness that evolves environments and a scheduler that orders them by difficulty.

Static environments quickly saturate frontier models, leaving them with no learning signal. To keep difficulty rising, the paper introduces a continual‑mutation loop that reshapes tasks as agents improve.

The harness runs two gated feedback loops: (1) a Plan Refinement loop that mutates a task plan, and (2) an Environment Refinement loop that applies the plan to the environment while multiple verifiers enforce correctness.

Pick the second pair (S₂, K₂) for replacement.

Replace scenario S₂ with a new scenario S₂′ while keeping skill K₂ unchanged, yielding $\xi'$_E = [(S₁, K₁), (S₂′, K₂), (S₃, K₃)].

Compute the plan as the difference between $\xi'$_E and $\xi_{E}$, which is “swap S₂ → S₂′”.

The Reviewer checks the plan against the rubric; it passes on the first pass, so the plan is accepted.

The Modifier creates $\Delta$E that encodes the swap and adds it to the environment, producing a new environment $E^{(g+1)}$.

Low‑effort edits affect only a single scenario–skill pair, keeping the mutation localized and preventing catastrophic task drift.

How does this two‑loop harness differ from a standard reinforcement‑learning training loop?

In a typical RL loop the agent interacts with a fixed environment and receives a scalar reward. Here the environment itself is mutated by a separate set of agents (Proposer, Modifier, Verifiers) before the learner sees it, and acceptance is gated by rubric‑based reviews rather than a single reward signal.

The scheduler presents environments in chronological order of their generations, advancing only when the current environment’s average reward exceeds a fixed threshold, thereby avoiding batches of unsolvable tasks.

Rollout rewards: $r_{0,1}$=0.7, $r_{0,2}$=0.6 → p₀($E_{1,1,1}$) = (0.7+0.6)/2 = 0.65 > $\tau$=0.75? No, so stay.

Second update: rewards 0.8 and 0.9 → p₁ = 0.85 > $\tau$, so move to next environment: (g₁, k₁) = (1, 2).

Third update on $E_{1,1,2}$: rewards 0.5 and 0.4 → p₂ = 0.45 ≤ $\tau$, stay.

Fourth update: rewards 0.9 and 0.95 → p₃ = 0.925 > $\tau$, k reaches $N_{1,1}$=2, so advance to next generation: (g₄, k₄) = (2, 1).

The scheduler guarantees that each generation is fully explored before moving on, and only environments that consistently exceed the threshold are promoted.

Why not sample environments uniformly at random instead of using this ordered scheduler?

Random sampling quickly yields many environments that are still unsolvable, producing rollouts with zero reward and no learning signal. The ordered scheduler ensures the agent always sees at least one environment it can solve, providing a steady gradient for improvement.

**Figure 2.** Loop-engineered multi-agent harness for environment evolution. It decomposes each generation into two gated feedback loops: (1) sequence-guided plan refinement, which generates and revises an evolution plan until it passes rubric-based review; and (2) plan-conditioned environment refinement, which evolves and repairs the candidate until strict solvability and quality checks pass.

Together, the loop‑engineered harness and the evolution‑lineage scheduler provide a self‑sustaining pipeline that continuously raises task difficulty while keeping learning signals informative.

Experimental Harness and Setup

We detail the Claude Code harness, benchmark, training setup, and evolution effort across generations.

High and max evolution effort drive the pass rate to zero within the 15‑generation limit.

Empirical results across the three effort levels show pass rate reaches 0% by generation 15 for high and max effort.

Compared to earlier single‑call interfaces, the parallel batching reduces tool‑call latency and cuts overall execution time, while the auto‑compact mechanism preserves context length for long‑horizon rollouts.

The harness batches multiple tool calls into a single assistant turn and automatically compacts the trajectory, cutting overhead and keeping long‑horizon context stable.

How does the Claude Code harness differ from a standard sequential tool‑call pipeline?

Standard pipelines invoke each tool call one after another, incurring latency per call and quickly exhausting the context window. The Claude Code harness instead issues all required calls together, processes them in parallel, and when the context approaches 16K it compresses the earlier trajectory into a summary turn, preserving credit while freeing space.

The Claude Code harness delivers up to a two‑fold reduction in execution time by parallelizing tool calls and managing context automatically.

Analyzing Evolution Dynamics

Ablation of evolution directions reveals their distinct impact on difficulty and mutation.

Recall that the paper iterates a multi‑agent evolution loop to keep tasks challenging. Here we ablate each evolution direction—scenario, skill, and length—while holding evolution effort high and applying each direction to identical seed environments.

**Figure 3.** Environment difficulty across 15 evolution generations under low, high, and max evolution effort, evaluated independently by Hy4 preview, Claude Opus 5, and GPT-5.6 Sol.

**Figure 4.** Instruction, environment, and verification system mutation rates across generations G1–G15 under low, high, and max evolution effort.

**Table 1.** Single-step consistency of evolution-direction effects under high evolution effort. The two difficulty-change columns are measured by Claude Opus 5 with xhigh thinking effort. The first block applies each direction once to the same seed environments. The second evolves each direction for 15 consecutive steps and averages the metrics over the 15 generation transitions. Agreement between the two blocks tests whether each direction preserves a stable, direction-specific change profile along longer lineages. Difficulty changes are computed as the later generation minus the previous generation, component columns report generation-level mutation rates, and Total is their unweighted mean.

Length direction produces the steepest pass‑rate decline.

Measured $Δ$ Pass Rate of –7.1 pp in the 1‑step block.

Scenario direction yields the highest total mutation.

Instruction mutation reaches 99.8 % (1‑step) and 97.1 % (15‑step) for scenario.

All three directions consistently reduce pass rate and increase average turns.

Negative $Δ$ Pass Rate values (–4.7 pp, –4.0 pp, –7.1 pp) and positive $Δ$ Avg Turns values (+13.5, +12.5, +9.4) across directions.

RL Training and Performance

Environment evolution outperforms baselines, delivering higher terminal‑bench accuracy after RL training.

Recall that the paper iteratively mutates terminal coding tasks so agents always face a harder problem.

RFT fine‑tunes a model only on trajectories that pass a verification filter, discarding the rest.

Environment evolution achieves the highest Terminal‑Bench 2.1 accuracy, surpassing co‑evolution by 8.6% and the ensemble baseline by 11.5%.

Figure 7 shows peaks of 71.5% and 64.9% for the two models, versus 62.9%/55.1% for co‑evolution and 60.0%/52.8% for ensemble.

**Figure 5.** Early training dynamics over the first 50 training steps. Solid curves show the fraction of rollout groups that are partially solved, with one to seven successful trajectories among eight rollouts; dashed curves show mean training reward.

**Figure 6.** Trajectory length over 200 RL training steps for Qwen3.6-27B (left) and Qwen3.6-35B-A3B (right); the left and right y-axes report average turns and tokens per trajectory, respectively, with annotations at step 200. Tokens per turn increase from approximately 951 to 1,221 for Qwen3.6-27B and from 947 to 1,103 for Qwen3.6-35B-A3B.

**Figure 7.** Checkpoint performance for Qwen3.6-27B (left) and Qwen3.6-35B-A3B (right) over 200 RL training steps, with offline evaluation every 10 steps; annotations mark each method's peak accuracy and corresponding training step. Step 0 denotes the shared checkpoint after RFT used to increase policy entropy before RL training.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers