ENVACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

Zishan Xu, Zhiyuan Yao, Yuxin Chen, Yifu Guo, Zhengxi Lu, Yuquan Lu, Jinyang Huang, Yan Xu, Yasheng Wang, Weinan Zhang, Xingshan Zeng, Weiwen Liu

EnvACE internalizes environment dynamics into a single policy, enabling agents to rehearse tool interactions internally before execution.

How can we train language agents to perform long-horizon tool use without relying on costly external simulators or real-world environment interactions?

Training language agents typically requires costly external environments or simulators to provide feedback, which limits scalability and often leaves environment modeling outside the policy itself. EnvACE replaces these external dependencies with "world rehearsal," where a shared policy acts as both the agent and the environment, generating its own tool responses to internalize environment dynamics. This unified approach allows the policy to anticipate outcomes and refine decisions before interacting with real tools, consistently outperforming standard reinforcement learning baselines across multiple benchmarks.

Paper Primer

EnvACE treats environment interaction as a self-contained process: the policy generates a tool call, then immediately generates the corresponding environment response, appending both to its history. By jointly optimizing these acting and rehearsal roles using role-wise Group Relative Policy Optimization (GRPO), the model absorbs the causal relationship between actions and responses directly into its parameters.

World rehearsal significantly improves task success rates across heterogeneous agentic benchmarks.

EnvACE-8B outperformed environment-scaling baselines on BFCL-v4, $\tau\$2-Bench, and VitaBench, achieving an overall score of 32.91%. On $\tau\$2-Bench, EnvACE improved the average score by 5.5% over standard GRPO training.

Internalized world models enable effective test-time scaling without additional external interaction.

Performing private rehearsal attempts before committed execution increased the overall benchmark score by 4.2% compared to non-rehearsal execution. Parallel rehearsal with EnvACE consistently outperformed base-model rehearsal, confirming that gains stem from internalized environment knowledge rather than just extra compute.

Why is parameter sharing between the acting and rehearsal roles critical?

Sharing parameters ensures that knowledge acquired during rehearsal—specifically how the environment responds to certain actions—is directly incorporated into the acting policy, allowing it to function as an agent world model.

How does this approach differ from using an external simulator?

External simulators remain separate from the policy, meaning the agent only consumes their output. EnvACE internalizes the dynamics, allowing the policy to "think" through consequences and repair invalid tool calls before they are ever executed in the real world.

Introduction and Motivation

EnvACE replaces costly external feedback with internal world rehearsal for LLM agents.

Training LLM agents for long‑horizon tool use currently depends on costly real or simulated environments; building and verifying these pipelines scales poorly.

External rollouts force every training step to query an outside system, making large‑scale learning prohibitively expensive.

**Figure 1.** Comparison of three agent rollout paradigms: real-environment rollout, external-simulator rollout, and EnvACE world rehearsal. Existing approaches obtain environment responses from external sources, whereas EnvACE internalizes environment responses within the policy through world rehearsal.

The key shift is moving from reliance on external environment feedback to internal simulation via world rehearsal.

Background and Related Work

Key prior ideas and the formal task model that EnvACE builds on.

Recent benchmarks push language agents toward long‑horizon, tool‑augmented tasks, prompting a surge of work on agentic reinforcement learning and on learning environment dynamics for LLMs.

Extends standard RL to language agents that interleave reasoning, actions, and observations, enabling long‑horizon tool use, web navigation, and software engineering.

Approaches that synthesize or learn environment dynamics so that language agents can obtain feedback without executing costly external processes.

It treats a language model as an RL actor that repeatedly reasons, issues tool calls, and consumes the resulting observations, closing the loop between thought and action.

The task is modeled as a finite‑horizon partially observable Markov decision process where the agent’s actions include tool calls and the environment’s observations include tool outputs.

We optimize the objective with Group Relative Policy Optimization (GRPO), which normalizes each trajectory’s reward against other rollouts for the same instruction.

The EnvACE Framework

Method introduces world rehearsal and role‑wise GRPO to internalize environment dynamics.

External environment feedback dominates the cost of training long‑horizon agents; EnvACE removes that bottleneck by letting the policy generate its own imagined observations.

World Rehearsal lets the policy rehearse the next observation before it ever talks to the real world — like a playwright runs through a scene privately before the live performance.

Step 0 (ACT): sample $a_0 = \\text{"move forward"}$ from $\\pi_{\\theta}(\\cdot \\mid h_0, \\text{ACT})$.

Step 0 (REHEARSE): generate $\\hat{o}_0 = \\text{"wall ahead"}$ from $\\pi_{\\theta}(\\cdot \\mid h_0, a_0, \\text{REHEARSE})$.

Update history: $h_1 = h_0 \\oplus (a_0, \\hat{o}_0) = [(\\text{move forward}, \\text{wall ahead})]$.

Step 1 (ACT): given $h_1$, sample $a_1 = \\text{"turn left"}$.

Step 1 (REHEARSE): generate $\\hat{o}_1 = \\text{"open corridor"}$.

Update history: $h_2 = h_1 \\oplus (a_1, \\hat{o}_1)$ now contains two action‑observation pairs.

The policy learns a mapping from “move forward” to “wall ahead” and from “turn left” to “open corridor” without ever querying a real simulator.

GRPO treats the ACT and REHEARSE outputs as two teams that each get its own score baseline, ensuring the policy improves both acting and internal simulation simultaneously.

Collect ACT outputs: $G_{x,\\text{ACT}}$ contains three tokens from Rollout 1.

Collect REHEARSE outputs: $G_{x,\\text{REHEARSE}}$ contains two tokens from Rollout 2.

Compute baselines: $\\mu_{x,\\text{ACT}} = (8+8+8)/3 = 8$, $\\mu_{x,\\text{REHEARSE}} = (5+5)/2 = 5$.

Advantages: each ACT token gets $A = 8-8 = 0$, each REHEARSE token gets $A = 5-5 = 0$ (illustrating a case where both roles already match their baselines).

Even with zero advantage, the clipped objective still updates $\\theta$ via the likelihood ratios, preserving stability.

Separate baselines ensure that a high reward from ACT does not inflate the advantage of REHEARSE outputs, and vice‑versa.

Why not use a single baseline for all outputs instead of role‑wise baselines?

A single baseline would conflate the very different distributions of ACT actions and REHEARSE observations, causing the policy to receive misleading credit signals; role‑wise baselines keep the credit assignment faithful to each role’s purpose.

Given instruction $x$, generate $N$ private rehearsal attempts from the act–rehearse loop, each yielding an imagined trajectory $\\tilde{\\tau}^{(n)}$.

Evaluate each trajectory with the policy’s internal feedback function to obtain $f^{(n)}$ (assessment + revision suggestion).

In PARALLEL mode, attempts are independent; in SEQUENTIAL mode, attempt $n$ conditions on all previous $(\\tilde{\\tau}^{(j)}, f^{(j)})_{j<n}$.

Aggregate all feedback into a compact rehearsal memory $m_x$ (e.g., by averaging token‑level embeddings).

The ACT role then conditions on $m_x$ and executes a single committed rollout in the real environment.

Attempt 1 produces trajectory $\\tilde{\\tau}^{(1)}$ and feedback $f^{(1)} = \\text{"turn right"}$.

Attempt 2 produces $\\tilde{\\tau}^{(2)}$ and feedback $f^{(2)} = \\text{"go straight"}$.

Rehearsal memory $m_x$ concatenates the two suggestions, yielding a weighted decision that prefers “go straight” because its confidence score is higher.

The ACT role receives $m_x$ and issues the final action “move forward” in the real environment.

Parallel rehearsals let the policy compare alternative strategies before committing, reducing costly missteps in the external world.

**Figure 2.** Overview of EnvACE and world rehearsal. Unlike conventional agentic RL, where an external environment provides observations after each action, EnvACE internalizes the interaction loop into a single policy. At each turn, the policy first produces an action, then rehearses the corresponding environment response, and conditions its next decision on the self-generated observation. This unified act–rehearse process enables the policy to learn environment dynamics during training and to simulate multiple candidate actions at test time before executing the most promising one in the real environment.

Performance Benchmarks

EnvACE outperforms all baselines on three benchmark suites, achieving the highest overall score.

EnvACE attains the highest overall benchmark score, beating the next‑best baseline by 0.99 %.

Table 1 shows EnvACE 32.91 % versus EnvScaler‑8B 31.92 %.

**Table 1.** Benchmark results across BFCL V4, $\tau^2$-Bench, and VitaBench. Overall is the arithmetic mean of the BFCL V4 Avg., $\tau^2$-Bench Avg., and VitaBench Avg. Blue cells indicate the highest result in each column, while red cells indicate the second-highest distinct result.

On BFCL V4, EnvACE reaches 46.04 %, 2.00 % higher than Qwen3‑8B and within 1.03 % of the top EnvScaler‑8B.

Table 1 reports 46.04 % for EnvACE versus 44.04 % for Qwen3‑8B and 47.07 % for EnvScaler‑8B.

EnvACE’s $\tau$2‑Bench average of 36.7 % is the second‑highest, exceeding EnvScaler‑8B, AWM‑8B, and AWM‑14B by 3.8 %, 5.5 %, and 6.0 % respectively.

Table 1 lists $\tau$2‑Bench scores: EnvACE 36.7 %, EnvScaler‑8B 32.9 %, AWM‑8B 31.2 %, AWM‑14B 30.7 %.

On VitaBench, EnvACE achieves the best 16.0 % average, edging out EnvScaler‑8B by 0.2 % and beating ScaleEnv‑8B by 1.0 %.

Table 1 shows VitaBench scores: EnvACE 16.0 %, EnvScaler‑8B 15.8 %, ScaleEnv‑8B 15.0 %.

**Table 2.** Performance comparison on FinMCP-Bench using TR, TP, and TF1 metrics. Values are reported as percentages (%). Blue cells indicate the highest result in each column, while red cells indicate the second-highest result.

EnvACE‑8B records the highest TF1 score of 46.78 %, 3.10 % above EnvScaler‑8B.

Table 2: TF1 – EnvACE‑8B 46.78 % vs. EnvScaler‑8B 43.68 %.

EnvACE consistently outperforms external‑environment baselines across diverse benchmarks.

Ablation and Scaling Analysis

World rehearsal lets EnvACE simulate feedback, boosting policy performance across scales.

Recall that EnvACE internalizes environment dynamics via world rehearsal, letting the policy learn from simulated feedback instead of costly external interaction.

EnvACE raises the $\tau$2‑Bench score from 31.2 % to 36.7 % at the 8B scale, a +5.5 % gain over standard GRPO.

Figure 3 shows the controlled comparison at the 8B scale.

Sharing parameters between acting and rehearsal improves the $\tau$2‑Bench score from 35.5 % to 36.7 %, a +1.2 % boost over the Per‑role Policy variant.

Figure 3 compares EnvACE with Per‑role Policy, which uses separate policies for the two roles.

EnvACE’s offline evaluation score climbs from 30.0 % at training step 50 to 36.7 % at step 470, a +6.7 % improvement.

Figure 5 tracks the training trajectory of EnvACE‑8B on $\tau$2‑Bench.

**Figure 3.** Ablation results on $\tau^2$-Bench. EnvACE achieves the best performance at both the 1.7B and 8B scales.

**Figure 4.** EnvACE performance across model scales. Scaling from 1.7B to 8B improves performance on both benchmarks.

**Figure 5.** Evaluation performance of EnvACE-8B throughout RL training on $\tau^2$-bench.

Test-Time Scaling Performance

Test-time scaling gains are quantified for parallel and sequential rehearsal with EnvACE.

Parallel rehearsal with EnvACE attains the highest overall test‑time score of $40.9\%$, a $4.2\%$ gain over the Non‑TTS baseline.

Table 3 shows $40.9\%$ vs. $36.7\%$ overall, with consistent improvements across both $\tau^2$‑Bench and BFCL Multi‑Turn.

Parallel and sequential modes both improve when the rehearsal policy is EnvACE rather than the base model, indicating that the gains stem from the internalized environment‑response knowledge rather than extra inference compute.

**Figure 6.** TTS scaling performance on BFCL Multi-Turn.

Qualitative Case Study

EnvACE secures a reward of 1 while baselines fail, showing proactive failure avoidance.

EnvACE obtains a reward of 1 while both EnvScaler8B and Vanilla receive 0, demonstrating successful anticipation and repair of tool‑call failures.

In both Case A and Case B the EnvACE agent avoids invalid actions and is credited with reward 1; the baselines either issue the invalid call or require extra recovery steps and receive reward 0.

Across the two scenarios, EnvACE’s world rehearsal predicts the tool‑call outcome, revises the call, and completes the task in a single turn. The baselines either proceed with the original call and later correct it (EnvScaler8B) or cascade through multiple clarification steps and even execute a forbidden write (Vanilla), incurring zero reward.

**Figure.** Case B: EnvACE

**Figure 7.** Case study of EnvACE compared with EnvScaler8B and Vanilla agents. EnvACE predicts potential tool-call failures before execution and automatically repairs invalid parameters, while baseline agents require additional recovery steps after failure.

World rehearsal simulates the call and predicts the response “Order not found – missing leading #”.

EnvACE rewrites the call to

The revised call is executed, returning a valid order status.

By simulating the tool response before execution, EnvACE avoids the error entirely, turning a potential failure into a successful interaction.

EnvACE’s ability to anticipate and repair tool‑call failures before execution yields reliable, single‑turn interactions.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers