AgentGarten: Code Worlds for Evolving Agents
Jiawei Chi, Shangchen Miao, Zhiyuan Shi, Kailu Wu, Hanyang Wang, Weiliang Chen, Qiyu Dai, Jinshan Ren, Jun Gao, Mingsheng Long, Yueqi Duan, Jiangran Lyu, Jialong Wu, Fangfu Liu
AgentGarten couples programmable simulators with a shared neural renderer to enable agents to evolve through interactive, real-time code worlds.
How can we create scalable, interactive virtual environments for agent training by using code-defined physics and neural rendering?
Simulators provide rigorous physics but require expensive 3D asset creation, while video world models synthesize rich visuals but lack inspectable, editable rules. AgentGarten separates these concerns: a simulator maintains persistent state and interaction rules, while a shared neural renderer synthesizes visual observations from structured geometry conditions. This framework allows agents to practice in diverse environments, diagnose failures, and refine their performance across rounds using written playbooks, achieving sophisticated tool-use strategies in a fraction of the time required by traditional reinforcement learning.
Paper Primer
The core mechanism is a geometry-conditioned neural renderer trained via Adversarial Forcing. It acts like a translator: it takes the "blueprint" of a scene (depth or surface normals) from a simulator and renders it into a consistent, high-fidelity visual stream for the agent.
To ensure the renderer remains stable and real-time, the authors use a two-pass training schedule: a no-gradient rollout records the trajectory, followed by a differentiable replay that updates the history-encoding parameters without the memory cost of a full rollout. This is paired with an exact adversarial objective that avoids double-backward passes, allowing the renderer to maintain visual consistency over long interactions.
Agents achieve complex physical tool-use strategies significantly faster than traditional reinforcement learning.
In a hide-and-seek environment, agents learned to build shelters by round 4 and use ramps by round 10. This represents a massive reduction in training episodes compared to the millions required by previous self-play reinforcement learning baselines.
Why use geometry (depth/normals) as the interface between the simulator and the renderer?
Geometry provides a universal, structured condition that any 3D engine can export, allowing the renderer to be shared across different environments without needing bespoke visual assets for every new scene.
What is the role of the "playbooks" in this framework?
Playbooks serve as a persistent, human-readable memory for agents; they allow agents to reflect on past failures, distill insights into written skills, and pass these refined strategies to subsequent agents in future rounds.
AgentGarten shifts the bottleneck of agent training from massive-scale tabula rasa exploration to the iterative refinement of strategies within inspectable, code-defined environments.
Introduction and Motivation
We expose the need for scalable, code‑defined environments to accelerate agent learning.
Existing virtual environments either lack the fidelity needed for reliable agent training or fail to provide realistic visual observations, making it hard to scale learning across diverse worlds.
Agents need environments that are both internally consistent and visually realistic, and these environments must be easy to create and scale as agents improve.
The need for scalable, code‑defined environments is the primary driver for advancing agent learning.
The AgentGarten Framework
AgentGarten unites programmable code worlds with a shared neural renderer to let agents learn and evolve in diverse, editable environments.
Building agents that generalize requires environments that keep a faithful record of past actions while offering rich, varied visual feedback. Existing simulators give control but demand expensive asset pipelines; pure video models provide visuals but hide the underlying state.
**Figure 1.** Code worlds for evolving agents. Top: Diverse environments spanning distinct geometries, visual appearances, and interactive tasks. Bottom: Agents continuously evolve through environmental interactions, developing increasingly sophisticated strategies such as shelter-building and ramp-crossing.
Exact replay schedules the rollout as a sequence of independent blocks, recomputing each block on demand so that the regenerated trajectory matches the original bit for bit.
How does exact replay differ from standard gradient checkpointing?
Standard checkpointing stores selected intermediate activations and recomputes the rest, which can introduce numerical drift because the recomputed activations are only approximations. Exact replay stores the full deterministic engine state for each block and re‑executes the block with identical inputs, guaranteeing bitwise‑identical results and eliminating drift.
Adversarial Forcing trains the neural renderer to match its own rollout distribution while simultaneously learning from real video data, using a discriminator that is regularized with an exact R1/R2 term.
Why not just use standard distillation without the adversarial component?
Pure distillation aligns the renderer’s output to a teacher model’s predictions, which preserves short‑term fidelity but accumulates errors over many blocks, leading to noticeable visual degradation. The adversarial term forces the renderer to match the true data distribution, correcting drift that distillation alone cannot address.
AgentGarten lets agents evolve by interacting with code‑defined worlds, using a shared neural renderer to provide consistent, high‑quality visual feedback.
Code Worlds Formulation
Code Worlds separate deterministic physics from neural rendering to enable error‑free, replayable simulations.
Video‑world models entangle rendering with the environment state, so visual errors corrupt future dynamics. Code Worlds avoid this by keeping the physics engine deterministic and delegating appearance to a separate neural renderer. This split eliminates error accumulation and enables arbitrary re‑rendering of the same underlying events.
A Code World is like a video game where a physics engine runs the world while a graphics engine draws the screen – the engine updates the true state, and the renderer only produces the visual observation.
How does this differ from a traditional video‑world model where rendering is part of the state?
In a video‑world model the rendered frame feeds back into the dynamics, so any visual artifact directly perturbs future states. In a Code World the engine’s transition $f_p$ ignores the rendered observation, guaranteeing that physics remain exact regardless of rendering quality.
Initial state $s_0$: ball $(0,0)$, wall $(2,0)$; camera pose $\pi_0$ looks straight ahead.
Engine computes $s_1 = f_p(s_0,\pi_0,a_0)$ → ball moves to $(1,0)$, wall unchanged.
Condition $c_0 = h_p(s_0,\pi_0)$ yields a depth map showing the ball at depth $1$ and wall at depth $2$.
Renderer produces $x_1 = R_\theta(x_{<1},c_{\le1},x_0,y)$ → a grayscale image of the scene with the ball shifted right.
Visual history $x_{<2}$ now contains $x_1$, ready for the next step.
The engine’s state update never consulted the rendered image, so even if $x_1$ were noisy the ball’s true position remains exactly $(1,0)$.
**Figure 2.** Interaction and rendering in a code world (Eqs. 1 and 2). The agent submits actions and receives generated observations. The engine maintains scene state and exports structured conditions, here surface normals; the neural renderer combines them with an appearance reference, text, and cached visual history. New observations return to the agent and join the history.
**Figure 3.** A coding agent builds a code world from a single image. A living-room example. The agent calls perception and generation models as tools, writes a scene program from their outputs, and repairs it after inspecting rendered views and rollout checks. The expanded scene keeps the observed room and adds connected rooms beyond the input view as authored extensions. The input image supplies the appearance reference $x_0$.
By decoupling deterministic state updates from learned rendering, Code Worlds give us a reusable, error‑free substrate for training agents that can be visualized with any appearance or camera configuration.
Neural Renderer Design
We detail the renderer’s core mechanisms: geometry‑conditioned rendering, teacher‑forcing adaptation, and inference caching.
The renderer must turn geometry‑conditioned latent tokens into realistic video frames in real time, while preserving the exact history needed for gradient‑based training.
Think of a painter who first sketches the 3‑D shape of a scene and then paints colors that respect that shape – the renderer first ingests geometry tokens, then generates RGB frames that stay aligned to them.
Pool the raw condition video $c$ into two patches per frame, yielding $[p_{0},p_{1}]$ for $t=0$ and $[p_{2},p_{3}]$ for $t=1$.
Encode each patch with $E$, producing latent vectors $e_{0},e_{1},e_{2},e_{3}$.
Apply $W_{\text{in}}$ (a $4\times4$ matrix) to each encoded vector and add $e_{\text{geo}}=0$, giving geometry tokens $G_{0}=W_{\text{in}}e_{0}$ and $G_{1}=W_{\text{in}}e_{2}$.
Result: two 4‑dimensional geometry tokens that will be concatenated with the corresponding RGB tokens before entering the Transformer.
The pooling and projection turn a high‑resolution video into a handful of geometry tokens, enabling the model to stay within a real‑time budget while still preserving coarse shape information.
How does this renderer differ from a standard neural video generator that takes only RGB frames?
Standard generators treat each frame as a flat RGB token sequence, ignoring explicit 3‑D cues. Here we inject geometry tokens $G_{t}$ that are processed jointly with RGB tokens, forcing the attention layers to respect spatial structure and enabling accurate motion prediction from depth or normal information.
The cache works like a sliding window of recently rendered frames: older frames are evicted, while the current block can attend to everything that remains, ensuring low‑latency access to history without growing memory indefinitely.
After rendering $B_{1}$, it is denoised and appended to the recent window: recent = $[B_{1}]$.
Rendering $B_{2}$ reads $x_{0}$ and $B_{1}$, then after denoising the cache becomes recent = $[B_{1},B_{2}]$.
Rendering $B_{3}$ reads $x_{0},B_{1},B_{2}$; after denoising the cache evicts $B_{1}$ (oldest) and stores recent = $[B_{2},B_{3}]$.
The cache guarantees that each block sees a bounded amount of history, preventing unbounded memory growth while still providing the necessary context for coherent video generation.
Why not simply keep all past blocks in memory instead of using a bounded cache?
Storing every past block would cause memory usage to grow linearly with rollout length, quickly exceeding GPU capacity for long videos. The bounded cache discards the oldest frames, which contribute little to the immediate attention context, thus preserving both fidelity and efficiency.
Streaming inference loop with bounded KV cache.
**Figure 4.** Block-causal teacher-forcing mask, shown for two target blocks. Clean copies $P_j$ encode history and see the reference, earlier clean blocks, and themselves; noisy copies $Q_j$ see the reference, clean blocks before $j$, and themselves, but never their own clean target. Text is visible to every row. The same layout is reused, with recorded latents, by the replay pass of Adversarial Forcing.
**Figure 5.** Exact replay, drawn as block-level attention masks. Row $i$ holds block $i$'s queries, and the columns are the key/value blocks it reads. The rollout runs one attention call per block and reads earlier blocks from a detached cache. Our replay runs the same calls but computes the history keys and values with gradients, preserving the rollout's execution structure. An SGF-style replay computes the same mask in one full-sequence call, which changes numerical execution. Table 1 evaluates the difference. In the implementation each block issues two calls, one for its noisy queries and one for the clean publication that later blocks read.
**Figure 6.** Exact R1/R2 with a frozen backbone $B$ and a trainable head $h_{\phi}$. (1) Differentiate with respect to $x$ only to obtain $g$ and $R$. (2) Evaluate the backbone JVP to obtain features $z$ and directions $v$. (3) Evaluate the explicit head JVP on detached $(z,v)$, returning both the logit and its directional derivative, which supplies $s$. An ordinary backward pass with respect to $\phi$ updates the head using the full discriminator objective.
**Figure 7. The inference cache.** It retains a sink prefix, which begins with the reference, and recent frames, with configurable capacities; older frames are evicted. The current block attends to every cached frame and to the text. After denoising, the clean block joins the recent window.
Together, geometry conditioning, parallel teacher‑forcing, exact replay, and the bounded inference cache form a cohesive pipeline that delivers high‑fidelity, real‑time rendering while preserving differentiable history for training.
Experimental Results
Agents learn complex tasks in code worlds using rendered observations and iterative playbooks.
Recall that AgentGarten defines virtual environments as executable code (“code worlds”) and renders them in real‑time with a geometry‑conditioned neural video model.
Block‑by‑block SDPA replay is bitwise identical to the original rollout, eliminating recomputation error.
Table 1 shows 0 % relative $L_2$ error for our method versus 3.99 % for SGF‑style FlexAttention.
The loop lets agents iteratively improve by observing rendered frames, proposing short Python programs, and recording successes and failures in a shared playbook.
How does this evolution loop differ from a standard reinforcement‑learning training loop?
Standard RL loops update a policy by back‑propagating through a differentiable simulator or environment, often using dense reward signals. Our loop keeps the environment deterministic and opaque—agents never see ground‑truth state or gradients. Instead, they improve by inspecting rendered frames, writing explicit Python programs, and sharing concrete skill files, which makes learning a discrete, symbolic refinement rather than gradient‑based optimization.
**Table 1.** Replay accuracy and cost. Relative $L_2$ error between rollout and replay, measured on one H100 with 36 layers, $480 \times 832$ resolution, 61 latent frames, and BF16. Times are medians of three warmed-up runs of the replay forward pass and exclude backward and optimizer updates.
**Table 2. Inference cost at steady state.** Measured with five sink and 44 recent latent frames in the cache and four current latent frames, using the hand-written kernels, CUDA graph capture, and the tiny decoder. Stage times are median GPU times; the total is the mean wall time per block and also covers work outside the three stages.
**Figure 8** Qualitative comparison with Self Forcing. Each pair shows the same 30-second condition trajectory at four moments.
**Figure 9** Five code worlds, each at five moments of one rollout. In every column, the untextured scene geometry (top) conditions the generated observation (bottom). Racing frames contain first-person views from two agents sharing one world state. The scene programs specify coarse geometry; the renderer supplies appearance from the initial image, text, and visual history.
**Figure 10.** Hide-and-seek in two settings. Top: self-play reinforcement learning trains a policy network on object state and rewards. Our pretrained agents see first-person frames rendered by the neural renderer, act through short Python programs, and keep what they learn in a playbook of skill files. Bottom: three games from the run, seen from an overhead camera.
**Table 3.** Milestones of physical tool use in hide-and-seek under two distinct paradigms.
**Figure 11.** One round of practice. Agents read the task file, which states the goal, the actions, and the limits but no solution; play in parallel within a fixed budget, seeing only camera frames; and write a playbook. Playbooks are archived, and the next round’s agents start in fresh conversations from the task file and every earlier playbook.
**Figure 12.** The same loop in four further worlds. A frame from round 1 (top) and round 4 (bottom) of each run, with the world's own measure. Table 4 lists every round.
**Table 4** Outcome of every round in the four additional worlds. Each row is the world's own terminal measure, one episode per round. For the bridge, lower is better.
AgentGarten achieves high replay accuracy and enables complex agent behaviors in diverse environments.
Related Work
We situate AgentGarten among prior executable and video‑based environments, highlighting how it differs in state handling and rendering.
Executable environments have long been the substrate of embodied learning. Habitat 2.0, ManiSkill2, and ProcTHOR each provide large‑scale simulated worlds, while the hide‑and‑seek study of Baker et al. shows that simple physics environments can support increasingly sophisticated strategies.
Language models now automate environment construction: Holodeck translates language into assets and spatial constraints, SceneCraft synthesizes scene code with visual feedback, and LLMR generates and revises interactive Unity experiences. Code2Worlds and SimWorld Studio extend this to dynamic scenes with standard interfaces for embodied learning.
A related line uses code to recover dynamics rather than author them: WorldCoder infers transition and reward functions, code‑world models translate game rules into executable programs, and Code as Worlds constructs executable hypotheses from multimodal observations.
Recent work also keeps the state of the world outside the video model and uses the model solely for rendering. Our interface supplies depth or surface‑normal maps, the renderer runs causally in short real‑time blocks, and the state remains a complete executable environment for pretrained agents.
Interactive video world models such as Genie, GameNGen, Matrix‑Game 2.0, WorldPlay, Matrix‑Game 3.0, SolarWM, Astronex‑World, Genie 3, Wonder, ActWorld, and LingBot‑World generate frames conditioned on actions, camera, or retrieved context, but they embed world dynamics within the generative model. In contrast, AgentGarten advances an executable environment first and delegates appearance to a neural renderer.
Autoregressive video generation and distillation methods (CausVid, Self Forcing, LongLive, Causal Forcing, Causal‑rCM, Mask Forcing, DMD2, Self Gradient Forcing) train students to mimic rollouts, often with score‑distillation or GAN losses. Our formulation introduces exact replay, eliminating numerical divergence and matching the rollout bitwise.
Agents that learn from written experience—Reflexion, ExpeL, Voyager—store knowledge in text. Our playbooks belong to this family but differ because agents perceive the world through camera frames rendered by the neural renderer, receive no solution file, and exchange lessons via fresh conversations.
Gradient penalties such as R1, R2, and R3GAN require differentiating input gradients, which fused kernels like FlashAttention cannot support. Forward‑over‑reverse products and fused attention JVP kernels provide the exact higher‑order derivatives needed for our stable penalty with a frozen discriminator backbone.
Conclusion and Future Directions
We outline open challenges, scaling directions, and recap the paper’s contributions.
Depth maps and surface normals provide a practical conditioning interface, yet dense geometry videos repeat information already captured by an object’s pose.
Spatial downsampling cuts cost but can lose thin structures, narrow gaps, and subtle contacts needed for precise control.
Pure geometry leaves many state aspects unspecified—e.g., a symmetric object can spin without changing depth or normals, and material or color remain unknown.
Future conditioning could use compact, abstract representations such as structured text describing object attributes (orientation, angular velocity) or high‑dimensional latent features aligned with the code world’s state.
A renderer that only receives geometry cannot distinguish a spinning wheel from a static one, limiting the tasks it can support.
Enabling richer interfaces expands the set of possible worlds, but worlds must still be authored; the next step is to let coding agents write and revise worlds while the shared renderer supplies appearance.
Automatic checks should verify whether a newly generated world offers a learnable task, avoids trivial failures, and yields knowledge transferable to subsequent rounds.
When many worlds exist, each playbook is short enough to read fully, but agents must decide which lessons to keep, merge, or discard for the current scene.
Code worlds can be reset and replayed exactly, allowing agents to test a lesson before passing it on, turning successful strategies into general knowledge.
In hide‑and‑seek, agents discovered shelters by round 4 and ramp crossings by round 10, and the same procedure improved performance in four additional worlds.
AgentGarten’s design separates a simulator that maintains persistent state from a geometry‑conditioned neural renderer that produces real‑time observations, enabling agents to act, observe consequences, and iterate.
Because each world is defined as code and rendered through the same interface, the number and difficulty of environments can grow alongside the agents that train within them.
Jiawei Chi, Shangchen Miao, Zhiyuan Shi, Kailu Wu, Hanyang Wang, Weiliang Chen, Qiyu Dai, Jinshan Ren, Jun Gao, Mingsheng Long, Yueqi Duan, Jiangran Lyu, Jialong Wu†, Fangfu Liu†
Affiliations: MirroS, Tsinghua University, Peking University (†Project lead).
Technical Appendices
Low‑level implementation details, training schedules, and task specifications.
This appendix records the concrete settings that enable reproducible experiments, from the renderer’s backbone to the exact task files used for evaluation.
**Table 5.** Training stages. Stages 1 and 2 train first on five-second and then on fifteen-second windows; each distillation run uses the teacher and fake score of the matching window length. The teacher is the stage-1 model and stays frozen. The text stream and the video autoencoder are frozen throughout.
During causal block processing the reference prefix is never assigned a diffusion timestep and is excluded from the loss; each target block receives four latent frames, each perturbed independently by one of four equally likely augmentations (contrast/exposure scaling, Gaussian noise, bilinear resampling, or identity).
Distillation follows a four‑step trajectory ℰ = (1600/1601, 15/16, 5/6, 5/8, 0); the teacher runs with classifier‑free guidance scale 6, while the student receives five fake‑score updates per teacher update, and the loss normalizes by the mean absolute error of the teacher’s velocity‑corrected predictions.
Freezing the backbone means its features z and Jacobian A are independent of the trainable parameters $\phi$, so the gradient g = Aᵀ u is obtained via a Jacobian‑Vector Product (JVP) and the generator loss reduces to a simple squared‑norm R = gᵀg, avoiding second‑order terms that would otherwise require back‑over‑backward.
The streaming cache stores recent latent frames and a limited number of “sink” frames; during distillation the cache configuration (e.g., five sink frames + 44 recent frames) determines throughput, and eviction removes both RGB and condition keys/values together.
Task files for the four additional worlds (Companion dog, Narrow bridge, Herding, Quarry loader) specify the goal, action set, fixed durations, and evaluation criteria; all actions last exactly one second, and world time advances only when the agent thinks or executes a batch.
All low‑level choices—hardware, learning‑rate schedule, cache policy, and task definitions—are fully specified to enable exact replication of the reported results.