Programmable World Model
Zheng-Hui Huang, Guixu Lin, Jiacheng Lin, Yi-Chuan Huang, Ruihan Yu, Muyao Niu, Siqi Yang, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang, Zhixiang Wang
Decoupling world-state logic from visual generation to enable persistent, programmable interactive environments.
How can we maintain persistent world state and enforce programmable rules in video world models?
Video world models generate realistic visuals but struggle to maintain persistent state, often losing track of entities or failing to enforce consistent rules during long interactions. The Programmable World Model (PWM) decouples state evolution from visual rendering: a lightweight engine executes logic programs to track entity states, while a generative renderer produces visuals conditioned on these explicit spatial controls. This separation achieves 94% Count Accuracy and 98% State Accuracy on the CombatStateBench, significantly outperforming existing models in maintaining world-state consistency over long horizons.
Paper Primer
The core mechanism hinges on state-augmented 3D oriented bounding boxes (OBBs) that act as a bridge between the engine and the renderer. The engine maintains a canonical world state—including off-screen entities and non-visual attributes like health—and a deterministic compiler projects these OBBs into pixel-aligned identity, semantic, and motion maps that guide the generative backbone.
PWM maintains significantly higher world-state consistency than implicit generative models.
On the CombatStateBench, PWM achieved 94% Count Accuracy and 98% State Accuracy, compared to 40.75% and 8.00% respectively for the LingBot-World-V2 baseline. A 53.25 percentage point improvement in Count Accuracy and a 90.00 percentage point improvement in State Accuracy.
Explicit structural conditioning improves visual consistency without degrading image quality.
PWM outperformed baselines across all VBench metrics, including Subject Consistency (94.74%) and Background Consistency (96.98%). Consistent gains across all four perceptual and temporal quality metrics.
Why is an explicit engine necessary if modern video models are already highly capable?
Generative models represent world facts implicitly, making them prone to "hallucinating" state changes, such as characters disappearing or killed entities continuing to move. An explicit engine provides an authoritative, verifiable record of state that remains consistent regardless of camera motion or occlusion.
Does this approach limit the generative model's ability to create complex visual details?
No; the OBBs only constrain the spatial support, identity, and motion of entities. The generative renderer retains the freedom to synthesize fine-grained details like articulation, texture, material, and secondary motion that are not explicitly defined by the box geometry.
Programmable World Models
We expose why current video models lack persistent, rule‑driven world state and propose a programmable solution.
Recent video world models can synthesize realistic, responsive environments, but they stop at generating plausible observations. They lack an explicit, persistent global state and a way to encode executable rules that govern how the world evolves.
Existing interfaces only let users move cameras, issue actions, or give high‑level prompts, offering no direct handle on individual entities or on the logic that drives interactions. Consequently, off‑screen entities, inventories, and task progress cannot be accessed or updated, and users cannot program conditions such as “open the door when the key is collected.”
A framework that separates an editable, persistent world state from the visual generator, letting a coding agent define entity states and rules that the renderer then visualizes.
**Figure 1** Overview of the programmable world model. Given a single reference image and a user description, a coding agent generates executable programs that specify entity states and interaction rules. A lightweight engine executes these programs to maintain and evolve an explicit world state, represented by state-augmented 3D OBBs. This representation is compiled into conditioning signals for a generative renderer, which produces realistic visual observations consistent with the world state. This design allows users to create playable games by defining game mechanics in advance, directly controlling individual entities, and maintaining persistent world state throughout gameplay.
The shift from passive video generation to rule‑based, persistent world state management unlocks controllable, consistent interactive environments.
Representation Trade-offs
We examine how representation granularity balances controllability, annotation cost, and training‑inference alignment.
The choice of representation directly shapes what a system can control, how costly that control is, and whether training‑time observations match inference‑time constructions.
**Figure 2** State representation trade-offs. Candidate representations range from lightweight semantic descriptions to detailed geometric structures [11]. Greater structural detail enables finer-grained control but increases the cost of training-data annotation and inference-time state construction and evolution. Moreover, representations extracted from observations or recorded from engines during training may differ from those constructed programmatically at inference time, creating a potential training–inference mismatch. We adopt state-augmented 3D OBBs as an intermediate abstraction that supports explicit world-space control while leaving fine geometry, articulation, and secondary visual dynamics to the generative renderer.
An OBB captures an entity’s position, size, and orientation in a shared world frame, giving enough structure for explicit control without the heavy annotation cost of full 3D geometry.
Project Entity A into a camera at $(0,0,5)$: the OBB projects to a 2D rectangle centered at $(1,2)$ with width 1 and height 1.
Rotate Entity B by updating its rotation matrix to $R_B' = R_B \cdot \text{Rot}_z(30°)$, yielding a new orientation.
Translate Entity A by velocity $(0.1,0,0)$ → new center $(1.1,2,0)$.
Re‑project both OBBs into a second camera at $(-2,0,5)$, obtaining new 2D boxes that respect the updated world‑space positions and orientations.
OBBs let the system manipulate high‑level pose and motion while guaranteeing that any rendered view remains geometrically consistent, all with only a few parameters per entity.
Thus, OBBs strike a sweet spot: they provide explicit, view‑independent control over entity geometry and dynamics while keeping annotation and inference costs low enough for open‑domain programmable worlds.
Related Work
Survey of prior video world models, explicit‑state approaches, and generative rendering techniques.
Interactive video world models have progressed to action‑conditioned generation, controllable camera motion, and long‑horizon rollouts, yet they are trained chiefly on pixel‑level objectives that do not enforce a persistent, authoritative world state.
Explicit‑state world modeling attempts to represent internal environment variables; StatePlay predicts both visuals and game‑state variables, while MASS introduces a typed shared state driven by a learned Logic Engine, both of which still rely on learned dynamics that can accumulate errors.
Generative rendering leverages learned priors to synthesize realistic frames from structured scene cues, ranging from diffusion‑based G‑buffer pipelines (DiffusionRenderer) to dynamic‑world renderers (AlayaRenderer series) and coarse‑to‑real pipelines that use simplified geometry to constrain scene layout while delegating fine details to the generative model.
The Programmable Architecture
How a coding agent builds and updates a persistent world state for controllable video generation.
Video world models can produce realistic frames but they drift because they lack a persistent, rule‑driven state. The method solves this by letting a coding agent maintain an executable canonical world state that survives across interactions.
The canonical world state is a structured snapshot that records every entity’s geometry, attributes, relations, and the executable rules that govern how actions change them.
Step 1: $s_0 = (E_0, A_0, Q_0; R_0)$ records the OBBs, attributes, empty relation set $Q_0$, and the opening rule.
Step 2: Player action $a_0$ = “move forward 1 m”. Engine $F$ checks the avatar’s position, updates its OBB, and leaves $A$ unchanged.
Step 3: After movement, the avatar’s OBB now overlaps the chest’s OBB, triggering the rule in $R_0$.
Step 4: Engine applies the rule, changing the chest’s attribute status to “open”. The updated state $s_1$ reflects this new attribute.
The canonical state cleanly separates visible geometry from hidden game logic, allowing the renderer to stay faithful to the visual scene while the engine enforces consistent gameplay rules.
The coding agent is a language‑conditioned orchestrator that translates a natural‑language description of the world into a concrete program that populates the canonical state.
How does this Coding Agent differ from a typical reinforcement‑learning agent?
A reinforcement‑learning agent learns a policy by trial‑and‑error interaction, whereas the coding agent directly synthesizes a deterministic program from language. It does not explore; it writes the exact state‑initialization and rule set that the engine will execute.
Detect 3D OBBs from the initial frame $I_0$.
Pass the OBBs and a natural‑language description $q$ to the coding agent, which produces a world program.
Engine executes the program to instantiate the canonical state $s_0$.
At each step $t$, the player issues an action $a_t$.
Engine transition $F$ updates the state: $s_{t+1}=F(s_t,a_t)$.
State compiler $P$ projects $s_{t+1}$ under the target camera $C_{t+1}$ into three aligned control maps (identity, semantic, direction) and concatenates them into $M^{\text{ctrl}}_{t+1}$.
Generative renderer $G$ consumes $I_0$, the camera trajectory $\{C_t\}$, and the control sequence $\{M^{\text{ctrl}}_t\}$ to synthesize the next video frame $I_{t+1}$.
Training: realized dynamics $\longrightarrow$ structural representation.
Inference: state transition $\longrightarrow$ structural representation $\longrightarrow$ generated dynamics
**Figure 3.** Architecture of the programmable world model. (1) Agent-orchestrated world programming starts from natural-language specifications to instantiate an executable canonical world state, which is maintained and advanced by a lightweight engine according to player actions and world rules. (2) Control compilation projects state-augmented 3D OBBs under the target camera into pixel-aligned identity, semantic, and motion-direction controls. (3) Generative rendering conditions a pretrained camera-controlled video generation model on these structured controls and visual history to synthesize the next video chunk. Completed chunks are incorporated into temporal and geometry-aligned spatial memories to support long-horizon generation.
Experimental Results
Quantitative and qualitative evaluation of our programmable world model.
Recall that the programmable world model keeps an explicit, executable canonical state while a video generator renders the visuals. We train on HUD‑free gameplay videos from Cyberpunk 2077, Forza Horizon 6, and GTA V, then evaluate on the newly built CombatStateBench benchmark of 50 controlled clips.
A prior interactive video world model that encodes world state implicitly inside its generative network.
An interactive video world model that augments generation with learned control signals but still lacks an external, executable state.
Our method improves world‑state consistency by over 50 % relative to LingBot‑World‑V2.
Table 1 shows the quantitative gap across all metrics.
**Table 1.** Video quality and world-state evaluation on 50 CombatStateBench clips. All values are reported as percentages. Count Accuracy is evaluated over 400 sampled frames (eight per clip), and State Accuracy over 50 death events, with three post-transition frames sampled per event.
**Figure 4.** Qualitative comparison of entity-death interactions over five generated frames. We compare LINGBOT-WORLD-v2 and YUME with our method under (a) a static camera and (b) a dynamic camera with moving entities. The input rows visualize the structured control signals provided to our renderer. Our method follows the specified death events while preserving the remaining entities and the scene evolution.
**Figure 5.** Additional qualitative results shown at five selected time steps. (a) Generalization to a novel minotaur scene with dynamic camera and entity motions. (b) Under a large-angle camera rotation, the model preserves scene consistency and correctly reveals characters located behind the initial camera view, following the persistent world state maintained by the engine. (c) Generalization to a racing game with moving cameras and vehicles. (d) Joint rendering of heterogeneous object categories, including humans and vehicles. (e) Selected frames from an 897-frame autoregressive sequence in which many NPCs progressively enter the scene. The input rows visualize the global scene state in (a)–(b) and camera-view 3D OBB maps with ground-grid references in (c)–(e). The generated results consistently follow the specified camera motion, spatial layout, and entity states.
PWM maintains state consistency significantly better than non‑programmable baselines.