From Pixels to States: Rethinking Interactive World Models as Game Engines

Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, Kaipeng Zhang

A framework for interactive world modeling that maps generative video models to the action-state-observation loop of game engines.

How can we bridge the gap between pixel-based video generation and the structured, rule-based dynamics of traditional game engines?

Generative video models can simulate interactive worlds, but they often struggle to maintain logical consistency because they treat the world as a sequence of pixels rather than a rule-governed system. The authors propose organizing these models around the recurrent action-state-observation loop used in conventional game engines, evaluating them across four dimensions: action control, state dynamics, persistence, and real-time generation. This analysis reveals that while current models excel at visual fidelity, they lack explicit state management, which is necessary for consistent, rule-based interactions over long horizons.

Paper Primer

The paper frames the challenge of interactive world modeling as a failure to replicate the "game engine loop." In a game like *Black Myth: Wukong*, an action (e.g., a sword strike) does not immediately produce pixels; it first updates an internal state (stamina, boss health, animation phase) which then dictates the visual outcome.

The authors categorize existing research into this loop: player action control (how intent is represented), game state dynamics (how the world evolves), state-observation persistence (maintaining consistency over time), and real-time generation (latency management). This taxonomy serves as a diagnostic tool to identify why current models diverge from true game engines.

Current generative models lack explicit state representation, leading to logical inconsistencies in long-horizon interactions.

The authors observe that models relying on implicit pixel-based dynamics fail to guarantee spatial and logical consistency when game rules (e.g., damage thresholds) dictate outcomes that are not visually obvious. The field has reached real-time generation and natural input control, but remains limited by the inability to preserve consequences beyond the current view or align outcomes with rule-defined moments.

Why is the "game engine loop" a better organizing principle than simply improving video generation quality?

Video generation models focus on pixel correlations, which are insufficient for games where the same input (e.g., a strike) must produce different outcomes based on hidden variables like boss health or cooldowns. The loop forces the model to account for these non-visual, rule-governed state transitions.

Does this paper propose a new generative architecture?

No, this is a systematic review and resource paper. It provides a unified framework to evaluate existing models and contributes a high-quality, state-annotated dataset to enable future research into explicit state-aware modeling.

Interactive World Models as Game Engines

We expose why pixel‑only video generation fails to enforce game rules and propose a state‑aware modeling shift.

Current video generative models predict future frames from player actions but lack an explicit representation of game rules, leading to inconsistent and brittle interactions. A genuine interactive world requires rule‑consistent dynamics, long‑term persistence, and real‑time generation, which conventional engines achieve via an explicit action‑state‑observation loop.

Pixel‑only prediction cannot guarantee that generated interactions obey the underlying game rules, so models must reason over an explicit game state.

**Figure 1.** Interactive game worlds as an action-state-observation loop. Player inputs update the internal game state according to game rules, and the resulting state is rendered into observations that continuously feed back to the player.

Shifting from pixel‑only prediction to state‑aware dynamics is the key to rule‑consistent, real‑time interactive game worlds.

Foundations of Video Generation

We survey video generation and interactive game models, highlighting their action‑control strategies.

Recent video generation and interactive game world research diverges along how language, vision, and player intent are coupled.

These models synthesize video frames from textual prompts, typically by feeding a diffusion transformer (DiT) with a latent representation of the scene.

The diffusion transformer architecture that generates video frames in a latent space conditioned on textual embeddings.

Extends DiT with modality‑specific attention weights, allowing simultaneous processing of text and visual tokens.

Fuses dual streams (text and video) into a single DiT pipeline, sharing parameters across modalities.

Uses a concatenated sequence of text and video tokens normalized by expert‑specific layers.

A neural game engine that learns to generate game frames conditioned on player actions for a single game.

Embeds frame‑wise keyboard and mouse states as conditioning vectors within a diffusion model.

Interleaves discrete action tokens with visual tokens, enabling the model to generate game frames conditioned on player actions.

Action control determines how a player’s intent is encoded before it influences video generation, shaping the interaction loop.

Conditions video generation on external camera‑pose trajectories encoded as rotations and translations.

Embeds Plücker ray representations of camera motion for image‑to‑video generation under epipolar constraints.

Extends camera‑control to image‑to‑video generation by conditioning on visual cues.

Applies epipolar‑constrained attention to transform a single image into a video sequence.

Localizes action effects (e.g., firing) to specific regions, embedding them as conditioning vectors.

Uses per‑agent natural‑language commands to steer individual entities during rollout.

Conditions multiple entities simultaneously through per‑chunk, per‑entity language prompts.

Steers high‑level NPC strategies using language, decoupled from direct player input.

Extends navigation‑centric models to object‑level interaction via action‑aware memory mechanisms.

The Neural Game Engine

Explicit game‑state tracking enables rule‑consistent video generation.

Video generators that predict pixels directly cannot guarantee that actions obey game rules, leading to implausible interactions. The remedy is to keep an explicit, up‑to‑date description of the world and let the generator consult it at every step. This section introduces the mechanism that makes rule‑consistent gameplay possible.

Think of the game state as a spreadsheet of variables—health, stamina, cooldowns, equipment—that the engine rewrites according to deterministic rules whenever the player acts.

Action arrives: “heavy strike”.

Rule checks stamina ≥ 20 → true, so stamina ← stamina − 20 = 30.

Rule checks health > 0 → true, so health ← health − 30 = 70.

Updated state (health = 70, stamina = 30) is emitted for the next frame.

Because the update is a pure arithmetic rewrite, the same action always yields the same state change, eliminating hidden drift.

Like a game master that watches the spreadsheet, applies the rulebook, and then hands the revised state to a Diffusion Transformer (DiT) to paint the next frame.

How does this differ from latent‑state world models that also update a hidden vector?

Latent models learn an opaque vector whose semantics are entangled with visual features; they can drift or misinterpret non‑visual causes (e.g., a strike that should kill but only wounds). The Neural Game Engine keeps the variables separate and interpretable, so the same action always produces the same rule‑driven update, guaranteeing logical consistency.

Receive player action aₜ and current explicit state sₜ.

Apply deterministic rule function f to obtain new state sₜ₊₁ = f(sₜ, aₜ).

Encode sₜ₊₁ into conditioning vector $c$ₜ₊₁ = W sₜ₊₁.

Feed cₜ₊₁ and recent frames into the Diffusion Transformer to generate frame xₜ₊₁.

Render xₜ₊₁ to the player and repeat.

State‑Observation Persistence and Real‑Time Interactive Generation build on this loop: the explicit state persists across frames, and the lightweight rule update satisfies the strict latency budget required for responsive gameplay.

Curating State-Annotated Gameplay

We built a large, fully annotated gameplay dataset to train neural game engines.

The paper’s core claim is that generative video models lack explicit game‑state modeling, leading to inconsistent interactions. This section shows how we obtain the missing state‑annotated data at scale.

Our data engine yields a 90‑hour, fully state‑annotated gameplay corpus, far surpassing existing video datasets in rule‑consistent supervision.

Over 90 hours of Black Myth: Wukong boss encounters captured at 1280×720, 30 FPS, with per‑frame actions, engine states, and RGB + depth observations.

The pipeline turns raw crowdsourced gameplay recordings into training‑ready samples with precise action, state, and visual streams.

How does this pipeline differ from typical video‑dataset collection pipelines?

Standard pipelines only capture raw pixels and maybe audio; they lack per‑frame engine actions and state vectors. Our pipeline injects a structured JSON stream from the game engine and augments the video with depth maps and dual captions, providing the explicit rule‑level supervision that generic video datasets cannot offer.

**Figure 2.** The dataset curation pipeline.

Explicit state annotations are essential for training neural game engines that respect game rules.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers