Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds
Jihai Zhang, Runjia Qian, Kaiqin Kong, Biao Jiang, Puyi Wang, Siru Chen, Tianle Li, Guanjie Chen, Junyi Wu, Chubin Chen, Yuanhang Li, Kaichen Huang, Haobo Zhang, Yangguang Li, Wei Yu, Guibin Chen, Yu Cheng, Yahui Zhou
Matrix-Game 4.0 separates game logic from visual rendering to enable real-time, playable game worlds on consumer GPUs.
How can we decouple game logic from visual generation to enable real-time, interactive, and persistent playable worlds?
Interactive video models struggle to maintain persistent world states and complex rules, often collapsing into inconsistent visuals as errors accumulate during long-running gameplay. Matrix-Game 4.0 solves this by decoupling the system: a lightweight game engine maintains the world state and rules, while a distilled video model acts as a real-time renderer for the engine's low-poly proxies. This architecture achieves 60 FPS gameplay with 112 ms latency on a single RTX 5090, enabling interactive, modifiable game worlds that respond to natural-language instructions.
Paper Primer
The system functions as a three-part pipeline: a multimodal coding agent translates player intent into executable game code, a game engine enforces physics and state, and a causal video model renders the engine's structural output into high-fidelity frames. The renderer is like a specialized film crew: it receives a storyboard of low-poly shapes and dynamic scene descriptions from the engine and paints the final visuals in real-time, ensuring the world remains consistent even when objects move off-screen.
Matrix-Game 4.0 enables real-time, playable video generation on a single consumer GPU.
The system achieves 480p, 60 FPS gameplay with a key-press-to-display latency of 112 ms on an NVIDIA RTX 5090. This represents a 19x increase in steady-state throughput and a 9.9x reduction in latency compared to the baseline configuration.
The system supports dynamic gameplay modification via natural language.
A multimodal coding agent edits game code and scene configurations in real-time, allowing players to change mechanics like health systems, NPC behavior, or game modes on the fly. The engine maintains persistent state for these rules, ensuring that visual generation remains grounded in the updated game logic.
Why separate the game engine from the video model instead of letting the model learn the world dynamics directly?
Pure video models struggle with long-horizon consistency and explicit rule enforcement because they rely on statistical correlations rather than causal understanding. By offloading state and rules to an external engine, the system prevents error accumulation and ensures that game mechanics like inventory or quest progress remain stable regardless of visual generation.
What is the scope of this system — can it render any game?
The current framework is designed for games where the engine can output structural proxies (low-poly geometry) and semantic metadata. While it demonstrates shooter, driving, and air-combat genres, it requires a paired dataset of low-poly and high-fidelity frames for training, meaning it is not a general-purpose engine for arbitrary existing games without adaptation.
Introduction
We expose the core gap: video generators lack persistent world state and low‑latency interaction, motivating Matrix‑Game 4.0.
Matrix‑Game 4.0 wraps a video generation model in an external harness that maintains persistent world state via a lightweight game engine, letting the model focus solely on high‑fidelity rendering of engine‑provided low‑poly proxies. A multimodal coding agent turns player ideas into game code and dynamic captions, while an autoregressive video renderer converts the low‑poly frames into detailed visuals conditioned on those captions and memory references.
With targeted inference optimizations the system runs at 480p, 60 FPS on a single RTX 5090, achieving a key‑press‑to‑display latency of 112 ms; single‑step generation reaches 83 FPS using 18.2 GiB of GPU memory. Steady‑state throughput also reaches 67 FPS on RTX 5080 and 58 FPS on RTX 5070 Ti.
Video generators face four key challenges: no continuous external correction, error accumulation, uncompressed frame‑based memory growth, and limited gameplay mechanics. These issues prevent long‑running, controllable worlds and hinder real‑time interaction.
**Figure 1.** **Matrix-Game 4.0.** From top: the system loop; diagonal split-screen comparisons between low-poly engine outputs and neural renderings on held-out validation examples; gameplay examples from games created by the coding agent in response to user requests; end-to-end latency comparisons with recent systems (Table 3) and the rendering throughput at 480p on a single RTX 5090.
System Overview
Describes the Matrix-Game 4.0 harness and its real‑time video generation trick.
The video generation model alone must remember all world rules and state, which quickly exhausts GPU memory and inflates latency. By offloading game logic to an external harness, the model can focus purely on high‑fidelity rendering.
It separates rule execution and structural rendering from pixel synthesis, letting a lightweight engine supply a low‑poly proxy while the video model renders the final frames.
Key A arrives → engine increments c from 0 to 1 → proxy square size 2 is emitted.
Video model receives size 2, past frame (empty), and description “square of size 2” → renders a detailed square with shading.
Key B arrives → engine increments c from 1 to 2 → proxy square size 3 is emitted.
Video model receives size 3 and description “square of size 3” → renders a larger, lit square.
The proxy conveys only the minimal structural change (size increment), yet the video model can synthesize a fully textured frame, showing how the harness offloads state while preserving visual richness.
How does this harness differ from simply feeding the video model its own previous frames?
Previous frames contain only pixel history; they do not encode explicit game rules or geometry. The harness provides a structured, low‑poly proxy and a deterministic world state, so the video model receives a clean, semantically meaningful cue rather than trying to infer rules from raw pixels.
**Figure 2.** **Roadmap of Matrix-Game V4:** The multimodal coding agent and the game engine form the harness for the video generation model. The player sends ideas and requests to the agent and actions to the engine (1). The agent writes the world, turning ideas into game code and scenes for the engine, and translates the world, turning the state and events reported by the engine into text descriptions. The engine maintains the world state and the rules in code (2) and sends control signals as a structural rendering of the scene (3). The video generation model renders frames in real time from the text description, its past frames, and the structural rendering (4). The generated frame returns to the player as visual feedback (5).
Beyond the harness, the paper contributes real‑time 480p 60 FPS gameplay on a consumer RTX 5090, a roadmap toward fully playable worlds, a reusable data‑to‑training‑to‑inference pipeline, and an open‑source release for the community.
The Multimodal Coding Agent
We detail the Multimodal Coding Agent that translates game state to text and code for dynamic video generation.
The system combines a game engine, a multimodal coding agent, and a video generation model. The agent is the bridge that turns engine state into natural‑language descriptions and turns player ideas into executable game code.
It works as a two‑way translator: it reads the engine’s structured state and writes natural‑language descriptions, and it takes player ideas or its own creative suggestions and writes code that updates the engine.
How is this agent different from a standard language model that only generates captions?
Unlike a pure captioning model, the agent also produces executable code that mutates the game world, and both its text and code are grounded in the same engine snapshot, guaranteeing that the description and the world stay synchronized.
The agent receives the engine snapshot and generates the caption “a blue car drives past a green tree”.
It parses the player request and emits code:
The engine registers the new lamp, updates its internal state, and returns an updated snapshot including the lamp.
The agent produces a new caption “a blue car drives past a green tree with a lamp nearby”.
This walk‑through shows how the agent keeps textual descriptions and world modifications tightly coupled, preventing drift between what is described and what actually exists in the simulation.
Receive the current engine snapshot (object list, positions, game rules).
Generate a natural‑language caption describing the scene for the video model.
Accept player or autonomous creative input expressed in plain language.
Translate that input into a short code snippet that mutates the engine state.
Send the code to the engine; the engine updates the world and returns a new snapshot.
Loop back to step 1, feeding the refreshed snapshot to the video model.
**Figure 4.** Dynamic captions follow the engine state. Top: low-poly proxy frames of one CitySample driving clip; numbered markers are the visible objects that the engine reports for this frame, each linked by its instance ID to a semantic description. Below: the caption that the rule-based template produces for the same frame, without any vision-language model. Orange text comes from object states and events, and blue text from object positions and distances. As the car moves, parked vehicles leave the description, pedestrians change from standing to approaching, and newly visible objects are announced under “Changes”.
System Workflow
The system renders high‑fidelity video by feeding low‑poly structural proxies to a fast engine.
High‑fidelity video rendering is computationally expensive, especially when many agents share a world. The trick is to let the engine run a lightweight simulation that only produces low‑poly structural proxies, while a separate model handles the visual detail. This division removes the bottleneck from the engine and concentrates the heavy rendering work in a dedicated video model.
Think of a wireframe sculpture that captures the pose of a character; the video model then paints a photorealistic skin over that wireframe.
Step 1: Object 1 receives BBOX coordinates $(0,0)$–$(1,1)$; RGB pixel block is filled with a red square.
Step 2: Object 2’s BBOX moves to $(1,0)$–$(2,1)$; the RGB model paints a blue square at the new location.
Step 3: All four BBOXes shift right by one unit; the video model renders four colored squares sliding synchronously.
Step 4: The final frame shows the squares aligned with the BBOX grid, confirming perfect pixel‑wise correspondence.
The proxy’s tiny vertex count lets the engine update positions in microseconds, while the video model learns to map those positions to full‑resolution textures.
How does Low‑Poly Proxy Rendering differ from a standard BBox‑to‑RGB pipeline?
Standard pipelines treat the BBOX as a rough visual cue and often discard precise alignment, leading to jitter between geometry and texture. Here the BBOX is an exact, pixel‑aligned structural signal; the video model never sees the original textures, so it must reconstruct them from geometry alone, which forces a tighter coupling between proxy and output.
Player input is captured and sent to the game engine.
The engine updates the persistent world state (positions, physics, AI).
Structural renderings (BBOX) and control signals are emitted for the current frame.
The video rendering model consumes the BBOX sequence and generates a high‑fidelity RGB frame.
The rendered frame is displayed to the player as visual feedback.
For each 181‑frame clip the system also builds a CSV of frame‑aligned states and object descriptions, then fills a natural‑language template to produce a prompt for the video model.
Dataset and Training Data
We describe the UE5 paired dataset and its dynamic caption pipeline that align structure with high-fidelity video.
The video generation model needs precise structural supervision, but existing datasets lack aligned geometry and semantics. We therefore construct a paired dataset in Unreal Engine 5 (UE5) that provides low‑poly proxies and high‑fidelity RGB frames for every timestamp.
Each clip contains a pixel‑perfect low‑poly structural render and a matching high‑fidelity RGB render, plus a depth map and per‑frame metadata, so the model can learn a deterministic mapping from geometry to photorealism.
Low‑poly render of Frame 0: pixel array of size 128 × 72 with gray boxes.
RGB render of Frame 0: identical resolution but with detailed textures and lighting.
Depth map of Frame 0: per‑pixel distance values matching the geometry.
Metadata record for Frame 0: object IDs {building → ID 12, tree → ID 7}, visibility flags {building = 1, tree = 1}.
Same three assets are rendered for Frame 1, but the camera pose changes, producing a new pixel‑aligned pair.
This toy example shows how every visual modality stays perfectly aligned across frames, letting the model learn a one‑to‑one correspondence between geometry and photorealistic appearance.
How does this dataset differ from standard video datasets that only provide RGB frames?
Standard datasets lack an explicit structural prior; here each RGB frame is paired with a low‑poly proxy and depth map, giving the model a clean geometric signal to condition on, which removes ambiguity about scene layout.
**Figure 3.** **UE5 paired training data.** (a) Each clip is captured twice in Unreal Engine 5 under the same camera and frame index: once as a low-poly structure render (BBox) that preserves scene layout and motion, and once as the high-fidelity RGB frame. (b) For every 81-frame clip, frame-aligned CSV records of camera pose, actions, visible objects, events, and environment are turned into a prompt: visible object IDs are looked up in a description table, spatial and temporal cues are composed, and a template produces the clip prompt.
Table 1 quantifies the scale of the dataset: the four scenes together provide 114 334 training clips (≈1 712 h of video) at 30 FPS, with a held‑out set that contains no overlapping capture sessions.
For every frame the system reads the current game state, maps object IDs to textual descriptions, and fills a template, so the caption always reflects the exact visual content and interactions.
Lookup: ID 12 → “building”, ID 7 → “tree”.
State extraction: building = intact, tree = on fire.
Spatial cue: tree is to the left of the building (derived from bounding boxes).
Template fill: “A building stands intact while a tree to its left burns.”
This example shows how the caption changes instantly when the tree’s state flips, keeping the textual description perfectly synchronized with visual changes.
Why can’t a static caption be used for these videos?
Static text would quickly become inaccurate as the camera moves or objects change state; the model would then receive contradictory supervision, leading to drift and visual artifacts in generated sequences.
Rendering Model Architecture
Turning low‑poly engine proxies into high‑fidelity video via a V2V architecture.
Generating high‑fidelity video from sparse engine proxies is costly because the model must infer both geometry and texture from minimal structural cues.
The model treats the low‑poly proxy as a structural guide and the noisy latent as a canvas; concatenating them lets a DiT backbone iteratively refine the canvas into clean frames.
Patch 1 receives proxy = [0.1, 0.2] and latent = [0.5, 0.6] → combined vector $[0.1,0.2,0.5,0.6]$.
Patch 2 receives proxy = [0.0, 0.1] and latent = [0.4, 0.5] → combined $[0.0,0.1,0.4,0.5]$.
DiT block processes the 4 tokens; self‑attention mixes information across patches, cross‑attention injects caption tokens.
Output head predicts velocity $v=[0.02, -0.01]$; latent update $z \leftarrow z - \sigma v$ (with $\sigma=0.5$) yields new latent values.
TAEHV decoder maps the updated latent to four RGB pixels, producing a $2\times2$ frame.
This tiny example shows how concatenation fuses structural and visual cues, and how a single DiT block can already propagate information across the whole patch grid.
Generation loop for a single chunk.
**Figure 5.** Architecture of the rendering model. The noisy latent of one chunk (one latent frame, four video frames) and the low-poly proxy encoded by TAEHV are concatenated into 148 channels and patchified into 405 tokens per latent frame at 480 × 864. Thirty causal DiT blocks apply self-attention over a KV cache that holds spatial-memory frames, reference frames, and the latest 20 chunks, cross-attention to the cached text keys and values of the dynamic caption, and a feed-forward network, each modulated by the timestep through AdaLN. The output head predicts a velocity, one or two flow updates produce the clean latent, and TAEHV decodes four frames. The finished chunk is then re-run at $t=0$ without the control input to write its keys and values into the cache.
Supervised fine‑tuning adapts the pretrained Wan2.2‑Fun‑5B weights to each game by training a bidirectional V2V model on 81‑frame windows (704 × 1280) with a flow‑matching loss.
Each window is conditioned on its low‑poly proxy, caption, first frame, and up to three context frames selected from at least 40 frames earlier based on camera‑pose overlap; the context frames are excluded from the loss.
To enable classifier‑free guidance, the training pipeline randomly drops the first frame (30 % chance), the caption (20 % chance), and the proxy control (20 % chance).
Causal few‑step distillation converts the bidirectional model into an autoregressive generator: Stage 1 trains a teacher with teacher‑forced diffusion; Stage 2 distills a causal student via EMA consistency; Stage 3 matches the student’s rollout distribution to the frozen bidirectional model using asymmetric DMD.
The final student rolls out 21, 41, 61, or 121 frames, caching the first frame and the latest 20 latent frames; TAEHV‑encoded control is used on 80 % of steps, with Wan‑VAE control on the remainder.
Real-Time Inference
Optimizations that turn tiny per‑chunk calls into smooth, high‑FPS generation.
Processing a video one chunk at a time forces the GPU to launch a kernel for every four‑frame segment. The launch latency dominates the compute, leaving the accelerator idle most of the time.
We keep chunks tiny (four frames) so the system can react instantly to player actions, but we then amortize the per‑chunk launch cost by fusing kernels, caching KV, and streaming I/O.
Chunk 1 arrives → GPU launch (0.5 ms) → compute (1.0 ms) → output first 4 frames.
Chunk 2 arrives → GPU launch (0.5 ms) → compute (1.0 ms) → output next 4 frames.
Total time = 2 × (0.5 ms + 1.0 ms) = 3 ms, yielding ≈ 333 FPS for this toy scenario.
When launch overhead dominates, fusing kernels and caching KV can shave off the 0.5 ms per call, multiplying the effective FPS by the number of chunks.
Record the DiT backbone as a CUDA Graph; use 8‑bit (FP8) linear layers.
Write fused Triton kernels for RoPE, normalization, and FP8 quantization; share one quantization across Q, K, V.
Cache the text‑condition KV tensors; recompute only when the caption changes.
Implement the video KV cache as a ring buffer that overwrites the oldest latent frame in place.
Split the KV window into parallel parts; compute each part’s self‑attention concurrently and merge the results.
Replace the full Wan VAE decoder with the lightweight TAEHV decoder.
Run model input and output through an asynchronous pipeline.
Apply split‑KV FlashAttention to further accelerate the attention computation.
**Figure 1.** (a) GPU time of one block, by kernel category (measured). (b) Frames out before the clean-KV update. (c) Rolling window: shift vs. ring. (d) Self-attention over seven window ranges.
How is chunk‑based inference different from processing the whole video sequence in one batch?
Instead of feeding all frames through the model at once, we feed a single latent frame (four video frames) at a time. This keeps the reaction latency per player action low and lets the engine provide structural context while the model focuses on high‑fidelity rendering of each chunk.
Evaluation and Results
Matrix‑Game 4.0 delivers record‑low latency and high‑FPS rendering on consumer GPUs.
Recall that Matrix‑Game 4.0 wraps a video generation model around a game engine, letting the model focus on high‑fidelity rendering of engine‑provided low‑poly proxies.
Matrix‑Game 4.0 achieves an end‑to‑end latency of 112 ms, the lowest among the surveyed systems and about one‑fifth of the next fastest.
Table 3 shows the latency of competing systems ranging from 517 ms to 1.55 s, while Matrix‑Game 4.0 runs at 112 ms.
Table 3 compares Matrix‑Game 4.0 with four recent interactive generation systems. Besides latency, Matrix‑Game 4.0 runs on a single consumer GPU, uses modest compute, and provides an open release of code and LoRA weights.
**Figure 7.** Re-rendering over 40-second held-out rollouts. For each game, the upper row is the low-poly proxy from the engine and the lower row is the frame generated by the distilled causal model at the same time step. The generated frames follow the proxy layout, camera, and object motion throughout the rollout, while the model fills in materials, vegetation, lighting, smoke, and muzzle effects.
**Figure 8.** Game modes written with the coding agent, rendered in real time. Top: in the third-person shooter, the agent adds health and ammunition control, shield durability, a faster shotgun, a game-over condition, and simple sound effects. Middle: in the driving game, the agent implements a time trial with route checkpoints and collision penalties. Bottom: the same driving game turned into a pursuit mode, in which the player must intercept a target vehicle and stay close to it for five seconds. The engine evaluates every rule; the video model renders each frame from the engine output.
Related Work
We position our approach among prior interactive video world models and code‑defined rendering systems.
Interactive video world models aim to predict future observations from user actions, while code‑defined worlds separate state logic from visual generation.
Genie learns a latent action space and an autoregressive dynamics model from raw video, enabling a controllable 3‑D environment without explicit supervision.
Early systems such as Genie and GameNGen embed both dynamics and rendering in a single network, limiting flexibility and performance.
Recent work shifts the burden of world dynamics to traditional game code, letting neural models focus solely on high‑fidelity visual rendering.
Code‑defined approaches (e.g., Code World Model, StreamRender‑H3, AgentGarten, Magpie) pair executable game logic with video generators, often using a low‑poly proxy video as conditioning input.
Neural rendering pipelines (e.g., vid2vid, ControlNet, DiffusionRenderer) condition video synthesis on structured graphics outputs such as G‑buffers, depth maps, or semantic masks.
Real‑time autoregressive generation techniques (Causal Forcing++, DMD, CUDA Graphs) reduce diffusion steps to a few or one, enabling frame‑wise video synthesis at interactive rates.