H3-World: Turning Language Understanding into World Control

Danze Chen, Zeqing Wang, Ziyue Lin, Xingyi Yang, Yeying Jin

H3-WORLD turns pretrained video generators into interactive world models by grounding compositional text instructions into temporal video latents.

How can we transform a large-scale, pretrained video generation model into an interactive world model that responds to specific action sequences?

Pretrained video generators can follow coarse text prompts, but they lack the precise, time-varying control required for interactive world modeling. H3-WORLD maps character and camera actions to compositional text instructions, binding each to a specific video latent interval via temporal attention routing. This lightweight adaptation achieves precise, interactive control while preserving the model's original generative quality and generalizing to unseen action combinations.

Paper Primer

The core challenge is that pretrained video models treat text as a global condition, making it impossible to specify different actions for different moments in a video. H3-WORLD solves this by treating the video sequence as a series of independent latent intervals, each paired with its own specific textual instruction.

The method hinges on single-egress routing: a mask that forces each action instruction to communicate only with its corresponding video latent. This prevents "control leakage," where an instruction intended for one moment accidentally influences the entire clip, and allows the model to learn action-conditioned dynamics using only lightweight Low-Rank Adaptation (LoRA).

H3-WORLD achieves precise, temporally grounded control with minimal parameter updates.

The model uses only 0.199% trainable parameters and 8,000 gameplay samples to learn effective character and camera control. The system successfully follows complex, multi-part action schedules that frozen models or global-prompting baselines fail to execute.

The model demonstrates strong compositional and visual generalization.

H3-WORLD correctly executes action combinations never seen during training and maintains control across diverse, out-of-distribution visual environments. It successfully composes character and camera primitives even when their joint occurrence was absent from the training data.

Why use language as the control interface instead of learning discrete action embeddings?

Language acts as a pre-existing semantic prior within large video generators. By using text, the model reuses the rich representations of motion and interaction already learned during large-scale pretraining, rather than forcing the model to learn a new control language from scratch.

Does this approach require a dedicated module to handle the actions?

No. H3-WORLD injects instructions directly into the model's native text pathway. The only architectural change is the routing mask, which requires no learnable parameters, keeping the system efficient and preserving the original generative backbone.

Motivation and Framework Overview

We introduce H3‑WORLD, turning language into precise control for interactive video generation.

Large video generators can already follow coarse natural‑language instructions, but they do not expose a fine‑grained action interface needed for interactive world modeling. This gap—lack of interactive control—limits their use as world models despite their strong generative capabilities. Language is beginning to serve as a high‑level abstraction for visual dynamics, suggesting a path to bridge this gap.

Current video generators provide vivid scenes but offer no precise way to steer characters or cameras with user‑defined actions.

MiniMax‑H3 is a 33‑billion‑parameter diffusion transformer that already interprets coarse textual motion commands to drive characters and cameras.

H3‑WORLD leverages the language understanding of MiniMax‑H3 to achieve precise, temporally grounded control. It translates each character or camera action into a structured textual instruction and aligns it with the corresponding video‑latent interval. Temporal attention routing then restricts the instruction’s influence to its intended time window, preventing leakage across actions. The whole system requires only lightweight LoRA adaptation—8 000 gameplay samples and 0.199 % trainable parameters—while preserving the generator’s visual quality.

**Figure 1:** H3-WORLD enables character and camera control across diverse visual environments.

**Figure 2.** Vanilla MiniMax-H3 already exhibits coarse zero-shot control through textual motion instructions.

Language emerges as a natural, high‑level interface for precise video control.

The H3-WORLD Architecture

We introduce latent-aligned prompts and a single‑egress routing scheme to control video generation.

Controlling video generation requires aligning discrete action commands with the latent video stream. The core trick is to bind each action prompt to its corresponding latent interval and restrict attention so that only the matched video tokens can read the action. This yields fine‑grained, compositional control without retraining the whole model.

Each scheduled action is turned into a short textual prompt that is attached to a specific video latent interval, so the model knows exactly when to apply that action.

How does this differ from simply concatenating raw control tokens?

Instead of feeding raw discrete keys, we translate each into natural‑language clauses, preserving the pretrained text‑conditioning of H3 and enabling the model to leverage its language understanding.

Map $u_{1}$ to the clause “walk forward”.

Map $c_{1}$ to the clause “camera pans right slowly”.

Concatenate to obtain $p_{1}$ = “walk forward and camera pans right slowly”.

Map $u_{2}$ to “jump”.

Map $c_{2}$ to “camera tilts up”.

Concatenate to obtain $p_{2}$ = “jump and camera tilts up”.

The prompts are independent per interval, so each action only influences its matched latent, enabling precise temporal control.

The interface aggregates discrete control keys within each video‑latent interval and turns them into textual prompts that the H3 model can ingest as part of its token stream.

Why use textual prompts instead of feeding the raw control vectors directly?

Textual prompts place the actions into the same modality as the pretrained language tokens, allowing the frozen H3 encoder to process them without additional architectural changes.

Collect all keys that appear in any frame: forward, backward, pan right, pan left.

Cancel opposing pairs: forward vs. backward → both removed; pan right vs. pan left → both removed.

Resulting active set is empty, so the interval contributes no action clause.

Aggregation yields a compact representation that captures any key occurring in the interval while discarding contradictory signals.

Encode the static semantic condition $s$ and the initial observation $I_{0}$ with the frozen H3 encoder to obtain tokens $S$ and $C_{0}$.

Convert each control pair $(u_{k},c_{k})$ into a textual prompt $p_{k}$ via the Action‑Conditioning Interface.

Encode each prompt $p_{k}$ with the shared H3 encoder $E$ and refine with the two‑layer token refiner $R$ to produce action tokens $A_{k}=R(E(p_{k}))$.

Pack all tokens into the sequence $X=[S;A_{1}\dots A_{K};C_{0};V_{1}\dots V_{K};P]$, where $P$ denotes masked padding.

Apply the deterministic single‑egress routing mask so that each $A_{k}$ can attend to its matched video latent $V_{k}$ and the static context, but not to other latents.

Inject LoRA low‑rank updates into the QKV and output projections of the H3 self‑attention blocks.

Run the frozen H3 backbone to denoise and generate the future video latents $V_{1:K}\sim p_{\theta,\phi}(V_{1:K}\mid I_{0},s,a_{1:K})$.

**Figure 3.** Overview of H3-WORLD. (a) Latent-aligned action prompts are independently encoded and packed with the static semantic condition, initial observation, and video latents. (b) MiniMax-H3 processes the packed sequence through single-stream self-attention. (c) LoRA adapts the attention projections under single-egress routing, which connects each action span directly to its matched video latent and retains bidirectional attention among video latents.

**Figure 4.** Training coverage of the action space. Of 135 valid character–camera pairs, 83 occur in 291,264 prompts and 52 remain unseen. The top 20 and 40 pairs account for 71.4% and 95.4% of prompts.

Empirical Evaluation and Generalization

H3‑WORLD demonstrates strong compositional generalization and precise action control.

We train rank‑32 LoRA adapters for $10^{4}$ steps ($\text{lr}=1\times10^{-4}$) on $7{,}872$ gameplay clips and evaluate on $128$ held‑out clips of $124$ frames each ($24$ fps, $832\times480$). All quantitative diagnostics use dense optical flow (Farneback) and report mean horizontal flow per video.

H3‑WORLD achieves compositional generalization on held‑out gameplay clips.

It follows unseen character‑camera action pairs in both held‑out and out‑of‑distribution observations, preserving scene layout and subject appearance.

**Figure 5.** Action response before and after adaptation. The scheduled camera pan switches from left to right after latent 15. Check marks and crosses summarize action control and temporal control for each condition.

**Figure 6.** Action-conditioning interfaces on a held-out clip. GT provides the recorded reference motion. Direct additive-bias and FiLM conditioning yield limited or inconsistent responses to the recorded controls, while text-based H3-WORLD produces coordinated character and camera changes.

**Figure 7.** Following recorded controls on held-out clips. Ground-truth videos appear above the corresponding H3-WORLD generations produced from the same initial observation and action sequence.

**Figure 8.** Controlled action comparison. The initial observation, seed, and sampling configuration are fixed across rows. Changing the action produces distinct character and camera motion, including stronger fast pans.

**Figure 9.** Compositional action generalization. Each group contrasts a seen action with an unseen composition of observed character and camera clauses on held-out gameplay (top) and out-of-distribution (bottom) observations.

**Figure 10.** Visual generalization to diverse initial observations. Each scene is evaluated with one character-control command and one camera-control command using the same learned action interface.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers