Reading, Writing, Sharing, and Interacting with the World through Video A Survey Toward Unified Video World Models
Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr, Hao Fei
A unifying world-modeling framework for video foundation models across understanding, generation, and interaction.
How can we unify the disparate fields of video understanding, generation, and agentic interaction into a single coherent framework of "video world models"?
Video foundation models have diverged into specialized silos: some models interpret visual content, while others synthesize it or simulate physical dynamics. This fragmentation obscures the shared challenge of connecting partial, lossy video observations to the evolving world states they depict. The authors propose a "world-modeling lens" that organizes the entire field into four complementary operations: reading (inference), writing (synthesis), sharing (unified modeling), and interacting (predictive simulation). This framework maps diverse architectures and tasks onto a common taxonomy based on how they represent state and handle temporal consistency. The survey identifies a clear shift in the field: research is moving away from simple perceptual fidelity toward persistent state, causal validity, and evaluation grounded in downstream decision-making.
Paper Primer
The paper addresses the lack of a unified account for video models by defining an observed video as a partial, lossy projection of a latent temporal world. By distinguishing between temporal coherence (local frame smoothness) and world consistency (compatibility with physical dynamics), the authors provide a rigorous basis for comparing models that were previously categorized only by their input/output modalities.
The core mechanism is the four-operation framework: Reading infers world properties; Writing synthesizes observations from conditions; Sharing couples these through shared representations; and Interacting rolls out state transitions under actions. This taxonomy allows the authors to classify world models along two axes: prediction space (observation-space vs. latent-state) and transition interface (passive vs. action-conditioned vs. interactive).
The field is transitioning from task-specific recognition to general-purpose world modeling.
The authors trace the evolution of video foundation models from early spatiotemporal architectures (e.g., I3D, SlowFast) to modern interactive simulators (e.g., Genie, GameNGen) and unified models (e.g., Emu3, Video-LaVIT). The survey covers hundreds of representative works, identifying a convergence where understanding and generation are increasingly coupled through shared token representations and latent spaces.
Why does the distinction between "temporal coherence" and "world consistency" matter?
Temporal coherence only requires neighboring frames to look similar, which can be achieved by simple interpolation. World consistency requires the sequence to be compatible with an underlying physical trajectory, which is the necessary standard for models intended to support reasoning, planning, or interaction.
What is the primary limitation of current evaluation practices in this field?
Many benchmarks rely on aggregate accuracy that can be solved via static appearance or language priors rather than actual video-based reasoning. The authors argue that evaluation must shift toward testing state persistence, controllability, and causal validity through interventions and long-horizon consistency.
Researchers should move beyond optimizing for perceptual fidelity and instead prioritize models that maintain persistent, physically grounded world states capable of supporting interactive decision-making.
The Unified Video World Model
We frame video intelligence via four core operations to define a unified Video World Model.
Video research has splintered into many task‑specific sub‑fields, making progress hard to compare. By treating video as a projection of an underlying temporal world, we can organize the entire literature around a single, coherent framework.
A Video World Model is a representation that predicts how the hidden, continuous world evolves over time and explains every observed frame as a noisy, partial view of that world.
The four operations capture the full life‑cycle of video‑world interaction: reading extracts world properties, writing creates video from world conditions, sharing aligns understanding and generation through common mechanisms, and interacting updates the world under actions.
**Figure 1.** Selected milestones in the parallel evolution of video understanding, generation, shared representations and workflows, and world prediction and control.
**Figure 2.** Knowledge graph of the four research fields. Reading, writing, sharing, and interacting branch into technical streams and representative works.
The field is moving from isolated, task‑specific models toward a single Video World Model that unifies perception, generation, and interaction.
Mapping the Video Intelligence Landscape
Video research is coalescing around a unified world‑model view, yet current systems remain split between understanding and generation.
The survey shows that video uniquely captures spatiotemporal dynamics, prompting a consensus on four core operations, but existing models still treat understanding and generation separately, leaving a conspicuous gap for integrated world‑modeling.
The architecture treats video as a bidirectional bridge between raw observations and an internal latent world, exposing shared components that can be reused for both comprehension and synthesis.
How does the Unified Video Architecture differ from simply stacking a separate encoder and decoder?
Stacking treats the encoder output as a one‑shot conditioning signal, discarding intermediate world‑state information after each forward pass. The Unified architecture retains a mutable latent state that can be queried, updated, and reused across multiple reading or writing steps, enabling consistent reasoning over long videos and tighter coupling between perception and synthesis.
Reading models infer latent world properties from observed video. The survey reports that spatiotemporal encoders (e.g., SlowFast, TimeSformer) combined with Video‑LLMs achieve strong semantic and temporal accuracy, but struggle with occlusion and long‑range memory, prompting the development of recurrent memory modules and hierarchical attention.
Writing models generate video from partial world specifications. Diffusion‑based generators (e.g., Sora, Video Diffusion Models) dominate high‑fidelity synthesis, while autoregressive token models (e.g., MAGVIT‑v2) excel at controllable generation. The survey highlights trade‑offs between perceptual quality and action‑conditioned controllability.
Sharing approaches unify reading and writing within a single model. The survey catalogs continuous‑latent unification (e.g., Decoupled Modular), discrete‑token unification (e.g., Chameleon, Emu3), and agentic/tool‑augmented pipelines (e.g., TUNA, Uni‑ViGU). Shared token spaces enable cross‑modal transfer, while modular bridges preserve task‑specific flexibility.
Interacting models roll out a transition function over a video‑centered world state. The taxonomy (Figure 10) classifies them by prediction space (Observation vs. Latent) and transition interface (Passive, Action‑conditioned, Interactive). The survey notes that latent‑state, interactive models achieve the strongest planning performance but require sophisticated action encoders.
**Figure 10** The $2 \times 3$ taxonomy of video world models. Rows give the prediction space: observations (frames, video tokens, or video-decoding latents) or latent states (video-derived features predicted without required rendering). Columns give the transition: passive prediction without action input, action-conditioned prediction with controls fixed before prediction, and interactive prediction that receives new controls after feedback. Each cell shows its prediction form and representative models; $x$ denotes observations, $z$ features, $a$ actions, $H$ the horizon, and $C$ and $Q$ the context and target positions of masked feature prediction.
The taxonomy of video world models (Figure 10) serves as the primary tool for evaluating model capabilities because it isolates the core modeling choice—what is predicted and how it is conditioned—allowing fair comparison across disparate systems.
Questions & answers
What is the main contribution of this paper?
The paper introduces a unified 'world-modeling lens' framework that organizes the entire video foundation model field into four operations: reading (inference), writing (synthesis), sharing (unified modeling), and interacting (predictive simulation), providing a common taxonomy for comparing previously siloed architectures.
What problem does this survey address?
Video foundation models have fragmented into specialized silos—some interpret visual content, others synthesize it or simulate physical dynamics—making progress hard to compare and obscuring the shared challenge of connecting partial, lossy video observations to the evolving world states they depict.
Why does the distinction between 'temporal coherence' and 'world consistency' matter?
Temporal coherence only requires neighboring frames to look similar, which can be achieved by simple interpolation, whereas world consistency requires the sequence to be compatible with an underlying physical trajectory—the necessary standard for models intended to support reasoning, planning, or interaction.
What are the four core operations in the proposed framework?
The four operations are: Reading (inferring world properties from video), Writing (synthesizing observations from conditions), Sharing (coupling reading and writing through shared representations), and Interacting (rolling out state transitions under actions for predictive simulation).
Along what two axes does the taxonomy classify video world models?
The taxonomy classifies models along two axes: prediction space (observation-space versus latent-state) and transition interface, allowing fair comparison across disparate systems based on what is predicted and how it is conditioned.
How does the Unified Video Architecture differ from simply stacking a separate encoder and decoder?
Stacking treats the encoder output as a one-shot conditioning signal, discarding intermediate world-state information after each forward pass, whereas the Unified architecture retains a mutable latent state that can be queried, updated, and reused across multiple reading or writing steps, enabling consistent reasoning over long videos and tighter coupling between perception and synthesis.
What overarching shift in the video research field does the survey identify?
The survey identifies a clear shift away from simple perceptual fidelity toward persistent state, causal validity, and evaluation grounded in downstream decision-making, with research moving from isolated task-specific models toward a single Video World Model that unifies perception, generation, and interaction.
What is the primary limitation of current evaluation practices in the video modeling field?
Many benchmarks rely on aggregate accuracy that can be solved via static appearance or language priors rather than actual video-based reasoning; the authors argue evaluation must shift toward testing state persistence, controllability, and causal validity through interventions and long-horizon consistency.
What core concept underlies the entire framework proposed in the paper?
The framework treats an observed video as a partial, lossy projection of a latent temporal world, providing a rigorous basis for comparing models that were previously categorized only by their input/output modalities.
What practical recommendation do the authors make for future research?
The authors recommend that researchers move beyond optimizing for perceptual fidelity and instead prioritize models that maintain persistent, physically grounded world states capable of supporting interactive decision-making.
What role does Figure 10 (the taxonomy of video world models) play in the paper?
The taxonomy of video world models serves as the primary tool for evaluating model capabilities because it isolates the core modeling choice—what is predicted and how it is conditioned—allowing fair comparison across disparate systems.
What unique property of video motivates the consensus on the four core operations?
The survey notes that video uniquely captures spatiotemporal dynamics, which motivates the consensus on the four core operations, though existing models still treat understanding and generation separately, leaving a gap for integrated world-modeling.
What datasets or benchmarks are evaluated in this paper?
The paper does not specify particular datasets or benchmarks that were empirically evaluated; it is a survey and framework paper that critiques existing evaluation practices rather than reporting results on specific benchmarks.
Who are the authors and where was this paper published?
The paper does not state the authors' names in the provided text, and the venue is not specified; it is available on arXiv at the identifier referenced in the source URL.
Key terms
- world-modeling lens
- A conceptual framework that treats video as a partial, lossy projection of an underlying latent temporal world, used to unify diverse video model architectures under a single taxonomy.
- reading (operation)
- The operation in which a model infers world properties—such as object states, scene semantics, or physical attributes—from observed video input.
- writing (operation)
- The operation in which a model synthesizes or generates video observations from given conditions, such as text descriptions or partial frames.
- sharing (operation)
- The operation that couples reading and writing through shared internal representations, enabling a single model to both interpret and generate video.
- interacting (operation)
- The operation in which a model rolls out state transitions conditioned on actions, enabling predictive simulation for planning or decision-making.
- temporal coherence
- A property of a video sequence in which neighboring frames look visually similar, achievable by simple interpolation without requiring physical plausibility.
- world consistency
- A property of a video sequence in which the entire sequence is compatible with an underlying physical trajectory, required for models that support reasoning, planning, or interaction.
- latent temporal world
- The unobserved, underlying state of a scene over time, of which any recorded video is treated as a partial and lossy projection.
- prediction space
- One of two classification axes in the taxonomy, distinguishing whether a model predicts directly in observation space (raw pixels or frames) or in an abstract latent-state space.
- transition interface
- One of two classification axes in the taxonomy, describing how a model conditions its state transitions, such as through actions, text, or other control signals.
- Unified Video Architecture
- An architecture that maintains a mutable latent state across multiple reading and writing steps, enabling consistent long-horizon reasoning and tight coupling between perception and synthesis.
- Video World Model
- A proposed unified model that integrates perception, generation, and interaction into a single system grounded in persistent, physically consistent world-state representations.
- perceptual fidelity
- A measure of how visually realistic or accurate generated or reconstructed video frames appear, without necessarily reflecting physical or causal correctness.
- causal validity
- The property of a model's outputs being consistent with cause-and-effect relationships in the physical world, as opposed to merely matching surface-level visual statistics.
- state persistence
- The ability of a model to maintain a consistent representation of world objects and their properties across time, even as the video evolves.