Video = World + Event Stream
Lianghua Huang, Zhi-Fan Wu, Yupeng Shi, Wei Wang, Mengyang Feng, Cheng Yu, Chen Liang, Junjie He, Chen-Wei Xie, Yu Liu, Jingren Zhou, Ang Wang, Bang Zhang, Baole Ai, Chongyang Zhong, Jinwei Qi, Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, Xinhua Cheng, Yitong Huang, Yun Zheng, Yuxiang Bao, Yuzheng Wang, Zhiwei Lin, Zoubin Bi
Wan-Streamer v0.3 reframes real-time interactive agents as a decomposition of persistent world context and time-varying event streams.
How can we improve real-time video streaming interaction by separating persistent scene context from transient event streams?
Interactive agents often struggle to balance stable character identity with dynamic, spontaneous behavior, leading to disjointed or limited responses. The authors introduce a world-event decomposition: the model treats a video as a persistent "world" (identity, environment, voice) plus an "event stream" (speech and free-form behavior) that unfolds over time. This framework enables open-vocabulary behavior generation while maintaining the same 200 ms model-side latency and 640×368 resolution as previous versions.
Paper Primer
The core mechanism treats the agent's output as an interleaved stream of spoken language and parenthesized behavior directives. By embedding these directives directly into the causal token stream, the model learns to synchronize complex physical actions—like reaching for an object or changing posture—with speech and environmental context without requiring a separate action controller.
The model achieves open-vocabulary behavior generation without increasing latency.
The system maintains a 200 ms model-side response latency and 550 ms total interaction latency, identical to the v0.2 baseline. Supports arbitrary, language-described actions rather than a fixed set of navigation or idle moves.
The world-event decomposition serves as a general-purpose pretraining objective. By training on large-scale video to predict how a world evolves given an event stream, the model acquires a foundational competence in scene dynamics that transfers to downstream tasks like full-duplex audio-visual interaction.
Why move to a "world + event stream" framing instead of just training on more interaction data?
This decomposition allows the model to learn from diverse, large-scale video corpora rather than being restricted to specific interaction domains, creating a more scalable pretraining objective that captures how scenes and subjects plausibly evolve.
Does this approach require a new, more complex architecture to handle the added behavior directives?
No; the modeling contract and serving topology remain unchanged from v0.2. Behavior directives are simply short language tokens that share the existing causal path, adding negligible computational cost.
Introduction and Motivation
We expose the inefficiency of monolithic video models and propose a world‑event split.
Current video interaction models repeatedly recompute the same background and acoustic conditions, wasting compute on parts of the scene that hardly change. This inefficiency becomes a bottleneck for real‑time, full‑duplex applications where latency is critical.
Monolithic video processing treats every frame as entirely new, so stable background, lighting, and voice traits are recomputed at each step; separating the immutable “world” from the time‑varying “event stream” lets the model focus compute on what actually changes.
The image displays a sequence of eight video frames from a game, each labeled with a timestamp and an event description: - [00.00s - 01.75s] [EVENT] (run on sidewalk) - [01.75s - 07.13s] [EVENT] (turn right onto road) - [07.13s - 17.38s] [EVENT] (run towards gold car) - [17.38s - 18.52s] [EVENT] (approach driver's side of gold car) - [18.52s - 19.46s] [EVENT] (open driver's door) - [19.46s - 20.57s] [EVENT] (enter car) - [20.57s - 27.70s] [EVENT] (drive gold car straight) - [27.70s - 30.93s] [EVENT] (turn gold car right) Below the frames, there is a text block: [WORLD #2] The environment features an outdoor poultry pen scene with a green awning above. The ground is dirt. There is a chicken coop ... The audio includes a female voice, speaking English, with a friendly and instructive tone, and you can hear the clucking of chickens and the crowing of roosters, as well as the sounds made by characters walking ... @character1 is a middle-aged white woman with blond hair wears a brown wide-brimmed hat. She wears a black T-shirt...
The key shift is moving from monolithic video processing to a world + event decomposition, which eliminates redundant computation on stable context.
The World-Event Model
World‑Event Decomposition splits video into a stable world and a fast‑changing event stream.
Re‑processing a static scene at every timestep wastes compute; the trick is to separate what stays the same from what changes.
The video is treated as a persistent “world” that holds the stable backdrop, plus a short‑lived “event stream” that carries every change.
World $W$ is created once: $W = \{ \text{environment}= \text{garden}, \text{character}= \text{Alice} \}$.
Event $e_1$: $\tau_1 = [0,2]$ s, $c_1 =$ Alice, $d_1 =$ “walks to coop”.
Event $e_2$: $\tau_2 = [2,4]$ s, $c_2 =$ Alice, $d_2 =$ “feeds chickens”.
The model receives $W$ once, then processes $e_1$ and $e_2$ sequentially without touching $W$ again.
Only the two short event records need to be streamed; the heavy world representation stays cached, cutting per‑step compute.
The “world” is a structured record that captures everything that stays roughly constant across a clip – layout, lighting, character identities, and ambient sound.
How is this different from simply feeding the whole video frames to a standard transformer?
A standard transformer would attend over every past frame at each step, recomputing the same background information repeatedly. Here the background lives in $W$, which is cached after one pass, so each streaming step only attends to the tiny event record.
An event is a short, time‑localized record that describes a change – a motion, a spoken line, or a sound – together with the character (if any) that caused it.
Why not merge $c_k$ into the description $d_k$ and drop the explicit character field?
Separating $c_k$ lets the model reuse a character’s persona embedding across many events, and it makes it easy to query “character‑independent” events (where $c_k = \emptyset$) without parsing free text.
**Figure 1.** Two examples of the general-purpose pretraining data: time-aligned events paired with a summary of the persistent world context, including the visual setting, acoustic conditions, and characters. Parentheses denote character behavior, while quotation marks enclose spoken words. For clarity, both examples contain only one character, so actor names are omitted from the event labels. Training supports multiple characters by associating each event with its corresponding character, or with an explicit no-character marker for character-independent events.
Attention is restricted to the current streaming block and all previously cached blocks, never to the entire history, so each step only looks at a bounded window of past information.
Is Block‑Causal Attention just a special case of sliding‑window attention?
No – sliding‑window attention discards everything outside the window, including any persistent world. Block‑Causal keeps the world prefix visible at every step, preserving global conditioning while still limiting the dynamic portion.
**Figure 2** The world context is tokenized and prefilled once before streaming. Language, audio, and video inputs and outputs then share a causal timeline coordinated by block-causal attention. The model maps this world and streaming multimodal user input to language-form speech and behavior actions. These predicted tokens condition synchronized audio-video generation. In the agent text stream, phrases inside parentheses denote free-form behavior, while text outside parentheses is spoken by the agent.
Streaming inference loop – world is prefixed once, then each block is processed online.
System Evolution and Performance
v0.3 boosts resolution three‑fold while preserving latency.
v0.3 raises output resolution by 3.3× compared with v0.1 while keeping model‑side and total latency unchanged.
Table 1 shows 192×336 px for v0.1 versus 640×336 px for v0.3; both report ≈200 ms model‑side latency and ≈550 ms total latency.
Each successive version adds a concrete capability while preserving the core streaming backbone.
**Table.** Comparison of system versions v0.1, v0.2, and v0.3 across various technical aspects.
v0.3 delivers a three‑fold resolution boost without any latency penalty.
How does v0.3’s “world + event stream” differ from the “scene + action” split in earlier models?
Earlier models treated the entire video as a single stream; v0.3 isolates the immutable background (“world”) from the rapidly changing user‑driven actions (“event stream”). This separation lets the model reuse the world representation across frames, cutting redundant computation.
Latency Analysis and Conclusion
Key latency numbers stay unchanged while the model gains real‑time expressive behavior.
Wan‑Streamer v0.3 continues to separate a Persistent World from a transient Event Stream, so stable scene context is processed once while only changes are streamed.
v0.3 maintains model‑side latency at ~200 ms, identical to v0.2.
Latency is measured from the moment a 160 ms user streaming unit becomes available to the thinker until the decoded audio‑video response unit is ready, using the same response boundary as prior versions.
total interaction latency stays at ~550 ms, matching the v0.2 runtime.
The 350 ms bidirectional network budget is kept as an external deployment assumption, isolating the model‑side path.
Beyond speed, v0.3 produces 640×368 video at 25 FPS with open‑vocabulary behavior that appears in sync with speech, driving facial expressions, posture shifts, and object interactions while the avatar idles or listens between explicit events.
This world + event‑stream formulation underpins general‑purpose video pre‑training and enables downstream fine‑tuning for roaming, audio‑visual interaction, and embodied manipulation, all without altering the low‑latency streaming contract.
The appendix records the core team (A.1) and the full alphabetical contributor list (A.2), acknowledging the collaborative effort behind the system.
Questions & answers
What is the main contribution of this paper?
The paper introduces a world-event decomposition framework that treats video as a persistent 'world' (stable identity, environment, voice) plus an 'event stream' (speech and free-form behavior directives), serving as both a general-purpose pretraining objective and the architectural basis for the Wan-Streamer v0.3 interactive agent system.
What problem does the world + event stream framework address?
Current video interaction models repeatedly recompute the same background and acoustic conditions at every timestep, wasting compute on parts of the scene that hardly change, which becomes a bottleneck for real-time, full-duplex applications where latency is critical.
Why is the 'world + event stream' framing preferred over training on more interaction data?
The decomposition allows the model to learn from diverse, large-scale video corpora rather than being restricted to specific interaction domains, creating a more scalable pretraining objective that captures how scenes and subjects plausibly evolve.
How does the world-event model differ from feeding full video frames to a standard transformer?
A standard transformer attends over every past frame at each step, recomputing the same background information repeatedly; in the world-event model, the background is cached in the world representation W after one pass, so each streaming step only attends to the small event record.
How are behavior directives represented in the model's output?
Behavior directives are represented as parenthesized tokens embedded directly into the causal token stream, interleaved with spoken language, allowing the model to synchronize complex physical actions with speech without requiring a separate action controller.
What is Block-Causal Attention and how does it differ from sliding-window attention?
Block-Causal Attention keeps the world prefix visible at every step while limiting the dynamic portion, whereas sliding-window attention discards everything outside the window including any persistent world context, losing global conditioning.
Why is the character field c_k kept separate from the event description d_k?
Separating c_k allows the model to reuse a character's persona embedding across many events and makes it easy to query character-independent events (where c_k = ∅) without parsing free text.
What are the key performance specifications of Wan-Streamer v0.3?
Wan-Streamer v0.3 produces 640×368 video at 25 FPS with a 200 ms model-side latency, representing a three-fold resolution boost over previous versions without any latency penalty.
Does the world + event stream approach require a new or more complex architecture?
No; the modeling contract and serving topology remain unchanged from v0.2, as behavior directives are short language tokens that share the existing causal path, adding negligible computational cost.
How does v0.3 differ from earlier versions of the model?
Earlier models treated the entire video as a single stream; v0.3 isolates the immutable background ('world') from the rapidly changing user-driven actions ('event stream'), allowing the world representation to be reused across frames and cutting redundant computation.
What downstream tasks does the world + event stream framework support?
The framework supports downstream fine-tuning for roaming, audio-visual interaction, and embodied manipulation, all without altering the low-latency streaming contract.
What kinds of behaviors can the v0.3 system generate?
The system produces open-vocabulary behavior synchronized with speech, including facial expressions, posture shifts, and object interactions, while the avatar idles or listens between explicit events.
What are the limitations or open questions acknowledged by the paper?
The paper does not explicitly enumerate limitations or open research questions; it focuses on describing the system design and its capabilities without a dedicated limitations section.
What datasets or benchmarks were used to evaluate the system?
The paper does not report specific datasets, benchmarks, or quantitative evaluation metrics used to assess the system's performance.
How does this work compare to prior approaches for interactive video agents?
Prior approaches treated video as a monolithic stream requiring full recomputation at each timestep; this work's world-event decomposition caches stable context once and streams only changes, reducing redundant computation while enabling open-vocabulary behavior generation.
Who are the authors and where was this paper published?
The paper acknowledges a core team listed in Appendix A.1 and a full alphabetical contributor list in Appendix A.2, but the paper does not specify individual author names, venue, or publication date in the provided text.
Key terms
- world-event decomposition
- A framework that separates a video into a persistent 'world' component (stable identity, environment, and voice) and a transient 'event stream' (speech and behavior that changes over time).
- event stream
- The transient, time-varying component of a video consisting of speech and free-form behavior directives that unfold over time, as opposed to the stable world context.
- persistent world (W)
- A cached representation of the stable background, identity, and environmental context in a video that is computed once and reused across all subsequent timesteps.
- Block-Causal Attention
- An attention mechanism that keeps the world prefix visible at every decoding step while restricting attention over the dynamic event portion, preserving global conditioning without full recomputation.
- behavior directives
- Short parenthesized tokens embedded in the model's causal output stream that instruct the agent to perform specific physical actions such as posture shifts or object interactions, synchronized with speech.
- open-vocabulary behavior generation
- The ability to generate a wide, unrestricted range of physical actions and behaviors described in natural language, rather than selecting from a fixed predefined set of actions.
- full-duplex interaction
- A mode of communication in which an agent can simultaneously send and receive audio-visual signals in real time, as opposed to taking turns speaking.
- character field (c_k)
- An explicit field in the event representation that identifies which character an event belongs to, kept separate from the event description to enable persona reuse and character-independent queries.
- causal token stream
- A sequence of tokens generated one at a time in order, where each token can only attend to previously generated tokens, forming the basis of autoregressive language and video models.
- Wan-Streamer v0.3
- The third version of the Wan-Streamer interactive agent system, which implements the world + event stream decomposition to achieve 640×368 resolution at 25 FPS with 200 ms model-side latency.
- model-side latency
- The time delay introduced by the model's computation before it begins producing output, measured here as 200 ms for Wan-Streamer v0.3.
- general-purpose video pretraining
- A training strategy that uses large-scale, diverse video data to teach a model foundational scene dynamics, enabling transfer to multiple downstream tasks without task-specific data collection.