EchoWM: Open and Enterable Omnimodal World Models

Songchun Zhang, Yaowei Li, Junhao Zhuang, Weiyang Jin, Haoyu Wang, Xin Lu, Yilang Sun, Shiyi Zhang, Haoran Li, Xiaoxiao Ma, Yuming Li, Yijun Liu, Yaofeng Su, Yanwen Ma, Haoyu Wu, Zihan Su, Yue Ma, Lvmin Zhang, Haoyang Huang, Zeyue Xue, Anyi Rao, Nan Duan

EchoWM is an omnimodal world model that generates synchronized video, audio, and speech while following continuous 6-DoF camera trajectories.

How can we build a single generative model that supports continuous, enterable navigation while simultaneously synthesizing high-fidelity video, environmental sound, music, and speech?

Generative world models often struggle to provide continuous, user-driven interaction, typically producing silent video or relying on rigid, subject-specific control schemes that fail to generalize across diverse scenes. EchoWM introduces a unified camera-intent interface that maps discrete commands and continuous poses to a shared relative 6-DoF trajectory, allowing the model to interpret navigation as either first-person observer motion or third-person camera-character evolution. This approach enables enterable, omnimodal generation where the model jointly synthesizes 720p video, environmental sound, and speech that remain synchronized over long-horizon rollouts.

Paper Primer

The core mechanism hinges on a unified camera-intent interface that decouples user navigation from specific actor dynamics. By representing all movement as relative 6-DoF trajectories calibrated to a global metric scale, the model treats first-person and third-person interaction as the same geometric problem, injecting this intent into the video backbone via a relative Unified Camera Positional Encoding (UCPE) branch.

The training process follows a progressive curriculum: Audio-Visual Continued Pretraining (AV-CPT) establishes the multimodal prior, Action Fine-Tuning (Action-SFT) isolates control responsiveness by freezing the backbone and training only the trajectory-conditioning branch, and Joint Fine-Tuning (Joint-FT) consolidates both capabilities.

EchoWM achieves state-of-the-art trajectory following and visual quality on public world-model benchmarks.

The model ranks first on the WBench Navigation benchmark and maintains top-tier visual quality on SANA-WM-Bench across both simple and challenging trajectory splits. The system supports multi-turn, long-horizon generation with synchronized audio-visual output, outperforming models that rely on domain-specific action spaces.

Why use camera trajectories instead of raw action logs for control?

Action logs are often system-specific and ambiguous, whereas camera trajectories provide a continuous, domain-agnostic representation of motion that can be recovered from diverse video sources, enabling broader data coverage.

How does the model handle the difference between first-person and third-person perspectives?

The model uses the same relative 6-DoF trajectory interface for both; the initial reference observation and static viewpoint description determine the observer-subject relationship, and the model learns the corresponding camera-character dynamics directly from the training data.

Introduction: Enterable Omnimodal Worlds

EchoWM introduces an enterable omnimodal world model that unifies navigation and multimodal generation.

Most existing world‑model systems generate silent visual rollouts or depend on rigid, subject‑specific action spaces, leaving interactive, multimodal experiences underexplored.

We need a generative medium that users can actively traverse while it simultaneously produces high‑fidelity video, environmental sound, music, and speech.

**Figure.** ECHO-WM: Enterable Omnimodal World Model.

**Figure 1.** EchoWM turns audio-visual generation into an enterable world. Given a reference observation, a structured world description, and a user control sequence represented as a relative 6-DoF trajectory, one model supports first- and third-person interaction, synchronized environmental sound and speech, and multi-turn continuation.

The key shift is moving from passive video generation toward active, enterable world models that couple navigation with synchronized multimodal output.

Related Work

Survey of prior interactive generative world models and their control interfaces.

Interactive generative world models have evolved from simple action‑conditioned simulators to high‑fidelity, multimodal generators. EchoWM builds on this lineage by introducing a unified camera‑centric interface that works for both first‑ and third‑person navigation.

Early works such as UniSim and Genie learned visual futures from discrete actions, while later systems like Genie 3, WorldPlay, and Matrix‑Game pushed resolution and horizon.

Action‑conditioned visual prediction model that learns latent dynamics for short‑term imagination.

Real‑time 720p exploration system that generates diverse environments conditioned on agent actions.

Streaming generative model that emphasizes long‑term geometric consistency across video frames.

Another streaming system that targets long‑horizon interactive video generation.

Control interfaces diverge between discrete action streams and continuous camera geometry, each with distinct representation choices.

Action‑conditioned world model that maps keyboard or controller inputs to future video frames.

Camera‑controllable video model that encodes dense Plücker coordinates via a dedicated encoder.

Incorporates relative camera geometry directly into the attention mechanism, bypassing a standalone encoder.

Joint audio‑visual diffusion models have begun to generate synchronized sound and speech alongside video, yet few expose geometric control.

Diffusion model that jointly synthesizes video and accompanying audio tracks.

Streaming autoregressive generator that maintains cross‑modal synchronization for long video sequences.

Omnimodal world model that unifies video, audio, language, and action for embodied AI.

Causal video diffusion and memory‑augmented retrieval have enabled longer interactive rollouts, but explicit retrieval adds overhead.

Causal adaptation of full‑sequence diffusion models for autoregressive video prediction.

Memory system that indexes past frames by pose or field of view for retrieval during generation.

LingBot‑World‑v2 is a predecessor world model that already supports joint video‑audio synthesis and basic camera‑based control, but its interface is limited to a single perspective.

The World Data Engine

How diverse recordings are merged into a unified training pipeline.

Training an enterable omnimodal world demands four properties—diverse appearance, natural audio, interactive motion, and reliable camera geometry—that rarely coexist in a single dataset. To obtain them we fuse four complementary sources, each covering a subset of the requirements.

The engine is an assembly line that ingests heterogeneous recordings, extracts the modality each source excels at, and stitches them into a single, camera‑intent‑aligned training example.

How is the World Data Engine different from simply concatenating raw video clips?

It does more than stack footage. The engine first extracts the strongest signal from each source (e.g., scripted actions, ground‑truth poses), runs separate audio‑visual and geometry processing streams, aligns them in time, and annotates each clip with a unified camera‑intent representation. The result is a structured example that the model can condition on, not an unfiltered mash‑up.

Audio‑visual path: the gameplay clip is sliced into a single short clip; the UE clip’s audio channel is discarded because UE provides no natural sound.

Geometry path: the UE clip’s 4‑second window is fed to the pose recovery module, yielding metric camera trajectories for all five views.

Alignment: timestamps from both clips are matched (they both start at t=0), and the action log from the gameplay clip is attached to the UE pose sequence.

Merging: a unified training example is created containing the RGB frames, the recovered poses, the action log, and the native audio from the gameplay clip.

Mixture assignment: the example is randomly placed into the “Joint‑FT” mixture because it contains both high‑quality geometry (UE) and natural audio (gameplay).

This toy walk‑through shows that the engine does not merely concatenate streams; it selects the best‑available signal from each source and fuses them into a single, consistently annotated example.

**Figure 3** Game-video dataset statistics. The combined training corpora contain predominantly third-person clips, a broad mix of game activities, and a smaller speech-present subset. Percentages are normalized within the reported collection.

**Figure 4.** World data engine. We combine internally collected game recordings, human-played Internet game recordings, Unreal Engine renders, and general Internet video to complement action logs, natural audio, interactive motion, reliable poses, and cross-domain appearance. A shared pipeline constructs synchronized clips, filters and annotates video and audio, recovers reliable metric poses, and forms AV-rich, control-clean, and balanced high-quality mixtures for AV-CPT, Action-SFT, and Joint-FT, respectively.

Because the audio‑visual and geometry objectives impose different continuity requirements, we split processing into two complementary paths.

The audio‑visual path first runs shot‑boundary detection on Internet and cinematic footage, then slices the detected continuous spans directly into short clips ready for appearance, speech, and environmental‑sound learning.

The geometry path retains roughly one‑minute windows for non‑UE data, runs metric pose recovery on the full window to obtain a shared geometric reconstruction, and only afterwards slices the window into short clips, ensuring neighboring clips inherit a consistent pose frame.

After both paths finish, we align their outputs temporally, attach structured audio‑visual annotations, and package the results into three stage‑aligned mixtures (AV‑CPT, Action‑SFT, Joint‑FT) that feed the downstream training stages.

Metric Pose Recovery

Recover metric camera poses, densify them, filter quality, and blend into staged data mixtures.

Heterogeneous video sources either provide metric camera‑to‑world poses (e.g., UE) or only raw video, forcing the system to reconstruct metric trajectories before training.

Recover a temporally coherent, metric‑scaled camera trajectory from raw video by fusing long‑range geometry with per‑frame depth cues.

ViPE receives VGGT‑Omega’s coarse geometry (relative rotations) and MoGe‑2 depth values for each frame.

Joint optimization adjusts the pose of frame 3 to a translation of 0.45 m forward and a 5° yaw, while scaling intrinsics to match the depth cue.

Frames 0‑9 converge to a smooth trajectory where inter‑frame translation never exceeds 0.12 m and rotation change stays below 3°.

The final metric scale aligns the whole window to real‑world meters, enabling downstream clipping.

Even a tiny window benefits from long‑range constraints: the optimizer propagates metric depth from a few reliable frames to the entire sequence, preventing scale drift.

How does this differ from a classic Structure‑from‑Motion pipeline?

Classic SfM treats each video independently and often yields only relative poses; our pipeline fuses a long‑range geometric backend (VGGT‑Omega) with per‑frame metric depth (MoGe‑2) and explicitly optimizes intrinsics per frame, producing absolute metric scale and temporally coherent trajectories.

Processing every frame is computationally heavy; subsampling then interpolating restores full resolution cheaply — like sketching a road map with a few landmarks and then filling in the missing streets.

Temporally subsample the 1‑minute window (e.g., keep every 5th frame).

Run metric pose recovery on the sparse set.

Interpolate rotations back to original timestamps using spherical linear interpolation (SLERP).

Interpolate translations linearly between recovered poses.

Convert the dense trajectory to a common camera‑to‑world convention, axis orientation, metric unit, and sampling rate.

Slice the dense sequence into short clips, re‑index timestamps relative to each clip start.

Before feeding trajectories to the model we discard any that fail three quality checks: unreliable reconstruction or unstable intrinsics, excessive jitter, and implausible motion or extreme scale outliers.

Audio‑visual filtering removes visually degenerate clips (poor exposure, low aesthetic score) and, when audio exists, discards tracks with unusable loudness or missing speech.

Structured annotation normalizes heterogeneous field names (e.g., `static_scene_caption` → scene) into a canonical schema: scene, style, viewpoint, subject, narrative, speech, sound.

To avoid leaking motion through text, the narrative field is omitted during trajectory‑conditioned training, leaving only static fields and the recovered trajectory as motion signals.

Stage‑aligned data mixtures form a curriculum: an AV‑rich pool for broad visual‑audio priors, a control‑clean pool for reliable metric trajectories, and a balanced high‑quality pool that combines both.

Collect all processed examples with their available signals.

Define the AV‑rich mixture: select examples with high visual/audio quality, regardless of pose reliability.

Define the control‑clean mixture: filter for reliable metric trajectories and smooth motion, then drop the narrative field.

Intersect the two sets to obtain the balanced high‑quality mixture, preserving both trajectory fidelity and audio‑visual richness.

Sample each mixture according to the stage schedule (AV‑CPT → Action‑SFT → Joint‑FT).

**Figure 5. Trajectory coverage of the control-clean mixture.** (a) Aggregate lateral–forward motion coverage over 28,605 sampled trajectories; the marked P90 range summarizes the robust spatial extent. (b) Representative straight, diagonal, return-loop, and orbit trajectories shown at a common spatial scale. Black and orange markers denote trajectory start and end points, respectively, and dashed circles show fitted orbits.

**Figure 8** Pose-magnitude distributions for global scale calibration. The plots show the peak-normalized distribution and empirical CDF of the per-clip maximum translation magnitude $m_i$ for the action-clean and AV-rich speech-present subsets. Action-clean clips contain broader motion magnitudes (mode 4.5, median 12.7), whereas speech-present AV-rich clips concentrate at smaller motions (mode 0.5, median 3.9). The heterogeneous support motivates one robust dataset-level percentile scale rather than independently normalizing each source.

In sum, metric pose recovery, densification, quality filtering, and staged mixtures together provide consistent, controllable trajectories that power the unified camera‑intent interface.

The Camera-Intent Interface

We map all user controls to a calibrated relative 6‑DoF camera trajectory that conditions video generation.

The bottleneck is that heterogeneous user controls—keyboard presses, gamepad actions, or motion capture—cannot be fed directly to a diffusion transformer without a common geometric language.

All navigation commands are expressed as a relative 6‑DoF camera trajectory, letting first‑person ego‑motion and third‑person camera‑character evolution share the same conditioning signal.

How does this differ from learning a mapping from keyboard commands to low‑level motor torques?

Instead of predicting actuator forces, we predict the resulting camera pose directly. This bypasses the ill‑posed inverse problem of inferring the exact control signals that produced a given motion, which varies across devices and game engines.

Integrating discrete navigation commands into the relative trajectory.

Frame 1: $u_1$ = forward‑only → $\xi_1 = (0.5,0,0,\,0,0,0.1)$. $T_1 = \exp(\xi_1)$ yields a 0.5 m forward shift and 0.1 rad yaw.

Frame 2: $u_2$ = forward + right → $\xi_2 = (0.5,0.2,0,\,0,0,0)$. Applying $\exp(\xi_2)$ to $T_1$ moves the camera further forward and right.

Frame 3: no keys pressed → $\xi_3 = \mathbf{0}$. $T_3 = T_2$ (pose stays unchanged).

Relative poses: $\Delta T_1 = T_0^{-1}T_1 = T_1$, $\Delta T_2 = T_0^{-1}T_2$, $\Delta T_3 = T_0^{-1}T_3$.

The trajectory encodes both translation magnitude and rotation direction; even a short key press sequence yields a smooth 6‑DoF path that the generator can follow.

Because different data sources use different units, we calibrate translation scale globally: the 90‑th percentile of per‑clip maximum displacements defines $s_{\text{global}}$, and all trajectories are divided by this constant.

**Figure 6.** Overview of EchoWM. Media context, structured text, and user controls share a unified camera-intent interface. Discrete controls or metric poses are converted into globally calibrated relative 6-DoF trajectories and serialized as an event stream. A relative UCPE branch injects this trajectory into video self-attention before audio-visual cross-attention, while the audio stream receives no direct trajectory condition. Four progressive training stages yield synchronized audio-visual generation and support multi-turn inference through synchronized tail-window conditioning.

**Figure 7** Camera intent unifies first- and third-person interaction. The same user intent is realized as observer motion in first-person scenes and as coordinated camera–character–world evolution in third-person scenes. The coupling is learned from heterogeneous data rather than specified by an explicit camera rig.

Progressive Audio-Visual Control

Progressive training aligns audio‑visual and trajectory supervision for reliable interactive generation.

AV‑rich training clips give strong audio cues but cover only a narrow set of camera motions, while control‑clean clips provide diverse, reliable trajectories but lack rich sound. Training a single model on the mixed data therefore suffers from conflicting supervision signals.

The model is first taught to see and hear, then to follow precise camera paths, and finally to combine both skills on a balanced subset—each stage builds on the previous one without overwriting it.

Stage 1: train $\theta$ on E₁ and E₂ only; the loss sees both video/audio and the full annotation, but the trajectory field is ignored.

Stage 2: freeze $\theta$, train ϕ on E₃ and E₄; the loss sees only static text fields and the trajectory, while audio loss is disabled.

Stage 3: fine‑tune both $\theta$ and ϕ on a balanced mini‑batch {E₁, E₃}; the learning‑rate multiplier is set to 0.1 of the previous stages.

After three passes the model can generate synchronized video‑audio while following an arbitrary trajectory supplied at test time.

The staged schedule lets the backbone retain rich sound‑visual patterns while the camera branch learns precise motion, avoiding the “one‑size‑fits‑all” collapse that occurs if both signals are trained together from scratch.

How does Progressive Audio‑Visual Control differ from simply fine‑tuning the whole model on the mixed dataset?

Joint fine‑tuning from the start would force the backbone to accommodate noisy trajectory signals while still learning audio‑visual cues, leading to a trade‑off where neither signal is fully captured. The three‑stage schedule isolates the learning signals, so the backbone masters audio‑visual fidelity first, the camera branch learns clean motion next, and only then are they combined under a gentle learning‑rate schedule.

After the three‑stage curriculum the model is still bidirectional, which is unsuitable for online generation. The next step rewrites the backbone into a causal, chunk‑wise streaming generator.

We concatenate clean and noisy versions of each chunk and mask the attention so that a noisy chunk can only see past clean chunks and its own noisy tokens—exactly the information available during streaming.

**Figure 9** Chunk-level causal attention patterns used for streaming post-training. (a) The audio-visual teacher-forcing pattern used by short-horizon reconstruction: each noisy query attends to the preceding clean context and the noisy tokens of its current chunk, while clean targets and future chunks are excluded. (b) The mask used by Forward 2 of long-horizon SGF, where access to clean history is additionally constrained by the sink-plus-FIFO KV-cache policy.

We first generate a rollout without gradients, then replay the same rollout with gradients attached to the cached clean context, letting the loss flow through the reconstructed attention maps.

We extend SGF to roll out many consecutive segments, keeping the same sink‑plus‑FIFO cache so the model sees a bounded but persistent history.

At inference the model can now operate in two modes: (1) bidirectional multi‑turn continuation, where each turn re‑encodes the tail of the previous output as clean context; and (2) causal streaming, where a persistent KV‑cache (sink + FIFO) supplies long‑term history without restarting.

WBench Navigation Results

WBench shows EchoWM’s leading interactive video quality and consistency.

The paper’s core premise is that a shared camera‑intent interface lets a single model generate coherent, enterable worlds from both first‑person and third‑person controls.

WBench is a benchmark that evaluates how well a world model can follow interactive navigation commands while preserving visual quality, scene layout, and temporal consistency.

Why does WBench report five separate dimensions instead of a single score?

Because interactive world modeling has multiple failure modes—visual degradation, loss of scene layout, command mis‑execution, temporal drift, and implausible physics—each of which can dominate the user experience. Reporting them separately lets researchers see which aspect a model struggles with and target improvements accordingly.

EchoWM attains the highest overall Average score of 81.7 on WBench, beating the next‑best method by +0.2 points.

Table 3 shows EchoWM’s Average of 81.7 versus the runner‑up (SANA‑WM) at 81.5.

**Figure 10** WBench Navigation comparison.

**Figure 11.** WBench Navigation comparison.

**Figure 12.** WBench Navigation comparison.

**Figure 13.** Qualitative comparison of causal world models on WBench Navigation. ECHOWM-Flash shows better action adherence and multi-turn consistency in scene structure, subject identity, and visual style.

Long-Horizon Evaluation

EchoWM sets new short‑ and long‑horizon benchmarks on SANA‑WM‑Bench.

We evaluate EchoWM on the SANA‑WM‑Bench under three protocols: a 241‑frame short‑horizon test, the official 961‑frame long‑horizon rollout, and a four‑step causal streaming variant.

The benchmark measures how well a world‑generation model follows prescribed camera trajectories while preserving visual fidelity over short and extended horizons.

EchoWM attains the highest short‑horizon VBench Overall, 83.91 on Simple trajectories and 83.96 on Hard trajectories, beating the runner‑up by up to 2.78 points.

Table 4 (short‑horizon results).

Beyond the headline quality, EchoWM also records the lowest translation (1.697 m) and camera‑motion consistency (0.160) errors on the Simple split, while remaining within 0.04° of SANA‑WM’s rotation error. On the Hard split, SANA‑WM yields the smallest rotation error (0.199°) but EchoWM still leads in visual quality.

Under the 961‑frame long‑horizon protocol (Table 5), EchoWM preserves strong visual quality (VBench = 81.36 / 81.72) and achieves the best revisit PSNR (15.10 dB) on the Simple split. Pose drift grows to 3.22° (Simple) and 12.05° (Hard), confirming that trajectory accuracy remains the primary long‑horizon challenge.

When evaluated as a four‑step causal streamer (Table 6), EchoWM‑Flash outperforms all competing causal models, attaining the highest VBench Overall (80.13 / 81.06) and the lowest rotation, translation, and CMC errors among causal methods.

**Figure 14.** Imaging quality comparison on the Simple- and Hard-Trajectory splits. We report the first-window, mean, minimum, and final-window IQ. Our method achieves consistently higher mean and minimum IQ than Evoke and SANA-WM with Causal Refiner. Although the competing methods exhibit smaller IQ drops, their overall IQ remains lower, indicating that a small drop can also result from consistently low imaging quality rather than stronger long-horizon quality preservation.

**Figure 15** Qualitative comparison of long-horizon causal streaming on representative SANA-WM-Bench trajectories under the 961-frame protocol. We compare ECHOWM-Flash with LingBot-World-v2 (LBW-V2), SANA-WM with and without Causal Refiner, and Evoke, using the same initial observation and camera-trajectory condition. Frames are shown at 12-second intervals. ECHOWM-Flash exhibits less accumulated autoregressive drift while better preserving trajectory adherence, scene identity, visual style, and major spatial structure over the extended rollout.

User Study

User study shows EchoWM is preferred over competitors across key criteria.

The pairwise study asked participants to judge 200 generated scenarios, comparing EchoWM against each baseline on six quality dimensions.

**Table 7.** Aggregate pairwise user-study judgments over 200 cases. Each cell reports the percentage of judgments. “Both good” and “both bad” preserve the two distinct same-quality outcomes.

Qualitative Analysis and Ablations

Qualitative probes reveal how the shared camera‑intent interface holds up across worlds, viewpoints, and modalities.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers