World Observer: Joint Actor-Observer Generation for Persistent World Modeling
Hyunwook Choi, Dahyun Chung, Hyunsung Kim, Siyoon Jin, Jinhyeok Choi, Junyoung Seo, Seungryong Kim
World Observer decouples acting from observing to maintain persistent world state for objects outside the actor's view.
How can we maintain persistent, evolving object states in a world model when those objects move outside the actor's current field of view?
Video world models are actor-centric, meaning they lose track of objects once they leave the camera's field of view. This leads to objects disappearing, freezing, or reappearing in incorrect states when the actor turns back. World Observer solves this by jointly generating a perspective actor and one or more panoramic observers that continuously track the surrounding environment. The actor and observers share a single world state, ensuring that objects remain dynamic even when unobserved. This approach significantly improves out-of-view dynamics, allowing objects to re-enter the actor's view with consistent, updated states while maintaining competitive visual fidelity.
Paper Primer
The core mechanism hinges on decoupling the observer from the actor: the system generates a 360° panoramic video stream alongside the actor's perspective view. By warping both streams from a shared initial panorama and using an "Observer Sink" of high-resolution perspective references, the model maintains geometric correspondence and fine appearance detail for re-entering objects.
World Observer significantly improves out-of-view dynamic consistency compared to existing world models.
The model achieves higher scores on OOV-Dgt (ground-truth dynamics) and OOV-Dself (self-consistency) metrics across both real and synthetic benchmarks. In synthetic benchmarks, the multi-observer variant achieves an OOV-Dgt score of 0.528 and OOV-Dself of 0.457, outperforming baseline models that struggle to maintain object state during out-of-view intervals.
The system uses a "joint generation" strategy where actor and observer latents are concatenated and processed by a shared Diffusion Transformer (DiT). This allows the observer to act as both a persistent memory of the world and a control mechanism, where prompts can drive events in regions the actor cannot currently see.
Why is a panoramic observer necessary instead of just using a larger memory buffer?
Existing memory-based approaches infer unseen evolution from previously observed data, which leads to state drift as intervals grow longer. World Observer maintains explicit visual evidence of the world's evolution, ensuring that objects continue to move and interact even when they are not in the actor's view.
What is the role of the "Observer Sink" in this architecture?
The panoramic observer is inherently distorted, which limits the fine appearance detail available to the actor. The Observer Sink provides fixed, high-resolution perspective crops from the initial panorama that the actor can attend to, restoring visual fidelity when a region re-enters the actor's view.
By treating the world as a persistent, multi-view entity rather than a single-view projection, World Observer enables more reliable long-horizon planning and interactive simulation for embodied agents.
The Problem of Actor-Centric Modeling
World Observer decouples acting from observing to keep out‑of‑view objects evolving.
Existing world models are actor‑centric: they update the world only where the actor currently looks, so when an object exits the camera view the model loses direct evidence of its evolution, leading to four characteristic failures—frame‑locked, lost, frozen, and impostor—each breaking continuity of the object’s state.
**Figure 2.** **Motivation.** When dynamic objects leave the actor’s view, existing models exhibit four failures: (a) frame-locked, (b) lost, (c) frozen, and (d) impostor state. These failures show the difficulty of maintaining and evolving states without direct observations during out-of-view intervals.
**Figure 1.** **Teaser.** World Observer jointly generates panoramic observers with the actor. (a) Decoupled observer enables control over actor's unseen regions. (b) Multiple observers can be placed flexibly to track occluded regions.
The four failure modes—frame‑locked, lost, frozen, impostor—highlight why actor‑centric world models cannot reliably maintain out‑of‑view dynamics.
Data Construction and Setup
We build synchronized actor‑observer video pairs and describe how a single Diffusion Transformer generates both streams together.
Actor‑centric models lose objects that leave the camera frustum, i.e., Out‑of‑View (OOV) regions, which hampers long‑term consistency.
Instead of generating only the actor’s view, the model simultaneously produces a panoramic observer stream that keeps track of everything the actor cannot see.
Start with actor latent $z_a^{(1)}$ and observer latent $z_o^{(1)}$ for frame 1.
Condition the DiT on $C_a$, $P_a$, $C_o$, $P_o$ and sample $z_a^{(2)}$, $z_o^{(2)}$ for frame 2.
Repeat for frames 3 and 4, each time feeding the newly sampled latents into the next step.
After frame 4, keep the last $H=2$ latents ($z_a^{(3)}$, $z_a^{(4)}$, $z_o^{(3)}$, $z_o^{(4)}$) as history for the next chunk.
History latents act as a short‑term memory that stitches consecutive chunks together, ensuring temporal continuity across the actor and observer streams.
For multi‑observer scenarios ($N>1$), each observer receives its own trajectory $C_o^{(i)}$ and prompt $P_o^{(i)}$, but all streams still share the same latent backbone.
Initialize actor latent $z_a^{(1)}$ and observer latent $z_o^{(1)}$ from the VAE encoder.
Feed $z_a^{(1)}$, $z_o^{(1)}$, $C_a$, $P_a$, $C_o$, $P_o$ into DiT to sample the next $T$ latents.
Decode the sampled latents into RGB frames for both streams.
Extract the last $H$ latents from each stream as history for the next chunk.
Repeat until the desired video length is reached.
**Figure 3.** **Dataset Overview.** Real-world (top) and synthetic (bottom) examples with moving and two-fixed-observer configurations, respectively. Colored boxes show cross-view correspondences, while the rightmost diagrams show camera configurations and trajectories.
Joint Actor-Observer Modeling
Jointly generate actor and observer streams so out‑of‑view objects stay consistent over time.
We generate the actor and observer streams together, like two synchronized video feeds that share the same timeline—each frame updates both views so they stay consistent.
Compute $\tilde{Z}_a^{k} = (1 - 0.5)\cdot 0.2 + 0.5\cdot 0.0 = 0.1$.
Compute $\tilde{Z}_o^{k} = (1 - 0.5)\cdot 0.5 + 0.5\cdot 0.0 = 0.25$.
Form the joint sequence $[\tilde{Z}_a^{k}, \tilde{Z}_o^{k}] = [0.1, 0.25]$ and feed it to the shared DiT.
The model predicts the residuals $(\epsilon - Z_a) = -0.2$ and $(\epsilon - Z_o) = -0.5$.
Mean‑squared error loss $L = \frac{1}{2}\big((-0.2)^2 + (-0.5)^2\big) = 0.145$.
Even with a tiny toy example, the joint sequence lets the model learn to denoise both streams simultaneously, enforcing consistency between actor and observer latents.
How does joint generation differ from training two separate models for the actor and the observer?
Separate models would process independent sequences and could not exchange information; the joint approach shares a single DiT, aligns matching positions on the RoPE axis, and allows cross‑attention so the observer’s memory directly informs the actor’s predictions.
Panoramic Observer and Sink
Introducing the Panoramic Observer and Observer Sink mechanisms for joint actor‑observer modeling.
Panoramic warping ties both streams to a common 360° view, but the raw panorama introduces geometric distortion that erodes fine appearance and weakens actor‑observer correspondence. To keep the observer lightweight while preserving detail for re‑entering regions, we introduce a dedicated Observer Sink.
Think of a single world map that both a driver and a drone consult: the map (the panorama) gives a shared reference, while each follows its own trajectory, staying geometrically aligned.
Unproject each pixel into a 3‑D point at distance 1, forming a flat square point cloud.
Rotate the point cloud 90° around the vertical axis to match the target pose.
Project the rotated points back to a $4\times4$ image, yielding the warped view $\text{warp}_t$ where the original left edge now appears at the top.
Encode the warped view with the 3D VAE, producing a latent tensor.
Concatenate the latent with a binary mask that marks all 16 pixels as valid (no occlusion).
Even with a tiny toy example, the warp preserves the global layout while re‑orienting the scene, illustrating how the observer stays aligned with the actor despite viewpoint changes.
How does this differ from simply re‑projecting the actor view at each timestep?
The Panoramic Observer uses a single global panorama and its depth to generate consistent warps for both streams, guaranteeing shared geometry. Re‑projecting the actor view would require separate depth estimates per view and would not ensure that actor and observer stay aligned.
Imagine a photographer who first captures a wide‑angle panorama and then takes four close‑up shots of the same scene; the close‑ups (the sink) provide the fine detail that the wide view lacks when objects re‑appear.
Crop 0 (front) yields a $4\times4$ patch centered on the forward direction.
Encode the patch with the VAE, producing latent $z^{\text{os}}_0$.
Repeat for crops 1‑3, obtaining $z^{\text{os}}_1$, $z^{\text{os}}_2$, $z^{\text{os}}_3$.
Append the four latents to $Z_{\text{seq}}$ at positions $H+L+\Delta^{\text{os}}$, $H+L+\Delta^{\text{os}}+\delta^{\text{os}}$, … respectively.
During generation, when an object re‑enters the actor view, attention heads can retrieve the matching high‑resolution latent, restoring fine appearance.
The sink supplies detailed texture exactly where the actor needs it, while the observer continues to operate on a coarse representation, keeping overall compute low.
Why not simply upscale the warped panorama instead of using a separate sink?
Upscaling would amplify the geometric distortion inherent in the panorama and offers no explicit perspective alignment. The Observer Sink provides true perspective crops that preserve local geometry and texture, enabling accurate rendering of re‑entering objects without extra distortion.
Warping and sink integration pipeline.
**Figure 4.** **Model Overview.** (a) World Observer jointly generates actor and panoramic observer streams with a shared DiT. (b) Observer Sink converts the high-resolution initial panorama into four perspective references for fine-grained appearance.
**Figure 5.** Visualization of self-attention. A query on a re-entering object attends to the observer region holding its out-of-view state, while the object fails to re-enter after out-of-view interval in non-joint model (Baseline).
Flexible Observer Placement
Flexible observers decouple monitoring from the actor, enabling independent tracking of out‑of‑view regions.
A fixed observer tied to the actor cannot watch distant regions where crucial changes happen. To keep those areas observable we let the observer move independently.
Think of a security camera that can be placed anywhere in the scene instead of being mounted on the protagonist – it can stay focused on a region of interest while the protagonist wanders elsewhere.
Frame 1: Actor at $(0,0)$. Closest observer is $p_1$, so $x_o^0$ = observer 1.
Frame 2: Actor moves to $(5,0)$. Distances to $p_1$ and $p_2$ are both $5$, tie‑break selects observer 1.
Frame 3: Actor reaches $(10,0)$. Closest observer becomes $p_2$, so the system switches to observer 2 for warping.
Switching observers when the actor crosses the midpoint keeps the geometric source near the actor, preserving stable camera control while still covering the far region.
How does a flexible observer differ from simply moving the actor’s camera?
Moving the camera keeps the viewpoint rigidly attached to the actor, so any region away from the actor becomes invisible. A flexible observer is an independent camera that can stay in a fixed location or move along its own trajectory, preserving visibility of out‑of‑view areas regardless of the actor’s motion.
Imagine several independent security cameras each covering a different zone, whose feeds are merged by a single processor that ensures all views stay consistent with the same underlying world.
Why generate all observer streams jointly instead of treating each observer as an independent model?
Joint generation forces the streams to share a common latent representation, guaranteeing that objects seen by different observers remain consistent in position and appearance. Independent models would drift, leading to contradictory world states across observers.
Empirical Results
World Observer dramatically improves out‑of‑view fidelity while preserving visual quality.
The paper’s premise is that a persistent observer stream can keep track of objects that leave the actor’s view. This section shows how World Observer delivers on that promise.
World Observer (Multi) raises Out‑of‑View fidelity (OOV‑F) by 0.20 over the strongest baseline.
Table 1 reports OOV‑F = 0.722 for World Observer (Multi) versus 0.522 for Dream‑X, the next best model.
**Figure 6.** **Qualitative results.** World Observer supports (a) state memory, (b) observer-prompted control, (c) long-horizon generation, and (d) multi-observer. Insets show the observer views.
**Figure 7.** Qualitative comparisons with other models. Red boxes mark target objects before exit and after re-entry. While existing models often lose, freeze, or inconsistently reconstruct them, World Observer better preserves object identity and state evolution across out-of-view intervals.
Quantitative Performance Analysis
World Observer sets a new record for out‑of‑view object tracking while preserving visual fidelity.
World Observer attains the highest Out‑of‑View (OOV‑F) score among all baselines.
Table 1 reports an OOV‑F of 0.674, outpacing the runner‑up by 0.12.
Baseline models often suffer: low OOV‑F means objects disappear permanently, while high OOV‑F paired with low OOV‑Dgt/Dself signals frozen or incoherent re‑entries. Conversely, high OOV‑Dgt with lower OOV‑Dself reveals impostor dynamics—objects follow the prompt but diverge from their true paths. World Observer preserves both OOV‑F and OOV‑D, ensuring re‑entered objects retain a consistent, continuing state. Adding extra observers further boosts OOV‑F without degrading OOV‑D metrics, confirming the benefit of multi‑observer coverage.
Ablation Study
We test how each component impacts out‑of‑view handling, camera control, and visual fidelity.
The core premise is that a persistent panoramic observer stream lets the model keep track of regions that leave the actor’s view. This section asks whether each ingredient of that design is essential.
Jointly generating actor and observer yields the strongest OOV scores while preserving camera stability and 3D consistency. Removing either side degrades the metric where that side contributed most, confirming that the two streams complement each other.
**Table 2.** Ablation study on Synthetic-OOV-Bench. We evaluate visual and temporal fidelity, camera-following accuracy, 3D adherence, and out-of-view dynamics. For fairness, all variants share the full model's conditioning. RotErr is (-) for w/o Actor generation, whose panoramas are cropped with the ground-truth rotation.
**Figure 11.** **Qualitative ablation results.** We show the effect of removing each component on generation quality. Without the observer, the model fails to maintain out-of-view dynamics, causing the dynamic object to disappear when it re-enters the view. Without the actor, panorama-based generation followed by cropping leads to reduced visual fidelity. World Observer preserves evolving out-of-view dynamics while maintaining high-fidelity generation.
Limitations and Future Work
World Observer keeps track of out‑of‑view objects by jointly generating a persistent panoramic observer stream.
The limitation subsection notes that World Observer currently requires panoramic observations as conditioning inputs, which confines its applicability to environments where such views are available.
One way to relax this requirement is to first outpaint a single perspective view into a full panorama, enabling the framework to operate with only perspective inputs.
The same outpainting idea can extend to multi‑observer scenarios: several panoramic conditions can be synthesized from perspective observations taken at different locations.
Because the observer budget is limited, capturing the entire world remains infeasible; however, flexible placement lets the system allocate its budget to regions of interest, preserving their dynamics regardless of the actor’s trajectory.
**Figure 12.** Qualitative ablation results in AR generation. We compare ground truth with variants without the observer and Observer Sink under a long camera trajectory with a full 360° sweep. Without the observer, out-of-view object dynamics are not maintained, while removing the Observer Sink degrades fine appearance under large viewpoint changes. Our full model preserves both dynamic objects (red/yellow) and appearance details (green) upon re-entry.
The societal impact subsection argues that decoupling observation from acting allows world models to maintain evolving states of selected regions even when the actor does not see them.
This capability benefits embodied navigation, interactive simulation, and planning, where persistent representation of relevant world parts is essential.
Flexible and multi‑observer configurations further enable a limited observation budget to be focused on regions of interest, supporting continuous world‑state maintenance without requiring the actor to revisit those regions.
**Figure 13.** Qualitative prompt ablation results. We compare (a) actor-concentrated, (b) unified, and (c) decoupled actor-observer prompting. Actor-concentrated prompting misplaces off-view events in the actor view, while unified prompting can hallucinate duplicate subjects in the observer. Decoupled prompting instead produces a consistent event across both streams.
Implementation Details
Implementation specifics of World Observer, training regime, and model architecture.
Recent world‑model approaches rely on a generative prior learned from large‑scale pretraining, optionally retrieving past observations or storing explicit spatial states. However, these components provide no direct evidence of how dynamic content evolves once it leaves the actor’s view, leading to frame‑locked, lost, frozen, and impostor failures (Fig. 2). In contrast, we jointly generate a panoramic observer that explicitly represents out‑of‑view dynamics.
Prior work has used panoramas either as intermediate representations for building coherent 3D worlds or as direct outputs via panoramic video generation. Producing panoramas as final output is challenging because pretrained backbones are optimized for perspective imagery, prompting tangent‑plane or cubemap formulations, and a fixed token budget reduces effective resolution per viewing direction. Our actor‑observer framework decouples observing from acting, allowing one or more observers to capture surrounding regions independently while the perspective actor remains high‑quality, sidestepping resolution and prior limitations.
Benchmarks such as STEVO‑Bench, WRBench, and MemoBench assess whether models maintain evolving states when parts of the world become temporarily unobserved, but they rely on VLM judgments of naturalness. Approaches like HyDRA and ReMind use dynamic memory and generative priors, whereas LiveWorld and WorldDirector advance hidden states through simulation, each facing issues of drift, limited evidence, or the need for structured entity modeling. We instead employ a panoramic video as a global observer that continuously captures out‑of‑view dynamics.
Implementation details of World Observer are described below.
World Observer is trained with teacher forcing; during training we inject Gaussian noise (scale 0.1) into conditioning history frames with 50 % probability to bridge the gap between clean inputs and self‑generated inference history. Each chunk conditions on $H$ = 2 history latents (5 frames) and mixes two data sources via ratio‑based interleaved sampling (60 % real panoramic videos, 40 % synthetic CARLA video), while the multi‑observer stage is trained on synthetic data only. Camera extrinsics are unified to a Right‑Down‑Forward ($RDF$) system, warping sources are sampled from surrounding frames, and a dropout schedule replaces captions and zeroes Observer Sink tokens 20 % of the time to enable classifier‑free guidance.
World Observer fine‑tunes Cosmos‑Predict2.5, a 2 B‑parameter Diffusion Transformer ($DiT$) with 28 blocks, 16 attention heads, and hidden dimension 2048, operating in a 16‑channel latent space compressed 4× temporally and 8× spatially. A chunk of $T$ = 77 frames yields $L$ = 20 latents per stream, rendered at actor resolution $R_a$ = 1280 × 704 and observer resolution $R_o$ = 640 × 320, giving the observer roughly a quarter of the actor’s tokens for a compact budget. The $DiT$ input concatenates the noisy target latent, warped video, warp validity mask, a learnable 6‑channel view embedding (distinguishing perspective and panoramic tokens), and per‑observer embeddings $e_{\text{obs}}$, while camera poses are encoded as a 6‑channel Plücker ray map added after patch embedding, and Observer Sink crops add four RoPE‑positioned tokens at offset $\Delta_{os}$ = 50 with stride $\delta_{os}$ = 5.
Evaluation Protocol
Evaluation protocol and metrics for the Out‑of‑View benchmark.
This appendix details how we assess the ability of world models to handle objects that disappear from view and later re‑appear. We first describe the construction of the Out‑of‑View benchmark, then list the quantitative metrics used for evaluation.
OOV‑Bench measures whether a model can keep track of dynamic objects that exit the camera’s field of view and correctly re‑integrate them when they return.
**Figure 8.** Out-of-view dynamics evaluation protocol. We track dynamic objects in 3D to evaluate their exit and re-entry (OOV-F) and motion consistency during out-of-view intervals (OOV-`D_self` and OOV-`D_gt`).
**Figure 9.** Comparison with existing Out-Of-View metrics. VLM-based metrics fail to identify frozen and impostor states as inconsistent, while our 3D displacement-based metrics quantitatively capture these failures.
Beyond OOV‑Bench, we evaluate overall generation quality using four families of metrics: visual fidelity (FID, FVD), image quality (VBench), camera‑following accuracy (RotErr, TransErr), and 3‑D adherence (masked PSNR, LPIPS). All metrics are reported separately for the real and synthetic OOV splits.
Evaluation Metrics and Details
Appendix C Continued defines out‑of‑view metrics, compares them to prior benchmarks, and details baseline adaptation.
Dynamic objects are excluded from reconstruction metrics; we segment them with SAM3 using text prompts for pedestrians and vehicles, then compute PSNR and LPIPS only on the remaining static regions.
Camera‑following accuracy is measured by extracting per‑frame extrinsics with Depth Anything 3 on both generated and ground‑truth frames, normalizing each translation trajectory by its maximum norm, and reporting mean geodesic rotation error (RotErr) and mean ℓ₂ translation error (TransErr).
For out‑of‑view dynamics we define OOV‑F as the fraction of objects that follow the ground‑truth exit‑and‑re‑entry pattern, and OOV‑D metrics that compare the 3‑D displacement of each object to either the ground‑truth trajectory (OOV‑`D_gt`) or its own pre‑exit motion (OOV‑`D_self`).
Existing benchmarks such as MemoBench and WRBench rely on VLM judges that cannot reliably infer 3‑D motion, leading to false “Yes” answers for frozen or impostor cases; our explicit displacement metrics expose these failures.
Baseline adaptation converts each benchmark’s camera trajectory to the format required by the target model—e.g., OmniRoam receives only translational components re‑based to the first frame, while PanoWorld receives relative translation and yaw in its panoramic coordinate system.
Similarly, HyDRA, HY‑World 1.5, Dream‑X, FantasyWorld, LingBot‑World, and Matrix‑GAME 3.0 each receive a trajectory transformed to their native pose or control representation, preserving motion direction and scale.
**Figure 10.** **Additional qualitative comparisons with other models.** We provide additional comparisons of World Observer with existing world models on real and synthetic scenes involving out-of-view dynamics. For synthetic scenes, we additionally include results from World Observer (Multi). Red boxes highlight the target objects before exit and after re-entry. Across diverse objects, motions, and scenes, existing models exhibit out-of-view failures such as lost, frozen, and impostor states, whereas World Observer consistently preserves object identity and state evolution throughout the out-of-view interval.
Additional qualitative results (Section D.1) confirm that both synthetic and real scenes benefit from the observer stream, with the multi‑observer variant handling actor motion across different observer regions.