WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang, Qingyan Bai, Ka Leong Cheng, Yue Yu, Yixuan Li, Yihao Meng, Zichen Liu, Yanhong Zeng, Yujun Shen, Qifeng Chen

WorldDirector decouples motion planning from video synthesis to maintain object permanence and identity in long-horizon world simulations.

How can we generate long-horizon, controllable video by decoupling 3D semantic planning from pixel-level synthesis?

Existing video world models struggle to maintain "object permanence"—when a dynamic entity exits the camera's view, the model often forgets its identity or physical state, leading to frozen objects or identity shifts upon re-entry. WorldDirector solves this by decoupling semantic motion planning from pixel generation: an LLM orchestrates 3D trajectories and camera paths, which are then injected as spatial and appearance conditions into a causal, chunk-based diffusion model. This approach achieves state-of-the-art performance on reconstruction and consistency metrics, enabling the synthesis of extended, controllable sequences where objects persist and evolve even when off-screen.

Paper Primer

The core mechanism hinges on explicit conditioning: the model receives a "Location Condition" (color-coded 2D bounding boxes) to guide movement and an "Appearance Condition" (RGB visual anchors) to preserve identity. To prevent the model from simply warping reference images, a "Temporal Drop Mechanism" sparsifies appearance references, forcing the model to synthesize natural, articulated motion driven by the LLM's trajectory plan.

WorldDirector achieves superior reconstruction fidelity and dynamic consistency compared to state-of-the-art causal world models.

The model outperforms baselines on PSNR (18.127), SSIM (0.502), and `DSC_CLIP` (0.917) on a test set of 100 novel, unseen scenes. The model improves PSNR by approximately 22% over the next best baseline (HY-World) while maintaining higher dynamic subject consistency.

The system operates in two stages: first, an LLM (Gemini) plans 3D trajectories and camera paths based on user instructions; second, the diffusion model generates the video in autoregressive chunks, retrieving historical context to anchor the current segment to the global scene.

Why is an LLM orchestrator necessary instead of letting the generative model handle motion implicitly?

Generative models lack the capacity to maintain long-term physical logic for off-screen entities; relying on internal priors leads to trajectory collapse or "frozen" states during prolonged occlusions. The LLM provides an explicit, semantic-level plan that ensures continuous, physically plausible movement.

What is the primary limitation of this approach?

The model relies on synthetic game data for training, which creates a domain gap that can result in reduced visual fidelity, such as blurry faces or unnatural locomotion, compared to real-world video.

WorldDirector demonstrates that persistent world simulation is achievable by treating video generation as a controllable planning problem rather than a passive pixel-prediction task.

Introduction and Motivation

We expose missing persistent dynamic memory in video models and motivate decoupling motion planning from synthesis.

Current video world models bind physical dynamics to pixel synthesis, which prevents objects from moving independently of the camera. This entanglement makes it impossible to guarantee that hidden entities preserve their motion and appearance when they re‑enter the frame.

To build a truly persistent video world, the system must separate semantic motion planning from visual generation, ensuring objects continue to move and keep their visual identity even when unseen.

**Figure 1.** Controllable world simulation with persistent dynamic memory via WorldDirector. By decoupling 3D semantic orchestration from latent video synthesis, our framework autoregressively generates long-horizon videos via causal chunks, ensuring rigorous dynamic memory and object permanence. Please refer to the video results on our project page for intuitive demonstrations.

The shift from pixel‑entangled models to decoupled semantic orchestration enables controllable, persistent world simulation.

Related Work and Context

Survey of video generation foundations, control methods, and memory mechanisms, introducing HyDRA.

Recent video synthesis work has moved from pure pixel fidelity toward treating generative models as rudimentary physics engines. Approaches such as Sora, Genie, Oasis, and DIAMOND model motion but struggle when objects leave the camera view, causing frozen or vanished entities.

Controllable generation builds on image‑level conditioning and extends it to video via spatial cues like bounding boxes (Boximator, GLIGEN). These methods excel on short clips but lack the autoregressive memory needed for long‑horizon simulations.

Memory mechanisms aim to preserve temporal coherence beyond the immediate context window. Prior solutions rely on sliding windows, static‑scene retrieval, or external monitors, yet they either assume a static world or become computationally prohibitive for dynamic object permanence.

HyDRA provides a lightweight, hierarchical token‑based memory that stores object appearances and trajectories, allowing the system to recall and update entities even when they are out of view.

The WorldDirector Framework

WorldDirector combines location conditioning and causal chunk generation to simulate controllable video worlds.

Long video synthesis suffers from drifting object identities because the model lacks a persistent memory of past frames. Without an explicit anchor, appearance cues become ambiguous, leading to sliding artifacts and inconsistent motion.

Location Conditioning supplies explicit 2D trajectories for every entity, anchoring where each object should be at each frame.

Frame 1: car mask occupies pixels (10‑30, 20‑40), person mask occupies (50‑60, 70‑80).

Frame 2: car shifts right by 5 px, person shifts down by 3 px – masks are updated accordingly.

Frame 3: car turns left, person stays still – masks reflect the new orientation.

Frame 4: both masks disappear (entity exits view) – $B$ entries become all zeros.

Each frame’s mask is concatenated with the latent $z_t$, so the transformer sees “where‑to‑draw” signals at every timestep.

Because $B$ is supplied explicitly, the model never has to infer motion from pixel statistics; it simply follows the prescribed trajectory, eliminating drift.

How does Location Conditioning differ from simply overlaying bounding‑box masks on the input frames?

Overlaying masks only provides a visual cue; Location Conditioning injects the masks as separate feature channels that the diffusion model can attend to directly, allowing it to condition generation on precise spatial coordinates rather than on pixel‑level appearance.

Causal Chunk-Based Generation splits the video timeline into fixed‑size chunks and generates each chunk sequentially, feeding back appearance and memory from earlier chunks to preserve identity and consistency.

Chunk 1 (frames 1‑2): $A$ is built from the initial frame $I_0$, $M=\emptyset$. The model produces frames 1‑2; frame 2 is kept, frame 1 is discarded.

Chunk 2 (frames 3‑4): $A$ is recomputed from frame 2, $M$ retrieves the cleaned context of frame 2. The model generates frames 3‑4; frame 3 is discarded.

Chunk 3 (frames 5‑6): $A$ uses frame 4, $M$ includes frames 2 and 4. The model generates frames 5‑6; frame 5 is discarded.

The final video $V$ consists of frames 2, 4, 6 (plus the initial $I_0$), preserving the ball’s trajectory without duplicate frames.

Discarding the first frame of each generated chunk prevents overlapping content, ensuring a smooth temporal flow across chunk boundaries.

Why does the algorithm drop the first frame of each generated chunk instead of keeping all frames?

Because the first frame of a new chunk would be identical to the last frame of the previous chunk (they share the same start), dropping it avoids redundant duplication and preserves a single continuous video stream.

**Algorithm 2** Causal Chunk-Based Generation **Input:** Location condition sequence $\mathcal{B}_{1:T}$, Captions $\mathcal{P}$, Initial reference frame $I_0$, Total frames $T$, Chunk size $K$, Camera Poses $\mathcal{C}_{1:T}$ **Output:** Generated continuous video stream $V$ 1: **procedure** `CAUSAL_GENERATION`($\mathcal{B}_{1:T}$, $\mathcal{P}$, $I_0$, $T$, $K$, $\mathcal{C}_{1:T}$) 2: Partition $\mathcal{B}_{1:T}$ into $N = T/K$ chunks: $\{\mathcal{B}^{(1)}, \dots, \mathcal{B}^{(N)}\}$ 3: Partition $\mathcal{C}_{1:T}$ into $N = T/K$ chunks: $\{\mathcal{C}^{(1)}, \dots, \mathcal{C}^{(N)}\}$ 4: Initialize video buffer $V \leftarrow I_0$ 5: **for** $n = 1, 2, \dots, N$ **do** 6: **if** $n = 1$ **then** 7: $\mathcal{A} \leftarrow$ `APPEARANCE_CONDITION_GENERATION`($\mathcal{B}$, $I_0$) 8: $\mathcal{M} \leftarrow \emptyset$ $\triangleright$ no historical context for the first chunk 9: **else** 10: $\mathcal{A} \leftarrow$ `APPEARANCE_CONDITION_GENERATION`($\mathcal{B}$, $V$) 11: $\mathcal{M} \leftarrow$ `CONTEXT_RETRIEVAL`($\mathcal{B}$, $\mathcal{C}$, $V$) 12: **end if** 13: $I_{start} \leftarrow V_{last}$ $\triangleright$ use the last frame of buffer $V$ as initial frame for current chunk generation 14: $V^{(n)} \leftarrow$ WORLDDIRECTOR($\mathcal{B}^{(n)}$, $\mathcal{P}$, $\mathcal{A}$, $\mathcal{M}$, $\mathcal{C}^{(n)}$, $I_{start}$) 15: $V \leftarrow V \cup V^{(n)}[1 :]$ $\triangleright$ append generated chunk without first frame to the buffer 16: **end for** 17: **return** $V$ 18: **end procedure**

**Figure 2.** **Overview of WorldDirector.** An LLM orchestrates 3D trajectories that are projected into 2D Location Conditions for causal chunk generation. Location ($\mathcal{B}$) and Appearance ($\mathcal{A}$) conditions are channel-concatenated with the noisy latent, while historical Context ($\mathcal{M}$) is sequence-concatenated. During generation, *temporal drop* is applied and an asymmetric attention routing prevents noise from polluting the context memory.

Quantitative Performance

Ours achieves dramatically higher fidelity while matching consistency across metrics.

Our method achieves substantially higher PSNR, SSIM, and lower LPIPS than all baselines.

Table 1 shows Ours reaches 18.127 PSNR, 0.502 SSIM, and 0.359 LPIPS, surpassing the next‑best PSNR of 14.782.

Other methods score in the mid‑14 dB PSNR range (14.1–14.8), SSIM between 0.35 and 0.455, and LPIPS around 0.40, while Ours improves these by several points. Our model runs at 16 fps with a context window of $N=10$ frames, trained for 3,000 steps at batch size 64 and learning rate 1×10⁻⁵. Evaluation uses VBench’s Subject and Background Consistency and DSC metrics, confirming comparable coherence across dynamic objects.

It is a pretrained diffusion model that generates short video clips from 3‑D scene tokens, providing a generic visual foundation for downstream planning.

Qualitative Analysis

Our method preserves object identity and motion while offering fine‑grained viewpoint control.

Location Conditioning encodes continuous object positions and camera poses, letting the generator align video content with the intended scene layout.

WorldDirector raises PSNR by +1.36 dB over the no‑appearance‑condition baseline.

Table 2 shows PSNR 16.764 for “No A” versus 18.127 for our method.

**Figure 3.** Qualitative comparison with baselines. Note that HyDRA uses the initial 10s of our results as a reference video for its generation. Please refer to the video results on our project page for intuitive demonstrations.

**Figure S2.** **Flexible Viewpoint Control.** WorldDirector supports diverse exploration paradigms. Top: A pure third-person view tracking a running dog with a 360° panoramic sweep. Bottom: A dynamic viewpoint switch from a third-person tracking shot to an independent first-person backward movement.

**Figure S3.** Qualitative comparison with baselines. Note that HyDRA uses the initial 10s of our results as a reference video for its generation. We highly recommend viewing the video results on our project page for a more intuitive demonstration.

**Figure S4.** Qualitative comparison with baselines. Note that HyDRA uses the initial 10s of our results as a reference video for its generation. We highly recommend viewing the video results on our project page for a more intuitive demonstration.

**Figure 5.** A generation example of Promptable World Events.

Baselines such as Yume, HY‑World, and Infinite‑World generate static subjects despite motion prompts, while LingBot‑World and HyDRA either degrade identity or replace the subject entirely. Without explicit location conditioning, they fail to synchronize camera and object dynamics, leading to incoherent scenes.

Ablation Studies

Ablations confirm the Appearance Condition is crucial for dynamic consistency.

WorldDirector decouples 3D semantic orchestration from latent video synthesis, letting an LLM plan trajectories while a causal chunk‑based generator preserves persistent object memory.

Removing the Appearance Condition degrades all quality metrics.

Subject Consistency drops from 0.917 to 0.878 (−4.3 %).

**Figure 4.** Ablation on Appearance Condition. We conduct experiments on a case involving complex character movements and multiple pose changes. The findings highlight the significance of the Appearance Condition for preserving dynamic consistency.

Promptable World Events let the LLM introduce novel objects at any time; their initial appearance is synthesized from text prompts and then added to the Appearance Condition pool to guarantee temporal consistency.

In summary, WorldDirector offers free exploration and flexible event design for video world models, preserving rigorous dynamic memory through spatial and appearance conditioning.

Extended Ablations

Ablation insights, viewpoint flexibility, qualitative comparisons, impact, and release safeguards.

WorldDirector separates 3‑D semantic orchestration from latent video synthesis, using an $LLM$ for trajectory planning and a causal chunk‑based generator for persistent object memory.

The mechanism drops the appearance reference for a short temporal window, forcing the model to synthesize motion from text rather than copying static frames.

How does the Temporal Drop differ from simply skipping frames?

Skipping frames removes visual information entirely, leaving a gap that the model cannot fill. The Temporal Drop, by contrast, replaces the appearance condition with a learned placeholder for a limited window, so the model still receives a continuous stream but must generate motion from the textual cue.

Ablating the dynamic‑context stream (leaving only the Appearance Condition) causes the model to lose object identity across causal chunks, producing semantically similar but visually distinct entities.

Removing the Temporal Drop Mechanism exposes the network to dense, frame‑by‑frame appearance references, resulting in severe motion rigidity—characters appear to “slide” instead of walking naturally.

By explicitly conditioning on spatial location, the system supports flexible viewpoint control: anchoring a target’s 2‑D bounding box near the camera center yields a third‑person view, while decoupling camera motion enables first‑person navigation.

**Figure S1.** Ablation on Dynamic Context and Appearance Condition Drop Mechanism. We highly recommend viewing the video results on our project page for a more intuitive demonstration.

Qualitative comparisons reaffirm earlier findings: prior baselines (Yume, HY‑World, Infinite‑World) generate little subject motion; LingBot‑World produces dynamic scenes but lacks fine‑grained control; HyDRA over‑emphasizes a front‑facing subject. Our method respects the textual layout, maintains precise interactive control, and preserves dynamic memory.

The work advances controllable video world simulation, which can benefit VR, gaming, and film production. The paper does not address potential negative uses such as deceptive video generation, fairness, privacy, or security concerns.

We plan a staged release of the inference code, $LLM$ prompt templates, and pretrained checkpoints for academic research. Users are urged to combine WorldDirector with external safety measures—prompt filters, video watermarking, provenance metadata, and runtime monitoring—to mitigate misuse.

Implementation Details

Training setup and context‑retrieval algorithm details for WorldDirector.

WorldDirector is trained on a high‑performance cluster of eight nodes, each equipped with eight NVIDIA A100 (80 GB) GPUs, giving a total of 64 GPUs. To keep the memory footprint manageable we use Fully Sharded Data Parallel (FSDP) together with activation checkpointing, which distributes the 3D VAE encodings, diffusion transformer blocks, and multi‑modal conditioning channels across the devices.

Training videos are rendered at 832 × 480 pixels and 16 fps, with a context length of $N = 10$ frames. We optimize the flow‑matching objective with AdamW at a constant learning rate of $1 \times 10^{-5}$, using BF16 mixed precision, a global batch size of 64, and 3 000 steps; the full training run finishes in about 72 hours.

Initialize a coverage counter $\text{cnt}[i] \leftarrow 0$ for every dynamic entity $i$ present in the training videos.

Repeatedly select the least‑covered entity $i^{*}$ (the one with the smallest $\text{cnt}[i]$) until $N$ frames have been chosen.

For static context, compute for each candidate frame $c$ the maximum field‑of‑view overlap between its camera pose $C_c$ and all training frame poses $C_v$; use this overlap as the frame’s static score and sort candidates descending to obtain $P_{\text{cam}}$.

For dynamic context, pick the candidate frame $f^{*}$ that contains the largest 2D bounding‑box area for the selected entity $i^{*}$, add $f^{*}$ to the selected set, and remove it from the candidate pool.

Compute $A_{\max}$, the maximum bounding‑box area among all entities in $f^{*}$, then update each entity $j$ present in $f^{*}$ by $\text{cnt}[j] \;+=\; \frac{\text{area}(B(j))}{A_{\max}}$.

After the loop, interleave frames from $P_{\text{cam}}$ and the dynamic list $P_{\text{box}}$ alternately, discard duplicates, enforce a minimum temporal stride of four frames, and stop when $|M| = N$.

Return the final context set $M$ sorted in chronological order.

Inference and Planning Details

Details of the inference pipeline: LLM‑driven planning and causal chunk generation.

Appendix C expands the inference pipeline into two concrete stages: a planning phase driven by a large language model (LLM) and a generation phase that consumes the planned trajectories in a causal, chunk‑wise fashion.

The LLM receives a structured description of the scene (objects, camera parameters, and desired narrative) and outputs coherent 3D motion paths for every dynamic entity.

The planning stage therefore turns high‑level narrative intent into a fully specified spatio‑temporal script that can be consumed by the video synthesis model.

The generator consumes the location script chunk by chunk, feeding each segment to the diffusion model while preserving continuity through a shared video buffer.

By iterating this procedure the system synthesizes arbitrarily long videos while keeping each segment anchored to the deterministic spatial constraints produced by the planning LLM.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers