Infinite Worlds with Versatile Interactions

Zelin Gao, Qiuyu Wang, Jiapeng Zhu, Jingye Chen, Zichen Liu, Qingyan Bai, Jiahao Wang, Yufeng Yuan, Hanlin Wang, Yichong Lu, Ka Leong Cheng, Haojie Zhang, Jian Gao, Tianrui Feng, Yuzheng Liu, Yao Yao, Yinghao Xu, Xing Zhu, Yujun Shen, Hao Ouyang

An open-source causal world model that sustains drift-free, 720p interactive simulation at 60 fps.

How can we generate high-fidelity, interactive video worlds that persist indefinitely without visual drift or loss of coherence?

Interactive world models typically degrade within minutes because autoregressive error accumulation causes textures to smear and geometry to warp. LingBot-World-Infinity solves this by training a causal backbone with a hybrid attention mask to resist drift, then distilling it into a real-time student model that uses a dual-agent harness to manage scene logic and user interaction. The resulting system maintains visual coherence for over an hour of continuous, high-fidelity generation at 60 fps.

Paper Primer

The core mechanism hinges on a two-stage training pipeline: a pre-training phase that uses a Mixture of Bidirectional and Autoregressive (MoBA) attention mask to regularize the model against overfitting, followed by a distillation phase that combines consistency distillation with distribution matching distillation (DMD). The MoBA mask acts like a safety rail: it forces the model to balance its reliance on past context with a bidirectional component, preventing the "drift" that occurs when a model relies too heavily on its own potentially flawed previous frames.

To move beyond passive video, the system wraps the generative core in a Director-Pilot harness. The "Director" (a Vision-Language Model) interprets user intent and scene state to propose logical events, while the "Pilot" (the Diffusion Transformer) grounds these instructions into pixel-level dynamics.

The model achieves structural stability over long-horizon rollouts without quality decay.

An hour-long uninterrupted generation session showed no perceptible visual degradation or geometric warping.

Why is an "agentic harness" necessary for a world model?

Standard video generators lack the causal reasoning to maintain a persistent, goal-directed world; the harness provides the "Director" logic needed to seed fresh content and events, turning a frame-predictor into a self-sustaining experience.

How does this approach differ from simply training a larger video model?

Prior models often sacrifice resolution or control to maintain interactivity; this system uses distillation to decouple high-fidelity generation from the computational cost of multi-step denoising, enabling real-time performance on a single GPU.

This paper shifts the bottleneck of world modeling from visual fidelity to long-term structural stability, providing an open-source framework for building persistent, interactive environments that do not drift over time.

Infinite Interactive Worlds

We expose stability and interactivity gaps in current video models and present LingBot‑World‑Infinity as a drift‑free, hour‑long solution.

Current interactive video models falter on two fronts: long‑horizon stability, where frame‑by‑frame conditioning causes error accumulation and visual drift, and high‑fidelity interactivity, where the compute required for real‑time, high‑resolution response forces severe compromises in quality or control.

**Figure 1.** LingBot-World-Infinity generates infinite worlds in real time, featuring versatile interactions.

The key shift is moving from short‑clip generation to a continuous, infinite world simulation that remains drift‑free and responsive.

LingBot-World-Infinity Architecture

Describes the causal video model, its MoBA attention, and distillation into a real‑time generator.

Long‑horizon video generation typically drifts because the model learns to rely on ever‑growing context rather than predicting forward, causing visual and physical inconsistencies.

It treats world simulation as a strictly causal process: each new frame is produced only from past frames and the current user command, guaranteeing that future states never influence the past.

Step 1: Predict $x_1$ using only the initial image (no past frames) and $a_1$.

Step 2: Predict $x_2$ conditioned on $x_1$ and $a_{\le2} = (a_1, a_2)$.

Step 3: Predict $x_3$ conditioned on $(x_1, x_2)$ and $a_{\le3} = (a_1, a_2, a_3)$.

Because each prediction never sees future frames, any error cannot be “corrected” by later information, which prevents the model from learning to hide drift.

How does this causal formulation differ from standard autoregressive video models that also generate frame‑by‑frame?

Standard autoregressive models often still attend to future tokens during training (e.g., via teacher‑forcing masks that leak future information). Here the factorization is enforced at every step, and the attention masks are explicitly designed to block any future context, guaranteeing true causality.

The mask blends a bidirectional component with the classic teacher‑forcing mask, letting the model see the full context early on while still producing strictly autoregressive outputs at inference time.

TF mask: token $t_3$ can attend to $t_1$, $t_2$, and itself, but not $t_4$.

BD mask: token $t_3$ can attend to all tokens $t_1\!-\!t_4$.

MoBA mask = TF $\cup$ BD, so $t_3$ sees $t_1$, $t_2$, $t_3$, $t_4$ (full attention) while the teacher‑forcing component still respects causality.

The bidirectional branch supplies a “global view” that regularizes the model, while the teacher‑forcing branch guarantees that the final autoregressive output never looks ahead.

Why not simply use a pure teacher‑forcing mask if the goal is autoregressive generation?

Pure teacher‑forcing forces the model to rely exclusively on the growing past, which leads to over‑fitting and visual degradation as the context expands. The bidirectional component injects a full‑context signal that stabilizes training and improves fidelity without breaking causality at inference.

We compress the multi‑step diffusion teacher into a student that predicts the same trajectory‑invariant target from any intermediate noisy latent, dramatically reducing the number of denoising steps.

What would happen if the EMA teacher ($\theta^-$) were omitted from consistency distillation?

Without the EMA teacher, the student would chase a moving target, leading to instability and preventing the model from learning a coherent few‑step mapping; the EMA provides a slowly‑changing reference that stabilizes training.

DMD aligns the student’s noised distribution with the data distribution by minimizing a KL divergence, but crucially it evaluates the student on its own rollout trajectories, directly reducing drift.

How does DMD differ from a standard KL‑based distillation that only uses teacher‑forced samples?

Standard KL distillation aligns the student to the teacher’s distribution on teacher‑forced samples, which does not expose the student to its own prediction errors. DMD evaluates the student on its own rollouts, directly penalizing drift that would accumulate during inference.

**Figure 3. Overview of LingBot-World-Infinity Pipeline.** An interactive world simulator is implemented as a causal video model. Our Infinity World is initialized from an initial image and its background description. The future world states are then autoregressively generated, conditioned on the historical context and user inputs (camera poses and prompts).

**Figure 4.** Overview of LingBot-World-Infinity DiT Block and MoBA Attention Mask. The action comprises camera poses and chunk-wise prompts, injected into the DiT block to enable user interaction. For self-attention, a bidirectional block is appended to teacher forcing mask, enabling autoregressive generation while preserving visual fidelity. For cross-attention, the autoregressive component attends to a background prompt and chunk-wise prompts of lower-triangular pattern to prevent access to future information, while the bidirectional component attends to a global prompt.

**Figure 5.** Overview of the Agentic Interaction Harness. Users can either interact with the existing world through semantic or object-centric actions, or intervene by introducing high-level textual events. The VLM (Director) performs causal reasoning and proposes coherent event updates, while the Video Generator (Pilot) grounds these semantic decisions into physically consistent video rollouts, enabling continuous interactive world simulation.

The Data Engine

The Data Engine turns raw videos into chunk‑wise, annotated training data.

Interactive video models suffer from drift because they are trained only on coarse, global captions. The Data Engine solves this by supplying fine‑grained, temporally localized supervision that matches the model’s real‑time instruction interface.

The Data Engine is a three‑stage pipeline that converts heterogeneous raw videos into chunk‑wise captions, giving the model both global context and fine‑grained local supervision.

How does this Data Engine differ from standard video‑dataset pipelines?

Typical pipelines attach a single global caption to each video. Our engine adds two novel ingredients: (1) a VLM‑driven semantic profiling that groups clips for balanced sampling, and (2) a multi‑granularity annotation step that creates per‑chunk captions aligned with control signals, thereby supplying the model with the fine‑grained supervision it needs for drift‑free generation.

Acquisition: the egocentric clip is labeled

Profiling: technical scores flag the egocentric clip as high quality (score 9) and the synthetic clip as medium (score 6); both pass the quality threshold.

VLM profiling: the egocentric clip receives tags

Annotation: each clip is split into 2‑second chunks; the egocentric chunks get local captions “hand reaches for cup” and “cup lifted”, while the synthetic chunks get “character jumps” and “character lands”.

Optimization: redundant verbs are removed, yielding concise chunk‑wise captions ready for training.

This toy walk‑through shows how the three stages cooperate to turn raw video into temporally aligned, locally grounded captions that the model can use to follow instructions step‑by‑step.

Data Acquisition – ingest videos from egocentric, synthetic, and web sources; normalize metadata.

Data Profiling – run technical quality filters, split long videos with TransNet V2, then apply VLM profiling to assign semantic tags.

Multi‑granularity Annotation – generate a global caption, then produce event‑level annotations for each temporal chunk, finally refine chunk‑wise captions for consistency.

**Figure 2.** Overview of the proposed data engine. Heterogeneous raw videos are temporally segmented, filtered, and routed to category-specific annotation pipelines, producing optimized chunk-wise captions.

By delivering chunk‑wise, visually grounded captions, the Data Engine removes the train‑inference mismatch that caused drift, enabling hour‑long, instruction‑aware video generation.

Real-Time Inference Stack

Deploying a low‑latency, interactive video system by stacking four tightly coordinated components.

Interactive video generation demands millisecond‑scale latency and high visual fidelity, yet naïve pipelines stall at the inference bottleneck.

The stack stitches four specialized modules—systematic optimization, agentic interaction harness, visual quality enhancer, and UI layer—into a single pipeline that delivers real‑time, coherent video.

How does the Inference Stack differ from a simple sequential pipeline?

Beyond chaining modules, the stack dynamically schedules the KV cache, overlaps latent generation with VAE decoding, and couples a high‑level Director (VLM) with a low‑level Pilot (DiT) so that causal consistency is enforced at every step.

Systematic optimizer compiles attention kernels, reducing per‑frame compute from 120 ms to 80 ms.

KV‑cache scheduler keeps only the most recent 4 tokens for each frame, cutting attention cost by 30 %.

VLM (Director) proposes “open door” after Frame 1; DiT (Pilot) renders the physical opening in Frame 2.

Spatio‑temporal refiner upsamples Frame 2 to 720p and interpolates an intermediate frame, delivering a smooth 30 fps playback.

UI streams Frame 1 (80 ms) → Frame 2 (80 ms) → interpolated frame (30 ms) with total latency ≈ 190 ms.

The stack’s overlapping stages keep the end‑to‑end latency well below a second, which is perceptually instantaneous for interactive use.

Apply compiler‑level optimizations and custom attention kernels to the distilled DiT backbone.

Distribute inference across multiple GPUs using a hybrid parallel strategy while preserving efficient inter‑GPU communication.

Pipeline latent generation and VAE decoding asynchronously: latent generation proceeds without waiting for image decoding.

Assign dedicated workers to VAE decoding, enabling parallel frame reconstruction.

Stream decoded frames incrementally as they become available, overlapping transmission with generation.

Deploy an efficient video streaming pipeline that delivers frames to the client with minimal buffering.

Beyond raw speed, the system must understand user actions and maintain causal consistency across long video horizons.

The Vision‑Language Model (VLM) acts as a high‑level Director that reasons about semantics, while the Diffusion Transformer (DiT) serves as a low‑level Pilot that renders physically plausible frames.

Why not use the VLM alone to generate video frames?

The VLM excels at high‑level reasoning but lacks the fine‑grained pixel synthesis capabilities of a diffusion model; the Pilot supplies the necessary low‑level physics and texture detail.

**Figure 6.** In the tracking-mode interface, the Vision-Language Model (VLM) comprehends interactive objects within the scene, while the tracking model continuously tracks these targets to display dynamic interactive floating windows (event cards) in real-time. Powered by the "Director-Pilot" co-simulation framework, the model demonstrates robust interactive capabilities by performing causal reasoning based on user actions (e.g., pushing a door open or rotating a soccer ball) and rendering physically logical, highly coherent spatio-temporal dynamics.

Interaction granularity is handled by two complementary modes within the Agentic Interaction Harness.

Mode A offers quick, whole‑scene reasoning without masks, while Mode B leverages SAM‑based tracking for precise object‑level manipulation.

Why introduce a tracking‑assisted mode when the direct semantic mode already works?

Direct semantic interaction cannot guarantee pixel‑accurate alignment for object‑specific actions; tracking supplies the missing spatial continuity, enabling reliable manipulation of individual items.

**Figure 9.** Interface of our interactive system. The center shows the live generation viewport; the bottom panel exposes the low-level control scheme (WASD for movement, IJKL for view control); and the right-hand panel is the agentic control surface, which lists the VLM-proposed “Event Proposals,” each bound to a hotkey. Fixed keys (Space, P) are always available, U/O and F/G carry context-aware character actions and environmental events proposed by the VLM, and the numeric keys are user-registered event slots.

After the Pilot decodes frames, a lightweight refiner first sharpens spatial detail then synthesizes intermediate frames to smooth motion.

Instead of a fixed attention window, the cache adapts each inference chunk to retain only the most informative history.

The user interface stitches together the generation viewport, low‑level controls, and an event‑proposal panel to give users immediate, hotkey‑driven interaction.

The UI divides responsibilities: a central viewport shows the world, a bottom panel maps familiar WASD/IJKL controls, and a right‑hand panel lists VLM‑generated event proposals bound to hotkeys.

Evaluation and Performance

We show that our distilled model keeps visual consistency over hour‑long, real‑time rollouts.

Our causal distilled model sustains high‑resolution generation in real time without visual degradation over hour‑long rollouts.

Fig. 10 shows a single uninterrupted 60‑minute session with no perceptible decay.

We compare our distilled model against the closed‑source HappyOyster and Genie 3 systems, as well as leading open‑source interactive world models, on visual fidelity, temporal stability, real‑time throughput, and interaction richness.

A compact, real‑time version of the causal pretrained backbone that preserves its visual quality and long‑term stability while running at 720p/60 fps.

How does the distilled model differ from the pretrained backbone?

The distilled model is smaller and includes a real‑time inference head, but it inherits the teacher’s anti‑drift training, so it keeps the same visual fidelity and long‑term stability while running orders of magnitude faster.

We also evaluate the causal pretrained backbone, which serves as the teacher for distillation, and find it outperforms prior causal world models on visual fidelity and long‑horizon stability (Fig. 12).

**Figure 11.** Qualitative comparisons. Our model maintains stable visual and physical consistency over long-horizon generation.

**Figure 10.** Hour-long world rollout. We sample frames from a single 60-minute generated session, covering 20 distinct scenarios. The timeline shows that the model can maintain coherent scene structure, visual quality over an extended uninterrupted rollout, providing a qualitative stress test for long-horizon stability.

**Figure 12.** Qualitative comparisons on causal pretraining. Our model shows stable performance compared with baselines.

The model’s ability to maintain visual consistency over hour‑long rollouts enables reliable interactive world generation.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers