AlayaWorld: Interactive Long-Horizon World Modeling - Full Technical Report
AlayaWorld Team, Kaipeng Zhang, Chuanhao Li, Yifan Zhan, Yongtao Ge, Yuanyang Yin, Jiaming Tan, Kang He, Liaoyuan Fan, Mingliang Zhai, Ruicong Liu, Xiaojie Xu, Xuangeng Chu, Zhen Li, Zhengyuan Lin, Zhixiang Wang, Zian Meng, Zihui Gao
AlayaWorld is an interactive video world model that generates persistent, long-horizon environments via autoregressive latent chunking.
How can we transform a large-scale video diffusion model into an interactive world simulator that maintains long-horizon consistency and follows user-defined camera and action controls?
Modern game development relies on labor-intensive pipelines for assets and physics, making it difficult to create rapidly evolving, personalized virtual worlds. AlayaWorld addresses this by generating interactive environments directly from user inputs, using a 15B video diffusion transformer that synthesizes video in autoregressive latent chunks. On the iWorld-Bench, the model achieves state-of-the-art performance in trajectory following and memory ability while reducing inference to four sampling steps per chunk.
Paper Primer
AlayaWorld functions as a causal generator: it processes a bounded visual context—comprising a persistent identity anchor, compressed temporal history, and geometry-aligned spatial memory—to predict the next video chunk. This architecture is like a navigator with a map: it keeps a fixed global reference (the sink frame) while constantly updating its local view by warping past observations into the current camera trajectory.
AlayaWorld achieves superior long-horizon stability and memory retention compared to existing video world models.
Performance on iWorld-Bench across generation quality, trajectory following, and memory ability metrics. Best overall performance among evaluated open-source models, including Cosmos and Matrix-Game 2.0.
The discrete autoregressive distillation method enables real-time interactive performance.
Reduction of sampling requirements from approximately 30 steps to 4 steps per chunk. ~7.5x reduction in sampling steps while maintaining control and memory stack integrity.
Why does the model use a "bounded" context rather than simply attending to the entire history?
A bounded context ensures that the computational cost per chunk remains constant as the roll-out grows, preventing the memory and compute requirements from scaling linearly with the length of the generated world.
How does the model handle the "drift" that typically plagues long-term autoregressive generation?
The model is trained with "Helios" drift simulation—which injects noise, blur, and saturation shifts into the context—and an error bank that replays the model's own reconstruction residuals, teaching it to recover from the specific failure modes it generates during inference.
Interactive World Modeling
We expose the gap between costly game pipelines and generative video world models, motivating interactive world modeling.
Traditional 3D game development strings together lengthy pipelines—concept art, asset creation, animation, physics, and gameplay programming—each demanding weeks or months of manual labor. By contrast, video world models promise to synthesize entire interactive environments on the fly from high‑level user inputs such as text, images, or video.
It is a generative system that produces a coherent, controllable video world while continuously responding to user‑directed camera motions and action prompts.
The image displays the logo for "ALAYA LAB", consisting of a stylized triangular icon above the text.
**Figure 1.** Interactive world simulation across diverse scenes.
The core shift is moving from labor‑intensive, handcrafted game pipelines to generative world models that create interactive environments on demand.
Data Composition and Annotation
We assemble a heterogeneous corpus and a unified pipeline to enable controllable video world modeling.
The model must learn both photorealistic evolution and precise camera‑conditioned navigation, but publicly available video datasets lack scalable, aligned camera signals.
To turn a wildly heterogeneous video dump into a clean, controllable training set, we first extract cheap universal features once, then run a fixed series of quality gates, and finally keep only clips that survive every gate.
Ingest writes a manifest entry for each clip and decodes frames once.
Run extracts optical flow with RAFT and computes frame‑level luminance variance, storing these in the shared cache.
Technical gate rejects clip B (resolution 480p < 720p); A and C pass.
Motion gate flags clip C as static (flow variance < 0.02) and discards it.
Select keeps only clip A, which passed all active gates.
The cache lets us evaluate six independent quality criteria without re‑decoding, turning a potentially hour‑long per‑clip workflow into a few milliseconds.
Beyond raw video quality, the model needs temporally aligned language that distinguishes subject motion from camera motion.
We split each clip into a global description (weather, location, style) and per‑segment tracks that separately record subject, environment, static scene, and camera cues, then fuse the tracks into a `full_prompt` and a `short_prompt`.
Segment 1: `subject_motion` = “person walks forward”, `camera_description` = “steady forward dolly”.
Segment 2: `subject_motion` = “person stops”, `camera_description` = “slow pan left”.
Segment 3: `subject_motion` = “person waves”, `camera_description` = “orbit around subject”.
Each segment receives a `camera_path` enum (e.g., FORWARD, `PAN_LEFT`, ORBIT).
All tracks are merged into a `full_prompt` of five sentences and a `short_prompt` of 22 words.
The explicit `camera_path` lets the model learn a discrete mapping from a high‑level control token to the underlying camera trajectory.
**Figure 2.** Representative training samples from the seven data sources, grouped by source type. Each row contains four temporally ordered frames from a single clip, making both the appearance diversity of the corpus and the camera motion available for supervision visible within one view.
Synthetic clips supply dense, accurate camera trajectories at scale, which is the only way to train the model to follow explicit camera‑control signals.
How does this pipeline differ from a standard video‑dataset cleaning script?
Typical scripts run independent checks per clip, often re‑decoding for each gate. Our design computes a shared feature cache once, then lets all six gates read the same cheap summaries, turning an O(#gates × decode) workflow into O(decode + #gates × lookup).
A single shared‑feature cache plus an all‑stages‑must‑pass filter yields a high‑quality, camera‑controllable corpus without redundant computation.
Architecture and Training Pipeline
We detail the LTX‑2.3 backbone, three‑stage training, and camera‑trajectory control that enable long‑horizon consistency.
Standard video diffusion models drift after a few seconds, breaking the continuity needed for an interactive world simulator. Our method tackles this by redesigning the backbone, conditioning, and training pipeline.
LTX‑2.3 is a 22 B multimodal transformer; we strip its audio head, leaving a ~13 B video‑only Diffusion Transformer (DiT) that serves as the core generator.
Does dropping the audio module degrade visual quality?
No – the audio branch only processes sound features. Since our downstream task never queries audio, the visual pathway remains unchanged, and the smaller model even trains faster.
Training is split into three stages so we first learn a generic video prior, then graft controllable memory and camera signals, and finally compress the sampler for interactive speed.
Why not train all three objectives jointly?
Joint training would entangle the generic video prior with control‑specific gradients, making it hard for the model to retain its broad generative knowledge while learning precise camera‑driven dynamics.
The per‑frame camera pose increments are injected directly into the transformer via an adaptive‑layer‑norm (AdaLN) module, letting the model steer the view without a separate cross‑attention bank.
How does AdaLN differ from cross‑attention conditioning?
Cross‑attention would require a separate key/value stream for the camera, adding O(N · C) cost. AdaLN instead modulates the existing token embeddings in‑place, keeping the self‑attention cost O(N²) unchanged.
Bidirectional pre‑training: fine‑tune the full DiT on domain video clips with a mixture of image, video, and text objectives.
Autoregressive training: freeze the backbone, train a LoRA‑adapted history‑compression module, then unfreeze everything and jointly train camera control, spatial memory, and a next‑forcing head.
Post‑training acceleration: distil the 30‑step teacher into a 4‑step student using Distribution‑Matching Distillation, self‑forcing++, and consistency distillation.
Roll‑out of a single chunk with camera‑trajectory control.
Step 1: Compute coverage for each cached frame. Frame 1 covers 40 % of the target pixels, frame 2 covers 15 %, frame 3 covers 45 %.
Step 2: Greedy selection picks frame 1 (largest uncovered area), then frame 3 because it adds the remaining 45 % without overlapping frame 1.
Step 3: Forward‑splat frame 1 into the target view, producing warped image Ĩ₁ and mask M₁.
Step 4: Forward‑splat frame 3, producing Ĩ₃ and mask M₃; combine masks (M₁ ∨ M₃) to obtain final coverage mask $M_i$.
Step 5: Encode the fused warped image (Ĩ₁ + Ĩ₃) with the VAE to obtain the spatial‑memory token $g_i$; feed $M_i$ as a self‑attention key bias so uncovered pixels are ignored.
The greedy max‑coverage selection guarantees that each added frame contributes the most new visual evidence, preventing redundant warps and keeping the memory budget low.
**Figure 1.** Overview of the proposed training pipeline, consisting of three stages: Bidirectional Pre-training, Autoregressive Training, and Distillation.
**Figure 4.** Formulation.
Evaluation and Results
AlayaWorld delivers top scores on most iWorld‑Bench metrics, confirming long‑horizon consistency.
AlayaWorld extends a diffusion video model into an interactive world simulator, letting users steer camera and actions while preserving long‑horizon consistency.
AlayaWorld leads on 87.5 % of iWorld‑Bench metrics, topping seven of eight evaluated scores.
Table 3 shows AlayaWorld achieving the highest value on seven metrics across Generation Quality, Trajectory Following, and Memory Ability.
iWorld‑Bench is a suite that probes a video model’s ability to follow action commands and retain visual consistency over long loops.
**Table 1.** Performance comparison of AlayaWorld against various video generation models across metrics for Generation Quality, Trajectory Following, and Memory Ability.
**Figure 5.** Qualitative results of camera-controlled generation. AlayaWorld follows diverse camera trajectories while maintaining coherent scene geometry and visual appearance.
**Figure 6.** Qualitative results of prompt-driven actions. AlayaWorld introduces newly requested actions and events during autoregressive generation while preserving the existing scene context.
**Figure 7** Qualitative results of consistent world generation under leave-and-return trajectories. AlayaWorld preserves scene structure and visual identity when revisiting previously observed regions.
**Figure 8** Qualitative results of long-horizon autoregressive generation. AlayaWorld maintains stable visual quality and coherent scene evolution over extended rollouts.
AlayaWorld’s dominant performance on long‑horizon memory tasks validates its spatial‑temporal memory mechanisms.
Questions & answers
What is AlayaWorld and what does it contribute?
AlayaWorld is a 15B video diffusion transformer that generates interactive virtual environments directly from user inputs (text, images, or video) in an autoregressive latent-chunk fashion, eliminating the need for traditional labor-intensive game development pipelines. Its main contributions include a bounded visual context architecture, spatial-temporal memory mechanisms, and drift-robust training, evaluated on the iWorld-Bench benchmark.
What problem does AlayaWorld address?
AlayaWorld addresses the difficulty of creating rapidly evolving, personalized virtual worlds, which traditionally require lengthy, labor-intensive pipelines covering concept art, asset creation, animation, physics, and gameplay programming. The goal is to replace these handcrafted pipelines with a generative world model that synthesizes interactive environments on demand.
Why do standard video diffusion models fail at long-horizon interactive generation?
Standard video diffusion models drift after a few seconds, breaking the continuity needed for an interactive world simulator. AlayaWorld addresses this by redesigning the backbone, conditioning, and training pipeline, including drift simulation and an error bank.
How does AlayaWorld's bounded context architecture work?
AlayaWorld processes a bounded visual context comprising a persistent identity anchor (a sink frame), compressed temporal history, and geometry-aligned spatial memory to predict the next video chunk. This design keeps computational cost per chunk constant regardless of roll-out length, preventing memory and compute requirements from scaling linearly with world length.
How does AlayaWorld handle drift in long-term autoregressive generation?
AlayaWorld is trained with 'Helios' drift simulation, which injects noise, blur, and saturation shifts into the context, and an error bank that replays the model's own reconstruction residuals. Together, these teach the model to recover from the specific failure modes it generates during inference.
How does AlayaWorld condition on camera control signals?
AlayaWorld uses Adaptive Layer Normalization (AdaLN) to inject camera control signals, which modulates existing token embeddings in-place rather than adding a separate key/value stream as cross-attention would require. This keeps the self-attention cost at O(N²) without the additional O(N·C) overhead of cross-attention conditioning.
Why is AlayaWorld trained in multiple stages rather than jointly?
Joint training would entangle the generic video prior with control-specific gradients, making it difficult for the model to retain broad generative knowledge while learning precise camera-driven dynamics. The staged approach keeps these objectives separated.
What data is used to train AlayaWorld, and how is it collected?
Training data includes both real-world video and synthetic clips; synthetic clips supply dense, accurate camera trajectories at scale, which is necessary to train camera-control following. The paper notes that publicly available video datasets lack scalable, aligned camera signals, motivating the use of synthetic data.
How does AlayaWorld's data cleaning pipeline work?
The pipeline computes a shared feature cache once per clip, then allows all six quality-gating checks to read from the same cached summaries, reducing the workflow from O(#gates × decode) to O(decode + #gates × lookup). All gates must pass for a clip to be included, yielding a high-quality, camera-controllable corpus without redundant computation.
Does AlayaWorld use an audio module?
AlayaWorld drops the audio branch from its backbone because the downstream task never queries audio, leaving the visual pathway unchanged. The paper notes this also results in a smaller model that trains faster without degrading visual quality.
What benchmark is used to evaluate AlayaWorld, and what are the key results?
AlayaWorld is evaluated on iWorld-Bench, where it achieves state-of-the-art performance in trajectory following and memory ability. The paper also reports that inference is reduced to four sampling steps per chunk, and that AlayaWorld's dominant performance on long-horizon memory tasks validates its spatial-temporal memory mechanisms.
What are the limitations or open problems acknowledged by the paper?
The paper does not explicitly enumerate limitations or open problems in the provided text. It focuses on contributions and results without a dedicated limitations section in the excerpted content.
How does AlayaWorld differ from prior video world model approaches?
AlayaWorld differs from prior approaches by combining a bounded visual context (preventing linear compute scaling), Helios drift simulation, an error bank for self-correction, and AdaLN-based camera conditioning within a single 15B video diffusion transformer. The paper does not provide a detailed quantitative comparison to specific named prior systems beyond iWorld-Bench rankings.
How many parameters does AlayaWorld have and what is its core architecture?
AlayaWorld is a 15B parameter video diffusion transformer that generates video in autoregressive latent chunks. It uses a causal generation paradigm with a sink frame as a persistent identity anchor, compressed temporal history, and geometry-aligned spatial memory.
Who created AlayaWorld and where was it published?
The paper is titled 'AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report' and is available on arXiv (arxiv.org/abs/2607.18367). The paper does not specify individual author names or a conference venue in the provided text.
Key terms
- AlayaWorld
- A 15B video diffusion transformer system that generates interactive virtual environments from user inputs using autoregressive latent-chunk video synthesis.
- video diffusion transformer
- A neural network architecture that generates video by iteratively denoising latent representations using transformer-based attention mechanisms.
- autoregressive latent chunks
- A generation strategy where video is produced in sequential segments, each conditioned on previously generated segments encoded in a compressed latent space.
- bounded visual context
- A fixed-size input window containing a persistent reference frame, compressed history, and spatial memory, designed so that computational cost per generation step does not grow with the total length of the generated sequence.
- sink frame
- A persistent identity anchor frame kept in the model's context throughout generation to maintain global visual consistency across long roll-outs.
- iWorld-Bench
- A benchmark used to evaluate interactive world models on metrics including trajectory following accuracy and long-horizon memory ability.
- Helios drift simulation
- A training technique that injects noise, blur, and saturation shifts into the model's context to teach it to recover from the visual degradation that accumulates during long autoregressive generation.
- error bank
- A training mechanism that stores and replays a model's own reconstruction residuals so the model learns to correct the specific errors it tends to produce during inference.
- AdaLN (Adaptive Layer Normalization)
- A conditioning method that modulates token embeddings in-place using scale and shift parameters derived from a control signal, avoiding the additional computational cost of cross-attention.
- cross-attention conditioning
- A method of injecting external signals into a transformer by adding a separate key/value stream, which incurs additional O(N·C) computational cost per attention layer.
- geometry-aligned spatial memory
- A component of the bounded context that stores past visual observations warped to align with the current viewpoint's geometry, enabling the model to recall spatially consistent information about previously seen areas.
- shared feature cache
- A pre-computed summary of a video clip's features that all quality-gating checks read from, avoiding redundant decoding and reducing data pipeline computational cost.
- camera trajectory conditioning
- The process of providing explicit camera movement signals to the model so that generated video follows a specified viewpoint path.
- long-horizon generation
- The task of producing coherent video or world simulations over extended durations, where maintaining consistency across many sequential steps is a core challenge.
- world model
- A generative model that simulates an interactive environment, predicting how the visual state of a scene evolves in response to user actions or camera movements.