Infinite Interactive World Rollout on a Single Desktop GPU

Fan Jiang, Zhaoxu Sun, Mengchao Wang, Ziyu Zhu, Chiyu Wang, Yunpeng Zhang, Wenlin Liu, Yun Wang, Xue Zheng, Rui Sun, Junfeng Ni, Hongyu Pan, Zhongxu Sun, Fei Yu, Zengye Ge, Mengmeng Du, Nianfei Fan, Mingchao Sun, Yu Liu, Yongchang, Yanqing Zhu, Jiahang Wang, Ning Ying, Yuze Xuan, Di Yang, Zhicheng Liu, Zhe Gao, Tingbing Xu, Jiacheng Sui, Wenjin Yang, Junnan Lai, Shufeng Liu, Yuan Liu, Zheng Zhou, Yingliang Peng, Dawei Cao, Kaifeng Sheng, Yuxiang Cai, Fei Lu, Mu Xu, Ning Guo

ABot-World-0 enables real-time, action-conditioned 720P world simulation on a single desktop GPU.

How can we enable real-time, infinite-horizon interactive world simulation on consumer-grade hardware?

Interactive world models struggle to balance visual fidelity with the low-latency, long-horizon stability required for persistent, user-controlled environments. Existing systems often suffer from accumulated autoregressive drift or require massive compute clusters that preclude local deployment. ABot-World-0 addresses this by distilling a bidirectional teacher model into a causal student, using a multi-stage pipeline that includes "LongForcing" to align long-horizon rollouts with teacher dynamics. The system uses raw keyboard inputs for unified character and camera control, supported by a custom inference stack that optimizes attention and VAE decoding. On a single NVIDIA RTX 5090, the model streams 720P video at up to 16 FPS with 1.2s action-to-first-frame latency, maintaining identity and motion coherence over extended interactive sessions.

Paper Primer

The core challenge is the "control–consistency–efficiency" coupling: generating a persistent world that responds to user actions without drifting into visual noise or requiring prohibitive hardware. ABot-World-0 solves this by treating data production as an active, feedback-driven loop and distilling a high-quality bidirectional teacher into a lightweight, causal student.

The method hinges on LongForcing: a distribution-matching stage that supervises long student self-rollouts against an extended-horizon teacher. This move forces the model to correct its own prediction errors before they compound, effectively bridging the gap between short-horizon training and long-horizon inference.

Achieves real-time interactive performance on consumer hardware.

Streaming 720P video at 16 FPS with 1.2s action-to-first-frame latency on a single RTX 5090.

Mitigates long-horizon autoregressive drift.

LongForcing provides corrective signals for long-horizon contexts that standard short-horizon objectives fail to cover. Enables coherent world evolution beyond the initial training horizon.

Why use raw keyboard inputs instead of latent actions?

Raw keyboard inputs are directly available from game recordings and align naturally with user intent at inference time, avoiding the label noise and complexity of inferring latent actions from passive video.

How does the system maintain character identity during third-person rollouts?

The model uses a reference-character memory module that prepends canonical identity tokens to the video sequence, allowing the DiT backbone to retrieve persistent appearance cues throughout the rollout.

The system treats data collection as a closed-loop servo system: a training monitor identifies under-performing scene-action combinations, and the WorldExplorer agent dynamically rebalances collection to target those specific weaknesses.

Interactive world modeling is now a full-stack engineering problem; by co-designing data infrastructure, distillation objectives, and inference kernels, high-fidelity simulation is achievable on a single desktop GPU.

Real-Time Interactive World Simulation

ABot‑World‑0 delivers real‑time, long‑horizon interactive video worlds on a desktop GPU.

ABot‑World‑0 is an action‑conditioned video world model that enables real‑time, long‑horizon interaction on a single desktop GPU. It unifies data from AAA games, simulation engines, and internet videos, and couples a bidirectional teacher with a causal student for efficient inference.

ABot‑World‑0 packs a bidirectional teacher‑student distillation pipeline into a causal model that can be queried with raw keyboard actions, producing video frames on the fly.

**Figure 1:** *ABot-World-0* turns a single NVIDIA RTX 5090 GPU into a real-time interactive world simulator, enabling infinite action-conditioned world rollout at 720P and up to 16 FPS with 1.2 s action-to-first-frame latency within a peak-VRAM budget of approximately 19 GiB.

The system’s streaming stack—lightweight VAE decoder, efficient low‑precision attention, and memory‑aware module scheduling—makes the full pipeline deployable on a consumer desktop without sacrificing visual fidelity.

ABot‑World‑0 proves that high‑quality, interactive world models can run in real time on a single desktop GPU.

Related Work

Survey of video generation approaches and their trade‑offs for interactive use.

Diffusion models now dominate video synthesis, moving from U‑Net backbones to diffusion transformers that attend spatiotemporally across the whole clip. This full‑clip bidirectional modeling yields strong visual quality and temporal coherence, and it naturally supports interpolation, infilling, and bounded generation where multiple temporal constraints coexist.

Because each frame can attend to both past and future frames, bidirectional attention couples the entire sequence, making incremental low‑latency or streaming generation difficult. This coupling motivates causal or autoregressive video diffusion methods that sacrifice global context for online efficiency.

Autoregressive (AR) video generation produces frames sequentially, conditioning each new segment only on previously generated content. Enforcing a causal constraint enables low‑latency streaming and, when combined with bounded context or caching, reduces memory overhead for long‑video synthesis.

AR models, however, suffer from temporal drift: quality degrades, identities change, and motion can stagnate after a modest number of rollouts. The drift is tied to exposure bias and error accumulation, prompting a suite of mitigation techniques such as Diffusion Forcing, PA‑VDM, Rolling Forcing, FIFO‑Diffusion, Self‑Forcing, LongLive, Self‑Forcing++, Causal Forcing, Context Forcing, HiAR, Diagonal Distillation, LongLive‑2.0, FLEX, and PackForcing.

Interactive video generation conditions dynamics on user actions; prior work encodes 6‑DoF camera extrinsics or dense ray maps, which can drift outside the training distribution during long rollouts. In contrast, we adopt a discrete keyboard‑action space that specifies local, incremental controls for character movement and camera motion, keeping the control representation bounded and naturally aligned with real‑time interaction.

Real‑time interactive generation balances two coupled metrics: latency (action‑to‑frame delay) and throughput (sustained FPS). Chunk‑wise generation amortizes attention and VAE decoding across multiple frames, while low‑bit DiT inference, memory‑aware scheduling, and KV caches reduce compute and memory, enabling 720p streaming at up to 16 FPS with ~1.2 s latency on a single RTX 5090 within a 19 GiB VRAM budget.

Data Infrastructure Overview

The section outlines a pipeline that ingests game, simulation, and internet video data, filters and annotates it for training.

ABot‑World treats data production as a core component of model development rather than a one‑off preprocessing step. The training corpus draws from three complementary sources—AAA game data, simulation engine data, and real‑world internet video data—each providing diverse environments, viewpoints, and motion patterns. Collection is handled by WorldExplorer, a unified agent‑driven system that navigates virtual worlds and captures synchronized multimodal signals, while internet videos are processed through a separate segmentation and pose‑estimation pipeline.

The infrastructure offers three decisive advantages. First, fully automated agent‑driven collection with training‑feedback‑driven adaptive rebalancing turns data production into an active servo system aligned with training objectives. Second, a comprehensive quality‑control pipeline applies 14 deterministic checks spanning six quality dimensions together with vision‑language‑model semantic assessment, maximizing the effective data ratio. Third, deterministic action annotation from game and simulation APIs yields high‑quality labels, while internet videos receive pose‑estimated pseudo‑labels, all unified under a canonical representation for downstream training.

**Figure 2.** Data Pipeline Overview. An end-to-end pipeline couples agent-driven and internet-scale collection, multi-stage quality filtering, and structured multimodal annotation. Curated data trains the world model, whose evaluation feeds a training-feedback loop that drives targeted re-collection and continual data refinement.

Data Collection Paradigms

WorldExplorer gathers interactive video data via a closed‑loop agent that adapts collection using training feedback.

Existing data‑collection pipelines trade off scalability, annotation quality, and behavioral diversity. Manual gameplay yields authentic behavior but cannot scale; rule‑based scripts are cheap yet rigid; agent‑driven collectors explore purposefully but risk narrow coverage and cannot handle passive video sources.

WorldExplorer turns data gathering into a self‑regulating thermostat: it watches the model’s training loss (the temperature) and automatically turns the data‑generation heater on or off to keep performance within a target band.

How does WorldExplorer differ from a traditional rule‑based scripted collector?

Rule‑based scripts follow a fixed itinerary and cannot react to the model’s learning state. WorldExplorer continuously measures training performance, diagnoses which scene‑action pairs are under‑represented, and retargets the agent’s goals toward those gaps, turning data collection into an active learning loop.

Phase 1 (unexplored priority) selects node 3 because it lies in the largest unexplored region.

Agent moves 1→2→3, marking nodes 1, 2, 3 as visited.

Phase 2 (nearby search) now targets node 5, the closest unvisited node to the current position.

Agent moves 3→4→5, visiting nodes 4 and 5.

All nodes visited; Phase 3 (forward fallback) would now simply patrol the loop to maintain coverage.

The multi‑phase selector guarantees that each new episode maximally expands the explored frontier before resorting to cheap forward motion, dramatically lowering revisit rates.

The loop works like a self‑adjusting assembly line: as the model stalls on certain tasks, the line automatically diverts more raw material (data) toward those bottlenecks.

Why not simply increase the static proportion of hard scenarios instead of using a closed‑loop?

Static over‑sampling wastes compute on scenarios the model already masters and cannot react to emergent failure modes. The closed‑loop continuously reallocates effort toward the *actual* weaknesses observed during training, achieving higher efficiency and better generalization.

Applying WorldExplorer to AAA games yields a unified corpus that mixes first‑person ego‑motion with third‑person contextual views, all captured via the game’s runtime API for exact control signals. This diversity lets a single model learn across viewpoints without separate architectures.

Simulation and Synthetic Data

We generate diverse trajectories with synthetic rendering and extract pseudo‑labels from internet videos, then filter them for training.

To broaden trajectory diversity beyond game recordings, we synthesize scenes with two rendering backends and extract pseudo‑labels from internet videos, then filter the data for training.

We combine Unreal Engine’s photorealistic rendering with 3D Gaussian Splatting reconstruction to create accurate urban and indoor scenes from proprietary aerial data.

How does the 3DGS backend differ from traditional mesh rendering?

3DGS stores scene geometry as a continuous Gaussian field, enabling smooth interpolation and compact storage, whereas mesh rendering relies on explicit polygonal surfaces that can be memory‑intensive for large environments.

WorldExplorer synthesizes geometric patterns and interpolates multiple points to produce smooth, collision‑free trajectories.

Compute cubic spline coefficients for the three points, producing a smooth curve.

Sample the curve at t=0.25 → (1.0, 1.5); t=0.5 → (3.0, 4.0); t=0.75 → (4.5, 4.75).

Check each sampled point against scene geometry; all are collision‑free.

Assemble the sampled points into a trajectory sequence ready for rendering.

Interpolation turns a sparse set of waypoints into a dense, physically plausible camera path, avoiding abrupt direction changes that would look unnatural.

Why not simply use a random walk to generate trajectories?

Random walks produce erratic, discontinuous motion that often collides with scene geometry and lacks the smoothness required for realistic video synthesis; our pattern‑based synthesis enforces structural constraints and smoothness.

We map motion paths captured from physical devices into the 3D assets, preserving authentic human browsing behaviors such as repeated scanning and handheld jitter.

Does importing real trajectories risk violating scene geometry?

Before rendering, each imported path undergoes collision detection; any segment that would intersect geometry is either trimmed or re‑projected onto a feasible surface, guaranteeing physical plausibility.

We project translational and rotational displacements onto the camera’s basis vectors and binarize them into discrete action signals aligned with each frame.

What happens if a projected displacement is near zero?

When the magnitude falls below the threshold, the corresponding action bit is set to zero, indicating no significant motion along that axis for the current frame.

We segment raw videos into coherent clips, then recover six‑DoF camera trajectories via pose estimation methods chosen according to scene characteristics, producing pseudo‑labels for training.

How reliable are pseudo‑labels compared to simulation ground truth?

While pseudo‑labels are noisier, the paper shows that the student model benefits from the added diversity; the filtering pipeline mitigates severe errors, and the distillation process tolerates moderate label noise.

A three‑stage pipeline applies 14 checks across six quality dimensions, progressively discarding low‑quality clips and annotating metadata for sample weighting.

Why keep clips with minor metadata issues instead of discarding them?

These clips still contain useful visual and motion information; the metadata flags enable the training scheduler to down‑weight them, preserving valuable diversity while mitigating their imperfections.

Together, these pipelines supply a rich, filtered dataset that fuels the student distillation process while maintaining high fidelity.

Data Annotation Pipeline

We annotate raw clips with unified action, language, and identity signals to fuel training.

Training a world model requires massive, diverse video data, yet manual labeling is prohibitively expensive. The pipeline therefore synthesizes annotations from whatever signals each source provides, while keeping a shared representation for downstream learning.

Each clip receives a precise, time‑aligned action token regardless of whether the source is a game, a physics simulation, or an internet video.

How do these action labels differ from a typical human‑annotated action dataset?

Human annotation supplies coarse, often ambiguous verbs, whereas our pipeline yields frame‑exact, low‑level control codes directly from the source or from a deterministic pose projection, eliminating subjectivity and scaling to millions of clips.

A large vision‑language model writes a concise natural‑language caption that describes the overall scene but deliberately skips camera motion.

Why not use human‑written captions instead of VLM‑generated ones?

Human captions would require labor‑intensive annotation for every clip; the VLM produces consistent, scalable captions while preserving the semantic richness needed for text‑conditioned generation.

Every clip is enriched with a set of discrete tags (e.g., indoor/outdoor, weather, vehicle type) extracted automatically from the VLM output.

What advantage do these tags give over simply random sampling of clips?

Random sampling would over‑represent common conditions (e.g., sunny daytime) and under‑represent rare ones (e.g., foggy night), leading to biased learning; tags enforce exposure to the full spectrum of contexts.

For third‑person videos we extract directional face thumbnails and synthesize a canonical frontal portrait to condition the model on a specific character.

How is this identity extraction different from generic person detection pipelines?

Generic detectors output a single bounding box per frame; our pipeline deliberately captures four directional views and reconstructs a canonical frontal face, providing a stable, view‑invariant identity embedding for consistent character generation.

Clip 1: the game API emits button IDs (0 = accelerate, 2 = steer‑left) at 30 fps; these are written directly as the action vector.

Clip 2: the known trajectory equation $x(t)=5t$ is discretized; each timestep yields a constant forward‑velocity action.

Clip 3: pose angles are projected onto a 2‑D displacement vector; if the magnitude exceeds 0.2 m the pseudo‑action “walk‑forward” is set.

All three vectors are mapped to the shared vocabulary entry “move‑forward” with optional steering sub‑token.

This toy example shows how heterogeneous sources can be homogenized into a single action token stream, enabling the model to train on a vastly larger, mixed‑origin dataset without manual labeling.

Model Formulation and Training

We detail the three‑stage distillation that turns a bidirectional teacher into a fast causal student.

Autoregressive video generation demands many diffusion steps, making real‑time interaction infeasible; the trick is to replace the expensive bidirectional generator with a lightweight causal student while preserving visual fidelity.

We first teach a full‑horizon teacher, then gradually force the student to act causally, compress its denoising steps with an ODE, and finally align long‑range rollouts with the teacher’s distribution.

How does this distillation differ from standard knowledge distillation?

Standard distillation matches logits on a per‑sample basis, assuming the same input distribution. Here we first enforce causal conditioning (Teacher Forcing), then replace the teacher’s multi‑step diffusion with a learned ODE (compressing the inference trajectory), and finally align entire rollout distributions over many steps (LongForcing), which addresses temporal drift that ordinary distillation ignores.

Teacher Forcing: the teacher outputs the exact pixel values for frames 1 and 2; the student receives these as targets while being fed only past frames.

ODE Distillation: instead of 5 diffusion steps per frame, the learned ODE maps the noisy latent after 1 step directly to the clean frame, halving compute.

LongForcing: after the student generates frame 1, we roll it forward and compare the distribution of the generated frame 2 against the teacher’s frame 2, adjusting the student to reduce the mismatch.

The three stages together turn a 5‑step per‑frame diffusion (≈20 ms) into a 1‑step ODE prediction (≈4 ms) while keeping long‑horizon consistency.

**Figure 3.** Overall training pipeline of ABot-World-0. We first adapt a pretrained video generator into a high-quality bidirectional action-conditioned teacher. The teacher is then progressively converted into a causal autoregressive student through Teacher Forcing, Causal ODE Distillation, and LongForcing, enabling few-step low-latency inference and stable long-horizon interactive rollout.

**Figure 4.** ABot-World-0 Model Architecture. ABot-World-0 incorporates action control and reference-character memory into a pretrained video DiT. Keyboard actions guide dynamics, while reference images help preserve character identity.

Bidirectional Teacher Training

We train a bidirectional video model that injects action signals and preserves character identity.

Pretrained video generators excel at visual fidelity but cannot be steered by user actions, so they fail to serve as interactive world models.

The teacher augments a pretrained diffusion transformer with two streams of conditioning: packed action tokens that drive motion, and identity‑memory tokens that lock the character’s appearance.

How does this action injection differ from latent‑action conditioning used in prior video models?

Latent‑action methods infer an abstract action embedding from video frames, which can be noisy and indirect. Here we inject the raw, user‑provided keyboard vector directly, preserving the exact control intent and avoiding any learned proxy.

Encode each frame’s raw keyboard input as an 8‑dim multi‑hot vector $a_t \in \{0,1\}^8$.

Group every four consecutive vectors and concatenate them to form a 32‑dim token $\tilde{a}_\tau = \operatorname{Concat}(a_{4\tau-3},\dots,a_{4\tau})$.

Pass the packed sequence $\tilde{a}_{1:T/4}$ through the Action Control Adapter $F_\psi$, which first applies PixelUnshuffle (kernel = DiT patch size) then residual convolutions.

Add the adapter output to the DiT patch embeddings of the noised latents: $\hat{z} = \operatorname{PatchEmbed}(z) + F_\psi(\tilde{a}_{1:T/4})$.

Feed $\hat{z}$ into the diffusion backbone for the remaining denoising steps.

Each $a_t$ is an 8‑dim one‑hot vector.

Concatenating the four vectors yields $\tilde{a}_1 = [1,0,0,0,\,0,1,0,0,\,0,0,1,0,\,0,0,0,1]$, a 32‑dim token.

The adapter $F_\psi$ reshapes $\tilde{a}_1$ via PixelUnshuffle, producing a spatial map that aligns with a DiT patch.

Adding this map to $\operatorname{PatchEmbed}(z)$ injects the action information directly into the latent representation.

Packing four frames into one token preserves temporal alignment with the VAE’s compression, letting the diffusion model treat the action as a single patch‑wise bias.

Think of the memory tokens as backstage crew that only watch the actors (video tokens) and never get distracted by the performance, keeping the character’s look steady throughout the scene.

Memory tokens are assigned RoPE indices $-2$ and $-1$, video tokens get $0,1,2$.

Attention scores are computed; video token $v_2$ attends to $m_1$ and $m_2$ (high scores) while $m_1$ receives no incoming attention.

After the attention update, $v_2$ incorporates identity features from $m_1,m_2$, preserving the character’s appearance.

Because memory tokens never attend to each other or to video tokens, their encoded identity remains unchanged across the entire rollout.

Causal Progressive Student Distillation

Transforms a bidirectional teacher into a fast causal student via three progressive stages.

The bidirectional teacher cannot run in real time because it generates the whole video at once and requires many denoising steps. We therefore compress it into a causal generator through a three‑stage pipeline.

We turn a high‑quality, non‑causal video model into a fast, causal one by progressively shrinking the training horizon and the number of diffusion steps.

Stage 1: Teacher Forcing – feed the clean history $v_0,v_1$ as conditioning, corrupt $v_2,v_3$ with diffusion noise, and train the student to reconstruct them.

Stage 2: ODE Distillation – pick a noise level $s=0.5$, compute $z_c^{s}$ from the Stage 1 model, and train $f_{\theta}$ to output the clean latent $z_c^{0}$ in a single step.

Stage 3: LongForcing – generate a self‑rollout $v_2',v_3'$ with the distilled model, then feed the longer sequence $v_0,v_1,v_2',v_3'$ to an extended‑horizon teacher and match the resulting distribution.

Result: after the three stages the student can produce $v_2',v_3'$ in two diffusion steps while maintaining visual fidelity comparable to the original teacher.

Progressively exposing the student to longer horizons and fewer denoising steps forces it to learn stable dynamics that would otherwise drift during long autoregressive rollouts.

How does this differ from ordinary knowledge distillation, which also transfers a teacher’s knowledge to a smaller student?

Ordinary distillation matches the teacher’s output logits on the same inputs, but it does not change the student’s generation order or reduce diffusion steps. Causal Progressive Student Distillation first reshapes the teacher into a causal model (Stage 1), then compresses its denoising trajectory (Stage 2), and finally aligns long‑horizon rollout distributions (Stage 3). Each stage addresses a distinct bottleneck—causality, inference latency, and drift—whereas standard distillation only addresses output similarity.

Full-Stack Inference Co-Design

Full‑stack inference design balances speed, latency, and memory for real‑time video rollout.

Few‑step distillation cuts denoising cost, but real‑time rollout still stalls on latent decoding, transformer work, and memory traffic.

Instead of speeding up a single operator, we redesign the entire inference pipeline so that latency, throughput, and memory constraints are satisfied together.

Decode three $4\times4$ maps → three $2\times2$ maps (LightVAE), cutting memory from $48$ KB to $12$ KB.

FP8 quantization makes each GEMM use half the bytes, halving DiT compute time.

Bounded KV cache stores only the last $2$ chunks, keeping cache size constant at $24$ KB.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers