ZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training

Xionghao Wu, Yijun Yang, Shiyang Zhou, Haoze Sun, Jianhui Liu, Songsong Yu, Jiyao Zhang, Wenbo Li, Bo Wang, Guoqing Ma, Lin Song, Renjie Liao, Shenghe Zheng, Wei Tang, Xiaojuan Qi, Yanwei Li, Yuan Zhang, Zhuotao Tian, Haoyang Huang, Nan Duan

ZimaBlue scales robot control by pre-training world models on 120,000 hours of egocentric video before grounding them in actions.

How can we scale generalist robot manipulation by leveraging massive, unlabelled egocentric video data to learn visual dynamics before fine-tuning on scarce action-labeled trajectories?

Robotic manipulation is bottlenecked by the scarcity of action-labeled data, forcing policies to learn physical and spatial priors from limited robot trajectories rather than the vast, readily available corpus of human egocentric video. ZimaBlue addresses this by decoupling learning into a three-stage curriculum: it first learns causal visual dynamics from unlabeled video, then grounds these dynamics in robot actions using a unified state-action interface, and finally specializes for real-time control. Scaling to 120,000 hours of embodied video improves zero-shot task success from 36.1% to 77.8% on real-robot evaluations.

Paper Primer

The core mechanism is a Slow-Fast dual-system architecture that separates deliberative world modeling from reactive control. The Slow branch acts as a high-capacity world model that predicts future visual states and exports K/V features, while the lightweight Fast branch consumes these features to generate 30 Hz actions without the latency of full video generation.

Large-scale video pre-training significantly boosts zero-shot generalization to unseen tasks.

Zero-shot success rates on real-robot manipulation tasks increased from 36.1% (target-robot data only) to 77.8% (with 120,000 hours of video pre-training). A 2.15x improvement in success rate over the baseline.

To handle heterogeneous robot interfaces, ZimaBlue maps all embodiments into a 100-dimensional semantic state-action space. This unified interface allows the model to ingest diverse robot data during mid-training, effectively turning the world model into a cross-embodiment controller.

Why use a dual-system architecture instead of a single end-to-end model?

Generative world models are computationally expensive and typically require iterative denoising, which introduces latency. The Slow-Fast design allows the model to maintain high-capacity world-aware representations while enabling the Fast branch to perform low-latency, 30 Hz closed-loop control.

How does the model handle the lack of action labels in egocentric human videos?

The model uses a three-stage curriculum where Stage I (video pre-training) learns causal visual dynamics from unlabeled video using a flow-matching objective. Action supervision is only introduced in Stage II, where it grounds the already-learned visual dynamics into motor-relevant representations.

ZimaBlue demonstrates that first-person video is a viable, scalable substrate for embodied intelligence, shifting the burden of robot learning from expensive teleoperation to the massive, untapped reservoir of existing egocentric video.

Scaling Robot Learning via Video

Robotic manipulation struggles because labeled trajectories are scarce while egocentric video is abundant.

Robotic manipulation faces a scaling bottleneck: robust generalization needs broad physical experience, yet collecting diverse, action‑labeled robot trajectories is costly and inherently limited. By contrast, egocentric videos capture rich object interactions, contact dynamics, tool use, and long‑horizon behaviors across many environments, offering a far more scalable source of embodied experience. The central challenge is turning this abundant, action‑free video into a model that can drive real‑time robot control.

**Figure 1.** Performance scaling with expanded data and acceleration. Left: Zero-shot task success rate monotonically increases as the training data scales up from specific-embodiment data (36.1% at 300 hours) to multi-embodiment datasets (46.1% at 6,000 hours), and improves up to 77.8% when further incorporating large-scale egocentric video data (120,000 hours). Right: Progressive acceleration techniques cut closed-loop latency from 450 ms to 33 ms on NVIDIA RTX 4090, achieving a 13.6× overall speedup.

ZimaBlue operationalizes this scaling insight with a three‑stage curriculum. First, it performs causal video pre‑training on over 120 k h of human and robot egocentric footage, learning visual dynamics and language grounding without any action labels. Second, it aligns those dynamics to heterogeneous robot trajectories via a unified action representation, preventing over‑fitting to a single embodiment. Third, it fine‑tunes the model on the target robot for deployment. To meet real‑time constraints, ZimaBlue adopts an asynchronous Slow‑Fast dual‑system: a large Slow world model supplies rich spatiotemporal representations, while a lightweight Fast branch reuses shallow features to predict actions at 30 Hz on an RTX 4090.

The key shift is moving from scarce, action‑labeled robot data to massive egocentric video pre‑training as the primary scaling axis for generalizable manipulation.

Prior Paradigms in Robot Control

We map the landscape of Vision‑Language‑Action, World‑Action, and dual‑system policies.

Related work clusters around three strands: Vision‑Language‑Action (VLA) policies, World‑Action models that predict video and actions jointly, and asynchronous dual‑system architectures that separate deliberation from fast control.

RT‑1 and its successor RT‑2 are large‑scale language‑conditioned policies that ingest heterogeneous robot demonstrations and output joint commands.

Vision‑Language‑Action (VLA) models extend this recipe by plugging in pretrained vision‑language backbones and richer action heads.

Integrates a pretrained vision‑language encoder with a diffusion‑based action generator, enabling zero‑shot instruction following.

A VLA system that couples a frozen CLIP visual encoder with a transformer policy, focusing on sample‑efficient fine‑tuning.

Extends $\pi$0 with a flow‑based action head that predicts continuous joint trajectories.

Combines a pretrained vision‑language model with a goal‑conditioned policy that plans via latent‑space optimization.

A commercial VLA pipeline that fuses large‑scale video‑language pretraining with robot‑specific action heads.

World‑Action Models (WAMs) go further by jointly predicting future video frames and the corresponding robot actions.

Adapts a pretrained video diffusion model to simultaneously generate future observations and actions, achieving zero‑shot manipulation.

Autoregressive video‑action diffusion that interleaves video and action tokens under causal attention.

Extends LingBot‑VA with an embodiment‑aware tokenizer and sparse expert capacity, improving multi‑embodiment transfer.

Asynchronous dual‑system policies separate a slow predictive planner from a fast reactive executor, enabling high‑frequency control without sacrificing long‑term reasoning.

An asynchronous architecture that runs a low‑frequency video‑DiT world planner alongside a high‑frequency action‑DiT executor, reusing planner context across multiple control steps.

The Slow-Fast Dual-System Architecture

Unified state space and a dual DiT system enable low‑latency, high‑capacity robot control.

Robot datasets expose wildly different control interfaces—single‑arm Cartesian commands, bimanual rigs, mobile bases, and dexterous hands. To train a single policy across this diversity we first map every robot into a common 100‑dimensional semantic state‑action vector, preserving physical meaning while allowing inactive coordinates to be masked.

The unified representation treats every robot as a fixed‑size vector where each slot always means the same physical quantity, so a policy can read and write the same indices regardless of embodiment.

Think of the Slow DiT as a scout that builds a detailed map of the environment (video dynamics), while the Fast DiT is a runner that follows that map in real time to produce precise actions.

Layer 1 Slow: compute K/V for video tokens $v_1\ldots v_4$ → store 4 K/V pairs.

Layer 2 Slow: repeat, producing another 4 K/V pairs (total 8).

Fast Layer 1: its action query attends to its own 2 action tokens and to the 4 K/V pairs from Slow Layer 1, producing an enriched query representation.

Fast Layer 2: attends to its updated queries and the 4 K/V pairs from Slow Layer 2, yielding the final action logits.

Because the Fast branch never recomputes video embeddings, the total attention cost is roughly (2 × 4 + 2 × 4) versus a full video generation cost of (4 × 4) per timestep.

The Fast branch gains high‑frequency responsiveness by reusing already‑computed visual context, while still benefiting from the Slow branch’s rich dynamics.

How does this differ from a simple cascade where a full video prediction feeds a separate action network?

In a cascade the action network would wait for the complete video prediction each timestep, incurring the full video generation latency. Here the Fast DiT consumes only the cached K/V features—already‑computed summaries—so it can react immediately while still being guided by the Slow branch’s world model.

**Figure 2.** Slow-Fast dual-system architecture. The Slow DiT processes observations, robot state, language instruction, along with noisy video and action tokens, to jointly predict future video latents and their corresponding actions (serves only as auxiliary supervision for video-action alignment). Concurrently, the Fast DiT ingests the updated observation and state, conditioned on the Slow DiT's video K/V caches, to generate fine-grained final actions for high-frequency closed-loop control.

**Table 1.** The table lists the slot ranges, their corresponding dimensions, and the descriptions of the robot's state components, including end-effectors, grippers, arm joints, torso, mobile base, and hand joints.

The Data Pyramid Curriculum

Stage I pre‑trains the Slow branch on massive egocentric video using a video‑only curriculum.

Robot learning suffers from a chronic shortage of diverse, action‑labeled data. To bootstrap a world model without this bottleneck, ZimaBlue first learns generic visual dynamics from massive egocentric video, postponing any action supervision.

The curriculum stacks three data tiers like a pyramid: a broad base of heterogeneous videos, a middle layer of robot‑centric video‑action pairs, and a narrow top of target‑embodiment data, letting the model acquire generic dynamics before grounding them in control.

How does the Data Pyramid differ from simply scaling up a single‑stage video‑action dataset?

Scaling a single dataset mixes diverse visual patterns with sparse action signals, causing the model to allocate capacity to noisy action supervision early. The pyramid isolates the visual learning phase, letting the model first saturate its visual capacity on abundant video, then inject actions only after a solid dynamics foundation exists.

Collect a heterogeneous video corpus (human, simulated, robot) and segment each recording into fixed‑length clips.

For each clip, construct a three‑row canvas: available camera views occupy canonical rows; missing views are padded with zeros.

Encode RGB frames with a VAE, producing a clean latent block $z$.

Sample a noise level $t$ and generate a noisy latent $z_t = (1 - \sigma^{\text{vid}}_t) \, z + \sigma^{\text{vid}}_t \, \epsilon$.

Apply the flow‑matching network to predict the denoising direction $\hat v^{\text{vid}}_{\theta}$ and compute the masked loss $L_{\text{vid}}$.

Back‑propagate only through the Slow branch; the Fast branch remains frozen.

Compute the noisy latent: $z_t = (1-0.5) \cdot (0.2,0.5,-0.1) + 0.5 \cdot (0.6,-0.3,0.4) = (0.4, 0.1, 0.15)$.

True denoising direction: $v^{\text{vid}} = \epsilon - z = (0.4, -0.8, 0.5)$.

The network predicts $\hat v^{\text{vid}}_{\theta}(z_t, c_{\le t}, \ell) = (0.38, -0.78, 0.48)$ (hypothetical output).

Apply the mask $m_{\text{vid}} = (1,1,1)$ (all tokens valid) and compute the squared error: $(0.38-0.4)^2 + (-0.78+0.8)^2 + (0.48-0.5)^2 = 0.0012$.

This toy illustrates how the model learns to map a noisy latent back toward the clean latent; the mask simply gates out any padded view rows, leaving the loss unchanged for valid tokens.

**Figure 3.** Data pyramid. Three-stage training progresses from broad visual diversity to deployment-specific embodiment. Stage I (Pre-training) learns general visual dynamics from diverse video sources without action supervision. Stage II (Mid-training) introduces robot actions from multiple embodiments, grounding visual dynamics in cross-embodiment control. Stage III (Post-training) specializes the model on target embodiments and benchmarks for deployment.

Aligning Video with Robot Actions

Mid‑training aligns video prediction with robot actions by jointly denoising shared tokens.

The scaling bottleneck stems from a mismatch: pre‑trained visual dynamics know how the world moves, but they lack any grounding in robot actions.

Mid‑training forces the shared backbone to predict both future video latents and corresponding action chunks, so the visual model learns to embed motor‑relevant information directly into its representations.

Corrupt the clean action a = (a₁, a₂) with Gaussian noise $\eta$ to obtain $a_t$ = (0.8 a₁ + 0.2 $\eta$₁, 0.8 a₂ + 0.2 $\eta$₂).

Compute the target `v_act` = $\eta$ – a, yielding `v_act` = ($\eta$₁ – a₁, $\eta$₂ – a₂).

Apply the binary mask `m_act` = [1, 0] (only the first joint is active for this embodiment) so the loss only penalizes the first dimension.

Evaluate the action loss `L_act` on the masked difference (ĥ`v_act` – `v_act`) and the video loss `L_vid` on the masked video error.

Combine them: sup = 0.6 `L_vid` + 0.4 `L_act`, producing a scalar gradient that updates the shared backbone.

The mask ensures that learning signals never flow into joints that do not exist for a given robot, preventing spurious gradients from corrupting the shared representation.

How does this joint video‑action denoising differ from standard video‑only diffusion?

Standard video diffusion predicts only future frames from noisy inputs; here the noisy future also includes action tokens, and the model must simultaneously reconstruct both modalities. The cross‑attention between video and action forces the backbone to learn a representation where a visual change and the motor command that caused it are inseparable, which does not happen in pure video diffusion.

Specialization and Deployment

Post‑training tailors the model to a robot by refining the Slow branch then training a fast reactive branch.

Deploying a pre‑trained model on a new robot demands adaptation without erasing the visual dynamics learned earlier. The post‑training stage solves this by first customizing the Slow branch’s interface and then adding a lightweight Fast branch for real‑time control.

We keep the visual backbone untouched and only learn small encoders/decoders that translate between the robot’s native state‑action space and the shared 100‑dim latent used by the backbone.

How does this differ from fine‑tuning the entire model on the new robot?

Fine‑tuning would modify the visual backbone, risking loss of the dynamics learned from billions of video frames. By training only the small encoders/decoders, we preserve those priors while still adapting to the robot’s control interface.

After the Slow branch is fixed, we train a slimmer Fast branch to predict short, shifted action chunks so it can react to fresh observations while the Slow rollout is still in progress.

Why not train Fast directly on full action sequences instead of using shifted, masked chunks?

Training on full sequences would require the model to wait for an entire Slow rollout before producing any output, eliminating the asynchronous advantage. The shifted‑chunk formulation forces Fast to predict from partial, out‑of‑phase observations, which is exactly the scenario it will face at deployment.

Fast receives the observation‑state pair at time $\delta$ and the input vector $[ \tilde a_0, \tilde a_1, \tilde a_2, \tilde a_3 ] = [0.5,0.7,0.9,\text{PAD}]$.

Because $p=2$, the first two positions are kept unchanged: $a^{F}_{t,0}=0.5$, $a^{F}_{t,1}=0.7$.

For positions $i\ge p$ we blend with the target velocity $\eta=[0.6,0.8,1.0,1.2]$ using $\sigma^{t}_{\text{act}}=0.3$:\n$a^{F}_{t,2}=0.7\cdot0.7 + 0.3\cdot1.0 = 0.79$,\n$a^{F}_{t,3}= \text{PAD}$ (ignored by the mask).

The mask $m^{\text{act},F}$ zeros out the padded entry and the prefix positions, so the loss is computed only on the blended third entry.

This toy walk‑through shows how Fast preserves already‑committed actions (the prefix) while learning to continue from a partially observed state, exactly the behavior needed for asynchronous control.

Fast acting while Slow is still rolling out is like a co‑pilot adjusting the steering wheel based on fresh sensor readings while the captain’s long‑range route plan is being computed.

Optimizing Closed-Loop Control

Accelerates closed-loop control via asynchronous Slow‑Fast inference and two‑step diffusion distillation.

ZimaBlue speeds closed‑loop control with three complementary tricks: a dual‑system that runs a low‑frequency “Slow” world‑model in parallel with a high‑frequency “Fast” action generator, a two‑step diffusion‑step distillation that halves the number of DiT evaluations, and a Torch‑compile pass that fuses the remaining operations.

The Slow stream predicts long‑horizon visual dynamics at a low rate, while the Fast stream continuously generates short‑horizon actions at a high rate, consuming the latest observation and the most recent Slow cache without ever waiting.

How does this differ from a naïve pipeline where the Fast controller waits for the Slow prediction each cycle?

In the naïve pipeline the Fast loop stalls until the Slow rollout finishes, limiting the control frequency to the Slow rate. Asynchronous inference decouples the two, letting Fast run at its own high rate and only replace future actions when a fresh Slow cache arrives, thus preserving responsiveness.

t = 0 ms: Slow generates a video K/V cache for the next 1 s and stores it.

t = 100 ms: Fast reads the cache, combines the latest observation, and emits actions for 0–200 ms.

t = 200 ms: Fast receives the same cache (still valid) and refines the 200–300 ms actions.

t = 300 ms: Fast again refines the 300–400 ms actions.

t = 400 ms: Slow finishes a new rollout, overwriting the cache with an updated prediction.

t = 500 ms: Fast now uses the fresh cache to generate actions for 500–600 ms, while the 400–500 ms actions remain from the previous Fast prediction.

Only the future, unexecuted portion of the action plan is replaced; already‑executed commands stay intact, guaranteeing smooth execution despite the asynchronous updates.

After establishing the asynchronous dual‑system, ZimaBlue compresses each branch’s diffusion inference from eight to two steps using Distribution Matching Distillation (DMD).

DMD teaches a lightweight student generator to produce the same video‑action distribution as a heavyweight teacher by matching the direction of the teacher‑student distribution mismatch.

Why not simply replace the teacher with the student after standard knowledge distillation?

Standard distillation aligns output logits but ignores the underlying diffusion dynamics. DMD explicitly matches the direction of the full teacher‑student distribution, preserving the temporal structure of the diffusion process while still cutting the number of steps.

Fake‑score re‑noises $x_S^{g}$ to $x_{S,0}^{f}=0.8 + \epsilon$, where $\epsilon\sim\mathcal{N}(0,\sigma^2)$; assume $\epsilon=0.1$, so $x_{S,0}^{f}=0.9$.

The fake‑score model outputs $\hat{x}_S^{\text{fake}}=0.85$ after denoising.

The frozen teacher (real‑score) outputs $\hat{x}_S^{\text{real}}=0.70$ for the same noisy input.

Compute $\text{DMD}_S = (0.85-0.70)/0.70 \approx 0.214$.

Use $0.214$ as the target direction for the student’s gradient update, nudging its next prediction toward the teacher.

The DMD direction captures both magnitude and sign of the mismatch, guiding the student to correct its bias rather than merely copying the teacher’s output.

**Figure 4.** Asynchronous inference in the Slow-Fast dual system. The Slow stream operates at a lower frequency and produces future video K/V caches. The Fast stream runs at a higher frequency, using updated observations, states, and Slow K/V guidance to generate action chunks. The generated actions are continuously executed and provide new observations for subsequent updates. “Outdated” marks a Slow video K/V cache based on past observations that is stale at the current timestep, while “prefix” denotes actions already committed for execution from the previous chunk and retained for continuity. The figure is purely schematic.

**Figure 10.** Attention masks of the Slow and Fast DiTs. Blue cells denote K/V blocks produced within the same branch, purple cells denote video K/V features imported from the Slow DiT, and white cells denote blocked dependencies. Rows are queries and columns are keys and values. In the Slow mask, $c_i$, $z_i$, $s_i$, and $a_i$ denote the clean video context, noised future-video latent, proprioceptive state, and noised action chunk at step $i$. In the Fast mask, $c_i$, $s_i$, and $a_i$ denote the updated observation, state, and prefix-conditioned action chunk at step $i$, while $\mathcal{K}_{\text{slow}}$ is the read-only Slow video K/V cache. Language condition $\ell$ is supplied through cross-attention and omitted from the self-attention diagram for clarity.

Real-Robot Performance

Zero-shot robot success rates across standard and perturbed tasks for ZimaBlue variants.

Recall that ZimaBlue overcomes data scarcity by pre‑training on massive egocentric video and then aligning those priors with robot actions. We now evaluate how those ingredients translate to real‑robot performance without any task‑specific demonstrations.

The full ZimaBlue configuration attains 77.8 % overall zero‑shot success, outpacing both $\pi$0.5 and DreamZero by a large margin.

Table 3 shows the full model’s average success across Standard and Perturbed suites at 77.8 % versus 63.6 % for $\pi$0.5 and 52.3 % for DreamZero.

**Figure 5.** Overview of zero-shot real-robot evaluation suites, showing initial/target states and instructions. Top: Standard suite (8 tasks) covering spatial grounding, stacking, object transport, articulated objects, and contact-rich control. Bottom: Perturbed suite (4 tasks) incorporating environmental shifts—appliance tasks feature dynamic glare, flashing light, and distractors, while bread/bowl tasks introduce unseen tablecloths and novel distractor objects.

**Figure 6.** Representative real-robot rollouts, ordered from left to right. Panel (a) contrasts a lateral placement failure with successful top-down placement under the same pink-target instruction. Panel (b) contrasts an incomplete bowl sequence with placement of the completed stack on the plate. Panel (c) contrasts a door that reopens after contact with a complete microwave closure. Panel (d) contrasts pulling the tablecloth with completed bread placement in the cluttered Perturbed scene.

Scaling trends (Table 2) reveal that adding 6 K hours of multi‑embodiment data lifts Standard success from 46.7 % to 57.9 % and Perturbed from 15.0 % to 22.5 %. Adding 60 K hours of egocentric video further boosts Standard to 82.9 % and Perturbed to 35.0 %.

Qualitative inspection shows two dominant failure modes: (1) the policy stalls after reaching a correct intermediate state, and (2) local interaction errors such as target displacement or loss of contact, especially under the Perturbed lighting and clutter conditions.

Benchmark: LIBERO-Plus

Key success rates and latency on the LIBERO‑Plus and RoboTwin benchmarks.

We now report ZimaBlue’s performance on two simulation suites: the robustness‑focused LIBERO‑Plus benchmark and the bimanual RoboTwin‑2.0 benchmark.

The benchmark measures how well a policy tolerates seven independent perturbations—camera view, robot start pose, language command, lighting, background texture, sensor noise, and object layout.

ZimaBlue attains an overall zero‑shot success rate of 86.7 % on LIBERO‑Plus, the highest among all reported methods.

Table 5 shows ZimaBlue’s 86.7 % average across the seven perturbation dimensions, surpassing the next‑best InternVLA‑A1.5 (85.8 %).

With supervised fine‑tuning on the LIBERO‑Plus training set, ZimaBlue reaches 92.0 % success, outpacing the second‑best CAC‑VLA by 1.9 pp.

Table 5 reports a 92.0 % average for ZimaBlue after SFT, while CAC‑VLA achieves 90.1 %.

**Table 5.** Evaluation results on LIBERO-Plus. “Average” represents the unweighted arithmetic mean of the success rates across all seven perturbation types. Within each evaluation protocol, the best and second-best performances are highlighted in bold and underlined, respectively.

**Table 6.** Evaluation results on the RoboTwin 2.0 benchmark. Success rates (%) are averaged over 50 tasks for both “Clean” and “Randomized” settings, with “Average” denoting their mean. The best and second-best results are highlighted in **bold** and <u>underlined</u>.

Benchmark: RoboTwin

ZimaBlue sets new state‑of‑the‑art success rates on RoboTwin 2.0.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers