DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

DreamX Team, Rui Chen, Xiangxiang Chu, Geng Li, Jifan Li, Qingfeng Shi, Datao Tang, Jing Tang, Jun Wang, Pengfei Zhang

DreamX-Phi 1.0 uses geometry-aware attention and object-centric supervision to predict faithful, action-conditioned robotic manipulation videos.

How can we leverage large-scale video diffusion priors to create a high-fidelity, action-conditioned world model for bimanual robotic manipulation?

Video world models often produce photorealistic rollouts that fail to respect commanded robot motions, such as moving the wrong arm or losing track of a manipulated object. DreamX-Phi 1.0 addresses this by injecting per-arm SE(3) transformations into attention via geometric encoding, while using depth and object-mask supervision to enforce physical consistency. On the WorldArena 2.0 benchmark, the model ranks first on Track 1 (video prediction) and ties for second on Track 2 (policy training).

Paper Primer

The core challenge is that standard video generators treat robot actions as generic tokens, which fails to preserve the rigid-body geometry of end-effector motion. DreamX-Phi 1.0 treats the robot as a structured geometric entity: it uses Projective Relative Positional Encoding (PRoPE) to inject SE(3) trajectories directly into the attention mechanism, ensuring the model understands where and how the arms move in 3D space.

To prevent the background from overwhelming the interaction, the model employs a "mask-and-supervise" strategy: it uses SAM3 masks to weight the loss toward the manipulated object and a frozen V-JEPA teacher to regularize object evolution over time. This is like a camera operator who ignores the static scenery to focus exclusively on the contact point between the gripper and the object, ensuring the grasp remains physically coherent.

DreamX-Phi 1.0 achieves state-of-the-art performance in action-conditioned video prediction.

Ranked 1st on WorldArena 2.0 Track 1 (EWMScore-P of 60.65) and tied for 2nd on Track 2 (67.19% success rate on Adjust Bottle). Outperformed 30 other entries on the Track 1 leaderboard snapshot.

Why is this approach better than simply feeding action tokens into a standard video model?

Token-based controls are geometrically implicit and often fail to preserve the rigid-body structure of robot motion. By using PRoPE, DreamX-Phi explicitly encodes the 3D end-effector trajectory, which forces the model to align visual changes with the specific commanded motion.

Does this model generate its own actions, or does it only predict what happens after an action is given?

It is a Forward Dynamics Model (FDM) that predicts future observations from an externally provided action sequence. It does not generate actions itself, though it can serve as a simulated environment for training a separate policy.

Introduction and Motivation

We frame the need for action‑conditioned, geometry‑aware video world models for robotic manipulation.

Video diffusion models can generate photorealistic rollouts, but without explicit action conditioning they may ignore the prescribed robot trajectories, moving the wrong arm or losing the manipulated object. To make rollouts faithful, we must preserve each arm’s SE(3) motion and enforce scene geometry consistency throughout the prediction.

Robotic planners need a world model that not only looks realistic but also guarantees that the predicted video respects the exact bimanual action trajectory and object interactions.

DreamX‑Phi 1.0 takes a single observed frame, a language instruction, and a prescribed bimanual action trajectory, then predicts a future video that follows the exact SE(3) motions while keeping the rest of the scene unchanged.

**Figure 1.** Overview of DreamX-Phi 1.0, an action-conditioned video world model that predicts future observations from a single frame and prescribed bimanual actions.

The shift from passive video generation to action‑conditioned robotic world modeling enables planners to evaluate manipulation strategies safely and efficiently.

Related Work and Foundations

Survey of prior video‑world models and the curated dataset that fuels DreamX‑Phi.

Action‑conditioned video world models treat a video generator as a learned simulator, letting a control signal steer future frames.

Scalable diffusion models that learn generic physical dynamics from massive video corpora, providing a visual prior for downstream control.

Extensions of video diffusion that incorporate an explicit action interface, allowing controllable visual dynamics across diverse domains.

Robot‑specific models that bind video frames to concrete robot trajectories, tightening the conditioning contract.

World‑action models (WAMs) connect video and control via three distinct interfaces.

Inject low‑dimensional action tokens into a video generator, offering a compact but geometrically implicit control signal.

Encode robot motion as rendered kinematic skeletons or optical flow, preserving spatial alignment in the image.

Geometry‑aware attention and depth supervision add physical consistency beyond raw control signals.

Injects relative SE(3) transforms into attention, preserving rigid‑motion structure across tokens.

Applies known camera geometry to queries, keys, and values, aligning attention with the 3‑D scene.

Wan2.2‑TI2V‑5B is a large‑scale video diffusion backbone that learns to generate realistic video frames from a latent representation, serving as the visual prior for DreamX‑Phi.

Data curation assembles three complementary sources—egocentric video, real‑robot demos, and simulated trajectories—to cover both visual diversity and precise action annotation.

**Table 1.** Data sources used for action-agnostic pretraining.

During curation we discard mobile‑base motion, dexterous‑hand actions, and stationary segments, but we keep failed attempts to expose failure modes.

Action‑agnostic pretraining retains each source’s native camera layout: single‑view streams stay single‑view, while synchronized multi‑view streams are concatenated spatially.

Action‑conditioned fine‑tuning pairs each video with its robot trajectory in the unified LeRobot v2.1 format; RoboTwin clips are first enhanced by DreamX‑Refiner for higher visual fidelity.

Methodology: Architecture and PRoPE

We introduce Pose-Relative Position Encoding (PRoPE) to inject structured bimanual robot trajectories into video diffusion transformers.

To adapt large-scale video diffusion models for robotic manipulation, we must bridge the gap between unstructured visual generation and the rigid constraints of physical motion. Our framework, DreamX-Phi 1.0, conditions video prediction on bimanual action trajectories, ensuring that generated motion respects the geometric relationships inherent in robot end-effector movement.

PRoPE injects robot motion into the transformer by treating end-effector poses as relative geometric transforms within the attention mechanism, allowing the model to learn how motion influences visual change without relying on absolute coordinate frames.

Compute the relative transform $R_{ij} = D_i D_j^{-1}$ representing the motion between the two frames.

Transform the query $Q_i$ by $D_i^T$ and the key $K_j$ by $D_j^{-1}$.

The attention score becomes $Q_i^T D_i D_j^{-1} K_j = Q_i^T R_{ij} K_j$.

The model now attends to token $j$ based on the relative motion $R_{ij}$ rather than absolute coordinates.

This formulation makes the attention mechanism invariant to the global coordinate frame, as only the relative transform $R_{ij}$ between tokens influences the attention weight.

**Figure 2.** Overview of the DreamX-Phi 1.0 framework. (1) Given an observed RGB frame, a language instruction, and an externally specified bimanual action trajectory, a Wan2.2-TI2V-5B video diffusion transformer predicts the future RGB video. Arm-grouped PROPE and a robot-only optical-flow cue provide complementary geometric and image-plane action conditioning. (2) During training, SAM3-derived masks reweight the RGB objective, Depth Anything 3 (DA3) depth maps provide targets for the depth-latent objective, and a frozen V-JEPA teacher supplies object-relational supervision. (3) DMD with adversarial training distills the multi-step teacher into a few-step student.

Geometric Consistency and Depth Supervision

We add lightweight depth and object‑aware supervision to enforce 3D consistency without changing inference.

RGB‑only prediction captures appearance but provides no explicit geometric cue; the background can dominate the loss while the robot‑object interaction is barely penalized. This makes a rollout look realistic even when the gripper penetrates the object or the object moves implausibly.

Attach a lightweight side‑branch that predicts depth in the same latent space, forcing the shared encoder to learn geometry.

Trunk blocks 1–2 process the RGB tokens, producing a shared representation $h^{\text{shared}}$.

Depth block 1 receives $h^{\text{shared}}$ as its input $h^{d}_{0}$ and cross‑attends to the RGB keys $K^{\text{rgb}}_{1}$, yielding $h^{d}_{1}$.

The output head linearly maps $h^{d}_{1}$ to $\mathbf{b}z_d$, a 4‑dimensional vector.

The encoder $E$ applied to the pseudo‑RGB depth produces $z_d$, also a 4‑dimensional vector.

Compute $L_{\text{depth}}$ as the squared Euclidean distance between the two 4‑vectors, divided by 4.

The depth loss forces the shared trunk to embed geometry early enough that a single replicated block can recover the correct depth latent.

How does this differ from simply adding a depth prediction head to the RGB transformer?

Here the depth branch reuses the same frozen encoder and receives cross‑attention from RGB features, while a naïve head would only see the final RGB tokens and would not influence the shared representation during training.

Weight the RGB loss toward the manipulated object and align its feature relations over time, so the model learns a coherent object response to robot contact.

For each object token $e_{w_i}=1+(2-1)\cdot1=2$; for each background token $e_{w_i}=1$.

Sum of $e_{w_i}$ over all 16 tokens: $8\times2 + 8\times1 = 24$.

Object token weight $w_i = 2/24 = 1/12$, background weight $w_i = 1/24$.

The weighted RGB loss therefore gives each object token twice the influence of a background token.

For JEPA, the eight object tokens are projected to $\mathbf{S}_b$, the teacher provides $\mathbf{Q}_b$, and the Frobenius norm of their Gram‑matrix difference is computed.

The mask weighting makes the loss robust to varying object sizes while still guaranteeing that object errors dominate the gradient.

Why is the mask used only for training and not required at inference?

During training the mask tells the model where to focus its error signal; at inference the model has already learned to embed geometry, so it can generate plausible object motion without an explicit mask.

Compress a multi‑step diffusion generator into a student that produces high‑quality videos in only a few denoising steps, using distribution matching and an adversarial head.

Compute the KL divergence between $q_{\eta,0.3}(\cdot\mid\mathbf{y})$ (student) and $p_{\text{data},0.3}(\cdot\mid\mathbf{y})$ (teacher), yielding $L_{\text{DMD}}=0.12$.

Pass $\mathbf{z}_f$ to the discriminator; it outputs $D(\mathbf{z}_f)=0.42$.

Generator loss $L_{\text{adv}}^{G}= -\log(0.42)=0.87$.

Discriminator loss $L_{\text{adv}}^{D}= -\log(0.78)-\log(1-0.42)=0.25+0.88=1.13$ (assuming $D(\mathbf{z}_r)=0.78$).

Student total loss $L_{\text{student}} = 0.12 + \lambda_{\text{adv}} \times 0.87$ (with $\lambda_{\text{adv}}=1$).

The KL term aligns the overall distribution, while the GAN term sharpens individual samples, together enabling a few‑step student to match a multi‑step teacher.

Why combine a KL loss with an adversarial loss instead of using only one?

The KL term guarantees that the student’s entire conditional distribution matches the teacher’s, preventing mode collapse; the adversarial term encourages realistic high‑frequency details that KL alone may smooth out.

WorldArena 2.0 Performance

DreamX-Phi 1.0 dominates the WorldArena 2.0 leaderboard across both tracks.

DreamX-Phi 1.0 adapts large‑scale video diffusion to robotic manipulation by conditioning on bimanual action trajectories and enforcing 3D geometric consistency.

The WorldArena Benchmark evaluates video‑conditioned robot policies on two tracks: overall visual‑motion‑content quality (Track 1) and precise physics‑driven task success (Track 2).

DreamX-Phi-1.0-FDM-0730 ranks first on WorldArena 2.0 Track 1 with an EWMScore‑P of 60.65.

The official leaderboard snapshot (commit cb8f9c2) lists our model at the top of the Top 3 ranking with a score of 60.65, surpassing all other entries.

**Table 2.** WorldArena 2.0 Track 1 leaderboard comparison at the August 12, 2026 snapshot. We report the official Top 3 and selected open-source reference systems. Scores use a 0–100 scale, and EWMScore-P averages the 15 component scores. Boldface and underlining denote the best and second-best results among the displayed systems, respectively.

**Table 3.** WorldArena 2.0 Track 2 results at the August 12, 2026 snapshot. Values are Adjust Bottle success rates (%) for the official Top 3 and selected open-source reference systems; higher is better. Our submission is highlighted. Boldface and underlining denote the best and second-best results among the displayed systems, respectively.

**Figure 3.** Qualitative WorldArena 2.0 Track 1 rollouts from DreamX-Phi 1.0. Each row is a predicted episode, with frames sampled in temporal order from left to right. (a) Standard RoboTwin 2.0 scenes. (b) Domain-randomized scenes with varied backgrounds, textures, lighting, and distractor objects.

WorldArena 1.0 Evaluation

DreamX-Phi-1.0-FDM-0730 tops the WorldArena 1.0 leaderboard with a 76.88 EWMScore-P.

DreamX‑Phi‑1.0‑FDM‑0730 achieves the highest EWMScore‑P of 76.88 on WorldArena 1.0 Track 1, out‑performing the official leaderboard top‑3.

Table 6 shows DreamX‑Phi‑1.0‑FDM‑0730 leading with 76.88, while the next best model scores lower on the same aggregate.

**Table 1.** Performance comparison of various models across Physics Adherence, 3D Accuracy, and Controllability metrics, with the final EWM Score-P.

On the complete 31-entry Track 1 leaderboard, our entry ranks first with an EWMScore-P of 60.65. On Track 2, DreamX-Phi-1.0-FDM-0730 achieves a 67.19% Adjust Bottle success rate and ties for second place in the full snapshot. Alpha-World and FlowWAM-FiveAges complete the Track 1 Top 3, while WOVR-PLUS leads Track 2 and Lute shares second place with our submission. These rankings are snapshot-specific and do not represent the final challenge standings.

**Table 4.** WorldArena 1.0 Track 1 comparison. Our offline evaluation of DreamX-Phi-1.0-FDM-0730 is shown alongside the official leaderboard Top 3 and selected open-source reference systems from the July 15, 2026 snapshot. Scores use a 0–100 scale, and EWMScore-P averages the 15 component scores. Higher is better; boldface and underlining denote the best and second-best results among the displayed systems, respectively.

Extended Model Comparisons

DreamX-Phi-1.0-FDM-0730 tops the WorldArena 1.0 leaderboard with a 76.88 aggregate score, beating the prior best by 3.24 points.

We evaluated a broad suite of 34 models, including DreamX-Phi-1.0-FDM-0730, UNIS, SisyphusWorld, and many others. Performance was measured across fifteen metrics such as Physics Adherence, 3D Accuracy, and Interaction Quality, each scored out of 100. The full set of Physics Adherence scores is listed in the appendix.

DreamX-Phi-1.0-FDM-0730 achieves the highest overall score on the WorldArena 1.0 benchmark.

It scores 76.88, surpassing the previous best UNIS (73.64) by 3.24 points.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers