GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

GigaWorld Team, Angyuan Ma, Boyuan Wang, Bohan Li, Chaojun Ni, Guo Li, Guan Huang, Guosheng Zhao, Hao Li, Hengtao Li, Jingyu Liu, Jiwen Lu, Qiuping Deng, Tingdong Yu, Xuancheng Xu, Xinyu Zhou, Xiuwei Xu, Xinze Chen, Xiaofeng Wang, Xiaoyu Tian, Yang Wang, Yifan Chang, Yukun Zhou, Yun Ye, Zhenyu Wu, Zhanqian Wu, Zheng Zhu

GigaWorld-1 provides a systematic roadmap and model for using world models as reliable, scalable evaluators for robot policies.

Evaluating robot policies currently requires slow, expensive real-world rollouts, creating a bottleneck that prevents rapid iteration. While world models can simulate these environments, it remains unclear which architectural and data properties make them reliable enough to serve as accurate policy evaluators. The authors introduce WMBench, a benchmark of paired real-world and simulated trajectories, to identify the design principles governing evaluator reliability. They find that long-horizon consistency and spatially aligned action control are more critical than raw visual realism. GigaWorld-1 instantiates these findings, outperforming state-of-the-art baselines by 14.9% on evaluator-alignment metrics.

Paper Primer

The core challenge is that standard video generation models prioritize visual fidelity, which often fails to capture the physical "success" or "failure" of a robot policy over long horizons. GigaWorld-1 addresses this by shifting the objective from photorealism to action-faithful, long-horizon consistency: it treats world modeling as an autoregressive continuation task, injecting robot actions through spatially aligned control maps and maintaining state via a hierarchical memory buffer.

GigaWorld-1 significantly improves the alignment between simulated policy outcomes and real-world robot performance.

Evaluator-alignment metrics on the WMBench benchmark. 14.9% improvement over competitive state-of-the-art baselines.

Evaluator reliability is dominated by long-horizon consistency and geometric alignment rather than short-term visual quality.

Correlation analysis shows that metrics like Subject Consistency ($\rho=0.88$) and Perspectivity ($\rho=0.86$) are strong predictors of outcome accuracy, while semantic alignment alone is weak ($\rho=0.11$). High correlation between specific geometric/consistency metrics and human-annotated success labels.

Why is this paper necessary if we already have high-quality video generation models?

Existing video models are optimized for visual appeal, not physical accuracy; they often suffer from viewpoint drift and object-identity collapse during long-horizon rollouts, making them unreliable for judging whether a robot policy actually succeeded.

What is the primary "design rule" for building these evaluators?

The authors conclude that evaluators must use explicit, spatially aligned control interfaces (like end-effector pose maps) and persistent memory modules to maintain object and scene stability across extended interaction sequences.

Researchers can now move beyond generic video metrics to evaluate robot policies using GigaWorld-1’s design roadmap, which prioritizes action-faithful dynamics and long-horizon stability over simple frame-level realism.

Abstract and Introduction

We expose why robot‑policy evaluation is a bottleneck and introduce WMBench to study world‑model evaluators.

Evaluating embodied robot foundation models is a critical bottleneck: unlike language models, assessing a robot policy requires repeated real‑world rollouts that are slow, hardware‑intensive, and demand continuous human supervision.

A learned world model that predicts the outcome of a robot policy without executing the robot on hardware.

A large‑scale paired dataset that aligns real‑robot executions with corresponding world‑model rollouts, enabling controlled comparison of evaluator quality.

**Figure 1.** This paper analyzes 324,000 world-model-simulated rollouts, 7 video world models, 4 action representation paradigms. Complemented by more than 12,000 hours of training data and guided by the roadmap we establish for building world models tailored to robot policy evaluation, we introduce GigaWorld-1.

Our systematic study yields three core insights: (1) long‑horizon, action‑faithful rollout consistency dominates evaluator quality; (2) pretraining benefits arise from balancing broad world knowledge with robot‑specific controllability; (3) architectural choices—action encoding, memory design, and evaluator‑focused post‑training—strongly determine alignment with real‑world behavior.

Guided by these findings we propose a practical design roadmap and instantiate it as GigaWorld‑1, which improves the core evaluator‑alignment metric by 14.9 % over strong baselines. We release code, pretrained checkpoints, datasets, and toolkits to enable scalable evaluation of embodied foundation models.

Contributions: (1) formalize world‑model policy evaluation as a research problem; (2) introduce WMBench for systematic benchmarking; (3) provide a large‑scale empirical study uncovering key design factors; (4) deliver GigaWorld‑1, a state‑of‑the‑art evaluator, and open‑source all resources.

Preliminaries

Real‑world policy evaluation is costly, so we need a simulator that preserves key outcomes.

Evaluating a robot policy $\pi$ on a physical robot is time‑consuming and risky. To speed up evaluation we let the policy interact with a learned world model M_$\theta$, but the simulator must preserve the decision‑relevant properties of the real trajectory, such as the ranking of policies, success predictions, and risk profiles.

The evaluator must assign scores that keep the ordering of policies unchanged between real rollouts and world‑model rollouts, thereby preserving success predictions and risk profiles.

What Matters Analysis

We expose which metrics, data, and design choices actually predict evaluator quality.

Question I asks how well existing visual and motion metrics predict the World‑Model Evaluation Score (WMES). Using the 324 k manually annotated rollouts, we compute Pearson correlations between each metric and WMES, with 95 % bootstrap confidence intervals.

We treat each metric as a predictor and measure how linearly its values co‑vary with the human‑rated WMES across all submissions.

Visual Fidelity group correlates 0.78 with WMES.

Correlation computed over 324 k rollouts; 95 % bootstrap CI excludes zero.

Geometry group correlates 0.71 with WMES.

Same protocol as Visual Fidelity; strong positive relationship.

Semantics group correlates 0.59 with WMES.

Correlation measured on the full submission set.

Subject Consistency metric correlates 0.88 with WMES.

Highest individual‑metric Pearson coefficient among all evaluated metrics.

Perspectivity metric correlates 0.86 with WMES.

Second‑best individual predictor after Subject Consistency.

Instruction Following metric correlates 0.84 with WMES.

Strong positive link indicates that following the task prompt is crucial.

Semantic Alignment metric correlates only 0.11 with WMES.

Weak positive signal; not a reliable evaluator.

Background Consistency correlates ‑0.45 with WMES.

Negative predictor; higher background consistency tends to lower WMES.

Photometric Consistency correlates ‑0.42 with WMES.

Negative relationship suggests that overly stable lighting harms perceived quality.

Interaction Quality correlates ‑0.11 with WMES.

Small negative coefficient; interaction cues are not helpful for WMES.

Finding 3 reveals that short‑horizon video generation quality does not guarantee reliable evaluation over long horizons; drift and texture accumulation degrade scores.

Instead of training the evaluator on per‑frame reconstruction, we teach it to predict the overall WMES token and to justify its decision with evidence tokens.

VLM evaluator achieves 87.80 % exact agreement with human WMES rankings.

Evaluated on 5 000+ videos; exact match counts where VLM score equals human score.

VLM evaluator attains 0.1304 MAE and 0.7574 Spearman correlation with humans.

Mean absolute error and rank correlation computed across the same 5 000+ videos.

Question II investigates whether larger pretrained backbones or richer training data yield better evaluators.

Cosmos‑Predict backbone attains average correlation 0.6123, outperforming larger general‑purpose models.

Average taken over all metrics on the evaluation set.

Wan 2.25 B backbone reaches 0.5948 average, beating LTX 22 B (0.5775) and SVD (0.5569).

Same metric suite applied to each backbone.

Adding broad physical videos (PhysData) to GigaData raises average score to 0.6144 (+0.0490).

Delta computed relative to GigaData‑only baseline.

Appending robot‑specific AgiBot data improves average to 0.5940 (+0.0286) but harms some motion metrics.

Observed trade‑off in per‑metric breakdown.

Finding 8 shows that robot‑domain data sharpens embodiment fidelity (e.g., Subject Consistency) while degrading structural metrics, highlighting a specialization trade‑off.

Question III asks how architectural choices affect evaluator reliability once data are fixed.

Control signals are rasterized into spatial maps and concatenated to the latent representation before denoising, ensuring early, spatially aligned guidance.

Channel‑concatenated control achieves Subject Consistency 0.8600.

Best among the four control variants on the full metric suite.

Trajectory Accuracy reaches 0.3528 with channel‑concatenated control.

Improvement over cross‑attention (≈0.30) and ControlNet (≈0.28).

Dynamic Degree improves to 0.3566 under the same setup.

Measured on the long‑horizon rollout benchmark.

Motion Smoothness climbs to 0.5747 with channel‑concatenated control.

Higher than all other control interfaces.

Finding 10 demonstrates that adding a hierarchical history buffer with persistent first‑frame anchoring dramatically lifts PSNR, FID, and FVD across 0–40 s intervals, confirming the need for long‑term memory.

**Figure 4.** Metric-group correlation with WMES. Metrics submitted to the WMBench are grouped into visual fidelity, geometry, semantics, dynamics, interaction, and appearance stability categories. Visual fidelity and geometry are the strongest group-level predictors, while appearance stability is negatively correlated with WMES.

**Figure 5.** Pearson correlation matrix over all submitted metrics. The full metric-level heatmap shows that Subject Consistency, Perspectivity, JEPA Similarity, Instruction Following, Image Quality, and Aesthetic Quality correlate strongly with WMES, whereas appearance-stability metrics and Interaction Quality are much less reliable.

**Figure 6.** **VLM-assisted Rollout Evaluator.** Given a three-view rollout video and a task-specific evaluation prompt, the LoRA-tuned Qwen3-VL evaluator predicts a WMES score and produces evidence-grounded rationales with structured aspect-level assessments of *overall video quality*, *instruction following*, and *physical adherence*.

Data Sources and Curation

Balanced, high‑quality multimodal data are essential for training reliable evaluator‑oriented world models.

Evaluator‑oriented world models require training data that is simultaneously diverse, physically realistic, and tightly aligned with robot actions. Existing robot datasets are either too narrow in embodiment or suffer from noisy, mis‑synchronized video, which hampers both visual fidelity and task relevance.

The pipeline discards low‑quality or misaligned clips, then balances the remaining videos across sources, robot types, and motion intensity, ensuring that every training example contributes useful physical and task information.

**Figure.** (a) Data Filtering and Balancing Pipeline, (b) Data Type and Robo Type Distribution, (c) Giga Data Crafter Pipeline.

Table 5 quantifies the curated corpus: ~12,980 hours total, split into ~1,298 h of generic physics videos, ~5,377 h of open‑source robot demonstrations, ~2,411 h of egocentric human‑hand recordings, and ~3,894 h of Giga‑collected robot demos covering humanoid, dual‑arm, single‑arm, and dexterous‑hand platforms.

After curation, each clip is automatically enriched with semantic masks, dense depth, and dual‑speed captions, providing the world model with geometry, object identity, and task‑level language without manual labeling.

**Table 1.** Data composition and characteristics.

Model Architecture and Training

We detail the autoregressive robot world model, its unified control, hierarchical memory, and training pipeline.

Fixed‑window diffusion models excel at short‑term denoising but cannot directly produce the multi‑second rollouts required for robot manipulation, because the context window is bounded.

We give the model a continuous, view‑specific description of what the robot will do (end‑effector pose) and how the camera will move (ray map), then stitch the two maps together so the diffusion network sees a single control tensor at every timestep.

Concatenate width‑wise: $C = [p_1, p_2, r_1, r_2]$ → a 4‑pixel control vector.

Encode with a tiny linear encoder $E$: $Z_{\text{ctrl}} = W_E \cdot C$ (where $W_E$ is a 2 × 4 matrix).

Slice a generation chunk of length $T_f=2$: $Z^{(1)}_{\text{ctrl}} = Z_{\text{ctrl}}[0:2]$.

Feed $Z^{(1)}_{\text{ctrl}}$ to the denoising network together with the noisy latent.

The concatenation lets the model attend to robot intent and camera motion in a single attention pass, avoiding the need for separate conditioning branches.

How does this unified control differ from conditioning on a single static pose vector?

A single pose vector cannot express view‑dependent geometry; the concatenated maps preserve spatial layout for each camera, allowing the diffusion network to align visual changes with the correct source of motion.

Instead of storing every past frame, we keep three memory slots (short, mid, long) plus an immutable first‑frame anchor, so the model sees both fine‑grained motion and coarse‑grained scene layout without blowing up memory.

Short memory $H^{(S)}_t = \{f_7, f_8\}$.

Mid memory $H^{(M)}_t = \{f_4, f_7\}$ (every 3 frames).

Long memory $H^{(L)}_t = \{f_1\}$ (the first frame).

Anchor $x_{\text{anchor}} = f_1$ is added, yielding $\tilde{H}_t = \{f_1, f_1, f_4, f_7, f_7, f_8\}$.

The denoising network receives $\tilde{H}_t$ together with the current noisy window.

Because the anchor repeats the original frame, the network can always reference the true initial colors and geometry, eliminating cumulative drift.

Why keep a permanent first‑frame anchor instead of just the latest short memory?

The anchor provides a fixed reference for object identity and lighting; without it, small errors accumulate and the model may gradually change colors or lose objects.

We give the self‑attention layer two separate streams: one for the noisy future window and one for the already‑generated history, then let them attend jointly while only the noisy stream receives cross‑attention from the task description.

Why not simply concatenate noisy and historical tokens and run a single standard attention?

Separating the streams lets us apply cross‑attention only where new semantic information is needed (the noisy window), avoiding redundant conditioning on already‑encoded history and reducing compute.

At each generation step we reset the temporal index to zero, so the model never sees absolute positions beyond its training horizon; the rotary operator is applied to these local indices.

Why not keep the absolute timestamps used during training?

Absolute timestamps would exceed the range seen during training once rollouts go beyond the original window, causing the model to encounter unseen positional embeddings and produce unstable motion.

To move smoothly between two textual conditions we rotate along the great‑circle on the unit hypersphere of embeddings, preserving angular relationships and thus semantic consistency.

Why not simply linearly interpolate the embeddings?

Linear interpolation can leave the intermediate vector off the unit sphere, distorting the semantic direction and causing abrupt changes in generated content; SLERP preserves the angular structure.

Stage 1 – Robot world foundation: start from a pretrained video backbone and continue training on a curated robot video corpus using the flow‑matching loss $\mathcal{L}_{\text{FM}}$.

Stage 2 – Autoregressive world modeling: convert the bidirectional model to an AR generator, train with the denoising loss $\mathcal{L}_{\text{AR}}$ while feeding hierarchical history and unified control.

Stage 3 – Optional scene adaptation: freeze the backbone and apply Low‑Rank Adaptation (LoRA) $W = W_0 + BA$ on in‑domain demonstrations.

Stage 4 – Few‑step distillation: first warm‑start with optional ODE distillation $\mathcal{L}_{\text{ODE}}$, then apply mandatory DMD2 distillation $\mathcal{L}_{\text{DMD2}}$ to obtain a fast student model.

**Figure.** The diagram illustrates a process for robot video generation using autoregressive windows and embedding manifold interpolation. Four distinct prompts (Grasp carton flaps, Gripper imposition, Press down carton lids, Reset) are mapped to specific points ($e_1, e_2, e_3, e_4$) on an embedding manifold. SLERP interpolation is performed along the manifold between these points to generate intermediate embeddings. These interpolated embeddings are then used to drive the robot video generation process, resulting in a sequence of visual frames corresponding to each prompt.

**Figure 10:** **Training pipeline of GigaWorld-1.** The model is first adapted into a robot world foundation model, converted into an autoregressive world generator, and finally compressed through optional ODE warm start and required DMD2 distillation for few-step rollout. Dashed branches denote optional modules that can be skipped.

WMBench Benchmark

WMBench quantifies world‑model fidelity and policy‑evaluation reliability via a four‑step benchmark.

WMBench provides a rigorously constructed benchmark that tests whether a learned world model can replace real‑world execution for evaluating policies. It combines a balanced dataset, a closed‑loop evaluation protocol, and a hierarchical metric system to measure both generation quality and evaluator reliability.

**Figure 3.** **WMBench evaluation pipeline.** The four-step protocol includes (1) collecting real-world policy rollouts, (2) training world models on a strict split, (3) executing closed-loop policy rollouts inside the learned world model, and (4) assessing metrics and outcomes to measure alignment with real-world conclusions.

Related Work

Survey of world‑model approaches and their role in evaluating robotic policies.

Recent work leverages video diffusion models as foundational world models, repurposing high‑fidelity video generators for predictive control in embodied tasks.

High‑quality video generators are adapted to predict future observations conditioned on control signals, enabling long‑horizon, low‑latency generation for downstream robotics.

World models have become a cornerstone of embodied AI, supporting four complementary paradigms that shape modern robotic learning pipelines.

World models are employed as (1) data engines for synthetic dataset generation, (2) embedded policies that output actions directly, (3) learned simulators for closed‑loop planning, and (4) value critics that estimate long‑horizon returns.

Evaluating robotic policies traditionally alternates between costly real‑world trials and fast but imperfect simulators, leaving a gap that world‑model‑based evaluators aim to fill.

Recent approaches treat world models as policy evaluators, offering scalable, visually realistic assessment without manual simulator engineering.

**Figure 2.** World model as policy evaluator framework. A world model serves as a policy evaluator by iteratively receiving policy actions and predicting future observations. Reliable evaluation requires not only visual quality, but also action-faithful rollout and agreement with real-world policy outcomes.

Discussion and Conclusion

We discuss why preserving policy‑relevant causality is the core challenge for robot world‑model evaluation.

Our study shows that the real bottleneck for learned robot evaluation is not video generation per se, but the ability to preserve policy‑relevant causality across iterative rollouts. Even models that produce visually convincing short clips can fail as evaluators because they drift away from the action‑conditioned dynamics needed for reliable policy comparison.

GigaWorld‑1 therefore targets a world model that remains controllable under arbitrary action inputs, retains broad world knowledge, and sustains long‑horizon consistency while staying practical for repeated use. This design cannot be reduced to a single factor such as model size or the amount of robot data.

These insights have concrete implications for benchmark design. WMBench is positioned as a reusable evaluator infrastructure rather than a static leaderboard, encouraging large‑scale experiments that reveal which modeling choices and evaluation signals truly matter for policy evaluation.

**Figure 15.** OOD generalization cases. From top to bottom, the rows evaluate generalization to object color and container appearance, object content changes, background and table-surface changes, and action-outcome variation covering both successful and failed executions.

**Figure 16.** Task-level success-rate alignment. Real-robot success rates are compared with generated success rates under closed-loop policy rollout. The gray dashed line indicates perfect agreement. GigaWorld-1 has a fitted line closer to the diagonal than the challenge baselines, indicating better calibration of task difficulty across the evaluated tasks and subtasks.

**Figure 17.** Success-rate bias across world models and subtasks. Bars show Gen - Real success-rate differences: green indicates overestimation of real-world success, red indicates underestimation, and values near zero indicate closer agreement.

We acknowledge several limitations: WMBench currently spans eight task families and does not cover the full spectrum of mobile manipulation, dexterous in‑hand tasks, or safety‑critical autonomy. Moreover, our focus on video‑centric world models leaves open the possibility that structured state‑space or hybrid 3D approaches could offer different trade‑offs.

Looking forward, we plan to expand the benchmark with more rollout data, develop finer‑grained metrics for outcome faithfulness and evaluator reliability, and explore richer multimodal training (e.g., Cosmos‑3) to improve controllability and physical grounding. Scaling model parameters will remain a complementary avenue to enhance capacity and long‑horizon robustness.

Ultimately, we hope these efforts accelerate the adoption of world models as policy evaluators in both research and industry, paving the way toward richer reward modeling, counterfactual evaluation, and certified uncertainty.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers