GigaWorld-Policy-0.5: a Faster and Stronger WAM Empowered by AutoResearch

Angen Ye, Angyuan Ma, Boyuan Wang, Chaojun Ni, Fangzheng Ye, Guan Huang, Guo Li, Guosheng Zhao, Haodong Yan, Hengtao Li, Jiwen Lu, Kai Wang, Mingming Yu, Qitang Hu, Qiuping Deng, Songling Liu, Xiaoyu Tian, Xiaofeng Wang, Xinyu Zhou, Xiuwei Xu, Xinze Chen, Yang Wang, Yejun Zeng, Yifan Chang, Yun Ye, Zhenyu Wu, Zhanqian Wu, Zheng Zhu

GigaWorld-Policy-0.5 uses a Mixture-of-Transformers architecture to decouple visual dynamics from action generation for low-latency robot control.

How can we make World Action Models (WAMs) faster and more effective for real-world robotics by automating the research and hyperparameter tuning process?

World Action Models (WAMs) improve robot policies by using future visual predictions as dense supervision, but they typically require expensive, real-time video generation during inference that makes closed-loop control sluggish. GigaWorld-Policy-0.5 solves this by using a Mixture-of-Transformers (MoT) architecture that separates visual dynamics modeling from action generation, allowing the model to skip video prediction entirely during deployment. This approach achieves 85 ms inference latency on an RTX 4090 while outperforming previous WAMs and Vision-Language-Action (VLA) baselines on long-horizon manipulation tasks.

Paper Primer

The core mechanism is an action-centered causal token structure that prevents future visual information from leaking into action prediction during training. By separating the model into a visual expert and a lightweight action expert, the system retains the physical grounding benefits of world modeling while enabling a streamlined, action-only inference pathway.

GigaWorld-Policy-0.5 significantly improves long-horizon task success rates compared to existing WAM and VLA baselines.

End-to-end evaluation on food heating and tableware arrangement tasks. An average success rate of 0.80, representing a 33% relative improvement over the strongest baseline.

The MoT architecture enables low-latency deployment suitable for real-time edge robotics.

Inference latency measured on a local RTX 4090 using a C++ runtime. 85 ms latency, which is 23% faster than $\pi\$0.5 and 53% faster than FastWAM.

Why use a Mixture-of-Transformers (MoT) architecture instead of a standard shared Transformer backbone?

The MoT structure allows the model to specialize separate experts for visual dynamics and action generation. This specialization enables a lightweight action-only inference pathway that avoids the computational cost of processing visual tokens during deployment.

How does the AutoResearch pipeline contribute to the model's performance?

AutoResearch automates the hyperparameter search process by running pilot training sessions to identify optimal learning rates and batch sizes. This systematic approach reduces manual intervention and ensures the model is tuned for both action prediction and visual dynamics objectives simultaneously.

Researchers can now leverage the dense supervision of world models for robot policy learning without the prohibitive latency costs of pixel-level future rollout, provided they adopt an action-centered, expert-separated architecture.

Introduction and Motivation

We expose the speed‑accuracy trade‑off of action‑centered world models for real‑time robot control.

World Action Models (WAMs) tightly couple robot actions with predicted future scenes, yielding richer supervision but incurring heavy compute and hyper‑parameter fragility that block real‑time closed‑loop use.

A WAM learns to predict both the next robot command and the ensuing visual frame, so the action is directly grounded in its imagined future effect.

**Figure 1.** Comparison of *GigaWorld-Policy-0.5* with baselines on inference frequency and success rate across hardware platforms in real-world settings.

The central tension is preserving predictive accuracy from future visual supervision while meeting real‑time latency constraints for robot control.

Related Work

Survey of prior world‑model approaches and their relevance to robot control.

This section situates our approach among prior world‑model systems, highlighting how they handle prediction, control, and efficiency.

Motus is a latent‑action world model that packs visual and action dynamics into a single transformer‑style backbone, enabling one model to predict both future frames and robot commands.

Fast‑WAM streamlines the World Action Model pipeline by discarding the explicit video rollout at test time and directly emitting actions from the latent world representation.

Uses free‑form text actions to steer video world models, enabling interactive prediction beyond passive generation.

Extends action‑conditioned video generation to continuous robot actions, introducing action‑scaled classifier‑free guidance.

Transfers visual experiences across domains by leveraging 3D spatial cues to improve geometric consistency.

Employs multimodal spatial conditions to generate cross‑domain embodied visual data.

High‑fidelity video‑and‑3D engine that produces temporally coherent embodied trajectories with fine‑grained control over appearance and action semantics.

Scales action‑conditioned embodied video generation across manipulation, navigation, and driving domains.

Unifies geometry‑aware world modeling by jointly optimizing 4D reconstruction, action‑conditioned video prediction, and goal‑conditioned visual planning.

Formulates control as future video generation followed by action recovery, using a single predictive model.

Combines large‑scale video pretraining with action prediction to achieve generalizable manipulation.

Jointly models robot actions and future visual dynamics, providing dense temporal supervision beyond sparse demonstrations.

Adapts pretrained video generation models for robotic manipulators, jointly predicting visual outcomes and action sequences.

Couples video and action diffusion within a unified framework, supporting scalable policy learning from heterogeneous data.

Extends Motus with multiview modeling and cross‑embodiment action representations for real‑time deployment.

Uses an Internet‑scale video backbone to produce latent video plans, decoded into actions via a flow‑matching inverse‑dynamics model.

Autoregressive video‑action world model with closed‑loop rolling prediction to mitigate error buildup.

Distills multi‑step video foresight into efficient geometric and semantic representations for fast inference.

Uses intermediate denoising features rather than decoded frames for action prediction, reducing computational load.

Exposes future dynamics through compact latent visual subgoals instead of pixel‑level video.

Incorporates predictive world knowledge into policies while avoiding explicit video rollout during execution.

Lightweight action‑centered world model that retains dense supervision but eliminates pixel‑level inference.

Action‑only execution pathway built on the World Action Model formulation, targeting real‑time closed‑loop control.

Collectively, these works illustrate a trajectory from pure video generation toward increasingly efficient, action‑focused world modeling, setting the stage for our GigaWorld‑Policy‑0.5 design.

The GigaWorld-Policy-0.5 Architecture

Method introduces the MoT architecture and AutoResearch to make action‑centered world models fast and robust.

World Action Models achieve strong policy performance but their dense future‑frame supervision makes inference expensive and their many hyper‑parameters are brittle in real‑time settings.

MoT routes each token to a specialist expert—visual tokens go to a visual expert, action tokens go to an action expert—while keeping a single shared token sequence for joint modeling.

Token classifier sends $v_1, v_2$ to the Visual Expert and $a_1, a_2$ to the Action Expert.

Visual Expert applies self‑attention on $\{v_1, v_2\}$, producing latent visual outputs $v'_1, v'_2$.

Action Expert applies self‑attention on $\{a_1, a_2\}$, producing latent action outputs $a'_1, a'_2$.

Multi‑modal self‑attention takes the concatenated sequence $[v'_1, v'_2, a'_1, a'_2]$ and allows each token to attend across modalities under the causal mask.

The final joint representation is fed to the policy head to predict the next action chunk.

By separating processing, MoT eliminates unnecessary visual‑to‑visual attention when only action information is needed, cutting compute roughly in half for token‑balanced inputs.

How does MoT differ from a standard mixture‑of‑experts (MoE) layer?

MoE selects a subset of expert networks per token but still mixes their outputs; MoT deterministically routes tokens to modality‑specific experts and then explicitly merges the streams with a shared self‑attention layer, preserving a single token order.

AutoResearch automatically searches over flow‑shift factors and other hyper‑parameters to find a configuration that yields low latency while keeping success rates high.

Candidate $(\gamma_a=0.5,\gamma_v=0.3)$ yields 12 ms latency but 78 % success.

Candidate $(\gamma_a=1.0,\gamma_v=0.8)$ yields 20 ms latency but 92 % success.

Bayesian optimizer proposes $(\gamma_a=0.8,\gamma_v=0.5)$, achieving 15 ms latency and 88 % success.

Full model is fine‑tuned with $(\gamma_a=0.8,\gamma_v=0.5)$ and deployed.

AutoResearch discovers a sweet spot where a modest increase in latency yields a disproportionate boost in success, something a manual grid search would miss.

Why not simply hand‑tune the flow‑shift factors instead of using AutoResearch?

Hand‑tuning requires exhaustive evaluation on the target hardware for each candidate, which is infeasible given the continuous nature of $\gamma$ and the interaction with other hyper‑parameters; AutoResearch automates this exploration and converges in far fewer trials.

The visual expert encodes the composite multi‑view observation into a set of latent visual tokens using a pretrained VAE.

The action expert takes action‑chunk tokens and denoises them to predict future robot actions.

The causal mask permits action tokens to attend to current visual and state tokens but blocks them from future visual tokens, preventing leakage of information that would not be available at control time.

**Figure 2.** Overview of GigaWorld-Policy-0.5, an MoT-based action-centered World Action Model. The model consists of a visual expert and an action expert: the visual expert specializes in processing video tokens, while the action expert focuses on action-token modeling. The two experts are connected through multi-modal self-attention, which follows the same causal masking strategy as GigaWorld-Policy [55] to preserve action-centered dependency modeling.

KV‑cached autoregressive action decoding.

Experimental Results

GigaWorld-Policy-0.5 delivers a 33% relative boost in long‑horizon success, reaching 0.80 SR.

World Action Models improve policy learning but incur high compute and hyper‑parameter sensitivity; the action‑centered Mixture‑of‑Tokens architecture and automated search aim to fix that.

GigaWorld‑Policy‑0.5 raises long‑horizon task success by 33 % relative to the strongest baseline.

Table 3 shows an average SR of 0.80 versus 0.60 for the best competing method.

**Figure 3.** Real-world demonstration of GigaWorld-Policy-0.5 on the Tableware Arrangement task.

**Figure 4.** Real-world demonstration of GigaWorld-Policy-0.5 on the Food Heating task.

Ablation and Sensitivity Analysis

We revisit the core premise and then show how each component impacts performance.

The paper showed that World Action Models benefit from dense visual supervision, yet their cost hinders real‑time use; here we isolate which design choices actually deliver the reported gains.

Mixing Action‑Conditioned World‑Model (AC‑WM) pretraining with standard WAM pretraining raises final success rate by roughly 10 %.

Figure 5 plots success rates over training steps, showing the AC‑WM‑augmented model reaches 85 % versus 75 % for the WAM‑only baseline.

**Figure 5.** Success rates at different training steps in the AC-WM ablation study.

The MoT‑based architecture cuts A100 inference latency from 229 ms to 189 ms, a 17.5 % speedup.

Latency measurements reported in the text compare FastWAM (229 ms) with GigaWorld‑Policy‑0.5 (189 ms) under identical torch‑compiled settings.

**Figure 6.** AutoResearch hyperparameter search and training progression on the *pick the fruit* task. Left: 1K-step pilot sweep over learning rate and batch-size configurations, where green bars indicate retained candidates and red bars indicate discarded configurations. Right: extended training progression under the selected configuration, showing that the model achieves the best validation action MSE at 30K steps.

AutoResearch selects a learning rate of $6\!\times\!10^{-5}$, achieving the lowest evaluation action MSE of 0.409764.

Table 5 lists the four candidates; the $6\!\times\!10^{-5}$ row has the smallest Eval Action MSE (0.409764) and the lowest training action loss (0.252476).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers