OpenWAM: An Open, Modular Exploration towards Systematic World–Action Model Pretraining

Yuran Wang, Siqiao Huang, Mingleyang Li, Chenhao Zhang, Jiaqi Liang, Weiyang Jin, Yue Chen, Xuemin Chi, Donghao Zhou, Qize Yu, Yu-Kai Wang, Yuhan Rui, Shenzhe Yao, Zhen Yuan, Zhenhao Shen, Kefei Zhu, Zijie Zhu, Ning Gao, Xiaowei Chi, Guanqi He, Shanghang Zhang, Hao Dong, Lin Shao, Hang Zhao

OpenWAM provides a modular research stack and a pretrained model to systematically study and scale World–Action Models.

How can we modularize World–Action Models (WAMs) to decouple generative backbones from embodied control, and what design principles govern their effective pretraining?

World–Action Models (WAMs) aim to bridge the gap between video-generative world knowledge and executable robot control, but existing systems are monolithic and tightly coupled. This makes it impossible to isolate which design choices—such as visual encoders, information flow, or training data—actually drive performance. OpenWAM factorizes the WAM design space into composable modules with unified training, inference, and evaluation protocols. This modularity turns WAM development into a controlled experimental program, allowing researchers to swap backbones and attention masks to isolate the mechanisms of world–action synergy. The resulting model, OpenWAM-$\alpha$, achieves top-tier performance across eight simulation benchmarks and real-robot tasks, demonstrating that principles derived from controlled studies scale effectively to heterogeneous data.

Paper Primer

OpenWAM-Infra decouples the WAM design space into four independent components: a composable model, a training runtime, a deployment runtime, and an evaluation protocol. The core move is to treat the architecture as a configuration of interchangeable encoders and stream backbones, connected by explicit visibility attention masks that control cross-modal information flow.

The system uses a joint flow-matching objective where video and action streams are noised independently, allowing the model to learn the relationship between physical dynamics and control signals. At inference, the deployment runtime serves these checkpoints through a unified policy server that supports both synchronous and asynchronous denoising schedules.

World–action synergy is driven by explicit world-to-action information flow during training and synchronized joint denoising at inference.

Controlled ablations on RoboTwin2.0 show that models with action-sees-video visibility outperform isolated baselines by ~5 points, while asynchronous denoising schedules fail to improve upon the synchronized diagonal. The tri-system architecture with joint self-attention achieves the best overall performance, with the 5B-parameter backbone providing the optimal balance of performance and efficiency.

Representation encoders can match or exceed the performance of reconstructive encoders when paired with dimension-contracting adapters.

DINOv3 and V-JEPA 2.1, when compressed via S-VAE to match the latent dimension of reconstructive encoders, achieve on-par success rates on manipulation tasks. The study identifies that latent compactness and information density—not the specific encoder objective—are the primary determinants of WAM performance.

Why is a modular infrastructure necessary for WAM research?

Existing monolithic systems couple architecture, training, and data, which confounds experimental results. OpenWAM-Infra isolates these variables, allowing researchers to determine which components actually transfer world knowledge and which interactions create synergy.

Does embodied pretraining improve in-domain fitting or generalization?

The study finds that embodied pretraining primarily improves out-of-domain generalization. Human egocentric video broadens world coverage, while robot trajectories provide the necessary action grounding.

Researchers can now treat WAM design as an empirical science, using the OpenWAM stack to isolate the impact of specific architectural choices and data recipes on real-world robot performance.

Introduction: The Monolithic Bottleneck

We expose the monolithic bottleneck of World–Action Models and introduce OpenWAM for modular, scalable embodied learning.

“Knowledge is the beginning of action, action is the completion of knowledge.” — Yang‑ming Wang (1963). Modern embodied agents must not only recognize the world but also anticipate and enact changes, yet existing World–Action Models are monolithic, tightly coupling generative backbones, representations, and control pipelines.

A WAM predicts how visual scenes evolve over time while simultaneously learning executable control signals that steer those predicted futures.

OpenWAM‑Infra factorizes the WAM design space into interchangeable modules—model architecture, visual encoder, action head, training data, inference pipeline, and evaluation protocol—enabling systematic comparison across designs. Guided by this infrastructure we pose three central questions: (1) What world knowledge should a WAM inherit? (2) How can world‑action synergy be created? (3) How can this synergy be scaled across domains?

Controlled experiments (OpenWAM‑Study) reveal three design principles: (1) Upstream knowledge transfers best through a sufficiently capable generative backbone and a compact, information‑rich visual latent space. (2) World‑action synergy requires dedicated action capacity, an explicit world‑to‑action information pathway, and a joint test‑time denoising schedule. (3) Embodied pretraining chiefly boosts out‑of‑domain generalization, with a single‑stage co‑training on egocentric and robot data yielding the strongest integration. Applying these principles, OpenWAM‑$\alpha$ is pretrained on ~518 M frames (≈6,400 h) of mixed human and robot data and attains top‑tier performance on eight simulation benchmarks and real‑robot tasks spanning single‑arm, bimanual, and dexterous‑hand embodiments.

**Figure 1** Overview of OpenWAM. OpenWAM-Infra (left) factorizes world-action modeling into composable modules with unified training, deployment, and evaluation; OpenWAM-Study (middle) resolves the design space through controlled questions and distills a pretraining recipe; OpenWAM-$\alpha$ (right), pretrained on 518.5M frames of egocentric and robot data, sustains top-tier performance from simulation to the real world.

Shifting from monolithic to modular World–Action Models unlocks systematic scaling of generative priors and embodied control.

Modular Infrastructure Design

OpenWAM‑Infra factorizes world‑action models into interchangeable modules with clean interfaces.

Monolithic world‑action models tie architecture, training, and deployment together, making extensions brittle and obscuring causal effects in controlled studies.

OpenWAM‑Infra splits a world‑action model into four independent pieces—visual encoder, stream backbones, visibility mask, and runtime utilities—so each piece can be swapped or upgraded without breaking the others.

Instantiate E and freeze its parameters; it maps a 64×64 RGB frame to a 256‑dim latent vector.

Pass the latent vector to W, which runs three transformer layers and outputs a predicted future latent sequence.

Simultaneously feed the same latent vector to A, which produces an action token sequence in parallel.

Apply M (mutual) so that video tokens attend to action tokens and vice‑versa during each transformer layer.

Collect the video prediction and action token outputs; the policy server later reads them to generate a robot command.

By swapping any of the four items—e.g., replacing E with DINOv3—researchers can isolate the effect of visual representation without retraining the backbones or altering the mask.

**Figure 2** OpenWAM Model Infra. Top: the three classes of interchangeable modules: the visual encoder $\mathcal{E}$ (left); the stream backbones $\mathcal{S}$ (middle); and the visibility attention mask $\mathcal{M}$ (right). The central Training Utils panel summarizes the utilities of the training runtime (Section 3.2). Bottom: the composition rule $C$ assembles the modules into six architecture variants across the Single-System, Dual-System, and Tri-System families.

How does OpenWAM‑Infra differ from a typical monolithic WAM that simply adds new layers?

In a monolithic WAM, adding a new encoder or backbone rewrites the whole forward graph, forcing a full retraining. OpenWAM‑Infra isolates each addition behind a stable API, so the new component plugs in without touching the other three pieces, and only the affected module needs fine‑tuning.

Training and Deployment Runtime

Unified training treats each architecture as a black box, enabling consistent runtime across models.

Training and deployment must run efficiently despite the many architecture families OpenWAM‑Infra supports. A monolithic trainer would need to know each backbone’s internals, breaking modularity and scaling. Instead, the authors design a single trainer that treats any architecture as a black box.

The trainer asks each architecture to prepare its inputs and run a forward pass, then optimizes a joint flow‑matching loss that samples independent noise timesteps for video and action, guaranteeing that any synchronous or asynchronous inference schedule stays in‑distribution.

Compute noisy video latent: $z_{tv}=0.5\cdot[1.0,\,-1.0]+0.5\cdot[0.2,\,-0.2]=[0.6,\,-0.6]$.

Compute noisy action: $a_{ta}=0.5\cdot[2.0]+0.5\cdot[0.1]=[1.05]$.

Forward pass yields predictions $\hat{v}_z=[0.58,\,-0.58]$, $\hat{v}_a=[1.00]$ (illustrative values).

With $\lambda_v=\lambda_a=1$ and $w(0.5)=1$, the video loss term is $\|[0.58,\,-0.58]-( [1.0,\,-1.0]-[0.2,\,-0.2])\|_2^2=\|[0.58,\,-0.58]-[0.8,\,-0.8]\|_2^2\approx0.097$.

Action loss term (mask $m=1$) is $\|[1.00]-( [2.0]-[0.1])\|_2^2=\|[1.00]-[1.9]\|_2^2=0.81$.

Total loss $L\approx0.097+0.81=0.907$.

Because $tv$ and $ta$ are sampled independently, the trainer simultaneously observes mixed‑noise states, guaranteeing that any later schedule—whether video leads, action leads, or they denoise together—remains covered by the training distribution.

How does this joint training differ from training each modality separately?

Separate training would sample a single noise timestep and optimize a loss for only that stream, leaving the other stream untouched. The joint formulation samples $tv$ and $ta$ independently, so every forward pass provides gradients for both video and action, and the loss simultaneously enforces consistency across the full two‑dimensional noise plane.

**Figure 3. Inference modes and denoising schedules of the deployment runtime.** (a) Illustration of the synchronous and asynchronous inference modes. (b) Illustration of the three denoising schedules (variance shift, linear offset, and sync); five denoising steps are drawn for illustration, and circles of the same color denote the timesteps that the two modalities reach at the same denoising step.

Initialize DeepSpeed ZeRO (stage 1 or 2) via Accelerate, enabling mixed‑precision (bf16) and gradient accumulation.

Activate gradient checkpointing; optionally offload checkpointed activations and optimizer state to CPU to reduce GPU memory.

Select a workflow: start from scratch (Pretraining), load a checkpoint and add a new configuration (Fine‑tuning), or resume the exact previous run (Resume).

Run the unified trainer: each architecture prepares inputs, performs a forward pass, and the joint loss $L$ is back‑propagated.

Save a self‑contained checkpoint containing model weights, full configuration, and action‑normalization statistics.

Deployment and Evaluation

Deployment runtime offers flexible inference modes, denoising schedules, and four accelerations.

Deployment runtime decouples when inference runs from how the two streams are denoised, and provides four orthogonal accelerations to shrink request latency.

OpenWAM‑Infra offers two orthogonal inference modes that trade latency for pre‑fetching work, much like a pipeline that loads the next batch while the current one is being consumed.

How does OpenWAM‑Infra’s asynchronous inference differ from a typical async request queue?

Typical async queues simply enqueue requests and process them later, leaving the client to wait for the next completed request. OpenWAM‑Infra’s asynchronous mode proactively prefetches the next action chunk while the current buffered actions are still being executed, so the client never experiences an empty buffer and latency is hidden behind the ongoing execution.

Denoising schedules define how the video and action streams traverse the joint noise plane, either together (synchronous) or with one stream leading (asynchronous), analogous to two runners on a track where one may start ahead.

Four independent tricks speed up the serving path without altering the model itself, similar to adding specialized shortcuts to a highway.

Latency is dominated by the inference mode; asynchronous prefetching and the four accelerations together shrink per‑request latency on the modular server.

Evaluation Protocol

Evaluation protocol measures latency speedups and integrates eight benchmarks via a unified client‑server pipeline.

The Dual‑System Joint Cross Attention architecture attains up to 2.63× inference speedup over the baseline.

Figure 4 demonstrates the latency breakdown and speedup factors across six architectures.

Benchmarks act as thin clients over a persistent WebSocket, sending observations (up to three camera views, language instruction, and optional robot state) and receiving a single action from the policy server.

A fixed 80‑dimensional vector can encode any robot’s action by mapping each embodiment’s native dimensions into predetermined slots.

**Figure 4. Serving latency across architectures.** Inference latency with Wan2.2-TI2V-5B as the video backbone on an RTX 5090. The prompt-embedding cache and video-decode skip are enabled by default; the figure ablates compilation and the DiT velocity cache.

Design Principles for WAMs

We ablate key design choices to see how each component impacts WAM performance.

We build on OpenWAM‑Infra to systematically probe which design choices matter for World–Action Models. Section 4.1 examines how upstream world priors should be inherited, while Section 4.2 explores synergy between world and action learning, and Section 4.3 tests cross‑domain generalization.

A binary mask that selectively blocks attention flow between modalities, ensuring that only allowed token pairs can attend to each other.

How does the Visibility Attention Mask differ from a standard causal mask?

A causal mask only blocks future tokens within the same modality; the Visibility Attention Mask can also block or permit cross‑modality attention, e.g., allowing video tokens to see action tokens or keeping them isolated.

**Figure 6.** WAM Performance with Different Video Backbone Size. With increasing video generation backbone size, performance of the resulting WAM consistently improves.

Using a 5 B video generation backbone retains most of the performance of a 14 B backbone.

Wan2.2‑TI2V‑5B trails Wan2.1‑I2V‑14B by only 1.40 percentage points on RoboTwin2.0‑Full.

**Figure 7. Visual representation priors.** We consider building WAMs with both reconstructive and representation encoders, and include a variant of representation encoders with S-VAE (Zhang et al., 2025a), an adapter that converts high-dimensional latents produced by representation encoders into low-dimensional vectors suitable for DiT processing.

Adding an S‑VAE adapter lets representation encoders beat the reconstructive FLUX.2‑VAE baseline.

DINOv3 w/ SVAE achieves 90.18 % success versus FLUX.2‑VAE’s 84.26 %.

**Figure 8. Attention Masking Strategies.** We control cross-modality information flow at training time via attention masking.

Effective inheritance of world knowledge relies on a capable generative backbone and a compact, information‑rich visual representation space.

Generative backbones improve with capacity, and representation encoders with S‑VAE match or exceed reconstructive encoders.

Synergy in World and Action Learning

Evaluating how architectural choices and information flow affect world–action synergy.

Synergy between world and action learning hinges on three decisions: where action‑specific capacity resides, which cross‑modal paths are active during training, and whether inference preserves the noise‑state relationship learned at training time.

Increasing architectural capacity from single‑ to dual‑ to tri‑system improves success rates, with the tri‑system joint self‑attention reaching the highest clean performance.

Table 1 shows the tri‑system joint self‑attention variant achieving 92.84 % clean success, the top value among all configurations.

**Table 1. Architecture Ablation.** Averaged success rates (%) on RoboTwin2.0-Full. Bold denotes best values.

World‑to‑action information flow during training is essential; the Action‑Sees‑Video mask yields 92.98 % clean success, outperforming isolated and video‑sees‑action masks by roughly five points.

Table 2 reports 92.98 % clean success for the Action‑Sees‑Video mask, the highest among the four masking strategies.

**Table 2.** Action learning requires access to world information. Success rates (%) on RoboTwin2.0-Full.

At inference, synchronized denoising—where video and action streams share the same noise level—outperforms all asynchronous schedules, showing that explicit world‑to‑action flow need not be enforced beyond training.

Cross-Domain Knowledge Consolidation

Cross-domain pretraining boosts out-of-domain success and favors mutual world–action visibility.

Embodied pretraining primarily expands out‑of‑domain (OOD) generalization.

Figure 10 reports OOD success rising from 14.5 % (from‑scratch) to 26.6 % (Ego + Robot), a +12.1 % gain.

Robot‑only pretraining delivers the highest in‑domain success (≈ 88.5 %), while both mixed strategies (sequential ego→robot and one‑stage ego + robot) achieve the strongest OOD performance. The two mixed approaches are virtually indistinguishable, so the simpler one‑stage co‑training is adopted as the default.

**Figure 9** Inference-time Information Flow via Denoising Schedule. Each curve traces action denoising progress against video denoising progress. Curves above/below the diagonal denoise action/video first, respectively.

**Figure 10.** Embodied Pretraining Primarily Improves OOD Generalization. Success rates on RoboTwin2.0-Clean2Random, ordered from lower to higher performance within each evaluation setting.

The OpenWAM-α Model

OpenWAM‑$\alpha$ factorizes the monolithic World–Action Model into modular video and action streams.

Recall that monolithic World–Action Models tie generative priors to embodied control, preventing independent scaling. OpenWAM‑$\alpha$ resolves this by factorizing the model into composable video and action modules.

OpenWAM‑$\alpha$ splits the monolithic WAM into two parallel diffusion streams—one for video, one for action—while letting them read each other through a shared attention mask, so each stream can be pretrained and scaled independently.

Patch embedding flattens each latent frame into a token; RoPE encodes (frame, height, width) coordinates.

Action tokens receive a 1‑D RoPE index (step 0, step 1).

Both streams are projected into the shared attention space; AdaLN adds timestep $t_v=0.6$ to video tokens and $t_a=0.4$ to action tokens.

At layer 1, the video token for frame 1 attends to the action token for step 0, while the first‑frame token is masked from future frames and actions.

After $30$ layers, the joint attention produces refined latent video and action representations ready for decoding.

The toy example shows how independent token sizes coexist in a shared attention matrix, and how the visibility mask protects the causal video token while still enabling cross‑stream information flow.

**Figure 12.** Overview of OpenWAM-$\alpha$. (a) The dual-system architecture: a video-generation DiT and an ActionDiT jointly denoise the future frames and the action chunk through shared attention under the mutual visibility mask, each conditioned on language and proprioception via cross-attention and carrying its own noise timestep. (b) The pretraining mixture: 518.5M frames (6,369 hours) of egocentric and robot data, co-trained in one stage. (c) Timestep sampling: training covers the full noise plane, while inference follows the synchronized diagonal. (d) The 80-D unified action space with fixed slot semantics shared across embodiments.

**Figure 11** Pretraining Changes the Preferred Information Flow. Central markers give the absolute success rate of Action Sees Video; arrows terminate at the matched Mutual result, with horizontal displacement reporting Mutual - Action Sees Video in percentage points. Green and red denote gains and drops, respectively.

**Table 3.** The recipe accumulated by OpenWAM-Study. Each evidence-backed choice becomes the default for OpenWAM-$\alpha$.

How does OpenWAM‑$\alpha$ differ from a standard dual‑stream diffusion model that simply concatenates video and action tokens?

Standard dual‑stream diffusion typically shares a single attention matrix without any masking, so future video frames can attend to each other and to action tokens, leaking information. OpenWAM‑$\alpha$ introduces the mutual visibility mask $M$, which explicitly blocks the first‑frame video tokens from attending to any noised future frames or action tokens, preserving causality while still allowing full cross‑stream interaction elsewhere. This mask is the key deviation that enables reliable joint prediction.

Pretraining Data and Scaling

We quantify the curated pretraining corpus and show OpenWAM‑$\alpha$’s benchmark edge.

OpenWAM‑$\alpha$ achieves state‑of‑the‑art performance on the mobile bimanual benchmark EBench, surpassing the runner‑up Qwen‑RobotManip by 4 points.

Figure 14 shows OpenWAM‑$\alpha$ leading the best VLA and WAM models across all eight benchmarks, with a 4‑point gap on EBench.

Embodied Pretraining teaches the model to predict future visual observations while simultaneously learning executable robot actions from a blended corpus of human, real‑robot, and synthetic robot data.

**Figure 14.** OpenWAM-$\alpha$ against the best of each family, per benchmark, grouped by embodiment.

**Figure 15.** Single-arm pretraining data of ABot-M0.5, Being-H0.7, and OpenWAM-$\alpha$. The dashed lines indicate that the single-arm data of OpenWAM-$\alpha$ amounts to only a small fraction of what ABot-M0.5 and Being-H0.7 consume.

Real-Robot Evaluation

OpenWAM‑$\alpha$ dominates real‑robot benchmarks, achieving the highest average success rates.

OpenWAM‑$\alpha$ attains the highest average success rate across all real‑robot benchmarks, outperforming both representative WAMs and VLAs.

Tables 6, 7, and 8 show OpenWAM‑$\alpha$ leading on single‑arm, bimanual, and dexterous‑hand tasks respectively.

The three embodiment evaluations probe complementary capabilities: single‑arm tests fine‑grained manipulation, bimanual tests long‑horizon coordination, and dexterous‑hand tests unseen action spaces. Across all settings OpenWAM‑$\alpha$ consistently leads, confirming its ability to generalize from pretraining to novel real‑world robots.

**Figure 16** ID and OOD comparisons between the two paradigms. (a) Fast-WAM versus StarVLA, two models without embodied pretraining, on ID and OOD splits. (b) OpenWAM-$\alpha$ versus the three strongest VLAs on ID splits. (c) OpenWAM-$\alpha$ versus the three strongest VLAs on OOD splits.

**Figure 17** Real-robot experimental setups across three embodiments. *Top*: the six single-arm tasks on the Franka-Research-3 platform. *Bottom left*: the three bimanual embodiments of the RoboDojo real-world track (Piper X, Piper, and ARX X5), covering 18 tasks in total. *Bottom right*: the four dexterous-hand tasks on the Wuji-hand and Tianji-arm platform, each illustrated by key intermediate stages of its execution.

Limitations and Implementation Details

Supplementary limits, training configs, evaluation protocols, and full benchmark scores.

This appendix expands on four pillars: (A) the work’s current limits and open research directions, (B) the exact training hyper‑parameters, (C) the real‑world evaluation setups, and (D) the complete per‑benchmark simulation scores behind Figure 13.

§A enumerates four concrete limitations: (1) we only explore the embodied pretraining phase, leaving post‑training adaptation largely open; (2) modality fusion is restricted to cross‑modality attention or hard‑routed MoE, while earlier‑fusion and soft‑routed MoE remain unexplored; (3) we omit UMI‑style data mixtures that could improve out‑of‑domain generalization; and (4) the chosen visual encoder, Wan2.2‑VAE, trades reconstruction fidelity for compactness but still struggles with viewpoint, noise, and scene variation.

§B.1 details the multi‑domain pretraining pipeline, which runs on 16 nodes (128 NVIDIA H200 GPUs) for roughly seven days, using a global batch size of 3,072 clips per optimizer step and the hyper‑parameters listed in Table 9.

§B.2 describes the downstream supervised fine‑tuning (SFT) stage; all downstream models start from the same OpenWAM‑$\alpha$ checkpoint and inherit the pretraining settings, differing only in batch size, training length, and whether image augmentation is applied, as summarized in Table 10.

§C outlines the real‑world evaluation protocols across three embodiment families: single‑arm, dexterous‑hand, and bimanual robots.

§C.1 evaluates six tabletop tasks on the Franka‑Research‑3 arm; each task’s natural‑language instruction is listed in Table 11, and after fine‑tuning on 100 demonstrations the policy is run for 20 trials per task, with success defined by task‑specific criteria (e.g., correct stacking or hanging).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers