Native Action-Prior Learning from Videos for World Action Models
Zhaochong An, Fei Zhang, Menglin Jia, Duncan Frost, Zijian Zhou, Yikai Wang, Xudong Wang, Aditya Patel, Belinda Zeng, Tao Xiang, Serge Belongie, Amir Bar, Sen He
NAVA-WAM pretrains robot action policies directly from observation-only videos using transition-structured joint attention.
How can we learn robot action priors from observation-only videos without needing intermediate vision-representation models?
World action models (WAMs) typically rely on costly, action-annotated robot trajectories, limiting their scalability. Existing methods attempt to bridge this gap by learning intermediate visual representations or latent actions, but these approaches often introduce indirect transfer or restrictive architectural constraints. NAVA-WAM introduces native action-prior learning: it uses future-video flow-matching supervision to directly optimize the action policy from observation-only videos. By structuring joint attention around visual transitions, the model learns action-relevant priors without requiring an intermediate interface. On the RoboTwin 2.0 benchmark, NAVA-WAM outperforms the strongest baseline by 17.9 points in out-of-distribution settings while maintaining high efficiency.
Paper Primer
The core mechanism is a Mixture-of-Transformers (MoT) architecture where a Video-DiT and an Action-DiT exchange information through transition-structured joint attention. During pretraining, the Action-DiT is optimized solely through future-video flow-matching, effectively treating visual transitions as the primary supervision signal for action priors.
To enable efficient deployment, the authors implement asymmetric attention: the visual stream is decoupled from the action denoising process during post-training. This allows the model to cache visual representations, enabling closed-loop control without the computational overhead of generating future video frames at inference time.
NAVA-WAM significantly improves generalization under distribution shifts compared to representation-based and latent-action-based approaches.
Evaluation on RoboTwin 2.0 (Random domain) shows a 17.9-point success rate improvement over the strongest baseline. 73.6% success rate on OOD tasks vs. 55.7% for the next best method.
The model demonstrates superior action-label efficiency, requiring fewer demonstrations to achieve high performance.
NAVA-WAM trained with 25 demonstrations per task outperforms baselines trained with 50 demonstrations. 72.0% success rate with 25 demonstrations vs. 65.0% for latent-action baselines.
Why does this approach avoid the "shortcut" problem common in latent-action models?
The model exposes the Action-DiT only to Gaussian-corrupted future states during pretraining. This prevents the model from simply copying the clean prediction target, forcing the action policy to learn meaningful transition dynamics instead.
What is the primary advantage of "native" action-prior learning over standard representation learning?
Standard methods learn visual representations that are only connected to control during downstream training. NAVA-WAM learns action-relevant priors directly within the policy, eliminating the need for an intermediate interface and enabling a more direct path from observation to control.
Introduction
We expose the data‑efficiency gap in robot learning and motivate native action‑prior learning from observation‑only videos.
Training large robot policies demands costly action‑labeled data, creating a pronounced data‑efficiency gap. Observation‑only videos are abundant and encode interaction dynamics, yet existing pipelines either pretrain vision encoders or infer latent actions, both of which introduce indirect interfaces that limit policy learning. NAVA‑WAM addresses this gap by directly pretraining the action policy from observation‑only videos, bypassing intermediate representations.
These are raw video streams that capture how objects and agents move and interact, but they contain no explicit robot action annotations.
**Figure 1.** Paradigms of leveraging observation-only videos for robot policy learning. (1) Vision representation learning first learns visual representations from videos within a vision encoder and then trains an action policy on top, introducing an indirect representation-to-control transfer. (2) Latent-action learning extracts latent actions from visual transitions as intermediate supervision for policy learning, introducing an additional modeling stage and a potential representational bottleneck. (3) Native action-prior learning directly uses video dynamics to optimize the action policy, avoiding additional intermediate interfaces and representational bottlenecks.
The primary obstacle is the data‑efficiency gap: robot learning must become less dependent on scarce action‑labeled demonstrations.
Related Work
Survey of prior world‑action models and observation‑only video learning approaches.
Prior work on robot policies can be split into two strands: models that predict actions directly from observations, and those that first learn a world model of visual dynamics.
A World Action Model (WAM) couples action prediction with an explicit model of how the visual scene will evolve, rather than treating dynamics as an implicit by‑product.
Cascaded and joint architectures dominate the WAM landscape, each with distinct trade‑offs.
Methods that first extract predictive information from a video or world model and then feed it to a separate action module.
Architectures that model future observations and actions within a shared network, allowing direct interaction between visual and action streams.
Joint WAM that simultaneously predicts future visual frames and generates actions in a closed‑loop setting.
Joint architecture that learns visual‑action coupling for language‑conditioned robot control.
Uses future‑video prediction only during training, but runs with action‑only inference.
Shows that co‑training with human videos improves cross‑embodiment generalization.
Vision‑Language‑Action (VLA) models extend pure visuomotor policies by conditioning on language, leveraging large‑scale vision‑language pre‑training.
Models that map visual inputs to actions while being guided by natural‑language instructions.
Observation‑only videos provide a scalable source of supervision for learning state transitions without any action labels.
Approaches that learn visual or predictive embeddings from videos and later reuse them for downstream policy learning.
Conditions robot policies on features extracted from predictive video models.
Learns visual representations by predicting future robot keypoints from video.
Uses predictive video representations for planning and VLA learning.
Jointly learns video prediction and language‑conditioned action generation.
Latent‑action learning methods infer intermediate action‑like variables from visual transitions before grounding them to real robot commands.
Frameworks that first learn a latent action space from video and then map it to executable motor commands.
Learns latent actions and decodes them into robot commands.
Pretrains a VLA by predicting discrete latent actions before grounding them.
Incorporates optical‑flow motion cues into the learning pipeline.
Learns transition‑oriented action representations in a semantic visual space.
Extends LingBot‑VA with improved transition modeling for action generation.
The NAVA-WAM Architecture
We describe how NAVA‑WAM learns action priors from videos and integrates them into robot control.
Training robot policies at scale is bottlenecked by the scarcity of action‑labeled data. NAVA‑WAM sidesteps this by extracting action‑relevant priors directly from observation‑only videos, then fine‑tuning on a modest set of robot demonstrations.
NAVA‑WAM is a world‑action model that embeds an action prior inside the policy itself, learning it from raw videos without any action annotations.
How does NAVA‑WAM differ from prior world‑action models that use an explicit interface?
Earlier models treat visual and action streams as separate modules linked by a hand‑crafted interface. NAVA‑WAM instead embeds the prior inside the Action‑DiT and lets joint attention transfer visual transition information implicitly, removing the need for a hand‑designed bridge.
Action‑DiT is a transformer that produces continuous robot actions while attending to both visual and action tokens, effectively learning a mapping from visual dynamics to motor commands.
Form the concatenated key matrix [$K_V$;$K_A$] = [[$k_{V1}$, $k_{V2}$], [$k_{A1}$, $k_{A2}$]].
Compute attention scores $Q_A$·Kᵀ, yielding a 2×4 matrix of dot products.
Apply softmax row‑wise to obtain attention weights.
Multiply the weights by the concatenated value matrix [$V_V$;$V_A$] to produce $H_A$, a 2×2 action representation.
Even with only two visual tokens, Action‑DiT can blend information from both streams, learning a transition‑aware action representation without any explicit inverse‑dynamics module.
Is Action‑DiT just a standard transformer decoder that happens to see visual tokens?
No. In a vanilla decoder the query stream attends only to previously generated tokens. Here Action‑DiT queries attend simultaneously to visual keys and action keys, and the visual stream itself remains static during inference, which is a distinct cross‑modal attention pattern.
During pre‑training the Action‑DiT learns an action prior by predicting future video frames, using only observation‑only videos and a transition‑structured attention mask.
Supervision comes from a flow‑matching loss that forces the model to predict a future video latent given a noised version, providing a dense learning signal without any action labels.
Why not train the Video‑DiT jointly instead of freezing it?
Freezing the Video‑DiT isolates the supervision signal to the Action‑DiT, ensuring that the learned prior truly reflects action‑relevant information rather than simply adapting the visual encoder to the loss.
**Figure 2.** Overview of NAVA-WAM. During native action-prior pre-training, the Action-DiT is optimized solely through future-video supervision, learning action-relevant priors directly from observation-only videos. During post-training, the Video-DiT and Action-DiT are jointly optimized on action-labeled robot demonstrations using video and action flow-matching objectives, adapting the pretrained Action-DiT to continuous robot control. At inference, NAVA-WAM denoises only robot actions without explicitly generating future videos, enabling efficient closed-loop control.
Pre‑training: freeze Video‑DiT, sample noise $\epsilon$ᵛ_k and $\tau_{v}$, construct noised future latent z_{k,$\tau_{v}$}, and optimize Action‑DiT via `L_pre`.
Post‑training: unfreeze both experts, sample video and action noises ($\epsilon$ᵛ_k, $\epsilon$ᵃ_k) and timesteps ($\tau_{v}$, $\tau_{a}$), and jointly minimize `L_post` = $\lambda_{v}$ $L_v^{FM}$ + $\lambda_{a}$ $L_a^{FM}$.
Inference: run a single joint forward pass to cache visual keys/values $C_V$, then iteratively denoise actions using only the Action‑DiT while reusing $C_V$.
Experimental Results
NAVA‑WAM delivers near‑perfect in‑domain success and strong out‑of‑domain generalization.
Recall that NAVA‑WAM learns an action‑relevant prior directly from observation‑only videos, sidestepping costly action‑labeled data.
NAVA‑WAM attains near‑perfect in‑domain success (99.0 % on LIBERO) while substantially out‑performing baselines on out‑of‑domain tasks.
Table 1 reports 99.0 % LIBERO, 83.5 % LIBERO‑Plus, 88.5 % RoboTwin Clean, and 73.6 % RoboTwin Random, the highest numbers across all methods.
On a physical Franka FR3 arm, NAVA‑WAM succeeds in 14/15 trials (93.3 %), far exceeding DreamZero‑DROID and $\pi$0.5.
Real‑robot evaluation reports 14/15 successes for NAVA‑WAM, 10/15 for DreamZero‑DROID, and 8/15 for $\pi$0.5.
With only 25 demonstration videos per task, NAVA‑WAM reaches 72.0 % success, outpacing representation‑based and latent‑action baselines by up to 12.4 points.
Table 2a shows 72.0 % for NAVA‑WAM versus 60.6 % and 65.0 % for the other pre‑training methods under the same budget.
**Figure 11.** Qualitative NAVA-WAM rollouts on RoboTwin 2.0. We show seven successful rollouts (top) spanning diverse manipulation skills and two representative failures (bottom). Each row contains four temporal keyframes, with the head-camera view on the left and the two wrist-camera views stacked on the right. The row header gives the RoboTwin task description, and the final keyframe is outlined in blue for success and red for failure. All examples are from the Random (OOD) domain.
**Figure 5** Qualitative visualization of motion transfer across domains. Each example transfers the transition encoded by a source pair (Source: current $arrow$ Source: next) to a different target observation. The source transition specifies the underlying motion cue (orange arrows), which is applied to Target: current to obtain Target: transfer. The Overlay visualizes the transferred state relative to the current target observation, while Target: real shows the corresponding real transition (blue arrows). Across simulation, real-world, and sim-to-real examples, the transferred states follow the source motion despite substantial changes in visual appearance and scene configuration.
Theoretical Analysis
Theoretical results show why the Action‑DiT transition representation and future‑state corruption are indispensable.
The appendix isolates the core theoretical mechanisms of the Action‑DiT pathway, stripping away architectural details to focus on the transition representation $r_k$ and the Gaussian corruption applied to the future visual state.
The representation $r_k$ carries information about the future state that cannot be inferred from the preceding visual state alone.
How does this differ from simply feeding the clean future state $z_k$ to the decoder?
Providing the clean $z_k$ would make $r_k$ a perfect copy of the target, collapsing the prediction error to zero without forcing the Action‑DiT to learn any transition dynamics. The theory shows that when $r_k$ is derived from a corrupted version, the decoder must rely on the information encoded in $r_k$, which is why the reduction $L_z - L_{z,r}$ is meaningful.
Adding Gaussian noise to the future visual state blocks a trivial shortcut that would otherwise let the model copy the target directly.
Why not simply train the model to denoise the corrupted future state instead of using the transition representation?
Denoising would still require the model to infer $z_k$ from $z_{k,\tau}$, but the Action‑DiT’s purpose is to learn a compact representation $r_k$ that captures the *transition* information. The theoretical bound shows that even with optimal denoising, the error cannot be eliminated, so the model must rely on $r_k$ to improve prediction beyond the baseline $L_z$.
Together, Propositions A.1 and A.2 demonstrate that the Action‑DiT pathway gains predictive power by encoding transition cues while the Gaussian corruption guarantees that the pathway cannot collapse to a trivial copy of the target.
Extended Analysis
Ablations reveal that native action‑prior pre‑training boosts success rates and visual prediction quality.
This appendix expands the ablation study by examining how the native action‑prior pre‑training influences optimization stability, prediction fidelity, and downstream task performance.
**Figure 8** Native action-prior pre-training dynamics. Training and held-out future-video flow-matching losses over pre-training. The training curve shows the raw per-step loss (faint) and its exponential moving average (solid), while the held-out loss is evaluated periodically without smoothing.
**Figure 9** Future-video prediction after native action-prior pre-training. Predictions on four held-out egocentric manipulation clips. Each example shows the ground-truth sequence (GT) and the corresponding prediction from NAVA-WAM (Ours) across block stacking, two-handed assembly, cloth folding, and card manipulation.
**Figure 4. Attention of learned action representations.** We visualize attention from action representations to visual tokens on held-out data, overlaid on the current and next input observations. Our pretrained Action-DiT attends to interaction-relevant regions and aligns with observed motion, while the CoMo latent-action baseline shows more diffuse attention over static regions.
**Figure 6** Scaling observation-only pre-training data. Success across data scales at different action-label budgets.
**Figure 7.** Transition-structured attention masks. The pre-training mask (left) and post-training mask (right) are shown for three consecutive transition segments $S_1-S_3$. Each segment contains a clean preceding visual state, a noised future state, and the corresponding Action-DiT tokens, with attention isolated across segments. Pre-training allows noised-future and action-token queries to attend to all tokens within their segment, whereas during post-training, noised-future video queries attend only to visual tokens.
**Figure 10.** Additional qualitative visualizations of motion transfer across domains. Each example transfers the transition encoded by a source pair (Source: current → Source: next) to a different target observation. The source transition specifies the underlying motion cue (orange arrows), which is applied to Target: current to obtain Target: transfer. The Overlay visualizes the transferred state relative to the current target observation, while Target: real shows the corresponding real transition (blue arrows). Across simulation, real-world, and sim-to-real examples, the transferred states follow the source motion despite substantial changes in visual appearance and scene configuration.
Benchmark Results
Action‑DiT reaches 97.0 % success on the full benchmark.
Action‑DiT attains 97.0 % success across the full suite of manipulation tasks.
Aggregated over 45 distinct tasks, the model solves the overwhelming majority.
Additional Benchmark Data
NAVA-WAM sets a new success benchmark on RoboTwin 2.0 while keeping inference fast.
NAVA‑WAM achieves the highest average success rate (93.3 %) on RoboTwin 2.0, surpassing all baselines.
Table 8 reports 93.3 % average success for NAVA‑WAM, while the next best baseline reaches only 85.0 %.
**Table 7.** Per-task success rates (%) on RoboTwin 2.0. All methods are trained exclusively on Clean-domain demonstrations and evaluated in both the Clean and Random (Rand.) domains. Baseline per-task results are taken from MV-WAM (Chen et al., 2026c). The final row reports the average across all 50 tasks. The best result for each task and domain is shown in bold.
The table compares three models: $\pi_{0.5}$, DreamZero, and NAVA-WAM, across metrics including latency (ms/call) and success rates on three tasks (T1, T2, T3), along with an average success percentage.
The qualitative rollouts (Section C.4) illustrate that NAVA‑WAM can robustly execute complex dual‑arm manipulations under random visual distractors, confirming the quantitative gains.
Efficiency and Failure Analysis
We evaluate NAVA‑WAM on real robots, compare its speed and success to baselines, and discuss remaining limitations.
Recall that NAVA‑WAM learns action‑relevant priors from observation‑only videos, enabling it to act without costly action‑labeled data. Here we test that claim on a physical Franka FR3 robot.
Successful rollouts demonstrate diverse manipulation skills: bimanual handover of a microphone and lifting a pot, dual‑shoe placement with orientation constraints, single‑arm placement of bread into a skillet, opening a laptop hinge, and long‑horizon block‑stacking tasks that require grounding color references.
Failure cases arise in fine‑grained, contact‑rich interactions. In Move Can Pot the arm knocks the can over and proceeds with an empty gripper; in Hanging Mug the left arm lifts the mug but the right arm never completes the placement, causing the episode to time out.
Limitations remain: (1) precise grasping and complex contact dynamics still challenge the policy, and (2) the current system lacks an explicit recovery mechanism for local execution errors.
Implementation Details
Implementation specifics for pre‑training, downstream training, benchmarks, and action‑only inference.
Native Action‑Prior pre‑training consumes 235,790 observation‑only manipulation episodes (≈1,054 h) drawn from Open X‑Embodiment, AgiBotWorld, and EgoDex, each split into 65‑frame clips and encoded by a frozen spatiotemporal VAE into five latent blocks.
Training runs for 14 k optimization steps on 128 GPUs using bf16 mixed precision; AdamW with $\beta$₁=0.9, $\beta$₂=0.95, weight decay 10⁻², a fixed learning rate of 1×10⁻⁴, and a global batch size of 512 updates only the Action‑DiT while Video‑DiT, text encoder, and VAE stay frozen.
This table compares training configurations for "Cold Start" and "Joint Post-training" across several parameters: Trainable components, Action horizon, Learning rate, Epochs, Scheduler, Global batch size, and Observation.
**Table 4.** Downstream training hyperparameters on RoboTwin 2.0. We first perform action-free target-domain cold start and subsequently jointly post-train the Video-DiT and Action-DiT using Clean-domain demonstrations.
Downstream adaptation begins with an action‑free cold start that fine‑tunes Video‑DiT on target‑domain videos; subsequently, both Video‑DiT and Action‑DiT are unfrozen and jointly trained using the combined loss ($\lambda$ᵥ = $\lambda$ₐ = 1) to align visual and action representations.
Transition‑structured attention masks partition each visual transition into an isolated segment containing the clean preceding state, the noised future state, and the corresponding Action‑DiT token group; within a segment, clean‑state queries self‑attend, while future‑video and action queries attend to all tokens in the segment.
Evaluation follows the standard LIBERO protocol (four task suites, 2 000 total trajectories, 50 rollouts per task) and extends to LIBERO‑Plus, which probes 10 030 perturbed test instances without additional training data.
RoboTwin 2.0 experiments train exclusively on 50 Clean‑domain demonstrations per task (2 500 trajectories) and assess performance on both Clean and Random domains, the latter serving as an out‑of‑distribution test.
Action‑only inference exploits the asymmetric transition attention: Video‑DiT representations are cached after a single joint forward pass ($\tau$ᵥ = $\tau$ₐ = 1), and subsequent denoising steps evaluate only the Action‑DiT, reusing the visual cache.
Action‑only inference with cached Video‑DiT representations (Algorithm 1)