Long-WAM: Scaling the Context of World-Action Models
Wei Huang, Bohan Zhang, Chenzhi Liu, Isabella Liu, Shuai Yang, Weian Mao, Luozhou Wang, Yicheng Xiao, Weifeng Lin, Qixin Hu, Bryan Chu, Sifei Liu, Linxi Fan, Xiaojuan Qi, Song Han, Yukang Chen
Long-WAM scales robot control context by using autoregressive video pretraining to learn causal history-to-future dependencies.
How can we scale the visual context of robot control models to improve performance on long-horizon tasks without incurring prohibitive real-time inference latency?
Robot control requires visual history to infer motion, but processing long sequences often introduces latency that breaks real-time performance. Long-WAM addresses this by using autoregressive video pretraining to learn causal temporal structure, which is then preserved during action adaptation to ensure the model learns to exploit history rather than just access it. On the RoboCasa GR-1 benchmark, increasing context from 0.0 to 19.2 seconds raises success from 63.3% to 78.7%, while asynchronous execution and hardware-specific optimizations maintain real-time response.
Paper Primer
The core mechanism is a staged adaptation: first, the model learns robot interaction dynamics through large-scale autoregressive video pretraining; second, it couples this video foundation to an action expert using an asymmetric interface. This "predict-then-act" approach uses the video expert to forecast future latent states, which then condition the action expert's denoising process.
To maintain real-time control, the system employs asynchronous execution—overlapping inference with robot motion—and streaming observation encoding to minimize post-trigger computation. The pipeline is accelerated via four-bit quantization (NVFP4) and device-specific kernel tuning, allowing the model to generate action chunks in 107.4 ms on an RTX 5090.
Autoregressive (AR) pretraining is essential for scaling context benefits.
On RoboCasa GR-1, the AR-pretrained model's lead over a bidirectionally pretrained initialization grows from 3.3 points at zero history to 17.1 points at 19.2 seconds of context. +15.4 percentage point gain in success rate (63.3% to 78.7%) when extending context to 19.2 seconds.
Strong execution quality amplifies high-level planning.
Pairing Long-WAM with a GPT-6 Astra planner raises overall success on composite tasks from 31.4% to 54.4%, a 23.0-point gain compared to a 13.6-point gain for the baseline $\pi\$0.5 policy. 23.0 percentage point improvement in overall success.
Why does the paper emphasize autoregressive pretraining over bidirectional alternatives?
The authors observe that bidirectional pretraining fails to yield gains from longer context because it lacks the causal history-to-future structure required for control. AR pretraining explicitly learns this temporal factorization, allowing the model to use history as a predictive resource rather than just an additional input.
What is the primary trade-off when scaling context in this system?
Context scaling is an execution resource: increasing history improves success rates but increases inference latency. The authors manage this by co-designing the model with asynchronous execution and edge-specific acceleration to ensure the controller responds within the required time window.
Context scaling in robotics is not a universal win; its value depends on the predictive quality of the foundation model and the system's ability to maintain low-latency execution. Future work should focus on adaptive context allocation that matches computation to the specific needs of the current interaction.
Introduction to Long-WAM
Long‑WAM scales visual history for real‑time robot control while keeping latency low.
Real‑time robot control demands enough visual history to infer dynamics, yet processing that history can delay the action. World‑Action Model (WAM) brings video prediction into closed‑loop control, making history a natural resource, but the latency bottleneck raises the question: how does WAM control scale with visual context while staying real‑time?
It separates learning of visual dynamics from action adaptation, letting a pretrained autoregressive (AR) video model provide long‑range predictions that the controller can use without slowing down.
**Figure 1 | Overview of Long-WAM.** (1) AR video pretraining. LongLive2.0-Robot learns predictive dynamics from approximately 10,000 window-equivalent hours of robot and egocentric videos. (2) Causal-to-causal adaptation. We preserve causal video dependencies while conditioning action-chunk denoising on observed history. (3) Context scaling. Relative to current-observation-only control, success increases from 63.3% to 78.7% on RoboCasa GR-1 and from 94.5% to 99.5% on LIBERO-Long, peaking at 19.2 and 2.4 seconds of history, respectively. (4) Efficient deployment. Asynchronous execution, streaming VAE encoding, decision-time prefill, within-call KV reuse, and hardware-specific acceleration enable real-time control on RTX 5090, DGX Spark, and Jetson AGX Thor.
The key trade‑off is between extending visual history to improve success and keeping end‑to‑end latency low enough for real‑time control.
Scaling Context via AR Pretraining
Scaling context lets a robot policy use longer histories without slowing down inference.
Real‑time robot control suffers when the model must ingest long visual histories, because each extra frame adds latency. The trick is to decouple history processing from action generation so the policy can look farther back without a speed penalty.
We first teach the model to predict future robot motions from long video clips, using only video supervision, so it learns a generic motion prior that later guides action generation.
Compute the noisy input: $x = (1-0.3)\times2.0 + 0.3\times5.0 = 2.9$.
The predictor $v_\theta$ outputs a velocity estimate $v_\theta(x)=0.9$ (simulated).
The target correction is $\bar{\epsilon}-z = 5.0 - 2.0 = 3.0$.
The squared error term is $\|0.9 - 3.0\|^2 = 4.41$.
With a scheduler weight $w(\sigma)=1$, the loss contribution for this chunk is $4.41$.
This toy example shows how the loss penalizes the predictor for failing to undo the injected noise, encouraging the model to learn a denoising mapping that works across any $\sigma$.
How does this teacher‑forcing loss differ from the standard AR loss used in language models?
Standard AR loss predicts the next token directly from the clean prefix, whereas here the model predicts a *difference* (noise‑minus‑data) from a *noisy* blend of the clean latent and Gaussian noise. This forces the network to learn an explicit denoising step, which is essential for later action conditioning.
After the video expert has forecasted future visual latents, the action expert consumes those latents together with the current robot state and language instruction to produce a chunk of actions, keeping the causal ordering learned during video pretraining.
Rollout produces future latents $\tilde{Z}_t = [1.0 + 0.9\times0.5,\; 2.0 + 0.9\times0.5] = [1.45,\; 2.45]$.
The cache $\mathcal{K}_t$ stores these latents as keys and values for the action expert.
Action noise $\epsilon_a = [0.2, -0.1]$, language command $c$ = “pick up the red block”, robot state $q_t = (x=0.5, y=0.3)$.
The denoising function $\text{Denoise}_\psi$ combines $\epsilon_a$ with $\mathcal{K}_t$, $c$, and $q_t$ to output $\hat{A}_t = [0.6, 0.4]$ (simulated).
The example shows that the action expert never recomputes video attention; it simply reads the pre‑computed cache, which is why the method scales to longer histories without extra latency.
Why isn’t the action expert just another autoregressive decoder that attends to the full video sequence?
Because attending to the full video at every action step would re‑run the expensive video attention for each token. By pre‑computing a shared cache $\mathcal{K}_t$, the action expert reuses the video representation once per control step, preserving the causal video dynamics while keeping inference fast.
First pass: train the video expert $\theta$ to match video flow conditioned on clean history (loss $\mathcal{L}_{\text{video}}$).
Second pass: freeze $\theta$, detach its cache, and train the action expert $\psi$ to denoise action chunks conditioned on the cached video latents (loss $\mathcal{L}_{\text{action}}$).
During inference, run the video rollout once to obtain $\tilde{Z}_t$, store $\mathcal{K}_t$, then feed $\mathcal{K}_t$ to $\psi$ to produce the action chunk $\hat{A}_t$.
**Figure 2.** Attention patterns and video–action inference strategies. (a) Action generation from the current observation only; (b) history-conditioned action generation without future prediction; (c) joint video–action co-denoising (CoD); (d) video prediction followed by action denoising (IDM). IDM conditions actions on both observed history and predicted future latents, reusing visual KV across action-denoising steps.
By precomputing a visual cache and reusing it for action denoising, the method achieves longer effective context without incurring the usual latency penalty, enabling real‑time control even on resource‑constrained platforms.
Edge Deployment Infrastructure
How the system meets real‑time constraints by overlapping inference with execution and streaming VAE encoding.
Real‑time robot control must deliver the next action chunk before the previous one finishes, yet the model still needs to ingest the latest visual history. The trick is to decouple history processing from action generation so that inference can run while the robot is already moving.
Instead of waiting for a full inference pass before the robot moves, the controller starts the next action as soon as the first $R\!-\!S$ steps of the predicted chunk are ready, overlapping the remaining inference with ongoing execution.
At $k=0$, the model predicts $\hat{A}_{t_0}$ covering steps $0\!-\!3$.
The robot immediately executes steps $0$ and $1$ (the first $R$ steps).
While the robot is moving, the model computes steps $2$ and $3$.
When step $2$ finishes, the controller discards the already‑executed prefix and hands over the remaining suffix (step $2$ onward) to the robot.
Because $O=1$, the handoff must happen within one control interval $\Delta t$ after the trigger.
The overlap $O$ is the only slack the system has to hide inference latency; making $O$ larger gives more safety margin but reduces the effective prediction horizon.
Instead of encoding the entire observation window before any inference, the VAE processes incoming frames incrementally, so that after the trigger the remaining frames are encoded on‑the‑fly and immediately fed into the action predictor.
Group 1 (frames 0‑1) is already encoded; its latent $\mathbf{z}_1$ is stored.
When the trigger fires, the encoder processes Group 2 (frames 2‑3) to produce $\mathbf{z}_2$.
Features $\mathbf{z}_1$ and $\mathbf{z}_2$ are concatenated and projected to a joint latent $\mathbf{z}_{12}$.
The joint latent is normalized and immediately supplied to the action predictor, which begins generating the next action chunk.
Group 3 (frames 4‑5) is processed in parallel with the predictor’s computation, so no idle waiting occurs.
By never waiting for the full video window, the streaming VAE keeps the inference pipeline continuously busy, turning what would be idle “WAIT” time into useful computation.
The figure illustrates the difference between a Streaming VAE and a Non-Streaming VAE approach. The top section shows the Streaming VAE, where inference tasks (Infer k+1, k+2, k+3) are interleaved with observation chunks, leading to overlapping execution and time savings. The bottom section shows the Non-Streaming VAE, where the system must wait for the "Full OBS" before starting inference, resulting in idle "WAIT" periods between chunks. The timeline shows the progression of chunks (k, k+1, k+2) and the corresponding execution of actions.
Asynchronous overlap loop – runs inference while the robot executes the previous chunk.
**Figure 4.** Optimizations for efficient edge deployment. Shared optimizations and device-specific tuning accelerate edge inference. “Base” denotes the Quant/GEMM implementation; shape-specific dispatch and autotuning remain enabled.
Empirical Performance
Long‑WAM (IDM) sets new success records across simulated and real robot tasks.
Long‑WAM (IDM) achieves the highest success on the longest RoboCasa GR‑1 tasks, reaching 78.7 %.
Table 4 shows 78.7 % success, a gain of 11.6 % over the strongest GR‑1 baseline.
Pretraining LongLive2.0‑Robot consumes ~30,720 GPU‑hours on 64 NVIDIA H100 GPUs; downstream WAMs train on 16 GB GPUs using AdamW (peak $10^{-4}$ learning rate, cosine decay, 5 % warmup), BF16 precision, and gradient clipping at $1.0$.
**Figure 5.** Simulation environments and real-world tasks. Our evaluation spans four simulation benchmarks—LIBERO, RoboTwin 2.0, DOMINO, and RoboCasa—and eight real-world task configurations on G1 and YAM. These include cup pickup at four conveyor speeds, dynamic cup stacking, bowl stacking, brick sorting by color, and dumpling placement into a pan.
Long‑WAM consistently outperforms baselines across simulation and real‑world tasks.
Hierarchical Task Composition
Long‑WAM + planner delivers a 23 pp overall success boost, surpassing baselines on atomic and composite tasks.
We evaluate Long‑WAM as the execution core of a hierarchical robot system, pairing it with a task planner that issues atomic sub‑instructions and occasional end‑effector corrections.
Planner augmentation yields a 23 pp overall success gain for Long‑WAM, far exceeding the 13.6 pp gain observed for the $\pi$0.5 baseline.
Overall improves from 31.4 % to 54.4 % with the planner, while $\pi$0.5 improves by only 13.6 pp under the same augmentation.
By contrast, Long‑WAM alone already attains 67.9 % Atomic‑Seen success, far above $\pi$0.5 (39.6 %) and GPT‑6 Astra (31.5 %). Querying the same policy every 15 steps raises success to 84.4 %, nearly matching the 85.6 % achieved by the full hierarchy.
Adding the planner dramatically improves compositional performance: Composite‑Seen jumps from 15.8 % to 38.8 % and Composite‑Unseen from 6.1 % to 35.0 %, demonstrating that strong atomic execution combined with high‑level planning unlocks unseen skill compositions.
**Figure 7.** Dynamic manipulation on Unitree G1. Long-WAM maintains 90–100% grasping success across conveyor speeds and achieves 95% success on dynamic cup stacking. Top: policy and human teleoperation comparisons. Bottom: Long-WAM and Fast-WAM rollouts.
**Figure 8.** Long-horizon execution on YAM: 20 trials per task.
Latency Analysis
Ablations show how each optimization cuts latency while preserving success rate.
Recall that real‑time robot control needs long visual context, but latency can bottleneck performance; Long‑WAM tackles this by decoupling history processing from action generation.
Long‑WAM achieves a 2.3× speedup over Fast‑WAM while preserving success rate.
Table 6 shows latency 107.4 ms vs Fast‑WAM 244.1 ms and SR 94.4 % vs 94.4 %.
V2/A2 reduces latency by 24% to 81.8 ms at the cost of a 1.0‑point SR drop.
Table 6 reports V2/A2 latency 81.8 ms and SR 93.5 % versus V4/A4 SR 94.4 %.
Pure asynchronous execution keeps SR at 94.2 % while cutting RMSE and jerk to about one‑third of Fast‑WAM.
Table 7 compares async SR 94.2 % (IDM) with Fast‑WAM’s larger RMSE and jerk.
Discussion and Limitations
We wrap up findings, outline limits, and point to future work on long‑horizon robot control.
Real‑time robot control needs long visual context to infer dynamics, yet standard models stall because latency grows with history. Long‑WAM tackles this by decoupling history processing from action generation, letting the policy use rich past information without sacrificing responsiveness.
A manipulation policy benefits from remembering recent motions and interaction states, while hour‑scale reasoning belongs to a higher‑level planner. The planner maintains semantic continuity; Long‑WAM supplies the low‑level controller with the physical context it needs for precise execution.
Each context window in our study is trained as a separate model, giving a clean view of how history helps at every length. The measured latency profile—$74.6\,$ms without history and $341.0\,$ms at $19.2\,$s of history—shows the compute cost that adaptive windowing could reclaim.
These trends motivate adaptive context allocation: keep the history that matters for the current interaction while matching computation to the required response speed. The design does not enforce a power‑law or a fixed memory optimum; instead it invites future work to balance what is remembered, how well it acts, and how quickly new observations are incorporated.
**Figure.** A sequence of frames showing a robotic arm performing a pick-and-place task. The arm moves an orange shoe from the table surface into a blue bin over the course of 252 frames.
**Figure 10.** Dynamic cup stacking with $\pi_{0.5}$. Time progresses left to right within each four-column block, then continues in the next block below; each overview frame is accompanied by two wrist views. The policy grasps the blue cup but misses the moving green cup (red circle), leaving the composite task incomplete.
**Figure 12.** Dynamic cup stacking with Long-WAM. Long-WAM grasps the blue cup, intercepts the moving green cup, and places the green cup inside the blue cup. The overview and wrist views reveal the transition from interception to alignment and insertion, illustrating coordinated execution across the stages of this dynamic task.
**Figure 16.** Long-horizon manipulation with Long-WAM on YAM. Top to bottom: brick sorting by color, placing dumplings in a pan, and stacking bowls. Each task contains six chronological snapshots, read left to right, with paired wrist views below each overview frame. The final snapshots show successful task configurations across all three tasks.
**Figure 9.** Task-conditioned video prediction with LongLive2.0-Robot. Five selected text-and-image-to-video (TI2V) examples, with the task prompt shown above each row. $f$ denotes the zero-based frame index. All frames retain the original field of view.
Asynchronous overlap experiments (Table 10) show that shortening the overlap reduces RMSE (0.0246 → 0.0058) and jerk (0.0436 → 0.0143), but the highest success rate (95.0 %) occurs at an intermediate stride $S\!=\!16$, illustrating that smoother chunk transitions do not always translate to higher task success.
Table 11 (Context P vs. History vs. Latency) reveals a roughly linear increase in latency as context grows: $0\,$P → $74.6\,$ms, $384\,$P → $341.0\,$ms, a $3.2\times$ rise for an eight‑fold history increase.
Long‑WAM shows that remembering physical history can boost robot performance while keeping inference fast enough for real‑time control.