World Action Modeling with Progressive Visual Planning
Fei Zhang, Zhaochong An, Duncan Frost, Yikai Wang, Pengfei Liu, Ya Zhang, Michal Drozdzal, Amir Bar
ProWAM uses progressive visual sub-goals to guide robotic control, enabling efficient long-horizon planning without dense video rollouts.
How can we improve long-horizon robotic control by replacing monolithic goal prediction with progressive, progress-conditioned visual sub-goals?
Robotic control models often struggle with long-horizon tasks because they either generate computationally expensive, dense video rollouts or lack the visual foresight to navigate complex sequences. ProWAM addresses this by predicting an ordered sequence of sparse visual sub-goals indexed by normalized task progress, which are then cached to guide action generation. This approach eliminates the need for iterative full-video generation during inference, setting new state-of-the-art performance on major robotic benchmarks.
Paper Primer
ProWAM integrates visual planning into a unified mixture-of-transformers architecture. It uses a video expert to predict sparse sub-goals at specific progress intervals, which are then injected as cached key-value features into a lightweight action expert that generates low-level control commands.
ProWAM significantly improves success rates on long-horizon and out-of-distribution robotic tasks.
Evaluations on LIBERO-Plus and randomized RoboTwin benchmarks. Achieved relative gains of up to +35.9% over the strongest baseline.
The model demonstrates superior zero-shot generalization in novel real-world scenes.
Comparison against DreamZero on Franka robot deployments. Outperformed the baseline by +15.0 absolute percentage points (70.0% vs 55.0%).
Why does ProWAM use sparse sub-goals instead of predicting the entire future video?
Predicting dense video rollouts is computationally inefficient and requires multiple denoising passes. By caching sparse sub-goal features, ProWAM reduces video-expert evaluations by approximately 10x while maintaining necessary visual guidance for long-horizon tasks.
How does the model handle tasks of varying lengths?
Sub-goals are parameterized by normalized progress coordinates $r \in [0, 1]$, where $r=0$ is the current state and $r=1$ is the endpoint. This relative scaling allows the model to anchor guidance consistently regardless of the total trajectory duration.
Introduction
We expose why compressing whole episodes into a single goal hampers long‑horizon control and introduce progressive visual sub‑goals as a remedy.
World Action Models (WAMs) aim to predict future visual dynamics and actions from a single observation, but they falter on long‑horizon tasks because generating dense video rollouts is computationally prohibitive.
A WAM treats a robot episode as a single prediction problem: given the current camera view and an instruction, it tries to output the entire future video and the corresponding action sequence.
ProWAM replaces the monolithic final‑goal prediction with a progressive sequence of sparse visual sub‑goals, each tied to a normalized progress value r, thereby providing explicit intermediate guidance without the cost of full video rollouts.
**Figure 1** Progressive Sub-goal Imagination for Long-Horizon Control. Given the current observation, ProWAM generates sparse visual targets at requested relative progress values (left). The resulting sub-goal features directly condition action generation for the corresponding execution rollout (right), providing explicit visual guidance without generating dense video rollouts.
By caching sub‑goal features after a single forward pass, ProWAM achieves efficient closed‑loop control, and its design naturally leverages large‑scale action‑free video pretraining to learn the visual planning component.
The key shift is from a single, monolithic goal prediction to a series of progress‑indexed visual sub‑goals that continuously anchor action generation.
Video Generation Foundations
We describe the diffusion video pipeline and introduce flow matching for both video and action generation.
The video generation pipeline first compresses a raw video $V$ into a latent tensor $X$ with a pretrained VAE, then learns a continuous‑time generative process that maps Gaussian noise $\epsilon$ to $X$ via flow matching. This same idea is later mirrored for action generation.
Flow matching treats generation as learning a deterministic velocity field that continuously transports random noise into the target data distribution.
Existing world‑action models differ in how they provide visual context $C(X)$ to the policy: full‑imagination models condition on the entire predicted video clip, offering rich spatial guidance but incurring high latency, while zero‑imagination models skip future latents entirely, enabling real‑time control at the cost of foresight.
The ProWAM Architecture
ProWAM replaces a single final‑goal target with a sparse sequence of visual waypoints that guide action generation throughout a long‑horizon task.
Final‑Goal Prediction compresses an entire episode into a single terminal frame, leaving the robot without intermediate guidance. ProWAM solves this by injecting a sparse sequence of progressive visual sub‑goals into the world‑action model, giving the action expert concrete waypoints throughout execution.
Instead of predicting only the final state, the model predicts a short ordered list of visual waypoints at increasing fractions of the task horizon, so the action expert can follow a concrete plan step by step.
Compute frame indices: $\tau(r_1)=\lfloor0+9\times0.3\rceil=3$, $\tau(r_2)=\lfloor0+9\times0.7\rceil=6$.
Extract frames $V_3$ and $V_6$ from the video and encode them: $\tilde G_1=E(V_3)$, $\tilde G_2=E(V_6)$.
Form the latent sequence $Z=\{X_0,\dots,X_{9},\tilde G_1,\tilde G_2\}$, where $X_0$ is the initial observation latent.
Map $r_1$ and $r_2$ through the sinusoidal embedding + MLP to obtain offsets $\Delta\text{mod}(r_1)$ and $\Delta\text{mod}(r_2)$, which are added to the base modulation for each sub‑goal slot.
This toy example shows how a handful of waypoints, placed at uniformly spaced progress fractions, can cover the entire trajectory regardless of its absolute length.
How does ProWAM’s progressive sub‑goal generation differ from classic hierarchical planners?
Classic hierarchical planners use separate modules to first generate high‑level goals and then invoke lower‑level controllers, often requiring hand‑crafted decomposition. ProWAM generates the whole sub‑goal sequence end‑to‑end inside a single transformer, shares weights across all sub‑goal slots, and conditions each slot on its progress coordinate, eliminating the need for an extra planning stage.
**Figure 2. Overview of ProWAM.** Left: Training architecture. Given a task instruction $T$ and reference observation $I_{ref}$, ProWAM jointly denoises future-video latents $X$, progressive sub-goal latents $G$, and action chunks $A$. The bottom timeline aligns these latent variables with physical execution, with sub-goals providing visual targets at specified relative progress levels. Right: Training and inference pipeline. ProWAM employs two-stage training: (1) Video-only Pretraining, where the video expert learns visual dynamics and progressive sub-goal prediction from action-free videos; and (2) Joint Fine-tuning, where the video and action experts are jointly optimized on robot demonstrations, enabling the action expert to condition action generation on predicted sub-goals. At inference, the video expert predicts sparse sub-goal features instead of a dense video rollout. These features are cached and reused during action denoising to avoid repeated video-expert computation.
Empirical Evaluation
ProWAM swaps a single final‑goal forecast for a sequence of visual sub‑goals.
ProWAM replaces a single final‑goal prediction with a chain of visual sub‑goals, letting the policy adjust actions as the task unfolds.
ProWAM attains the highest real‑world success rate among evaluated methods.
ProWAM (Ours) reaches 70.0 % SR on the Franka deployment, while the next best method achieves 55.0 %.
LIBERO evaluates long‑horizon manipulation tasks in a simulated kitchen, measuring how often a policy completes a full sequence of sub‑tasks.
RoboCasa365 is a large‑scale simulated household suite containing 365 distinct tasks, many of which require multi‑step planning and object manipulation.
**Figure 13** Visual Comparison between ProWAM and Baseline. Top shows the baseline policy stalls early, leaving bottles unplaced (4/6). Bottom shows that our policy clears the entire table (6/6).
**Table 1.** Success rate (%) on LIBERO-series. Plus* is strictly used for zero-shot evaluation.
**Table 2.** Success rate (%) on RoboTwin. Random* is used for OOD evaluation.
**Table 3.** RoboCasa365. Table 3 reports results on RoboCasa365. ProWAM achieves a 48.1% overall success rate and ranks second overall among the evaluated methods. It also delivers competitive performance on long-horizon composite tasks. Notably, ProWAM achieves the second-highest success rate on the challenging long horizon Composite-Unseen split, substantially outperforming prior world-model baselines. These results support the effectiveness of progressive visual sub-goals for long-horizon execution and generalization to held-out task compositions.
**Table 4.** Real-world performance.
**Figure 3. Failure case on “Cube Stacking.”** The failed sub-goal (bottom) exhibits more ambiguous contact geometry than the successful one (top), and the gripper subsequently opens prematurely.
**Figure 4.** Efficiency and sub-goal ablations. (a) OOD success versus inference computation across WAMs. (b) Performance and computation across different sub-goal counts, where $k = 4$ yields peak OOD performance with efficient inference. (c) Performance across Near, Mid, and Far progress schedules. Near achieves the best OOD result while ID performance remains stable.
**Figure 5.** Closed-loop sub-goal visualization. For each observation at $t_0$ and $t_1$, the model generates sub-goals in a single pass at progress levels $r \in \{0.3, 0.5, 0.7\}$ to guide action generation.
Related Work
We position ProWAM among prior world‑action and vision‑language‑action models.
World Action Models (WAMs) use generative video priors to condition robotic policies, typically via a cascaded pipeline that first predicts future visual states and then recovers actions.
Modern WAMs unify prediction and action generation with joint diffusion objectives, forming an imagination spectrum ranging from full‑imagination WAMs to zero‑imagination WAMs.
ProWAM distinguishes itself by predicting a variable‑length sequence of future visual states indexed by relative progress, providing guidance at multiple stages of the remaining trajectory.
Vision‑language‑action (VLA) models transfer multimodal foundation knowledge to robotics, often representing actions with discrete tokens or continuous diffusion/flow‑matching objectives.
Hierarchical VLAs address long‑horizon tasks by introducing textual sub‑instructions, multimodal reasoning traces, or visual sub‑goals, but these pipelines often rely on decoupled modules that increase computational complexity and error propagation.
ProWAM integrates progressive visual sub‑goals directly into its video‑generation pathway and couples them to action generation within a unified WAM, avoiding separate planners or online trajectory search.
Prior approaches that insert intermediate foresight either use standalone goal‑generation modules or perform online model‑based trajectory search, both of which suffer from error propagation or prohibitive latency for real‑time control.
Pilot Study and Ablations
Ablation study of Final‑Goal Prediction across imagination regimes and data fractions.
The pilot study inserts the Final‑Goal Prediction (FGP) module into both zero‑imagination and full‑imagination WAM regimes and measures success rates at three training‑data fractions (0.3, 0.5, 1.0).
All model variants share the same MoT flow‑matching diffusion backbone (Wan2.2‑TI2V‑5B) and text conditioning (T5 embeddings), differing only in how the action expert cross‑attends to video and whether an auxiliary goal‑prediction target is added.
Table 5 lists the observation and action spaces for the three simulation benchmarks; each uses a 33‑frame window, a 4× video‑to‑action ratio, and language conditioning, with differences only in camera count, video resolution, and dimensionality.
Training data are subsampled deterministically: for a ratio R∈{0.3, 0.5, 1.0} the first R fraction of shuffled episodes (fixed seed) is kept, yielding nested subsets (D0.3 ⊂ D0.5 ⊂ D1.0). Optimizer steps scale linearly with R, and hardware allocation varies (LIBERO on 8 GPUs, RoboTwin/RoboCasa on up to 64 GPUs).
By allowing the action expert to co‑attend to a native goal representation (FGP), the model supplies long‑horizon guidance without extra overhead, improving sample efficiency especially when training data are scarce.
Real-World Experiments and Limitations
We examine real‑world transfer, a simulated ABC benchmark, and discuss current limitations.
World Action Models compress an entire episode into a single final‑goal prediction, which hampers long‑horizon tasks; ProWAM instead generates progressive visual sub‑goals to steer actions step‑by‑step.
Implementation Details. We train on the DROID dataset, following the two‑stage recipe used in earlier experiments. Stage 1 pre‑trains the video expert with the progressive sub‑goal objective; Stage 2 jointly fine‑tunes the video expert and the action expert using a differential learning rate. The three DROID views are composited into a $576\times640$ image, each clip spans $33$ frames at stride $1$, and actions are predicted as a chunk of $H=24$ absolute joint‑position targets, of which the first $H_{ol}=24$ are executed open‑loop at $15\,$Hz before replanning.
**Table.** Training hyperparameters for the two stages of the model.
Real‑world Experimental Settings & Analysis. We deploy the policy on a single Franka FR3 arm equipped with a Zed Mini wrist‑mounted camera and two Zed 2i exterior cameras; monocular images are taken from the left‑rectified sensor of each camera to match the original DROID data. Evaluation uses five trials per task on randomized object configurations across four tasks: basic pick‑and‑place, long‑horizon multi‑object assembly, contact‑rich placement, and spatially grounded placement.
Physical Setup (real robot). The hardware mirrors the DROID setup: the wrist view occupies the upper two‑thirds of the composite image, while the two exterior views are tiled below, yielding the $576\times640$ input used for inference.
Real‑world Experiment Performance. Zero‑shot evaluation on the Franka FR3 shows ProWAM achieves an average success rate of $70.0\%$, outperforming the strongest baseline by an absolute $15.0\%$ margin.
**Figure 7.** Pre-training curves for the sub-goal video expert. Both training losses (sub-goal and clip) and the validation (Val) loss decrease overall throughout pre-training.
Implementation Details (ABC experiment). We follow a training strategy similar to LIBERO and RoboTwin, using the ABC dataset’s mix of $ \approx 43\%$ real‑robot and $ \approx 57\%$ simulated demonstrations. The three cameras (head + two wrists) are composited into a $384\times320$ image, actions and proprioception are $z$‑score normalized, and we train with a single sub‑goal $k=1$ at progress $r=0.7$.
Physical Setup (ABC simulation). The benchmark models a dual‑arm system with two $7$‑DoF arms and head‑ and wrist‑mounted cameras, providing a realistic bimanual manipulation environment.
Evaluation protocol. The official ABC evaluator runs closed‑loop rollouts on $20$ seed‑matched worlds, each rollout consisting of $120$ action chunks of $15$ executed steps. We report the strict AVG (all six bottles placed) and the mean maximum number of bottles placed out of $6$.
Results. Table 15 shows that our policy succeeds in $7$ of $20$ worlds versus $1$ for the official ABC policy, and achieves a higher mean bottle count ($4.85$) compared to $4.10$.
**Table 15.** Performance on ABC put-bottles-in-bin Task.
**Figure 8.** Visualized sub-goal prediction of our pretrained video expert. Each example shows the initial frame (Initial) and predicted sub-goals at five progress values $r \in \{0.1, 0.3, 0.5, 0.7, 0.9\}$, with our prediction (Ours, top) above the ground truth (GT, bottom). Left: in-domain training embodiments (DROID, AgiBot). Right: out-of-domain environments unseen during pre-training (LIBERO, RoboTwin). The selected examples qualitatively illustrate transfer to environments not used during video pretraining.
DROID is a collection of real‑world robot demonstrations captured from three synchronized camera views, providing dense visual supervision for progressive sub‑goal learning.
Limitation: Manually Specified Inference Schedule. Although ProWAM can handle variable sub‑goal counts, the current inference schedule is hand‑crafted rather than dynamically inferred from task state, limiting adaptability.
Limitation: Model and Pre‑training Scale. We use a moderately sized video expert and a compact action expert; the impact of scaling these components or enlarging the pre‑training corpus remains unexplored.
Limitation: Task Diversity. Evaluated horizons are still modest compared with long‑term real‑world deployments, and more dexterous tasks have yet to be examined.
Training Hyperparameters
Training hyperparameters, video‑only pretraining details, and pilot FGP results.
The video‑only pretraining stage equips the policy with a strong visual dynamics prior before any action head is attached, allowing the downstream action expert to build on an already‑capable video predictor.
This table lists the hyperparameters used for training, including the optimizer (AdamW), peak learning rate ($1 \times 10^{-4}$), learning rate schedule (cosine), number of epochs, global batch size, gradient accumulation, weight decay ($1 \times 10^{-2}$), max gradient norm (1.0), flow-matching timesteps, and loss weights.
Progressive sub‑goal conditioning drives the video predictor: for each clip we sample up to $K=8$ future sub‑goal frames at evenly spaced progress positions and condition the model on them jointly with the current observation. Sub‑goals are injected with a zeroed rotary position and the loss is computed bidirectionally, with the sub‑goal and clip losses weighted equally.
**Table 7.** Pilot evaluation of FGP. Average SR (%) under (30%, 50%, 100%) training-data regimes.
Simulation Implementation Details
Simulation training setups, benchmark details, and efficient inference tricks.
Training follows a two‑stage recipe: Stage 1 (Cold‑start) adapts only the video expert on downstream video data, preserving its future‑prediction prior; Stage 2 (Joint SFT) attaches the action DiT and co‑trains video and action with mixed‑attention coupling, using equal loss weights $λ_{video}=λ_{sub}=λ_{act}=1$.
In‑domain environments test three commands—placing a blue block in a green bowl, picking up a left sneaker, and retrieving corn—each evaluated at progressive progress values $r=0.1,0.3,0.5,0.7,0.9$.
Out‑of‑domain environments assess generalization with commands such as putting a bowl on a plate, grabbing a synthetic leather shoe, and grasping a light wood roller, again across the same $r$ schedule.
LIBERO comprises four single‑arm suites (Spatial, Object, Goal, Long) with 10 tasks each and 500 expert demonstrations; we evaluate average success rate (ASR) over 2000 trials (50 per task).
LIBERO‑Plus adds perturbations (camera, robot, language, lighting, background, sensor noise, layout) and is used solely for zero‑shot testing, keeping training data identical to LIBERO.
RoboTwin features 50 bimanual tasks with 2,500 clean‑scene and 25,000 randomized‑scene trajectories; we train on the clean set and evaluate cross‑domain generalization on the randomized split.
RoboCasa365 contains 365 everyday tasks across 2,500 kitchen scenes; training uses the Human300 dataset (300 tasks) and evaluation reports success on 18 Atomic‑Seen, 16 Composite‑Seen, and 16 Composite‑Unseen tasks.
Table 8 lists the LIBERO two‑stage hyperparameters: Stage 1 trains the video expert with random sub‑goals $k\sim U\{1,\dots,8\}$ at learning rate $1\!\times\!10^{-4}$, batch 64, 10 epochs; Stage 2 adds the action head, fixes horizon = 16, uses $k=1$ at $r=0.7$, batch 128, cosine schedule.
Table 9 shows RoboTwin hyperparameters: Stage 1 runs a single epoch of video‑only adaptation (learning rate $5\!\times\!10^{-5}$); Stage 2 co‑trains with $k=4$ sub‑goals at progress points $r=0.1,0.2,0.35,0.5$, using batch 1024 and AdamW optimizer.
Table 10 details RoboCasa365 settings: Stage 1 uses a constant learning rate $1\!\times\!10^{-5}$ and batch 64; Stage 2 trains with horizon $H=32$, $k=6$ progressive sub‑goals ($r\in\{0.3,0.4,0.5,0.65,0.8,0.9\}$), batch 1024, cosine schedule, and AdamW optimizer.
Table 11 reports closed‑loop success rates on RoboCasa365, where ProWAM ranks fourth overall (48.1 % overall, 76.0 % Atomic‑Seen, 46.5 % Composite‑Seen, 18.2 % Composite‑Unseen).
Table 12 compares KV‑cached inference variants: KV‑1 and Joint‑10 achieve identical OOD success (85.8 %) while KV‑1 cuts computation by 86.7 %; ProWAM‑Light removes explicit sub‑goal tokens, preserving in‑domain performance but dropping OOD success to 80.3 % with modest compute savings.
Efficient inference decouples video and action branches: the video expert runs once per replanning step to cache sub‑goal key‑value features, which the action head reuses across all diffusion denoising steps, reducing video‑expert calls from $T\approx10$ to $1$.
Beyond explicit sub‑goal conditioning, ProWAM‑Light masks sub‑goal tokens during training and inference, forcing the action head to rely solely on the current observation cache; this yields comparable in‑domain results but a modest OOD drop, highlighting the generalization benefit of explicit sub‑goals.