InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization

Haoxiang Ma, Junhao Cai, Xiaoxu Xu, Hao Li, Yuyin Yang, Yang Tian, Jiafei Cao, Hongrui Zhu, Zherui Qiu, Zhaxizhuoma, Yuqiang Yang, Jiaqi Peng, Xueyuan Wei, Yangkun Zhu, Jiahao Jiang, Xing Gao, Hanqing Wang, Feng Yuan, Kailin Li, Xueyue Zhu, Tai Wang, Yan Ding, Jiangmiao Pang, Jia Zeng, Jingjing Zhang, Bowen Zhou, Yao Mu, Chunhua Shen, Weinan Zhang

InternVLA-A1.5 unifies VLM-based reasoning with latent foresight to enable robust, real-time robot control.

How can we unify VLM semantic priors with physical dynamics prediction in a single robot policy without degrading the VLM's original reasoning capabilities?

Unified robot policies often struggle to balance semantic understanding with physical dynamics, frequently degrading the vision-language backbone or incurring high latency by generating future pixels from scratch. InternVLA-A1.5 addresses this by using a native vision-language model backbone and a lightweight expert that queries a frozen video generator for future states via compact latent tokens, rather than predicting pixels. This design preserves semantic instruction-following while inheriting world-model dynamics, achieving state-of-the-art performance on six simulation benchmarks and superior compositional generalization in real-world tasks.

Paper Primer

The model architecture uses a Mixture-of-Transformers (MoT) design: a pretrained vision-language backbone handles multimodal perception, while a lightweight unified expert generates continuous actions. The core move is latent foresight: learnable tokens query a frozen video generation model to distill spatiotemporal priors into a compact latent code, which conditions the action head without requiring pixel-level generation at inference.

InternVLA-A1.5 achieves superior compositional generalization and long-horizon execution compared to existing VLA models.

In real-world held-out instruction binding tasks, the model consistently outperformed baselines, and on the long-horizon MOF chemistry procedure, it reached a 76.4% success rate compared to 29.3% for the next-best baseline. 20.8 percentage point lead on specific insertion tasks and significant gains in long-horizon sequential execution.

The model maintains real-time inference speed by discarding the video generation branch at deployment.

Inference takes approximately 0.1s per step on an NVIDIA RTX 5090, avoiding the per-step generation cost associated with world-model-based methods.

Why use a frozen video generator instead of training a dynamics model from scratch?

Training from scratch is computationally expensive and often fails to capture complex physical dynamics. By querying a frozen, pretrained video model, the policy inherits powerful spatiotemporal priors without the overhead of pixel-level generation.

How does this model avoid the semantic drift common in unified VLA models?

The model keeps the vision-language backbone training on VQA and subtask prediction throughout policy learning, ensuring that the semantic and instruction-following capabilities are preserved rather than eroded by the action-generation objective.

By decoupling latent foresight from pixel-level generation, InternVLA-A1.5 demonstrates that robots can leverage large-scale generative world models for control without sacrificing real-time performance or semantic grounding.

Motivation and Problem Framing

We expose semantic erosion and present InternVLA‑A1.5, a VLM‑based policy with a lightweight unified expert for latent foresight.

Unified robot policies aim to combine the semantic priors of pretrained Vision‑Language Models (VLMs) with the physical dynamics learned through future prediction. Existing designs, however, let the semantics of the frozen backbone drift—a phenomenon we call semantic erosion—while also training future prediction from scratch in pixel space, missing the dynamics priors of large video generators.

When a pretrained VLM is forced to serve heterogeneous objectives (language, vision, action), its internal representations gradually shift away from the original semantic space, weakening instruction‑following ability.

InternVLA‑A1.5 solves this by (1) keeping the VLM backbone frozen and training it on VQA and sub‑task prediction throughout policy learning, and (2) attaching a lightweight unified expert that handles continuous action generation and latent foresight queries without disturbing the backbone’s semantics.

**Figure 1. Overview of InternVLA-A1.5.** InternVLA-A1.5 unifies understanding, latent foresight, and action by attaching a lightweight expert to a pretrained VLM backbone. It is co-trained on vision-language and robot manipulation data, and introduces learnable foresight tokens to extract task-relevant future information as a compact latent representation. This representation is supervised by a frozen video generation model, which is used only during training and removed at inference. Extensive simulation and real-world experiments validate the effectiveness of the design.

The core trade‑off is preserving semantic reasoning from the VLM while injecting world‑dynamics via latent foresight, achieving both robust instruction following and long‑horizon physical control.

The Unified Expert Architecture

How the model learns to predict future‑aware continuous actions.

Stage 1 gives the frozen VLM a token‑level interface for robot and VQA data, but its discrete FAST tokens are too slow for real‑time control. Stage 2 therefore adds a lightweight “unified expert” that shares the VLM’s full‑attention layers while providing its own linear‑attention streams, and equips it with a foresight‑reasoning module that injects future‑state priors from a frozen video model.

The unified expert sits alongside the frozen VLM, re‑using the VLM’s full‑attention layers but adding three gated “DeltaNet” layers that operate on its own token groups, so the model can keep the VLM’s semantic reasoning while learning a fast, continuous control head.

How is the unified expert different from a typical adapter layer?

An adapter merely adds a small feed‑forward bottleneck on top of a frozen backbone. The unified expert, by contrast, introduces its own token groups and three gated linear‑attention streams that can attend bidirectionally within the group and causally to the VLM context, giving it a separate computational path for continuous control rather than a thin residual tweak.

Learnable foresight tokens query a frozen video‑generation model, pulling in spatiotemporal priors that describe how the scene is likely to evolve, and feed those priors into the expert’s action generator.

Concatenate $H_t$ and $Q_f$ → matrix of shape $3\times4$ (one $H_t$ row plus two $Q_f$ rows).

Apply the shared transformer $\Phi_\theta$ (here a single linear layer for illustration) → $Z_f^t$ becomes $\begin{bmatrix}0.6&-0.3&0.2&0.2\\ 0.2&0.1&-0.1&0.3\\ -0.1&0.2&0.0&0.4\end{bmatrix}$.

Project with $P_{\text{WAN}}$ (identity for the toy example) → $C_f^t = Z_f^t$.

Feed $C_f^t$ into the frozen video model; the model’s cross‑attention produces a future‑conditioned embedding that later guides action flow‑matching.

Even with only two tiny tokens, the transformer can blend the current observation $H_t$ with the learnable queries, producing distinct conditioning vectors that the video model treats as “future hints”. This illustrates how a small number of parameters can inject rich spatiotemporal priors.

Why not replace the foresight tokens with a simple concatenation of raw video frames?

Raw frames would explode the token budget and break the VLM’s fixed‑size embedding table. The learnable tokens act as a compressed query that extracts only the most useful future information from the frozen video model, keeping the overall architecture lightweight and trainable.

Inference loop for continuous action generation using flow‑matching.

**Figure 2.** Framework of InternVLA-A1.5. The architecture adopts a Mixture-of-Transformers design comprising a pretrained VLM for multimodal perception and a lightweight unified expert that shares full attention layers with the VLM while maintaining separate linear attention layers. The unified expert uses learnable foresight tokens for training-only visual foresight supervision and action query tokens decoded via flow matching for continuous action prediction.

**Figure 3. The chat-template layout.** Robot action samples and VQA samples share the same prompt-and-label structure. The prompt concatenates $K$ image blocks, the task instruction, the control mode $m$, and the discretized state. The label part carries the supervised target: a subtask description and a FAST action token segment for robot data, or the answer span for VQA. Only label tokens contribute to the next-token cross-entropy.

**Figure 4.** The foresight reasoning mechanism. Learnable foresight query tokens attend to the current visual-language context through the unified expert and produce conditioning embeddings for the frozen video generation model. The video supervision loss encourages these tokens to encode future-relevant information, which is further reused to guide continuous action generation.

**Figure 5.** Attention masking pattern of InternVLA-A1.5 during training and inference. The VLM tokens follow the causal attention pattern of Qwen3.5. The unified expert uses group-wise causal attention across token groups and bidirectional attention within each group. During training, the unified expert is prevented from attending to FAST action tokens to avoid information leakage and gradient interference.

Training proceeds in three stages: Stage 1 learns a token‑level VLM executor; Stage 2 adds the unified expert, the foresight pathway, and flow‑matching losses; Stage 3 fine‑tunes the whole system with a smaller batch size and a cosine‑decay learning‑rate schedule.

Stage 1: AdamW, LR $5\times10^{-5}$, batch 1024, 300 K steps, no foresight tokens.

Stage 2: AdamW, same LR and batch, 600 K steps, 50 foresight tokens, loss $L_{\text{stage2}} = L_{\text{stage1}} + \alpha L_{\text{video}} + \beta L_{\text{action}}$ with $\alpha=1$, $\beta=10$.

Post‑train: AdamW, batch 128, cosine LR decay from $5\times10^{-5}$ to $5\times10^{-6}$ over 60 K steps, keep 50 foresight tokens and 50‑step action chunks.

Data Composition and Scaling

How the training data are assembled, balanced, and sampled for InternVLA‑A1.5.

Training a policy that predicts both actions and future observations requires a balanced data mix; otherwise the abundant synthetic frames drown out the scarce real‑world diversity that is crucial for robust manipulation.

The robot corpus stitches together six heterogeneous datasets—one large simulation and five real‑world collections—into a single, massive stream of episodes and frames that the model can learn from uniformly.

Raw frame counts: Sim = 300 k, Real‑A = 50 k, Real‑B = 20 k.

Without balancing, a random batch would contain ~85 % Sim frames, starving Real‑A/B.

Applying the two‑level sampler (see later) first groups each source, then draws sub‑samples proportional to (#frames)$^{γ}$ with $γ=1$, yielding per‑source probabilities proportional to 300 k, 50 k, 20 k.

After inter‑group re‑mix weighting (e.g., up‑weight Real‑A and Real‑B by factors 2.5 and 3.0), the effective sampling probabilities become roughly 0.55 (Sim), 0.30 (Real‑A), 0.15 (Real‑B).

A batch of 1000 frames now contains ~550 Sim, 300 Real‑A, 150 Real‑B frames, preserving diversity.

The sampler converts a raw 15:1 simulation‑to‑real frame ratio into a much more balanced 3.7:1 ratio, ensuring the model sees enough real‑world experience to learn robust manipulation.

Why not simply discard most simulation frames instead of re‑weighting them?

Simulation provides coverage of rare poses and failure modes that real‑world data rarely capture. Discarding them would reduce the diversity of state‑action pairs, harming the model’s ability to generalize to unseen situations.

The multimodal stream supplies language‑grounded visual tasks so the frozen VLM backbone retains its broad semantic and spatial reasoning while the robot stream teaches action.

Even with balanced streams, the raw frame counts across robot sources are wildly skewed, which would still let the largest dataset dominate every training batch.

The sampler first treats each data source as a separate group, then draws sub‑datasets inside each group proportionally to their frame counts raised to a tunable exponent, finally mixing groups with manually‑adjusted weights.

Compute intra‑group probabilities: Sim‑1 = 250/400 = 0.625, Sim‑2 = 150/400 = 0.375.

Apply Re‑Mix to obtain raw group weights proportional to 1/√(frame count): Sim ≈ 0.05, Real‑A ≈ 0.18, Real‑B ≈ 0.45.

Manually boost Real‑A and Real‑B by factors 2.0 and 2.5 respectively, yielding final group weights: Sim = 0.05, Real‑A = 0.36, Real‑B = 1.13 (renormalized to sum = 1).

During batch construction, first pick a group according to these weights, then pick a sub‑dataset inside the group using the intra‑group probabilities.

The resulting expected frame contribution per batch is roughly 10 % Sim, 30 % Real‑A, 60 % Real‑B, dramatically increasing exposure to the scarce real‑world data.

The two‑level scheme lets us preserve the richness of large simulation datasets while guaranteeing that every real‑world source appears frequently enough to influence the learned policy.

How does this sampler differ from naïve oversampling of the small datasets?

Naïve oversampling would duplicate the same few episodes, leading to overfitting. The grouped sampler preserves the original distribution within each source (so diverse scenes are retained) while only adjusting the overall frequency of each source.

**Figure 6.** Overview of the training data for InternVLA-A1.5. (a) Robot manipulation data. Six sources, one synthetic (InternData-A1) and five real-world, are unified into a single action space and together provide 1.2M episodes and 861M frames. Each card reports the episode count, frame count, and sampling weight of a source, and the outer ring shows its share of frames. By frame count the corpus is split roughly evenly between simulation and real-world data, while the sampling weights shift the effective mixture toward the real-world sources. (b) Multimodal data. The InternVLA-M1 corpus contributes about 3M samples across four QA categories, one for general vision-language ability and three for robotics-oriented spatial grounding, with representative examples shown on the right.

The robot manipulation corpus’s sheer scale (1.2 M episodes, 861 M frames) combined with a carefully balanced sampling scheme delivers both breadth and diversity for robust policy learning.

Real-World Manipulation Performance

Real‑world robot trials reveal InternVLA‑A1.5’s success rates across four tasks.

InternVLA‑A1.5 attaches a lightweight “unified expert” to a frozen VLM, letting the policy learn latent foresight while preserving semantic reasoning.

InternVLA‑A1.5 attains 76.4 % success on the long‑horizon MOF task, far outpacing $\pi$0.5 (29.3 %) and Motus (0 %).

Figure 8 shows the MOF success rates for the three methods.

Benchmarks that test a robot’s ability to map a language instruction to a concrete manipulation, where only a subset of (object, target) bindings are seen during training.

**Figure 13.** The long-horizon MOF task. The first panel gives the full instruction, and the remaining panels show the 13 sequential subtasks together with the final completed state, from inserting the funnel and pouring the liquid to stoppering the flask and switching on the stirrer. White outlines mark the object or target involved in each step.

**Figure 7.** Overview of the four real-world tasks. The first three evaluate instruction following, where the robot brings a specified tube to a specified target, namely a box in Sort Tubes, a hole of a tube rack in Insert Tubes, and a position on the opposite side in Move Tubes. MOF is a long-horizon chemistry procedure. In each panel, colored outlines mark the task-relevant tubes and targets.

**Figure 8. Real-world results across instruction-following and long-horizon tasks.** We report overall success rates on Sort Tubes, Insert Tubes, Move Tubes, and MOF, comparing $\pi_{0.5}$, Motus, and InternVLA-A1.5.

**Figure 9.** Generalization to seen and out-of-distribution instruction bindings. We decompose performance into seen vs. OOD combinations across the three instruction-following tasks.

**Figure 12. Test-tube sorting task.** From left to right: (left) the raw scene, containing orange- and blue-tipped plastic tubes and a left and right box; (middle) the seen bindings present in the training demonstrations; and (right) the held-out bindings, evaluated only at test time, where the arm-to-color assignment is reversed. In (middle) and (right), each tube and its target box are drawn in matching outline colors.

Table 3 reports RoboTwin results; InternVLA‑A1.5 achieves 93.3 % (clean), 93.0 % (randomized), and 93.2 % average, surpassing all baselines.

Table 4 (DOMINO) compares zero‑shot and fine‑tuned settings; InternVLA‑A1.5 improves SR by 27.7 % (zero‑shot) and 29.3 % (fine‑tuned) over the next best baseline, confirming its robustness to dynamic manipulation.

Simulation Benchmark Evaluation

InternVLA‑A1.5 sets new simulation performance records across six benchmarks.

InternVLA‑A1.5 beats the strong baseline $\pi$0.5 by 23.7 percentage points on the SimplerEnv benchmark.

Average success rate 80.8 % versus 57.1 % for $\pi$0.5 (Table 2).

**Table 3.** Results on RoboTwin. We report success rates under clean and randomized evaluation settings, together with their average.

**Table 6.** Results on LIBERO-Plus. The best and second-best results are highlighted in bold and underline, respectively.

**Table 7.** Results on EBench. SR (%) is the primary metric, Score denotes continuous task progress (%). The best and second-best results are highlighted in bold and underline, respectively.

**Figure 11.** Foresight-conditioned future rollouts from the frozen world model. The model predicts future frames ($t \sim t + 4$) conditioned on foresight embeddings extracted from the VLM context, demonstrating both accurate motion prediction and physically consistent scene evolution.

Ablation Studies and Mechanism Analysis

We dissect how each component influences performance and training speed.

We run ablations to isolate the contribution of the foresight reasoning mechanism by disabling individual components while keeping the pretrained backbone fixed.

**Table 8.** Ablation studies on the foresight reasoning. All experiments are conducted on the two-stage pretrained InternVLA-A1.5.

These drops indicate that video‑level supervision supplies invariance to visual perturbations and reinforces motion consistency, whereas the foresight tokens act as the conduit that injects structured dynamics from the frozen video model into the unified expert.

**Figure.** Training loss curves for different models over 60K training steps.

The faster convergence translates into fewer fine‑tuning steps needed to achieve comparable performance, confirming that the pretrained representations reduce the difficulty of policy adaptation.

Visualizing rollouts from the frozen WAN model conditioned on the learned foresight embeddings reveals coherent multi‑step predictions of both articulated robot motion and downstream physical effects such as liquid level changes, demonstrating that the tokens successfully distill spatiotemporal priors into actionable control knowledge.

Technical Details and Reproducibility

Appendix details the real‑world tasks and simulation benchmarks used for evaluation.

A.1 Real‑world Task Details enumerates three instruction‑following tasks and one long‑horizon synthesis task used to probe the robot policy.

Sort Tubes requires the robot to pick a tube of a specified color with a named arm and place it into the same‑side box; four (arm, color) bindings exist, evaluated in a full‑coverage setting and a held‑out setting where bindings are reversed.

Insert Tubes asks the robot to insert a tube of a given color into a specified hole; training covers three holes per color, while the out‑of‑distribution test uses the unseen (color, hole) pair.

Move Tubes involves moving a tube from the left rack to a hole on the right rack; each color is trained on a subset of right‑side holes, and the held‑out evaluation swaps the target holes.

MOF (Multi‑Object Foresight) reproduces a metal‑organic framework synthesis procedure consisting of 13 sequential subtasks, requiring the robot to track progress across subtle scene changes such as liquid level.

A.2 Simulation Benchmark Details lists six benchmark suites that stress different aspects of generalization and robustness.

LIBERO comprises four task suites (Spatial, Object, Goal, Long) with 10 tasks each and 50 human‑teleoperated demonstrations per task, yielding 2 000 demonstrations total; a single model is fine‑tuned on the mixture and evaluated per suite over 500 rollouts.

LIBERO‑Plus augments LIBERO with systematic visual and layout perturbations; the pretrained LIBERO checkpoint is evaluated zero‑shot on these perturbed tasks.

RoboTwin 2.0 contains 50 bimanual manipulation tasks, each with Easy (clean) and Hard (domain‑randomized) variants; InternVLA‑A1.5 is fine‑tuned on 27 500 demonstrations (50 clean + 500 randomized per task) for 100 k steps on 24 GPUs (batch size 16 per GPU, peak LR $1 \times 10^{-4}$, decay over $140\text{k}$ steps).

DOMINO evaluates precise action execution under dynamic constraints using a continuous Manipulation Score (MS); InternVLA‑A1.5 is tested zero‑shot after RoboTwin fine‑tuning and also after additional fine‑tuning on DOMINO Level‑1 data (35 tasks, 100 rollouts each).

EBench comprises 26 mobile‑manipulation tasks annotated along five capability and four generalization dimensions; InternVLA‑A1.5 is fine‑tuned with batch size 128, AdamW optimizer, cosine LR schedule (peak $5 \times 10^{-5}$) for 100 k steps, achieving faster convergence than baselines.

SimplerEnv assesses cross‑embodiment generalization on four manipulation tasks; training follows the same AdamW, batch‑size‑128, cosine schedule (peak $5 \times 10^{-5}$) protocol as EBench.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers