PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models
DeepCybo Team, Yu Bin, Haipeng Cao, Zheng Chang, Kai Chen, Youning Chen, Kailin Deng, Yichao Du, Xiaotong Fu, Haoyang Ge, Yunlong Guo, Chenliu Hao, Jiyan He, Xuguo He, Yakun Hou, Kai Hu, Cong Huang, Tuopusen Huang, Yu Huang, Hong Li, Peize Li, Shijie Lian, Xiaopeng Lin, Yun Lin, Haibao Liu, Haochen Liu, Qiuzhi Liu, Shengcai Liu, Zhiqiang Liu, Tao Luo, Peng Ren, Shuo Ren, Chaoyi Ruan, Zhaolong Shen, Yukun Shi, Qiyuan Su, Yuxuan Tian, Yining Wang, Changti Wu, Hao Wu, Xueyin Xu, Ruoqi Yang, Zhaoyang Yang, Hang Yuan, Zhaoyang Zeng, Hanwen Zhang, Ruimeng Zhang, Yao Zhang, Yibo Zhang, Yuxiang Zhang, Zhirui Zhang, Ziyi Zhang, Zubin Zheng, Zishen Zhuang
PhysBrain 1.5 unifies embodied understanding, action generation, and future-state prediction into a single autoregressive model.
How can we unify embodied action generation, physical state prediction, and visual-spatial understanding into a single vision-language model architecture?
Embodied agents struggle to bridge the gap between high-level semantic understanding and low-level physical interaction, often requiring separate, specialized models for perception, planning, and execution. PhysBrain 1.5 unifies these tasks by encoding language, end-effector motion, and future visual states as discrete tokens, training them jointly through a shared autoregressive backbone. The 8B model achieves a 72.5 average score across 28 embodied benchmarks, setting a new open-source state of the art that rivals leading proprietary systems.
Paper Primer
The model treats the physical loop—observation, interaction, and environmental change—as a unified sequence prediction problem. It extends the language vocabulary with dedicated action tokens (representing end-effector trajectories) and visual tokens (representing future RGB, depth, and robot-mask states), allowing a single model to output text, motion, or future frames based on task-specific loss masks.
To handle diverse robot and human motion data without expensive global canonicalization, the model uses the preceding action chunk as a local motion context. This "conditional trajectory continuation" allows the model to infer embodiment-specific motion scales and conventions implicitly from the input history.
PhysBrain 1.5 sets a new open-source state of the art for embodied understanding.
Average score of 72.5 across 28 benchmarks, outperforming the next best open-source baseline by 6.5 points. Performs on par with proprietary models like Gemini 3.6 Flash (73.0) and GPT-6-Astra (73.3).
The model maintains broad multimodal competence alongside specialized physical skills.
Outperforms the base Qwen3-VL-Instruct (8B) model on all 28 embodied benchmarks while retaining general vision-language capabilities. Ranks first on 14 benchmarks and second on 10 among all open-source models evaluated.
Why use discrete tokens for actions and visual states instead of continuous regression?
Discrete tokenization allows the model to use a shared autoregressive backbone and output head for all modalities, enabling joint optimization of semantic, motion, and future-state supervision within a single training objective.
How does the model handle the heterogeneity of different robot and human motion datasets?
Rather than forcing a global coordinate frame, the model uses the preceding action chunk as a local motion prior, allowing it to adapt to source-specific motion scales, directions, and control frequencies dynamically.
The model's reliance on human interaction videos for pre-training is the primary driver of its physical priors, effectively transferring human-centric commonsense to robotic embodiments.
PhysBrain 1.5 demonstrates that a unified autoregressive interface is sufficient to capture the full physical loop, effectively collapsing the distinction between perception, planning, and execution into a single sequence-modeling task.
PhysBrain 1.5: Overview
We introduce PhysBrain 1.5, a unified model that learns the physical loop of observation, action, and prediction.
PhysBrain 1.5 extends a vision–language foundation by treating actions and future visual states as tokens, enabling a single model to perceive, act, and predict within physical environments. It is trained first on human‑interaction videos that provide embodied supervision, then refined with supervised fine‑tuning on a mix of demonstrations, robot trajectories, and simulated episodes. Across 28 embodied benchmarks the 8 B model scores 72.5, leading open‑source results and rivaling proprietary systems.
**Figure 1.** Leading overall performance on the report's embodied benchmark suite. PhysBrain 1.5 achieves an overall score of 72.5 with 8B parameters, ranking first among all open-sourced models and delivering performance on par with leading proprietary systems.
**Figure 8.** Examples of object pointing across diverse complex scenarios.
**Figure 9.** Examples of predicting the movement trajectories of target objects based on action instructions.
**Figure 10.** Examples of challenging spatial reasoning question answering.
**Figure 11.** An example of object scale estimation in a scene with multi-view observations.
**Figure 12.** An example of spatial imagination and orientation reasoning a scene with multi-view observations.
**Figure 13.** An example of object motion trajectory understanding in a short video.
**Figure.** A sequence of 24 frames showing a person organizing items into a green storage bin. The sequence depicts picking up a carrot, placing it in the bin, and then proceeding to clear other items from the table surface.
PhysBrain 1.5 matches proprietary models while setting the open‑source benchmark for embodied performance.
Architecture and Design
PhysBrain 1.5 unifies language, action, and visual tokens into a single autoregressive model.
Physical intelligence requires a model that can simultaneously understand a scene, decide how to act, and anticipate the resulting visual change. Existing VLMs excel at perception, but they lack a unified way to generate actions and future visual states, creating a bottleneck for end‑to‑end physical reasoning.
Instead of treating language, motion, and images as separate output spaces, we give the model a single “dictionary” that contains words, action commands, and visual symbols—all indexed by the same embedding table. The model then predicts the next token from this shared pool, regardless of modality.
How does this differ from simply appending new tokens to a vanilla VLM?
In a vanilla VLM the added tokens would sit on a separate output head, requiring distinct projection matrices and preventing gradients from action or visual supervision from updating the language backbone. Here the tokens share the same embedding and output projection, so learning signals from any modality flow through the same parameters, yielding true multimodal grounding.
Token 0 → “go”, Token 1 → “stop”.
Token 2 → “`move_forward`”, Token 3 → “`turn_left`”.
Token 4 → “
Given the prompt “go”, the model predicts Token 2 (“`move_forward`”) as the next action token.
After emitting the action token, the mask switches to the visual subset, and the model predicts Token 4 (“
The same softmax layer can emit a language word, an action command, or a visual symbol simply by changing the loss mask, demonstrating that the unified vocabulary truly bridges perception, action, and prediction.
Generating a multimodal sequence with a shared LM
**Figure 2. PhysBrain 1.5 model architecture.** Built on a pretrained vision–language model, PhysBrain 1.5 jointly learns embodied understanding, action generation, and visual-state prediction through a shared backbone, token embedding, and output head over a unified language–action–visual vocabulary. Embodied understanding covers foundational visual-spatial perception, spatial and multi-view understanding, embodied cognition and reasoning, spatial pointing and affordance understanding, and visual trajectory reasoning. Action tokens are decoded by ActionPiece into end-effector trajectories, with optional action history providing motion context; visual tokens are decoded by a Vision decoder into future RGB images, depth maps, and robot masks. The three task families provide complementary supervision under a shared autoregressive training objective.
By unifying the token space, PhysBrain 1.5 learns a single set of parameters that can reason about what to say, what to do, and what will happen next, eliminating the need for separate perception, planning, and prediction modules.
Embodied Understanding
PhysBrain 1.5 learns to continue robot actions by tokenizing short wrist trajectories and conditioning on recent motion.
Because datasets use wildly different robot frames, a single global coordinate system is impractical. Enforcing a universal frame would require costly calibration and risk corrupting data that lack such metadata. Instead, the model must infer the local motion convention from recent observations.
Training signals combine visual perception with the physical consequences of actions, so the model learns the loop of “see → act → observe change”.
How does Embodied Supervision differ from ordinary supervised learning on images?
Ordinary supervision only penalizes static label errors, while Embodied Supervision also penalizes mismatches between predicted actions and the resulting visual change, forcing the model to capture cause‑effect dynamics.
Each wrist motion is expressed as a short, relative trajectory that the model predicts token‑by‑token.
Relative translation: $p_{t+1} - p_t = (0.1,0,0)$.
Relative rotation matrix: $R_t^{\top} R_{t+1}$ is a 10° $z$‑rotation; its first two columns are $[ \cos10°, -\sin10°; \sin10°, \cos10°; 0,0]$.
Applying $\rho_6$ yields the 6‑D vector $(\cos10°, \sin10°, -\sin10°, \cos10°, 0,0) \approx (0.985,0.174,-0.174,0.985,0,0)$.
Gripper state $g_{t+1}=0.2$ is appended, forming $a_{t,1} = (0.1,0,0,\;0.985,0.174,-0.174,0.985,0,0,\;0.2)$.
Because all steps are expressed relative to the initial pose, the representation stays consistent even if the robot’s global frame differs across datasets.
Why predict a whole 16‑step chunk instead of regressing each step sequentially?
Predicting the chunk as a single token sequence lets the model capture temporal coherence and leverage the transformer’s parallel attention, whereas step‑by‑step regression would accumulate errors and break the autoregressive parallelism.
The tokenizer discretizes a 16‑step wrist trajectory into a short string of 32 tokens drawn from a 512‑token vocabulary.
How does the ActionPiece tokenizer differ from simply quantizing each joint angle?
Instead of naïvely rounding each joint value, the tokenizer learns a data‑driven codebook that captures common motion primitives, yielding more compact and semantically meaningful tokens that the transformer can attend to efficiently.
**Figure 3.** Qualitative examples of embodied understanding with PhysBrain 1.5. Representative cases illustrate visual-spatial perception, 3D and multi-view understanding, embodied reasoning and planning, spatial grounding and affordance, and visual trace and trajectory reasoning. Responses include language answers, spatial coordinates, and waypoint sequences, with predicted points and trajectories overlaid on the corresponding observations.
By conditioning on the recent action chunk $z^{-}$, the model treats trajectory continuation as a conditional language modeling problem, preserving continuity across heterogeneous datasets without explicit global alignment.
Visual-Foundation Generation
Predicts future RGB, depth, and robot mask as a unified token sequence.
The model must anticipate how a scene will look after an action, which requires predicting three tightly coupled visual streams at once. Treating RGB, depth, and robot mask as separate outputs would break spatial consistency and inflate the prediction burden.
Instead of predicting three independent images, the model emits a single sequence of tokens that jointly encode the future RGB frame, depth map, and robot mask, preserving spatial alignment across modalities.
Interleave the codes per location: $(2,7,1)$, $(5,0,1)$, $(1,4,0)$, $(3,6,2)$.
Prepend the start token and append the end token, yielding the sequence: \<|`future_joint_vq8_start`|>, $(2,7,1)$, $(5,0,1)$, $(1,4,0)$, $(3,6,2)$, \<|`future_joint_vq8_end`|>.
The total token count is $3N + 2 = 14$ for this toy example.
During generation the model predicts each triplet in order, preserving the spatial relationship across modalities.
After generation, the triplets are de‑interleaved back into three $2 \times 2$ grids and decoded to reconstruct the three future images.
Interleaving forces the language model to attend to RGB, depth, and mask tokens from the same pixel simultaneously, which yields coherent cross‑modal predictions despite using a single shared head.
**Figure 6.** Qualitative results of future visual prediction across diverse robot embodiments. Each row presents the current RGB observation and the predicted future RGB image, depth map, and robot mask. The examples shown use a one-second prediction horizon. The model preserves the dominant scene layout and object identity while anticipating task-consistent changes. Predictions across the three modalities are spatially coherent, jointly capturing future appearance, workspace geometry, and robot occupancy.
Training Objectives
We define a unified token scheme and an autoregressive loss that jointly train perception, action, and future‑state prediction.
Training separate heads for perception, action, and future‑state prediction forces the model to learn three disjoint token spaces, which hampers shared physical reasoning.
Actions, future visual states, and ordinary words are all encoded as the same kind of token, so the model can predict any of them with the same autoregressive decoder.
How does this unified tokenization differ from simply appending action vectors to the language sequence?
Because actions and visual frames are first discretized into learned tokens, the model predicts them with a softmax over a fixed vocabulary, just like words. Directly appending raw vectors would require a regression head and a different loss, breaking the shared autoregressive framework.
Action token lookup: “move‑right” → embedding $e_a$.
Future visual frame is rasterized into four visual tokens $v_1,\dots,v_4$ and each is looked up to embeddings $e_{v_i}$.
All embeddings are concatenated: $[e_a, e_{v_1}, e_{v_2}, e_{v_3}, e_{v_4}]$.
The decoder receives this sequence and predicts the next token, which in this toy case is the end‑of‑sequence token.
The example shows that heterogeneous modalities are reduced to a homogeneous token stream, enabling a single cross‑entropy loss to train them together.
**Table 1.** Composition of the two-stage training data. Physical-aware pre-training data are constructed from human interaction videos; embodied SFT combines curated human, real-robot, and simulation data. Both stages also include general language and vision-language instruction data.
Pre-training Data
Training data splits into physical pre‑training and embodied SFT, providing millions of samples per supervision type.
We organize training data into two stages: physical‑aware pre‑training and embodied supervised fine‑tuning (SFT). The pre‑training stage supplies perception, action, and future‑state supervision from large‑scale human interaction videos, while SFT expands the pipeline with real‑robot and simulated interactions.
Human action data provides the largest supervision set with 31.2 M samples.
Physical perception: 24.3 M; Human action: 31.2 M; Future‑state: 26.8 M; General instruction: 14.9 M.
Embodied SFT Data
Supervised fine‑tuning uses over three million hours of embodied action data.
PhysBrain 1.5 leverages a total of 3.12 M hours of embodied action data for supervised fine‑tuning.
Table 3 aggregates 2,620 h of robot/simulation trajectories with 500 h of high‑quality human motion.
The SFT corpus also mixes 1 M high‑quality general language‑vision instruction samples to preserve broad multimodal capabilities during embodied adaptation. Capability‑group scales are 500 K for foundational perception, 2.36 M for spatial and multi‑view understanding, 1.65 M for embodied reasoning and planning, 1.8 M for grounding and affordance, and 300 K for visual trajectory reasoning.
Before inclusion, all sources undergo resampling to raise temporal resolution, removal of invalid poses or timestamps, and a motion‑informativeness filter that down‑samples static segments while retaining novel interaction changes. The resulting embodied future‑state corpus contains 1.2 M training samples, each pairing current observations with aligned RGB, depth, and robot‑body mask targets.
Evaluation Benchmarks
PhysBrain 1.5’s embodied performance is quantified across a broad benchmark suite.
Recall that PhysBrain 1.5 treats observation, action, and future visual states as a unified token stream, enabling it to learn the physical loop of perception‑action‑feedback.
PhysBrain 1.5 attains an overall average of 72.5 across 28 embodied benchmarks.
Overall Average row in the evaluation results table.
PhysBrain 1.5 sets a new performance baseline for open‑source embodied multimodal models.
Multimodal Performance
PhysBrain 1.5 matches the baseline on general multimodal benchmarks while excelling on several tasks.
We independently re‑evaluate all comparison models with a unified metric per benchmark and a common preprocessing pipeline, ensuring a fair head‑to‑head comparison.
PhysBrain 1.5 outperforms Qwen3‑VL‑Inst. on ChartQA and V* benchmarks.
Table 6 shows 86.24 % vs 84.96 % on ChartQA and 87.96 % vs 83.77 % on V*.
Action and Prediction Analysis
We assess how dropping key prediction modules impacts model performance.
The paper’s core idea is to treat observations, actions, and future visual states as a unified token stream, enabling the model to learn the “physical loop” of perception‑action‑feedback.
**Figure 7. Action-token perplexity during pre-training.** The PhysBrain 1.5 model is evaluated on ID test data and OOD data from RoboDojo, which is excluded from training. ID test episodes are disjoint from training episodes.
Removing the action‑token prediction component hurts pre‑training perplexity by roughly 12 points.
Figure 7 shows perplexity falling from about 17 to 5.07 when the component is present.
**Figure 4.** Image-plane visualization of action trajectory predictions on held-out episodes. Predictions are generated by the post-trained model. Given a task instruction, the current image, and a preceding action chunk, PhysBrain 1.5 predicts the next 16 action steps for the requested wrist or wrists. Predicted and ground-truth end-effector positions are projected into the image plane and overlaid on the current observation: solid lines denote ground-truth trajectories, and dashed lines denote predictions. The corresponding task instruction is shown below each example. All examples are drawn from test episodes unseen during training. These visualizations compare predicted trajectories with ground truth rather than report executed robot rollouts.
**Figure 5** Three-dimensional visualization of action trajectory predictions in ID and OOD test scenes. Predictions are generated by the post-trained model. Each example shows the current observation and task instruction, alongside ground-truth (blue) and predicted (orange) trajectories for the left and right wrists. Black diamonds mark the trajectory origins, and axes are measured in centimeters. The $x$, $y$, and $z$ axes follow the coordinate conventions of the original robot action data for each example; coordinate systems are not aligned across sources. The top row presents in-distribution (ID) examples, while the bottom row presents out-of-distribution (OOD) examples from RoboDojo, which is excluded from the model's training corpus. These examples provide a qualitative comparison of predicted and ground-truth motion within and beyond the training distribution.
Training Configuration
Describes the two‑stage training pipeline and reports embodied‑understanding results.
We train PhysBrain 1.5 with ms‑swift1 leveraging the Megatron‑Core backend, initializing from the 8 B‑parameter Qwen3‑VL‑Instruct model and augmenting its token set with dedicated action and visual‑state symbols. Training follows a two‑stage schedule: first a physical‑aware pre‑training phase that familiarizes the model with embodied dynamics, then a supervised fine‑tuning (SFT) stage using curated interaction data. The implementation and scripts are publicly available at the provided GitHub link.
**Table 4.** Results on Embodied Understanding.
Supplementary Material
Appendix details on evaluation protocols, metrics, and visualizations.
The contributors are listed alphabetically by family name.
Section B documents the evaluation protocols and visualizations used for both embodied and multimodal benchmarks.
EmbodiedEvalKit3 provides a unified framework that aggregates a broad collection of embodied perception, reasoning, and planning benchmarks under a consistent interface.
We reuse each benchmark’s input protocols, prompting templates, and evaluation metrics; for RoboVQA we extend the visual input from eight to sixteen frames, and for ShareRobot‑Trajectory and VABench‑Visual‑Trace we report $100\!-\!\text{NRMSE}$ so that higher scores uniformly indicate better performance.
All point‑localization tasks are evaluated with a micro‑averaged F1 score that jointly measures precision and recall.
For a sample $i$, let $P_i$ be the set of predicted points and $G_i$ the set of ground‑truth target regions; the number of valid predictions is $H_i = \sum_{p \in P_i} \mathbf{1}\big[\exists g \in G_i \text{ such that } p \in g\big]$.
We construct a one‑to‑one match set $M_i \subseteq P_i \times G_i$ where each pair $(p,g)$ satisfies $p \in g$, and compute precision $P = \frac{\sum_i H_i}{\sum_i |P_i|}$, recall $R = \frac{\sum_i |M_i|}{\sum_i |G_i|}$, and F1 $= \frac{2PR}{P+R}$.
Six additional benchmarks (ERQA‑PLUS, ViewSpatial‑Bench, MindCube, MMSI‑Bench, 3DSRBench, Q‑Spatial‑Bench) are integrated from their official releases and evaluated using top‑1 accuracy or the prescribed success‑rate metric.
The general multimodal suite includes MME, MMStar, RealWorldQA, VideoMME, MVBench, POPE, AI2D, ChartQA, DocVQA, TextVQA, V*, and ScreenSpot, covering perception, reasoning, and GUI grounding.
Image inference uses non‑thinking mode, up to 32,768 generated tokens, temperature 0.7, top‑p 0.8, top‑k 20, repetition penalty 1.0, presence penalty 1.5, seed 3407, and a maximum image area of 4,014,080 pixels; the minimum area is 602,112 pixels for RealWorldQA and 200,704 pixels for other tasks.
Video preprocessing samples at 2 FPS, caps clips at 128 frames for VideoMME and 32 frames for MVBench, and enforces a per‑frame pixel ceiling of $768 \times 322 = 786{,}432$ pixels while aligning dimensions to multiples of 32.
Action‑token perplexity on the ID test set declines from 17.12 to 5.07, and on the held‑out RoboDojo corpus from 12.39 to 6.57, indicating improved prediction of reference actions despite the latter being excluded from training.
Qualitative visualizations illustrate PhysBrain 1.5’s capabilities in pointing, trajectory generation, and spatial‑relationship Q&A on examples drawn from sources beyond the 28 embodied benchmarks.