Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories

Xiaomi Robotics Team, Jun Guo, Piaopiao Jin, Jason Li, Peiyan Li, Yingyan Li, Futeng Liu, Wanli Peng, Optimus Qin, Yifei Su, Nan Sun, Qiao Sun, Runze Suo, Heyun Wang, Yunhong Wang, Rujie Wu, Caoyu Xia, Lina Zhang, Jack Zhao, Guoliang Chen, Wenlong Chen, Xinze He, Bin Li, Qing Li, Zhuorong Li, Heng Qu, Wenxuan Song, Diyun Xiang, Yifan Xie, Peiran Xu, Hangjun Ye, Wen Ye, Han Zhao, Quanyun Zhou

A foundational vision-language-action model trained on 100k+ hours of real-world trajectories to enable scalable robot manipulation.

How can we scale vision-language-action (VLA) models to achieve robust, out-of-the-box mobile manipulation using massive real-world trajectory data?

Robot learning is bottlenecked by the high cost and limited diversity of teleoperated data, which prevents policies from scaling like large language models. Xiaomi-Robotics-1 addresses this by training on over 100,000 hours of real-world manipulation trajectories, using an automated pipeline to annotate scene state transitions at scale. This massive pre-training translates into consistent performance gains as data and model size increase, setting new state-of-the-art results across multiple simulation benchmarks.

Paper Primer

The model uses a two-stage recipe: pre-training on massive, auto-labeled trajectory data to learn general action generation, followed by post-training to align these capabilities with specific robot embodiments and human-like imperative instructions. The architecture is a Mixture-of-Transformers (MoT) that couples a pre-trained Vision-Language Model (VLM) with a Diffusion Transformer (DiT) for action generation.

The core mechanism is a flow-matching objective: the DiT generates action chunks conditioned on the VLM's visual and language representations, while the VLM itself is supervised with auxiliary action-generation tasks to steer its internal features toward action-relevant representations.

Xiaomi-Robotics-1 establishes a new state-of-the-art on the RoboCasa365 benchmark.

The model achieved a 57.6% success rate, significantly outperforming the previous best of 46.6%.

Scaling pre-training data directly improves out-of-the-box performance in unseen environments.

Post-training success rates on novel tasks rose monotonically from 26% (no pre-training) to 75% (100% of pre-training data).

Why is an auto-labeling pipeline necessary for this approach?

Traditional manual segmentation and language annotation of robot trajectories are labor-intensive and prohibitively slow at the 100,000-hour scale required for effective scaling laws.

How does the model handle the difference between UMI handheld grippers and actual robot hardware?

The post-training phase uses cross-embodiment data to bridge the gap, aligning the general action-generation capabilities learned from UMI data to the specific kinematics and imperative instruction sets of mobile manipulators and robot arms.

The authors observe that model capacity at the billions-parameter scale may already be sufficient for the current dataset, suggesting that data volume—rather than model size—is the primary bottleneck for further generalization in robot learning.

Robot foundation models can follow the scaling laws of large language models if provided with massive, diverse, and auto-labeled trajectory data, enabling robust out-of-the-box performance in unseen environments.

Introduction

Introducing a scalable VLA model that overcomes robotics’ data bottleneck.

Robotics research is throttled by a unique data bottleneck: real‑robot teleoperation is slow, expensive, and yields highly redundant trajectories. Collecting robot data is like filling a bathtub with a teaspoon—each drop takes time, and the tub never fills fast enough.

A single, foundational model that ingests visual observations and language commands and directly outputs robot actions, enabling out‑of‑the‑box manipulation in previously unseen environments.

Training follows a two‑stage recipe: (1) pre‑training on >100 k hours of real‑world manipulation trajectories collected with UMI devices, using an auto‑labeling pipeline that generates natural‑language descriptions of scene‑state transitions; (2) post‑training that aligns the learned generative capability with robot embodiments and imperative task prompts, enabling efficient fine‑tuning for downstream tasks.

**Figure 11.** Examples of UMI data in the Pre-training Dataset.

**Figure 12.** Examples of UMI data in the Post-training Dataset.

The key shift is moving from narrow, task‑specific policies to a foundational Vision‑Language‑Action model that can be adapted efficiently to many new manipulation tasks.

System Architecture Overview

Describes the two‑stage training recipe and the Mixture‑of‑Transformers core that links vision‑language encoding to action generation.

The primary obstacle is that annotating hundreds of thousands of real‑world manipulation trajectories with task‑level language is prohibitively expensive. Our solution is a two‑stage recipe: first we pre‑train on massive auto‑labeled data, then we post‑train to align the model with specific robot embodiments and imperative instructions.

The model fuses a vision‑language encoder (VLM) with a Diffusion Transformer (DiT) so that visual and linguistic context steer a dedicated action generator, while the action generator never attends to its own action tokens.

How does this Mixture‑of‑Transformers differ from simply stacking a VLM encoder followed by a standard Transformer decoder?

In a standard encoder‑decoder stack the decoder attends to all encoder outputs *and* to its own previously generated tokens. Here the DiT (decoder) is prevented from attending to the action‑related tokens it is currently generating, which eliminates information leakage about future actions and lets the same cached VLM representation be reused without recomputation.

VLM produces KV cache: $K = [k_{v1}, k_{v2}]$, $V = [v_{v1}, v_{v2}]$.

DiT receives input sequence $[k_{v1}, k_{v2}, a_1, a_2]$.

During self‑attention, the attention mask zeros out entries where query or key corresponds to $a_1$ or $a_2$, so $a_1$ and $a_2$ cannot attend to each other or to themselves.

DiT predicts $a_1$ from the visible context $[k_{v1}, k_{v2}]$ using flow‑matching.

After $a_1$ is sampled, the mask is recomputed so $a_2$ can attend to $a_1$ (if desired) but still cannot see its own future token.

DiT predicts $a_2$ conditioned on $[k_{v1}, k_{v2}, a_1]$.

Masking action tokens forces the generator to rely exclusively on visual‑language context for the first step and only on already‑generated actions for subsequent steps, preventing leakage of future action information.

**Figure 1. Overview.** Xiaomi-Robotics-1 is pre-trained on over 100k hours of real-world UMI trajectories with auto-labeled state-transition language prompts. It is then aligned to robot embodiments and imperative instruction prompts via cross-embodiment post-training. Xiaomi-Robotics-1 scales effectively with data and model size. It is able to perform multiple tasks in unseen environment out-of-the-box and learn new tasks efficiently.

**Figure 6. Qualitative Results of Pre-training.** After pre-training, Xiaomi-Robotics-1 is able to predict action trajectories for UMI grippers on a held-out validation set according to the language description of state transitions.

Empirically we observe that scaling either the amount of auto‑labeled data or the model size consistently reduces validation action error, and those gains survive the post‑training alignment step, yielding higher success rates on out‑of‑the‑box robot tasks.

Mixture-of-Transformers Architecture

How the Mixture‑of‑Transformers architecture balances capacity and speed.

Training a Vision‑Language‑Action model at hundred‑thousand‑hour scale quickly exhausts memory and latency budgets. A monolithic transformer that processes vision, language, and action tokens together would require prohibitive compute. The authors therefore split the workload across two cooperating transformer streams.

Think of the architecture as a kitchen with two specialist chefs: the Vision‑Language Model prepares the ingredients (visual and textual embeddings), while the Diffusion Transformer cooks the final action sequence. The two chefs share the same pantry of token embeddings, so the dish stays coherent while each chef works at its own speed.

How does Mixture‑of‑Transformers differ from a single, large transformer that processes all tokens together?

In a single transformer the attention matrix must span vision, language, and action tokens simultaneously, giving an O(N²) cost where N is the total token count. MoT isolates the generative stage (DiT) to a smaller token set and a reduced hidden size, so its attention cost scales with the much smaller action‑token subset while still accessing the full context via the shared cache.

The Diffusion Transformer (DiT) is a transformer‑based generative model originally designed for image diffusion. Here it is repurposed to emit “flow‑velocity” tokens conditioned on the VLM’s cache, enabling flow‑matching based action chunk generation.

VLM parameters = 2 × 128 × 128 = 32 768.

DiT parameters = 2 × 64 × 64 = 8 192.

Total parameters = 32 768 + 8 192 = 40 960, i.e. roughly 4× fewer than a single 128‑hidden transformer with the same depth (which would need 2 × 128 × 128 = 32 768 × 2 = 65 536).

Reducing the DiT’s hidden size cuts its quadratic matrix‑multiply cost, so the overall model retains the expressive VLM while the generative stage runs at a fraction of the compute.

**Figure 2. Model Architecture.** Xiaomi-Robotics-1 adopts a Mixture-of-Transformers [44] architecture that couples a pre-trained VLM with a DiT. The VLM encodes the observation and language instruction, and additionally predicts action chunks via Choice Policies [59] to accelerate training convergence. Conditioned on the robot state and the VLM's KV cache of the observation and language tokens, the DiT generates the action chunk via flow matching. Note that the action-related tokens from the VLM are excluded from the DiT's attention computation.

By allocating most parameters to the VLM and keeping the DiT lightweight, Mixture‑of‑Transformers scales to billions of parameters without sacrificing real‑time inference, which is essential for mobile manipulation in unseen environments.

Action Generation and Flow-Matching

Defines the flow‑matching loss and auxiliary supervision that train the DiT and VLM to generate coherent action chunks.

The DiT tends to shortcut the learning problem by simply copying the VLM’s action predictions, which prevents it from grounding generation in visual and linguistic context.

Instead of denoising a noisy action sequence, the model learns to predict the instantaneous velocity that would transform a noisy chunk into the ground‑truth chunk at a chosen flow‑matching timestep.

How does flow‑matching differ from the diffusion‑denoising objective used in standard diffusion models?

Diffusion denoising asks the model to reconstruct a clean sample from a noisy one, which requires learning a full reverse trajectory. Flow‑matching instead asks for the instantaneous velocity that would move the noisy sample toward the clean one at a specific time $\tau$, turning the problem into a single‑step regression rather than a multi‑step denoising schedule.

Sample $\epsilon=[0.1,-0.2,0.05,0.0]$; compute $\tilde a_{\tau}=0.6\cdot[0.2,0.5,0.1,0.4]+0.4\cdot[0.1,-0.2,0.05,0.0]=[0.16,0.22,0.07,0.24]$.

Compute the true flow $u$ analytically as $(a-\tilde a_{\tau})/(\tau(1-\tau))$ → $[0.083,0.625,0.083,0.417]$.

The DiT predicts a velocity $\hat v=[0.08,0.60,0.09,0.40]$.

Flow‑matching loss = $\| \hat v - u \|_2^2 \approx 0.0012$, a small error indicating good alignment.

Flow‑matching turns a multi‑step denoising problem into a single‑step velocity regression, dramatically simplifying training while still guiding the model toward the correct action trajectory.

Initialize a random noise chunk $a_{\tau}=0$ and sample a Gaussian $t:t+H\sim\mathcal N(0,I)$.

For each of 5 integration steps, evaluate the DiT velocity $v_{\theta}(o_t,l,s_t,\tilde a_{\tau})$.

Update the chunk with Euler’s rule: $a_{\tau}\leftarrow a_{\tau}+\Delta\tau\,v_{\theta}$, where $\Delta\tau=0.2$.

Repeat the velocity evaluation and update until the 5th step.

Return the final $a_{\tau}$ as the clean action chunk.

adaLN injects the sampled flow‑matching timestep $\tau$ directly into the DiT’s normalization layers, letting the model modulate its computation based on how far it is from the clean action.

In addition to training the DiT, we teach the VLM itself to propose candidate action chunks and scores, steering its visual‑language representations toward features useful for downstream action synthesis.

Appending robot‑state and action‑query tokens, then selecting the best candidate.

**Tab. 1.** The VLM takes the current observation $o_t$ and language instruction $l$ as inputs. Conditioned on the robot proprioceptive state $s_t$ and the KV cache produced by the VLM, the DiT generates the action chunk via flow-matching [49].

Scaling Data for Generalization

We describe the massive UMI-driven pre‑training and post‑training pipelines that endow the VLA model with general manipulation skills.

Scaling annotation for hundreds of thousands of hours of robot trajectories is the bottleneck that prevents general‑purpose manipulation learning.

UMI is a handheld gripper that records both the robot’s end‑effector pose and a first‑person view, letting us capture manipulation data at scale without bespoke instrumentation.

At t=0 s the pose is (0.2, 0.1, 0.5); the video shows the block on the table.

At t=2 s the pose updates to (0.35, 0.15, 0.45); the video shows the hand reaching toward the shelf.

At t=5 s the pose reaches (0.5, 0.2, 0.3); the block rests on the shelf and the video shows the final placement.

Because the gripper is human‑operated, the recorded trajectory already encodes a sensible manipulation plan, eliminating the need for costly motion‑planning annotation.

The auto‑labeling pipeline first cuts each trajectory into equal‑length clips, then dispatches them to a Qwen3.5‑27B model that captions the gripper and object state changes within each clip.

A producer–consumer design separates CPU‑heavy clip extraction (producer) from GPU‑heavy caption generation (consumer), keeping hundreds of caption requests in flight and finishing labeling of the full 100 k‑hour corpus in roughly two weeks.

During each training step we sample vision‑language and UMI trajectories at a 1:9 ratio, pack all VLM tokens into a single sequence, and process four flow‑matching timesteps per sample in one DiT pass conditioned on the cached VLM key‑value memory.

**Figure 3. Pre-training Dataset.** The pre-training dataset of Xiaomi-Robotics-1 contains over 100k hours of real-world manipulation trajectories collected with UMI devices.

**Figure 4. Post-training Dataset.** The post-training dataset of Xiaomi-Robotics-1 comprises about 10k hours of cross-embodiment trajectories, including over 7.2k hours of in-house robot data collected with mobile manipulators and dual-arm robots, over 1k hours of instruction-labeled UMI data, and open-source robot datasets.

**Figure 5. Scaling of Pre-training.** We show the validation action errors (MSE) from the data-scaling and model-scaling pre-training experiments. We terminate the training for 12.5% and 25% data in the data-scaling experiment early as the validation loss indicates overfitting.

End-Effector Pose Representation

We encode arm motions as relative end‑effector pose deltas to unify actions across robots.

Robotic platforms differ in arm kinematics and sensor mounting, so a naïve policy that predicts absolute end‑effector poses would produce inconsistent commands across robots. This mismatch forces the model to relearn the same motion for each embodiment, wasting data and compute. The section therefore introduces a representation that abstracts away those hardware‑specific offsets.

Instead of asking “where should the hand be in world coordinates?”, we ask “how should the hand move relative to its current base frame”. The same delta works for any robot because the base frame is already aligned.

Compute the inverse of the base pose: $(EE_{Base_t})^{-1}= \begin{bmatrix}0 & 0 & 0^\circ\end{bmatrix}$ (identity for a zero‑rotation base).

Multiply the inverse with the future pose: $\hat{T}_{t+1}= (EE_{Base_t})^{-1} EE_{t+1}= \begin{bmatrix}0.10 & 0.00 & 0^\circ\end{bmatrix}$.

The predicted delta tells the controller “translate 10 cm along the x‑axis, no rotation”.

If the same arm were mounted on a different robot whose base at $t$ is rotated $90^\circ$, $EE_{Base_t}= \begin{bmatrix}0 & 0 & 90^\circ\end{bmatrix}$, the inverse rotation would cancel the base orientation, yielding the identical delta $\hat{T}_{t+1}= \begin{bmatrix}0 & 0.10 & 0^\circ\end{bmatrix}$ in that robot’s frame.

The delta $\hat{T}_{t+i}$ abstracts away the base’s orientation, so the same numeric command produces the same physical motion on any robot after the frames are aligned.

How does predicting a relative delta differ from predicting an absolute end‑effector pose?

Absolute poses are tied to a global coordinate system; a change in robot mounting or base pose would require the model to output a completely different value for the same physical motion. A relative delta is expressed in the current base frame, so the same numeric delta yields the same motion regardless of where the robot sits or how its base is oriented, eliminating the need for the network to learn embodiment‑specific offsets.

For mobile platforms we treat the chassis as the “base” and encode its motion as a 2‑D velocity vector, while the waist is represented by the change in its position relative to the base. All robots therefore share a single concatenated action vector; dimensions that a robot lacks (e.g., a wheeled base has no arm joints) are simply masked out during loss, ensuring a uniform training signal.

Out-of-the-Box Evaluation

Post‑training scaling yields up to 79% out‑of‑box success on unseen tasks.

Scaling the amount of pre‑training data dramatically improves out‑of‑box performance.

Overall success climbs from 26 % without action pre‑training to 75 % when using the full 100 % of the 20 k‑hour corpus; 12.5 % of the data already yields 53 % success.

Increasing model size further boosts success rates, especially on the hardest tasks.

The 2 B variant attains 61 % overall success; the 5 B and 10 B variants reach 75 % and 79 % respectively, with shoe‑tidying improving from 58 % (2 B) to 92 % (10 B).

**Figure 7. Post-training Evaluation.** We evaluate the post-trained model out-of-the-box across four tasks in novel environments. Crucially, both the environments and object instances are unseen during training.

**Figure 8. Quantitative Results of Post-training.** We showcase the success rates of post-trained models across different pre-training data scales and model sizes.

Benchmark Performance

Simulation results show Xiaomi‑Robotics‑1 vastly outperforms baselines in low‑data fine‑tuning.

Xiaomi‑Robotics‑1 achieves a 75 % average success rate in low‑data fine‑tuning across four simulation tasks.

Low‑data (<10 h per task) average success 75 % versus 40 % for the $\pi$0.5 baseline and lower for other methods.

RoboCasa365 expands the original RoboCasa suite to 365 tasks and 2 500 scenes, stressing long‑horizon mobile manipulation and zero‑shot composition of unseen task templates.

Table 3 shows that on this large‑scale benchmark Xiaomi‑Robotics‑1 attains the highest average success (74.5 %) among nine published methods, confirming its scalability.

VLABench evaluates language‑conditioned manipulation across five tracks (In‑distribution, Cross‑Category, Commonsense, Instruction, Texture) using success (SR), progress (PS) and intention (IS) scores.

Table 4 reports that Xiaomi‑Robotics‑1 achieves the best SR and PS in almost every track, with the second‑best scores consistently held by ERVLA or $\pi$0.5, highlighting its superior general‑purpose manipulation.

Across all three benchmarks the model also shows strong data efficiency: performance degrades far less than baselines when the fine‑tuning budget is reduced from <10 h to <40 h per task.

Chain-of-Thought and Action Loss

We evaluate our model on RoboCasa and RoboCasa365, reporting per‑method success rates.

We train with 500 demonstrations per task, using chain‑of‑thought labeling half the time, and evaluate each task on five tracks with 50 episodes per track.

Xiaomi‑Robotics‑1 attains the highest average success on the RoboCasa benchmark.

74.5 % average success, surpassing the second‑best World2Act at 72.6 %.

Xiaomi‑Robotics‑1 also leads the RoboCasa365 benchmark in overall average success.

57.4 % average success, well above the next best ABot‑M0.6 at 46.6 %.

Related Work

Related work situates our approach among benchmarks, scaling studies, and robot foundation models.

This section surveys the benchmarks, scaling literature, and foundation‑model families that contextualize our work.

RoboDojo is a unified simulation‑and‑real‑world benchmark that evaluates generalist robot manipulation policies across a broad set of tasks and capability dimensions.

Models that predict future observations or environment dynamics and use those predictions to generate robot actions, typically built on pre‑trained video backbones.

Models that leverage pre‑trained vision‑language backbones to predict robot actions directly from multimodal inputs, capitalizing on rich semantic knowledge.

Empirical observations that model performance improves predictably when data, compute, and parameter count are scaled together.

Portable devices that capture real‑world manipulation trajectories without requiring a robot, enabling in‑the‑wild data collection.

Human‑centric recordings of hand‑centric actions that are later retargeted to robot embodiments via representation alignment.

**Table 4.**

**Table 1.** Comparison of various robot foundation models across different evaluation metrics.

Conclusion

We present Xiaomi‑Robotics‑1, a scalable VLA model and acknowledge contributors.

Xiaomi‑Robotics‑1 leverages more than 100 k hours of real‑world manipulation trajectories, using an auto‑labeling pipeline to turn scene transitions into language prompts. Scaling data and model size consistently boosts performance, and the model transfers out‑of‑the‑box to unseen environments after post‑training. It also adapts to new real‑robot tasks with minimal data and attains state‑of‑the‑art results on four simulation benchmarks.

Core contributors (alphabetical): Jun Guo, Piaopiao Jin, Jason Li, Peiyan Li, Yingyan Li, Futeng Liu, Wanli Peng, Optimus Qin, Yifei Su, Nan Sun, Qiao Sun, Runze Suo, Heyun Wang, Yunhong Wang, Rujie Wu, Caoyu Xia, Lina Zhang, Jack Zhao.

Acknowledgment contributors: Guoliang Chen, Wenlong Chen, Xinze He, Bin Li, Qing Li, Zhuorong Li, Heng Qu, Wenxuan Song, Diyun Xiang, Yifan Xie, Peiran Xu, Hangjun Ye, Wen Ye, Han Zhao, Quanyun Zhou. We thank the broader team—including Li Jiang, Xiaohan Yu, Meichen Mu, Xiaoke Xilinjueluo, Qingyi Li, Qi Liu, Yayun Liu, Jun Xia, Feng Qiu, Donghao Wang, Yan Hou, Dong Wang, Liangliang He, Jiaxin Liu, Kang Zhou, Rui Cai, Shuoxue Bi, Yingchao Zhou, Kun Ma, Yiwei Zhou, and Dongsheng Li—for their tremendous support.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers