Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

Shuailei Ma, Jiaqi Liao, Xinyang Wang, Jingjing Wang, Chaoran Feng, Zijing Hu, Chong Bao, Zichen Xi, Yuqi Gan, Weisen Wang, Yanhong Zeng, Qin Zhao, Zifan Shi, Wei Wu, Hao Ouyang, Qiuyu Wang, Shangzhan Zhang, Jiahao Shao, Yipengjing Sun, Liangxiao Hu, Lunke Pan, Nan Xue, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Ka Leong Cheng

LingBot-Video is a sparse Mixture-of-Experts video foundation model designed to bridge digital video generation with embodied physical intelligence.

How can we scale video generative models to support embodied intelligence by replacing dense compute with sparse Mixture-of-Experts (MoE) and structured data curation?

Video generative models are typically optimized for aesthetic fidelity, leaving them poorly suited for the physical constraints and action-oriented dynamics required for robotics. LingBot-Video addresses this by replacing dense transformer blocks with a sparse Mixture-of-Experts (MoE) architecture, allowing the model to scale parameter capacity for complex physical priors while keeping active computation efficient. The model achieves superior scaling efficiency compared to dense baselines, maintaining competitive inference speeds even at million-token sequence lengths.

Paper Primer

The core mechanism is a single-stream Diffusion Transformer (DiT) that uses fine-grained expert segmentation to decouple total parameter capacity from active FLOPs. By routing tokens to specialized experts, the model learns distinct physical and motion patterns without the subtask interference common in dense architectures.

The system is like a specialized library: instead of every librarian reading every book, the router acts as a clerk that directs specific queries to experts who hold deep knowledge in that domain, ensuring the model can handle diverse tasks—from fluid dynamics to robot manipulation—without increasing the cost of every single inference step.

Sparse MoE scaling provides superior representation capacity for physical priors compared to dense models under equivalent compute budgets.

The MoE 13B-A1.4B model consistently outperforms a dense 1.3B baseline in training and validation loss, with larger variants (MoE 30B-A3B) approaching the performance of dense models twice their active size. At 1M tokens, the MoE architecture achieves a 3.18× latency advantage over a dense 30B model while maintaining near-parity with a 3B dense baseline.

Why is a sparse MoE architecture preferred over a standard dense transformer for this specific application?

Dense models force all tokens to share the same parameters, which causes interference when trying to model asymmetric domains like spatial textures versus temporal motion. Sparse MoE allows the model to scale its total capacity to capture complex physical laws while keeping per-token active computation low enough for high-resolution, long-sequence video generation.

How does the model handle the gap between training on structured captions and real-world user prompts?

The authors use a two-stage Caption Rewriter: an "Expand" stage turns brief prompts into action-centric prose, and a "Map" stage uses a fine-tuned model to convert that prose into the structured JSON format the generator expects, preventing the hallucinations common in single-step expansion.

Motivation and Overview

Video models excel at visual fidelity but miss the physical realism needed for embodied robotics.

Current video generative models are built for visual appeal; they excel at producing photorealistic, aesthetically pleasing sequences but ignore the physical constraints that embodied agents must obey.

Embodied robotics needs video predictions that respect contact stability, rigid‑body dynamics, and long‑horizon state consistency—properties that pure visual fidelity does not guarantee.

Three tightly coupled shortcomings prevent this transition: (1) dense architectures activate every parameter for every token, inflating inference cost; (2) training data consist mostly of internet videos that lack robot‑centric actions; (3) loss functions reward aesthetics and text‑video alignment, not dynamics or task success.

We close the gap with an integrated stack: a sparse Mixture‑of‑Experts (MoE) video frameword for efficient scaling, a data profiling engine that injects robot manipulation, navigation, and egocentric footage, and a multi‑dimensional reward system that penalizes physically implausible outcomes while rewarding task completion.

**Figure 1.** Samples of *Text-to-Image* and *Text-to-Video* tasks generated by LingBot-Video. LingBot-Video can produce images and videos with high visual fidelity, rich details, and strong text-prompt alignment across diverse scenes and subjects.

The core shift is from chasing visual fidelity toward enforcing physical plausibility for embodied agents.

Task-Unified Architecture

Unified diffusion transformer processes visual latents and multimodal conditions in a single token stream.

Prior video generators treat each task—text‑to‑image, text‑to‑video, image‑to‑video—as a separate pipeline, inflating parameters and breaking scaling. The core obstacle is a structural mismatch between condition tokens and visual latents that forces task‑specific encoders.

All modalities become a single token stream, so every transformer layer mixes them directly—like a single conveyor belt that carries mixed parcels instead of parallel lanes for each type.

Layer 1 computes self‑attention over the 7‑token sequence, producing mixed representations R₁…R₇.

Layer 2 repeats self‑attention on R₁…R₇, further blending condition and visual information.

The final token corresponding to V₄ is fed to the velocity head, yielding a predicted motion vector.

The same parameters that attend to text also attend to video frames, so learning a visual concept automatically improves the corresponding textual description.

How does this single‑stream design differ from dual‑stream video generators?

Dual‑stream models keep condition and visual tokens in separate pathways and only fuse them at a late stage, which limits early cross‑modal interaction and forces duplicated attention kernels. The single‑stream design interleaves them from the first layer, allowing every attention head to see both modalities simultaneously.

Condition tokens occupy a temporal‑only slice of a 3‑D coordinate grid, while visual patches occupy the spatial‑temporal volume, preventing overlap without extra encoders.

Why not simply prepend a special token to separate conditions from visuals?

Appending a delimiter leaves the rotary embeddings ambiguous, causing condition and visual tokens to share the same frequency spectrum and interfere during attention. The 3‑D RoPE explicitly allocates disjoint frequency subspaces, guaranteeing that attention scores respect the intended modality boundaries.

Before dot‑product attention, each head’s queries and keys are RMS‑normalized, keeping their magnitudes in a narrow band.

Would a standard LayerNorm on queries and keys achieve the same effect?

No. LayerNorm centers the vector around zero, which can shift attention scores and introduce bias. RMSNorm preserves the direction of the vector while only scaling its magnitude, which is sufficient to control logit growth without altering the relative geometry of queries and keys.

A single shared timestep embedding is projected once, then each layer adds a tiny learned offset, avoiding a full MLP per layer.

Why not compute a separate timestep embedding for every layer?

Per‑layer embeddings multiply the number of parameters and require an extra MLP, which adds compute and memory traffic. The shared‑plus‑offset scheme retains the expressive power to modulate each layer while keeping the overhead comparable to a single linear projection.

**Figure 2.** Overview of the task-unified single-stream diffusion transformer. Unified inputs are processed by stacked transformer blocks, where timestep modulation controls the attention and Sparse MoE branches; the attention branch applies QK-Norm and multi-modal 3D RoPE, while the MoE branch combines always-on shared experts with top-$K_r$ routed experts before predicting velocity.

The base generator described above is followed by a cascaded refiner that takes the coarse velocity prediction and iteratively sharpens it, completing the video generation pipeline.

Sparse MoE Scaling

Scaling video diffusion by replacing dense feed‑forward layers with a sparse MoE that decouples capacity from compute.

Instead of forcing every token through the same dense feed‑forward network, the model lets each token pick a small set of experts that specialize on its visual or temporal pattern, keeping per‑token compute low while expanding total capacity.

Compute shared expert outputs: $E^{(s)}_1(u_t)= [0.5,0.5,0.5,0.5]$, $E^{(s)}_2(u_t)= [0.2,0.2,0.2,0.2]$.

Sigmoid router yields $g_{t}=[1,0,1,0]$, activating routed experts 1 and 3.

Routed expert outputs: $E^{(r)}_1(u_t)= [0.3,0.3,0.3,0.3]$, $E^{(r)}_3(u_t)= [0.1,0.1,0.1,0.1]$.

Aggregate: $m(u_t)= [0.5+0.2+0.3+0.1,\dots]=[1.1,1.1,1.1,1.1]$.

Only two routed experts fire, so the extra parameter pool ($N_r=4$) does not increase the per‑token FLOPs, yet the model can still draw from a larger set of specialized functions.

How does this MoE differ from a standard dense FFN that simply widens the hidden dimension?

Widening a dense FFN increases both total parameters and per‑token FLOPs proportionally, because every token still passes through the full matrix multiply. In contrast, the sparse MoE keeps the active FLOPs constant by routing each token to only a few experts, so capacity can grow without a corresponding compute increase.

**Figure 3.** Expert-count recipe comparison using training and validation loss. Training curves show raw logged losses in the background and smoothed curves for visualization; validation curves are unsmoothed and aligned to the training-loss step range.

**Figure 7.** MoE-to-dense speed ratio, computed as dense latency divided by MoE 30B-A3B latency.

Router Mechanics

Load balancing via group‑limited routing, bias correction, and a sequence‑wise auxiliary loss.

Router affinity measures how strongly a token wants to be processed by a particular expert, acting as the raw compatibility score before any routing decisions.

How does router affinity differ from a plain dot‑product score?

The dot product alone can be arbitrarily large, making the top‑K selection unstable across batches. Applying a sigmoid caps the score, preserving the relative ordering while ensuring all affinities lie between 0 and 1, which stabilizes the bias‑correction step.

To keep communication cost low in distributed training, the experts are first partitioned into groups, and routing decisions are made at the group level before selecting individual experts.

Divide the $N_r$ routed experts into $N_g$ groups.

Add the online correction bias $b_j$ to each affinity, yielding \tilde{\alpha}_{t,j}=$\alpha_{t,j}$+$b_j$.

Score each group by summing the top‑2 bias‑corrected affinities within the group.

Select the top $K_g$ groups based on these group scores.

Within the selected groups, pick the top $K_r$ experts to form $R_b$($u_t$).

Compute bias‑corrected affinities \tilde{$\alpha$} = $\alpha$ + b = [0.25, 0.68, 0.10, 0.41, 0.57, 0.34, 0.50, 0.19].

Group A top‑2 corrected scores: 0.68 (expert 2) and 0.41 (expert 4) → sum = 1.09.

Group B top‑2 corrected scores: 0.57 (expert 5) and 0.50 (expert 7) → sum = 1.07.

With $K_g$=1, the higher‑scoring group (Group A) is selected.

Within Group A, the top $K_r$=2 experts are 2 and 4, forming $R_b$($u_t$) = {expert 2, expert 4}.

Group‑level scoring lets the router discard an entire half of the experts with a single comparison, saving bandwidth while still allowing fine‑grained expert selection inside the winning group.

Online bias correction injects a dynamic scalar $b_j$ for each expert, nudging the routing probabilities toward a balanced load without adding an explicit auxiliary loss.

Compute sign($n_j$ - \bar{n}) for each expert: [+1, -1, 0, 0].

Update each bias: $b_j$ ← $b_j$ - $\eta$ sign(...). Resulting biases become [0.02-0.1= -0.08, -0.01+0.1= 0.09, 0.00, 0.03].

Apply bias centering: compute mean bias = (-0.08+0.09+0.00+0.03)/4 = 0.01.

Subtract the mean from each bias: final biases = [-0.09, 0.08, -0.01, 0.02].

The centering step preserves the relative differences that drive load balancing while preventing the biases from drifting arbitrarily.

Because video generation works on long token sequences, we add a sequence‑wise auxiliary loss that penalizes imbalance within each packed video rather than only across the whole batch.

Training Objectives

Introduces an auxiliary balance loss that equalizes expert usage across sequences while keeping per‑token cost low.

Without an explicit regularizer, the router tends to concentrate tokens on a few experts, leaving most of the expert pool idle and wasting capacity.

The loss nudges the router so that, within each packed sequence, every expert receives roughly the same share of tokens, preventing “starvation” of any expert.

Token 1 affinities: $\alpha_{1} = [0.9, 0.2, 0.1, 0.4]$ → top‑2 experts are 1 and 4.

Token 2 affinities: $\alpha_{2} = [0.3, 0.8, 0.5, 0.2]$ → top‑2 experts are 2 and 3.

Token 3 affinities: $\alpha_{3} = [0.6, 0.7, 0.2, 0.1]$ → top‑2 experts are 2 and 1.

Counting selections gives $c_1(s)=2$, $c_2(s)=2$, $c_3(s)=1$, $c_4(s)=1$.

Normalize: $f_j(s)=\frac{N_r}{K_r T_s}c_j(s)=\frac{4}{2\cdot3}c_j(s)=\frac{2}{3}c_j(s)$ → $f_1=1.33$, $f_2=1.33$, $f_3=0.67$, $f_4=0.67$.

The auxiliary loss penalizes the deviation of each $f_j$ from 1, encouraging the router to spread tokens more evenly.

Even with a tiny toy, the loss reveals that two experts are over‑selected while the others are under‑used; the auxiliary term will push the router toward a balanced $f_j=1$ for all experts.

How does this auxiliary balance loss differ from the classic load‑balancing loss used in earlier MoE papers?

The classic loss directly penalizes the soft routing probabilities $p_{t,j}$ (e.g., via an entropy or KL term) while gradients flow through the router. Here we first compute a hard top‑$K_r$ count $c_j(s)$, detach the resulting frequency $f_j(s)$ from the graph, and only then apply a scalar penalty $L_{\text{seq}}$. This avoids interfering with the diffusion gradient signal and isolates the regularizer to the sequence‑level expert usage.

With this auxiliary objective the model can grow the expert pool arbitrarily large—boosting representational capacity—without increasing the per‑token compute budget, which is essential for scaling video generation to physically plausible scenes.

Scaling Results

Scaling experiments show sparse MoE’s efficiency and predictable gains up to 120 B parameters.

Sparse MoE with 13 B total parameters matches a dense 1.3 B baseline while using the same active compute budget.

Figure 5 shows consistently lower training and validation loss for the MoE model across the training steps.

**Figure 4.** Active-capacity recipe comparison using training and validation loss. Training curves show raw logged losses in the background and smoothed curves for visualization; validation curves are unsmoothed and aligned to the training-loss step range.

**Figure 5.** Comparable active-parameter scaling comparison between Dense 1.3B and MoE 13B-A1.4B using training and validation loss. Training curves show raw logged losses in the background and smoothed curves for visualization; validation curves are unsmoothed and aligned to the training-loss step range.

Overall, the scaling study confirms that sparse MoE architectures retain inference efficiency while delivering predictable performance improvements as total capacity grows.

Cascaded Refiner

A two‑stage cascade refines low‑resolution video with conditional rectified flow to add high‑frequency detail.

Generating high‑resolution video directly explodes the number of spatio‑temporal tokens, making inference prohibitively slow.

We first quantify the inference advantage of the sparse MoE by defining the speed ratio $r = T_{\text{dense}}/T_{\text{MoE}}$; values greater than 1 indicate the MoE is faster.

**Figure 6.** Training-loss comparison for compute-comparable scaling experiments. The faint background traces show the raw logged training losses; the bold curves are smoothed only for visualization and are not used to alter the underlying measurements.

To keep generation tractable we split the pipeline into a low‑resolution base stage and a high‑resolution refiner, a pattern common in diffusion‑based super‑resolution.

The refiner takes a degraded low‑resolution latent, runs a short conditional rectified‑flow denoising, and outputs a high‑resolution latent that restores fine details.

Compute the noisy start: $\mathbf{x}_\tau = (1-0.9)[0.2,-0.1] + 0.9[0.4,-0.3] = [0.38,\, -0.28]$.

Pick an intermediate timestep $t = 0.45$ (halfway to $\tau$). Interpolate: $\mathbf{x}_t = \frac{1-0.45/0.9}{1-0.9}[0.8,0.5] + \frac{0.45}{0.9}[0.38,-0.28] = [0.59,\, 0.11]$.

Target velocity: $\mathbf{v}^\star_{\text{ref}} = (\mathbf{x}_\tau - \mathbf{x}_0)/0.9 = ([0.38,-0.28] - [0.8,0.5])/0.9 = [-0.466,\,-0.867]$.

This tiny toy shows how the refiner’s training signal is a simple scaled difference, yet the flow‑matching loss forces the network to learn a continuous vector field that can be integrated back to high‑quality video.

How does this conditional rectified‑flow differ from the standard diffusion denoising used in the base generator?

Standard diffusion starts from pure Gaussian noise and traverses the entire noise schedule. The refiner instead starts from a *degraded* latent that already contains most of the coarse structure, and it only walks a short segment of the schedule (from $\tau$ down to 0). This focuses capacity on restoring high‑frequency details rather than re‑learning the global motion.

**Figure 8. Refiner Generation.** We compare the base video generation against the refined video generation. The left example's comprehensive prompt is “Three dancers, a man in the center and two women on either side, are performing a traditional Indian dance on a stage...”. The right example's comprehensive prompt is “A humanoid robot named 'AURORA' walks steadily down a gravel path in a meticulously manicured formal garden...”.

Overall, the cascaded refiner adds a lightweight high‑frequency restoration stage that preserves the base model’s global semantics while delivering crisp, legible high‑resolution video.

Data Profiling Engine

The Data Profiling Engine structures raw media into multi‑dimensional records that drive efficient curation.

Collecting ever more video and image data quickly hits a wall: each additional gigabyte costs money, storage, and preprocessing time, yet the marginal gain in model performance shrinks.

The engine projects every raw image or video onto a compact, five‑dimensional schema so that downstream modules can query, filter, and balance the corpus without ever parsing free‑form captions.

Structural Metadata: duration $=4$ s, frames $=120$, shot boundary detector finds a single shot.

Semantic Labels: VLM tags “dog” (0.92), “park” (0.87), “running” (0.81), and extracts the text “Playground sign”.

Motion signals: camera motion score $0.15$ (mostly static), subject motion score $0.78$, tracked‑motion tracker confirms a moving bounding box.

Camera attributes: color tone “warm”, shot size “medium‑wide”, angle “high‑angle”, lens “35 mm”.

Quality signals: aesthetic score $0.73$, synthetic‑image likelihood $0.04$, artifact flag “no watermark”.

This toy profile shows how heterogeneous signals collapse into a single record that can be queried (e.g., “high‑motion, warm‑toned clips”) without re‑running heavy detectors.

The World‑Knowledge Topological Graph consumes the semantic tags from the profiling engine, arranging them into a hierarchical concept tree that supports distribution‑aware sampling across both images and videos.

**Figure 9.** Overview of the Data Profiling Engine. Each image or video sample is annotated across five complementary dimensions (structural, semantic, motion, camera, and quality) into a structured profile record, which then drives downstream filtering, balanced sampling, and captioning.

Data Curriculum

We reshape the training pool through a five‑stage curriculum that concentrates on embodied, high‑motion data.

Generic, low‑motion footage dominates most video corpora, leaving the model starved of the physical interactions required for embodied intelligence.

We treat the world‑knowledge graph as a control surface: nodes that are rare or yield high denoising loss are sampled more often, while over‑represented easy nodes are down‑weighted.

How does this differ from simple class‑frequency reweighting?

Class‑frequency reweighting only looks at how many samples each label has. Distribution‑Aware Sampling also considers the node‑wise denoising loss, so a node that is frequent but still hard for the model gets up‑weighted, while an easy‑to‑learn frequent node is down‑weighted. The graph further lets us spot entirely missing actions and deliberately inject them.

Stage 1: Train on a curated pool of 192p images only, discarding the lowest‑tier samples.

Stage 2: Add 192p video clips (including 70 k h of embodiment footage) while tightening image quality filters.

Stage 3: Upscale both streams to 480p and apply stricter aesthetic and motion criteria.

Stage 4: At 480p, aggressively down‑weight generic video while preserving scarce manipulation, navigation, and egocentric clips.

Stage 5: Train a cascaded refiner on a tiny 1080p subset (<1 % of the original video pool) for final high‑quality refinement.

Stage 1 keeps 61 % of images → 610 images; videos are not yet introduced.

Stage 2 adds videos (47 % retained) → 235 videos, and tightens images to 27 % → 165 images.

Stage 3 upsamples to 480p, retaining 27 % of both streams → 45 images, 64 videos.

Stage 4 further down‑weights generic video to 0.2 % → 0 video (only high‑value clips remain), while keeping the same 27 % of images.

Stage 5 extracts a 1080p refinement set (≈1 % of the remaining video pool) → 0 video samples, but the high‑quality image pool stays at 45 samples for the refiner.

The curriculum shrinks the overall corpus dramatically while concentrating the remaining data on high‑motion, embodiment‑rich content, which drives better physical plausibility without inflating compute.

**Figure 10. World-Knowledge Topological Graph.** Structured tags from the Data Profiling Engine link each sample to a semantic tree of visual concepts and, for videos with actions, to an action tree of dynamic behaviors. The graph serves as a control surface for data curation: node statistics and training-feedback signals are used to up-weight rare or difficult concepts, down-weight saturated easy modes, and identify under-covered domains for targeted data expansion.

**Figure 11.** Data curriculum across the five progressive pre-training stages. The image stream (blue) and video stream (green) are decomposed into their constituent sources, with band width indicating each source's relative proportion; percentages denote the fraction of data retained at each stage relative to the modality's initial pool.

RL Design

Infrastructure that lets massive video diffusion models learn via reinforcement learning without choking on memory.

Reinforcement‑learning post‑training for video diffusion faces two brutal bottlenecks: gigabyte‑scale latent trajectories and millions of expert parameters that overwhelm memory and communication.

Instead of keeping every intermediate activation in GPU memory, the system selectively recomputes them, choosing the coarsest granularity that fits the current memory budget.

Step 1: The system picks block‑level granularity because 2 checkpoints can cover the 8‑token sequence with two 4‑token blocks.

Step 2: Forward pass stores activations for tokens 1‑4, then discards them.

Step 3: During back‑propagation, the discarded block 1 activations are recomputed on‑the‑fly.

Step 4: The same process repeats for tokens 5‑8, using the second checkpoint.

Step 5: Total recomputation cost equals one extra forward pass per block, while peak memory stays within the budget.

Choosing the coarsest checkpoint that satisfies the budget minimizes extra compute while guaranteeing that no activation ever exceeds the allocated memory.

How does this differ from the activation checkpointing used in large‑scale language models?

Language‑model checkpointing typically fixes a single granularity (e.g., layer‑wise) for the whole run. Here the granularity is dynamic and can be driven by an explicit memory budget, allowing the system to adapt on‑the‑fly to different model sizes and parallel configurations.

Compiling the computation graph before any model parallel sharding lets the compiler fuse whole‑block operations, turning many small kernels into a few large ones.

Why compile before sharding instead of after?

Compiling after sharding would see each shard as an independent subgraph, preventing cross‑shard kernel fusion. By compiling first, the optimizer can see the whole block as a single unit and generate a fused kernel that runs efficiently on each device after sharding.

Logging and checkpoint saving run in a background thread behind a bounded queue, so the training compute never stalls waiting for I/O.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers