ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Yihui Yang, Yuanhong Yu, Jiawei Wang, Zhongrui Yu, Jia Shi, Kun Zhan, Ruizhen Hu, Xiaowei Zhou, Yu Liu, Sida Peng
ScaleNav uses Internet video pretraining to learn navigation priors, enabling robots to reach language-specified targets.
How can we leverage large-scale, action-free Internet video to improve language-conditioned robot navigation?
End-to-end navigation policies struggle to generalize because collecting diverse real-world robot data is expensive, while simulated data often fails to bridge the sim-to-real gap. ScaleNav addresses this by using a unified world-action model that learns navigation priors from Internet egocentric videos before fine-tuning on action-labeled simulation data. This approach achieves a 94.9% success rate on the HabitatGS-LN benchmark, outperforming existing baselines by leveraging large-scale human navigation experience.
Paper Primer
The core mechanism is a three-stage training procedure that transfers visual navigation priors from Internet videos into executable action chunks. The model uses a shared Diffusion Transformer (DiT) to jointly predict future observations and action sequences, conditioned on navigation instructions and an initial-frame anchor that maintains target context.
The routing layer is like a translator: it maps the visual patterns of human movement learned from Internet videos into the specific motor commands required for a robot to reach a target.
Incorporating Internet egocentric videos significantly improves navigation success rates.
Comparing the full ScaleNav framework against a variant trained only on simulated data. Success rate improved by 5.3 percentage points on Short-Horizon OVON and 4.5 percentage points on HabitatGS-LN.
Why use Internet egocentric videos instead of just collecting more simulated data?
Internet videos provide a scalable source of diverse real-world scenes and human navigation behaviors that are difficult to replicate in simulation, helping to bridge the sim-to-real gap.
What is the primary role of the "initial-frame anchor" in this architecture?
The anchor retains the initial frame's video tokens throughout navigation, allowing the model to maintain accurate interpretation of the instruction even as the robot's viewpoint changes.
ScaleNav demonstrates that large-scale, action-free Internet video data can be effectively distilled into actionable navigation policies, reducing the reliance on costly, manually collected robot datasets.
Introduction and Motivation
We expose the data‑scarcity bottleneck in language‑conditioned robot navigation and outline ScaleNav’s three‑stage solution.
Robust navigation to visible targets described in natural language demands diverse training data, yet collecting action‑labeled robot trajectories is expensive and simulated data suffers a sim‑to‑real gap. ScaleNav addresses this scarcity by first learning visual dynamics from large‑scale Internet egocentric videos, then aligning those dynamics to executable robot actions using a modest amount of simulated action data, and finally fine‑tuning the whole system jointly. This three‑stage pipeline enables end‑to‑end policies that succeed on both simulated benchmarks and a real quadruped robot.
The robot receives a natural‑language instruction that names a target object visible in its initial view and must move to that object and stop exactly where the instruction says.
**Figure 1.** **Overview of ScaleNav.** ScaleNav learns navigation priors from Internet egocentric videos through action-free video pretraining, then uses action-labeled simulated data for action alignment and joint fine-tuning. It jointly predicts future observations and action chunks to follow navigation instructions. The right panel illustrates deployment on a real quadruped robot.
The primary obstacle for language‑conditioned navigation is the scarcity of action‑labeled robot data.
Related Work and Architecture
We position our approach among prior work and outline the model components.
Related work on language‑conditioned navigation falls into modular pipelines, learned vision‑language‑action policies, and joint world‑action models. We also introduce the key components of our method before detailing the training pipeline.
An Action Chunk is a short sequence of low‑level motion commands that the robot executes as a unit.
Egocentric Video refers to first‑person recordings captured by a wearable camera, showing exactly what the robot (or human) sees.
The model jointly predicts future video frames and action chunks using a shared DiT backbone, so visual and motor predictions reinforce each other.
To keep the navigation instruction grounded, the model retains a noise‑free token representation of the first observation (anchor) and a sliding window of recent chunks.
**Figure 2.** Overview of ScaleNav. (a) Language-conditioned local navigation. The model predicts video and action chunks from the instruction, current observation, and memory using chunk-wise causal attention. The memory contains an initial-frame anchor and a sliding window of recent video and action chunks. (b) Three-stage training. Video pretraining uses action-free real videos, followed by action alignment and joint fine-tuning with action-labeled simulation data. During action alignment, the self-attention layers within the DiT, the action head, and the action embedding MLP (not shown) are trainable. The snowflake on the outer DiT block marks the remaining DiT layers as frozen. Joint fine-tuning updates the DiT and both heads. For clarity, the initial bidirectional adaptation in Stage 1 and Internet egocentric video replay in Stages 2 and 3 are omitted.
The Three-Stage Training Pipeline
Three training stages progressively learn navigation priors, then align actions, and finally fine‑tune jointly.
We train the model in three successive stages. Stage 1 learns navigation priors from raw egocentric videos paired with navigation instructions, using video‑instruction pairs while keeping the VAE and text encoder frozen. Stage 2 introduces action‑labeled simulated data, training only the action‑embedding MLP and action head, and Stage 3 jointly fine‑tunes video and action prediction, still with the VAE and text encoder frozen, completing the three‑stage training pipeline.
We first teach the model to predict future egocentric observations from video alone, conditioning only on the navigation instruction and the current observation, without any action tokens.
How does Action‑Free Video Pretraining differ from standard video pretraining that uses full clips?
Standard pretraining predicts an entire clip bidirectionally, requiring the model to infer both past and future frames simultaneously. Here we first train on full clips, then switch to chunk‑wise causal attention so the model only predicts forward in time, matching the robot’s online perception.
Compute noisy latent: $z_v^{t_v}=z_v^{0}+t_v\epsilon_v = 2.0 + 0.5\times1.0 = 2.5$.
The model predicts flow velocity $v_v^{\theta}=0.4$ (hypothetical output).
True residual: $\epsilon_v - z_v^{0}=1.0 - 2.0 = -1.0$.
Loss contribution: $\|v_v^{\theta} - (\epsilon_v - z_v^{0})\|^{2}= \|0.4 - (-1.0)\|^{2}=1.96$.
This tiny example shows that the loss penalizes the mismatch between the model’s predicted correction and the actual noise‑induced deviation, driving the model to accurately denoise video latents.
Stage 1: Pretrain on video‑instruction pairs using the video flow‑matching loss; keep VAE and text encoder frozen.
Stage 2: Freeze the video backbone; train the randomly initialized action‑embedding MLP and action head on action‑labeled simulated data, jointly predicting video and action chunks.
Stage 3: Unfreeze all trainable components and jointly fine‑tune using the combined loss $L = (1-\lambda)L_{\text{video}} + \lambda L_{\text{action}}$, still keeping the VAE and text encoder frozen.
After the model has learned to predict future observations, it produces short sequences of discrete actions—action chunks—that drive the robot toward the instructed target.
Why generate actions in chunks instead of predicting a single long trajectory?
Chunking limits the horizon of each prediction, keeping the conditioning context manageable and allowing the model to correct errors after each short segment, which aligns with the incremental nature of robot control.
Loss Functions and Fine-Tuning
How the model balances video and action losses and adapts to navigation data.
We let the model decide how much to trust the video prediction versus the action prediction for each batch, by weighting the two losses so they sum to one.
How does this differ from simply fixing $\lambda$ to a constant value?
With a constant $\lambda$ the model would always prioritize one loss, so when the video loss is small but the action loss is large the gradients from the action term could be overwhelmed, or vice‑versa. Computing $\lambda$ from the batch’s losses lets the weighting adapt to the current difficulty, preserving both signals.
During Stages 2 & 3 we replay Internet egocentric videos from the Stage 1 dataset, but we feed only the video loss for these samples. This prevents the action loss from pulling the pretrained video representation away from its original distribution.
When an episode ends early we extend the chunk to the fixed length by repeating the last frame and inserting a zero‑motion action $(0,0,0)$. The repeated frames are masked out of the video loss so they do not corrupt the learned video prior, while the zero actions remain in the action loss to teach the model to stop.
We repeat the last real frame $f_2$ twice, obtaining the padded sequence $[f_1, f_2, f_2, f_2]$.
For each padded frame we add a zero‑motion action $(0,0,0)$, so the action sequence becomes $[a_1, a_2, (0,0,0), (0,0,0)]$.
A max‑pooled mask $m=[1,1,0,0]$ marks the first two positions as valid; applying $sg$ to the masked video loss yields $L_{\text{video}} = \text{loss}(f_1,f_2)$ only.
The action loss still sees all four actions, encouraging the model to output zero motion for the padded steps.
Masking protects the video prior from artificial repetitions, while the zero‑action supervision teaches the model that “no motion” signals arrival.
Stage 3 performs joint fine‑tuning: we keep the action modules trainable while freezing most of the video‑pretrained backbone, then we unfreeze the remaining DiT modules and the video head and continue training with the same inverse‑loss weighting objective.
We build an Internet egocentric video dataset by mining ~60 K indoor clips from YouTube, selecting keyframes with a vision‑language model, and generating navigation instructions for each segment. After automatic filtering and human‑in‑the‑loop review, the final set contains 18.2 K navigation segments.
For the action‑labeled simulated data we generate episodes in Habitat‑GS using indoor scenes from InteriorGS, plan collision‑free trajectories with SCAN‑Planner, and record aligned video and robot actions. After review, we obtain ~2.6 K episodes spanning 650 scenes.
Benchmark Results
ScaleNav’s simulated performance on two navigation benchmarks.
ScaleNav pretrains on action‑free egocentric video, then fine‑tunes on a small action‑labeled set, enabling better navigation without massive labeled data.
It measures how well a robot follows a language instruction to reach a specific object in a photorealistic indoor scene.
How does HabitatGS‑LN differ from standard goal‑category navigation benchmarks?
Standard benchmarks ask the robot to reach any instance of a category (e.g., “go to a chair”). HabitatGS‑LN requires the robot to identify the exact instance described by attributes in the instruction, forcing joint language grounding and precise spatial reasoning.
ScaleNav outperforms all baselines on both benchmarks, achieving the highest success rates.
Table 1 shows ScaleNav’s SR of 94.9 % and SPL of 86.1 % on the full model, exceeding the next best baseline by several points. +5.3% SR over the variant without video pretraining
**Figure 3.** Qualitative results of ScaleNav in an exhibition hall (a), an open lounge area (b), and an office area (c). The robot follows language instructions to reach the specified targets.
ScaleNav delivers the best simulated navigation performance while keeping inference latency under 100 ms per action.
Ablations and Real-Robot Performance
Ablations reveal each Internet video component’s impact on navigation performance.
We evaluate the contribution of each Internet egocentric video component by removing it and measuring success rate (SR), success‑weighted by path length (SPL), and amortized latency on HabitatGS‑LN.
**Table 2.** Internet egocentric video utilization and component ablations on HabitatGS-LN. Video pretraining and Alignment denote Stages 1 and 2, respectively. Pseudo-action denotes action labels estimated from these videos, Anchor denotes the initial-frame anchor, and Replay denotes Internet egocentric video replay during Stages 2 and 3. Best results are highlighted in bold.
The full model (“Ours”) outperforms all ablations, demonstrating that the combination of video pretraining, alignment, pseudo‑action, anchor, and replay is essential for peak performance.
Appendices
Action‑labeled robot data are scarce, so the paper pretrains on large‑scale, action‑free egocentric videos.
Robotic navigation systems need dense, action‑labeled trajectories to learn how to move, yet collecting such data at scale is expensive and time‑consuming. Consequently, existing datasets are small and limit the generality of learned policies. ScaleNav tackles this bottleneck by first pretraining on massive collections of action‑free Internet egocentric videos, then fine‑tuning on the comparatively tiny set of action‑labeled episodes.