SuperNav: An Agentic Navigation System for Any Task in Any Scene
Jinkai Zhang, Jingyi Xu, Yuanhong Yu, Jiarui Guo, Ruizhen Hu, Hujun Bao, Xiaowei Zhou, Sida Peng
SuperNav decouples high-level navigation reasoning from motion execution using a pretrained MLLM and a specialized agent harness.
How can we enable general-purpose navigation agents to handle diverse, complex human requests without requiring navigation-specific fine-tuning of large multimodal models?
General-purpose navigation systems often struggle to generalize because they either rely on rigid, predefined workflows or require navigation-specific fine-tuning that limits their performance in new environments. SuperNav addresses this by equipping a pretrained Multimodal Large Language Model (MLLM) with an agent harness that manages task state, context, and tool use, allowing the model to direct navigation without being fine-tuned on navigation data. The system achieves 78.00% success on single-object navigation and 59.50% on demand-driven tasks, significantly outperforming existing action-prediction baselines.
Paper Primer
SuperNav functions as an agentic loop: the MLLM interprets the user request and visual observations, then selects from a suite of Navigation Skills and Tools to execute motion and track progress. The core move is the unified visual-point interface: the MLLM specifies a destination by pointing directly in an observation image, which a backend executor then translates into physical movement.
SuperNav significantly improves success rates across diverse navigation tasks compared to end-to-end action-prediction models.
In instance-level navigation, SuperNav achieved 78.00% success, compared to 34.00% for the strongest baseline (UniNaVid). A 2.3x improvement in success rate for single-object navigation.
The system’s modularity allows for interchangeable motion backends, such as a geometric executor for simulation or a learned policy for real-world deployment. Navigation Skills—markdown-based procedural guidance—provide the MLLM with reusable strategies for search, recovery, and verification, which the model reads on-demand to handle complex, multi-stage requests.
Why does this approach use a harness instead of just fine-tuning the MLLM on navigation data?
Fine-tuning limits the model to the specific environments and tasks present in the training data. By using a harness, SuperNav preserves the MLLM's general-purpose reasoning capabilities, allowing it to adapt to new scenes and complex, multi-stage requests without retraining.
What is the primary role of the Navigation Skills in this framework?
Skills provide the MLLM with reusable, on-demand procedural guidance for recurring challenges like blocked passages or target verification. They do not programmatically enforce actions, but rather inform the MLLM's decision-making process by entering the context window.
SuperNav demonstrates that navigation can be treated as a general-purpose agentic task, shifting the burden from navigation-specific model training to effective harness design and tool-use interfaces.
Motivation and Problem Framing
SuperNav decouples high‑level reasoning from low‑level motion by wrapping a pretrained MLLM in a navigation‑specific harness.
General‑purpose service robots must satisfy a wide range of human requests in environments they have never seen, yet existing approaches either fine‑tune multimodal large language models (MLLMs) for specific navigation actions or rely on rigid modular pipelines that cannot adapt to new tasks.
Robots need both task generality—handling arbitrary requests—and scene generality—operating reliably in unseen spaces—without retraining the core language model for each new navigation problem.
**Figure 1.** Skill-guided sofa search. The Agent rechecks candidates, recovers from a blocked doorway, and verifies completion.
The key shift is moving from task‑specific fine‑tuning to a general‑purpose, agentic navigation harness that lets a pretrained MLLM direct motion through reusable tools.
The SuperNav Agent Architecture
The SuperNav Harness coordinates MLLM reasoning with navigation tools and skills.
The core difficulty is that a single pretrained MLLM cannot directly control a robot’s low‑level motion. Without a dedicated harness, the model would have to learn every motor primitive, which is inefficient and brittle. The SuperNav Harness solves this by delegating execution to specialized tools while the MLLM focuses on high‑level reasoning.
The harness is a thin orchestration layer that keeps the MLLM’s dialogue context alive, routes its high‑level commands to concrete navigation tools, and feeds back observations so the model can keep planning.
How does the SuperNav Harness differ from simply fine‑tuning the MLLM for navigation?
Fine‑tuning forces the model to internalize low‑level motor commands, which is data‑inefficient and hard to generalize across robots. The harness keeps the MLLM pure as a reasoning engine and plugs in external, replaceable executors, so the same MLLM can drive many hardware backends without retraining.
Initialize a session and obtain the first four‑view observation.
Feed the observation and user request to the MLLM, which parses goals and termination conditions.
Iteratively: (a) construct the current context (history, goals, latest images); (b) let the MLLM choose a Tool or Skill; (c) execute the chosen operation; (d) append the tool’s output to the context.
When the MLLM judges the request satisfied or blocked, invoke the termination Tool to close the session.
The Geo‑based Executor translates a pixel‑level point selected in an image into a 3‑D world coordinate, then plans a collision‑free path to that location using geometric information.
Back‑project the centre pixel: $\mathbf{X}=K^{-1}[0.5,0.5,1]^\top \times 2\text{ m} = (0,0,2)$ m in camera space.
Transform $\mathbf{X}$ to world coordinates using the robot’s pose, yielding a target point 2 m ahead.
Run A* on the occupancy grid; the planner finds a straight‑line path with no obstacles.
Execute the trajectory; the robot moves forward 2 m and captures a new four‑view observation.
Because the executor relies only on geometric depth, it can precisely reach any pixel‑selected location as long as the depth map is accurate, but it fails when the scene contains reflective or transparent surfaces that corrupt depth.
Why not use the Learned Executor for every navigation task if it can predict motion?
The Learned Executor depends on a trained policy that may not generalize to unseen geometry or lighting conditions, whereas the Geo‑based Executor uses explicit depth and geometry, guaranteeing safe, deterministic paths in novel environments.
The Learned Executor treats the selected image point as a goal image and predicts short motion increments that gradually bring the robot toward that visual target.
The encoder extracts a feature vector for the current view and a separate vector for the goal patch.
The policy predicts a displacement of (0.2 m forward, 0.1 m right) and a stop flag = false.
The robot moves accordingly, captures a new observation, and the policy re‑evaluates.
After three iterations the stop flag becomes true, indicating the robot has aligned with the doorway.
The learned executor can reach visually defined goals without explicit depth, but its success hinges on the visual similarity between current and goal patches.
What failure mode distinguishes the Learned Executor from the Geo‑based Executor?
If the visual goal patch is occluded or the lighting changes dramatically, the learned policy may predict incorrect displacements, whereas the Geo‑based executor would still compute a valid 3‑D target from depth.
Navigation Skills are reusable markdown packages that encode procedural advice (search, recovery, completion) which the MLLM can load on demand.
The Skill instructs the MLLM to select a new point slightly left of the original target (e.g., PointNav(front, 0.45, 0.5)).
The chosen point is passed to the Geo‑based Executor, which replans a path around the obstacle.
The robot follows the new path, obtains a fresh observation, and the MLLM re‑evaluates the candidate.
If the candidate is still inaccessible, the Skill advises back‑tracking to the last known open corridor.
Recovery relies on the ability to dynamically re‑select points and re‑plan, which both executors support, but the Geo‑based executor guarantees a collision‑free path while the learned executor may need multiple iterations.
Why does the MLLM still need a separate Recovery Skill if the executor can already handle failures?
The executor can only react to a concrete motion failure; the Recovery Skill provides higher‑level strategy (e.g., choose a new point, backtrack, or switch executors) that the MLLM must decide based on context.
Table 1 (referenced here) reports the navigation performance of SuperNav with both executors across single‑object, multi‑object, and demand‑driven benchmarks. The numbers illustrate that the Geo‑based Executor achieves higher success rates, while the Learned Executor offers comparable efficiency on some tasks.
Empirical Evaluation and Results
SuperNav’s navigation performance beats baselines on multiple benchmarks.
SuperNav + Geo‑based Executor achieves the highest Success Rate of 73.33 % on the 1 m threshold of the Demand‑Driven benchmark.
Table 1 shows SuperNav + Geo‑based Executor outperforming all baselines.
SuperNav outperforms task‑specific baselines across diverse benchmarks.
Ablation Studies
Ablations reveal how each navigation component reshapes tool usage and overall success.
Removing Navigation Skills reduces tool‑call frequency from 11.17 to 7.43 calls per episode (≈33 % drop).
Table 6 shows 7.43 calls for the “w/o Navigation Skills” variant versus 11.17 calls for the full SuperNav system.
Removing the agent‑oriented Tool design inflates tool calls to 30.28 per episode (≈171 % increase).
Table 6 reports 30.28 calls for the “w/o Agent‑oriented Tool Design” variant, far above the 11.17 baseline.
Disabling both Skills and Tool design yields 15.37 calls per episode (≈38 % increase over the full system).
Table 6 lists 15.37 calls for the “w/o Skills and Tool Design” variant.
Substituting the reasoning model with GPT‑6 Astra drops success to 86 / 120 episodes (71.7 % SR) at the $0.25\,$m threshold.
The decision‑model substitution experiment reports 86 successful episodes out of 120 when using GPT‑6 Astra, compared to the full SuperNav configuration.
Learned Executor Architecture
Learned Executor turns a marked image goal into a sequence of robot motions.
Turning a pixel in an image into a concrete robot motion is the bottleneck that prevents end‑to‑end navigation from scaling to real‑world tasks.
How does Learned Executor differ from the original NoMaD goal‑conditioned policy?
NoMaD predicts motions toward a full destination image; Learned Executor instead marks a single pixel in a reference image, keeps that pixel fixed, and augments the current view with a bounded, age‑aware visual history. This design lets the executor operate with a constant 12‑frame budget and reuse cached features, whereas NoMaD would recompute the entire goal image at every step.
Encode each selected frame with DINOv2, obtaining token $z_i$ for every $i\in S_t$.
Compute the age embedding $\psi(t-i)$ for each token and add it to $z_i$, forming $h_i = z_i + \psi(t-i) + e_o$.
Append the goal token $g_t$ (derived from the marked reference image) to the set $\{h_i\}$ and feed the whole collection into the 4‑layer Transformer, yielding fused representation $c_t$.
Condition the flow head on $c_t$ and sample a noisy sequence $x_\tau$, then denoise to obtain the predicted increment sequence $\hat{A}$.
Clip $\hat{A}$ to $[-1,1]$, rescale by $0.25\,$m, and execute the first three increments as turn‑then‑translate primitives.
The history selection keeps recent observations densely sampled while older frames are sparsified, ensuring the executor always has fresh visual context without exceeding the 12‑frame budget.
Closed‑loop execution loop for a single point‑goal request.
**Figure 6.** Learned Executor architecture. A point selected in a fixed reference image is fused with a bounded, age-aware RGB history. The policy predicts displacement increments and a stop score; the executor converts the increments to motion and returns fresh observations for replanning.
**Figure 7.** **Variable-length visual memory.** Each row shows every stride-two candidate on a shared observation-index scale, from the start to the current frame. Colored frames are selected; gray frames are omitted from this fusion pass but remain available in the history. Repeated thinning of the older candidate half limits the input to at most 12 observations, selecting 5, 10, and 11 frames at $t = 8$, 24, and 60, respectively.
Conclusion and Summary
Appendix A details the SuperNav implementation, runtime tooling, and evaluation limits.
SuperNav wraps a pretrained MLLM in a navigation harness, delegating low‑level motion to specialized executors without any navigation‑specific fine‑tuning.
In the Unitree Go2 platform the backend fuses four RGB cameras with LiDAR odometry; the mapper builds a voxel‑grid height map (voxel size 0.05 m) and an A* planner generates paths that a proportional heading controller follows at 10 Hz.
For simulation, Habitat‑Sim supplies 640 × 480 RGB‑D frames (90° HFOV, camera height 1.25 m); the Geo‑based Executor back‑projects pixel anchors using depth, queries a precomputed navigation mesh via Recast/Detour, and a GreedyGeodesicFollower executes discrete forward and turn actions with a 0.7 m standoff and 0.3 m goal radius.
The agent loop runs through the Codex CLI, exposing environment operations via the Model Context Protocol (MCP); each task launches a dedicated working directory containing the Skill snapshot and logs, with read‑only access to the Skill bundle and strict tool whitelisting.
Both executors share a visual‑point contract (view, normalized image point, feedback); the Learned Executor uses hab local navigate with status and stop calls, while the Geo‑based Executor uses hab visual point navigate without a preview round‑trip.
Navigation calls return four directional RGB images; a global‑task JSON filter strips absolute pose, maps, and evaluator metadata, leaving only the observations needed for downstream decision making.
The Skill manifest records file inventories, hashes, and a bundle hash; a visibility probe validates the manifest before loading, ensuring that the same backend, entry Skill, and snapshot hash can be reused safely.
Benchmarks enforce hard timeouts (3 800 s per Codex execution, 180 s MCP startup, 300 s per tool call) and require an explicit hab close session outcome; failures are logged and the batch runner proceeds to the next task.
Evaluation uses geodesic‑distance thresholds (≤ 1 m for single‑object and multi‑object tasks, ≤ 2 m for demand‑driven tasks) and demands an explicit STOP command; ordered arrival is required for sequential demand‑driven benchmarks.
**Table 5.** Baseline observation settings in Habitat-GS and AI2-THOR. Dimensions refer to environment-rendered images before each model's preprocessing. Camera height is 1.25 m. All policies receive the complete instruction and retain their native history mechanisms.
The table compares four navigation methods (NaVid, UniNaVid, StreamVLN, and OmniNav (Action Former)) across four metrics: RGB views, resolution per view, HFOV, and precision.
Benchmark Construction Details
Evaluation protocols and extra results for instance‑navigation benchmarks.
We build the instance‑navigation benchmark in Habitat‑GS using InteriorGS scenes, generate object‑level semantic annotations with a vision‑language model, and manually verify them. The final set comprises 150 single‑object tasks and 150 ordered multi‑object tasks, each containing two to five targets per instruction.
Success is measured by reaching an approved viewpoint within 1 m: $\min_{v \in V_g} d_{\text{nav}}(p, v) < 1 \text{ m}$. SPL is computed as $S_i L_i / \max(L_i, P_i)$, where $S_i$ indicates success under the viewpoint criterion and $P_i$ is the executed path length.
Stage denominators count only completed goals; SRₖ measures the fraction of tasks that finish the first k goals without requiring a final STOP. Multi‑object tasks contain 38, 37, 37, 38 tasks for k = 2–5, while demand‑driven tasks have 70, 48, 45, 26, 8, 1, 1, 1 tasks for k = 1–8. No method reaches six or more stages, so SR₆‑SR₈ are zero for most baselines.