SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?
Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Xu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, Wentao Zhang, Weikang Li, Yunfang Wu
SpatialScout evaluates VLM agents on their ability to autonomously explore 3D scenes and construct precise metric object maps.
Can current Vision-Language Model (VLM) agents effectively explore 3D environments and map object locations using active visual observation?
Spatial intelligence benchmarks typically rely on question answering, which allows models to guess correctly using language priors without actually understanding the 3D environment. SpatialScout forces agents to actively explore 80 indoor scenes, select viewpoints, and triangulate object centers into a shared coordinate frame using a fixed budget of camera movements and observations. Even the strongest models struggle to bridge the gap between coarse layout recovery and precise metric mapping, with only 3.25% of targets achieving high-precision localization.
Paper Primer
The benchmark requires agents to perform closed-loop active perception: they must decide where to move, capture images, associate objects across views, and refine their 3D map estimates. The core move is the integration of a shared tool interface—including camera control and geometric triangulation—that forces the model to demonstrate spatial reasoning through explicit coordinate submission rather than multiple-choice selection.
Frontier vision-language models fail to achieve high-precision 3D mapping despite successful coarse layout recovery.
Claude Opus 5.5, the top-performing model, achieves a mean IoU of 41.71% across all targets, but only 3.25% of its predictions meet the stringent 0.9 IoU threshold. A massive drop-off in precision occurs when moving from coarse placement to strict geometric alignment.
Failure analysis reveals that incomplete coverage is the primary bottleneck, followed by incorrect visual correspondences that persist even when geometric triangulation residuals are low. Models that perform well exhibit extensive multi-view grounding and incremental refinement, whereas weaker models often waste interaction budgets on uninformative viewpoints or fail to update their maps as they explore.
Why is this benchmark a better test of spatial intelligence than existing question-answering datasets?
QA benchmarks allow models to succeed via language biases or lucky guesses; SpatialScout requires agents to build a persistent, metric 3D representation that must be updated through active, multi-step interaction.
What is the primary constraint on the agent during the mapping process?
Agents operate under a strict interaction budget of 100 model calls and 100 image captures, forcing them to prioritize informative viewpoints and efficient map updates over exhaustive exploration.
SpatialScout shifts the evaluation of spatial intelligence from static answer accuracy to the active construction of metric maps, revealing that current VLMs lack the geometric grounding necessary for reliable 3D scene understanding.
Introduction and Motivation
We expose the gap between passive QA and active 3D mapping by introducing the SpatialScout benchmark.
Spatial intelligence hinges on agents actively gathering visual evidence to resolve occlusions and perspective ambiguities. Existing benchmarks rely on spatial question answering, which only tests whether a model can produce a correct answer, not whether it can build an accurate metric map through autonomous exploration. Consequently, they leave open whether Vision‑Language Models (VLMs) can truly construct reliable 3D representations.
SpatialScout turns the evaluation of VLM agents into an active 3D mapping problem, requiring them to move a camera, fuse multiple views, and output object centers in a common coordinate system.
When evaluated on SpatialScout, the strongest frontier VLM, Claude Opus 5.5, attains a mean object‑scale overlap of 41.71 %, with only 47.23 % of targets exceeding an IoU of 0.5 and a mere 3.25 % surpassing IoU 0.9. These numbers reveal a substantial gap between coarse layout recovery and the construction of precise metric maps.
The key shift is moving from passive question answering toward active 3D exploration and mapping.
SpatialScout Benchmark Design
SpatialScout tests VLM agents on active 3‑D exploration and object‑center mapping.
SpatialScout measures how Vision‑Language Models (VLM) actively explore a 3‑D scene and build an object‑center map, moving beyond static question‑answering.
**Figure 1.** Overview of SpatialScout. A VLM agent observes, explores, and localizes the position of objects in a 3D environment.
Unlike prior QA‑style tests, SpatialScout requires agents to actively acquire visual evidence, continuously update a metric map of object centers, and be judged on the final spatial reconstruction rather than a single answer.
Agent Interface and Mapping
Construction of the SpatialScout benchmark and its agent interaction protocol.
We now detail how the SpatialScout benchmark is built and how agents interact with it during evaluation.
**Table 2.** Shared agent interface. No tool reveals ground-truth target positions or evaluation scores.
**Figure 3.** Evaluation of VLM agents on SpatialScout. A VLM agent navigates through the 3D space by controlling a camera, and applies observation and mapping tools to finally submit the positions of identified objects.
Instead of passively reconstructing a scene from a single snapshot, the agent actively moves the camera, gathers multiple views, and fuses them to infer the 3‑D centers of objects—much like a treasure hunt where each new viewpoint reveals a clue that narrows down the hidden location.
How does active object‑center mapping differ from the passive reconstruction approaches used in earlier 3‑D perception benchmarks?
Passive methods typically ingest a single, fully‑observed point cloud and output a reconstruction; active mapping requires the agent to decide where to look, collect multiple partial observations, and iteratively fuse them. The key difference is that the agent must control the camera (via `move_camera`, `rotate_camera`) to reduce uncertainty, rather than relying on a static observation.
Fix the scene catalog (80 indoor scenes, 3,410 targets) and the initial camera pose $o_0$, $R_0$.
Hold constant the tool interface (Table 2) and the target‑region radius $r = \max_i \|p_i\|_2$.
Vary the agent’s policy for selecting views, moving the camera, and invoking
Measure the accuracy of the submitted center map using Intersection over Union (IoU) between the predicted and reference object volumes.
Record the completeness as the fraction of targets for which the agent successfully submitted a position.
It moves the camera to a new pose and captures an RGB image.
Using
It submits $p'_1$ with
Repeating the process for targets 2 and 3 produces estimates $p'_2$ and $p'_3$.
Finally, it calls
This trial shows how the agent’s view selection and triangulation directly affect the final center estimates; errors in camera pose or correspondence will propagate to $p'_i$, impacting both accuracy and completeness.
Evaluation Metrics and Results
Standardizes tool use and constraints to compare VLM agents on object‑center mapping.
SpatialScout measures VLM agents by how well they actively explore scenes and build an object‑center map, rather than merely answering static questions.
Introducing a unified evaluation protocol removes variability in tool usage and coordinate limits across agents.
All seven agents are run with the same ten tools, identical prompts, and the validator enforces $||\pi||_2 \le r + 0.5 m$ for every submission.
All agents receive the identical task prompt and tool descriptions, and the same movement bounds defined in Appendix A.
**Table 3.** Object-micro localization results on 80 scenes. All values are percentages, and threshold columns use strict IoU comparisons. The > k% columns measure the percentage of box overlaps that are greater than k%.
**Figure 4.** Examples of accurate, incomplete, and mislocalized object-center maps. The red arrows indicate a missing or mislocalized object in the reconstructed scene.
Geometric Grounding and Triangulation
Defines the geometric triangulation tool, budgets, and logging used to evaluate agents.
We evaluate each VLM agent by running a fixed‑budget episode that interleaves perception tools and reasoning steps.
It reconstructs a 3‑D point from the lines of sight that the agent marks in two calibrated camera views — like intersecting two sight lines to locate a point in space.
Compute $A_1 = I - d_1 d_1^\top = \begin{bmatrix}1&0&0\\0&1&0\\0&0&0\end{bmatrix}$ and $A_2$ identically.
Form the objective $\|A_1(p-o_1)\|^2 + \|A_2(p-o_2)\|^2$ and differentiate w.r.t. $p$.
Solving the linear system gives $p = (0.5,0,5)$, the point equidistant from both sight lines.
Residual $\sqrt{0}=0$ and ray angle $0^\circ$, indicating perfect parallax.
This toy case shows how the projection matrices enforce perpendicularity and how the minimizer balances the two lines; in real scenes, non‑zero residuals flag weak parallax or mismatched features.
How does this triangulation differ from the classic linear‑triangulation used in structure‑from‑motion?
Classic linear‑triangulation solves a homogeneous system directly, assuming noise‑free correspondences. Here the formulation explicitly projects onto the orthogonal plane ($A_k$) and minimizes perpendicular distances, which makes the residual a diagnostic of geometric consistency rather than just a solution error.
Initialize episode with budget $B = 100$ model calls and capture limit $V = 100$ images (the initial image counts toward $V$).
Agent receives the first image and may issue a model call; the call can request any subset of tools (e.g., triangulation, depth estimation).
Each tool execution consumes no additional model call but may produce new images that increment the capture count.
When the capture count reaches $V$, further tool requests are ignored; remaining model calls can still be used for reasoning.
Agent may invoke
All attempts (model, prompt, tool schema, budgets) are logged; failures (service or runtime) are recorded separately and excluded from the final score.
After the episode ends (either by exhausting $B$, calling
Table 3 reports object‑micro localization percentages for all models, using strict $IoU$ thresholds (0 %, 50 %, 90 %).
Geometric Grounding Performance
Quantitative and qualitative evaluation of Active Mapping performance.
The section reports how well Active Mapping agents construct object‑center maps and highlights the impact of tool‑call accuracy on overall performance.
$IoU$ measures the volumetric overlap between a predicted object box and its ground‑truth counterpart, indicating how accurately the map captures object extents.
Astra outperforms DS‑V4.1 across all geometric grounding metrics on the shared 80‑scene benchmark.
Table 5 shows Astra achieves 99.33% successful triangulations versus 63.48% for DS‑V4.1, with fewer geometric warnings and lower median center error.
**Table 5.** Geometric grounding on 80 shared tasks. Both models use the xhigh setting. Triangulation support is measured over submitted objects, warnings over successful triangulation calls, and median center error over submitted objects.
The accuracy of tool‑call execution directly governs the quality of the final object‑center map.
Tool-Calling and Failure Analysis
We revisit how models actively use tools to map 3‑D scenes and what that reveals.
Recall that SpatialScout measures how agents actively explore 3‑D scenes rather than merely answering questions. This section examines the tools they invoke and how those actions relate to mapping performance.
Agents decide when and where to invoke external tools (e.g., move a camera or request a triangulation) as part of a purposeful exploration loop.
How does “Agentic Tool-Calling” differ from simply issuing many tool commands?
It is not about quantity; the agent chooses tools based on the current uncertainty and the expected reduction in mapping error. Blindly issuing many calls wastes time and context, whereas an agentic policy targets the most informative viewpoints.
**Figure 5.** Tool-use patterns across interaction steps for four models. Claude Opus 5.5 makes the most tool calls, whereas GPT-6 Astra achieves competitive mapping performance with fewer calls.
**Figure 6.** Illustration of tool-call frequency of multiple models.
**Table 4.** Success factors and failure patterns across 40 trajectories. Each trajectory may exhibit multiple factors (non-exclusive).
**Figure 7.** All-target mean IoU vs. average inference cost per task.
Related Work
We position our benchmark among prior works on active 3D reconstruction and embodied evaluation.
Agentic inverse graphics and scene construction have been explored in several recent benchmarks. IR3D‑Bench evaluates scene understanding by reconstructing a scene from a single image, while BlenderGym studies code‑based graphics editing and its impact on performance. BlenderAlchemy, VIGA, and BlenderBench further examine interleaved execution, inspection, memory, and exploratory editing.
Active reconstruction and next‑best‑view planning have a long history in 3‑D perception. SCONE predicts surface‑coverage gain, MACARONS couples RGB mapping with online coverage anticipation, and GenNBV learns a generalizable view‑selection policy. Recent systems such as NARUTO, ActiveSplat, ActiveGAMER, FisherRF, AIR‑Embodied, AREA3D, and SightCAD integrate exploration with neural scene mapping or vision‑language guidance.
Spatial and embodied benchmarks assess reasoning from visual observations. VSI‑Bench and SITE focus on spatial reasoning, OpenEQA adds episodic memory and active‑exploration settings, and Explore‑until‑Confident studies stopping criteria for embedded QA. E3VS‑Bench, ESI‑Bench, S3‑Bench, and EmbodiedBench evaluate viewpoint‑dependent evidence acquisition, active observation, and multi‑object tasks.
Memory, geometry tools, and process diagnosis underpin many of these systems. ConceptFusion and ConceptGraphs build semantically rich 3‑D representations; 3D‑Mem and SpaMEM provide scene memory for embodied reasoning. DUSt3R, VGGT, NeRF, and Gaussian splatting offer geometric inference and rendering capabilities, while 3DCodeBench investigates inference‑time computation in graphics tasks.
Long‑horizon agentic benchmarks evaluate sustained reasoning and tool use. AgentBench and GAIA test interactive decision‑making; WebShop, WebArena, VisualWebArena, and OSWorld extend evaluation to web and desktop environments. $\tau$‑bench, DySQL‑Bench, SWE‑bench, Terminal‑Bench, NL2Repo, RepoZero, and AgenticRAGTracer probe tool use under evolving interactions. SpatialScout adds a spatial‑intelligence focus for VLM agents.
Limitations and Future Work
We discuss the reliance on simulated data and the upcoming real‑world dataset release.
SpatialScout currently depends on data generated by a physics simulation engine, which may not capture the full variability of real‑world scenes. Consequently, the benchmark’s conclusions might not fully transfer to physical environments.
To address this limitation we will release a real‑world dataset within the next month and welcome collaborations to expand the evaluation.
Call
Capture an initial RGB image with
Optionally reposition the camera using
Capture additional images from new viewpoints.
For each visible object, record pixel coordinates of a surface point (or silhouette center) in the original PNG frame.
Use
Infer the object’s axis‑aligned bounding‑box center from its observed extent and the triangulated point.
Submit the estimated center via
When all desired objects are submitted, call
The gap between simulated physics and real‑world scene complexity may cause agents that perform well in the benchmark to struggle when deployed in physical environments.
Benchmark Implementation Details
Full reproducibility details for scene generation, interaction, and evaluation.
Scene preparation records the random seed, render size, simulation and rendering back‑ends, input‑image encoding, dependency versions, and SHA‑256 fingerprints of the source configuration, benchmark and simulator code, and asset tree. Cached preparations are reused only when their fingerprints, initial camera poses, and target counts agree with the requested configuration.
Movement transforms absolute positions from the task frame to world coordinates by $o_0 + R_0 p$, with each world coordinate bounded by $\pm15$ m. Relative translations are limited to $1$ m, yaw and pitch increments to $180^\circ$ per axis, final pitch clamped to $[-85^\circ,85^\circ]$, and roll remains fixed.
Prompts share a common episode loop, task prompt, and tool descriptions; the prompt also encodes the model‑call and capture budgets, the fixed task axes, and the world‑AABB center convention. When $V=1$, the model is asked to estimate from the initial image and its calibration, with budget reminders inserted before model requests (10 and 3 calls remaining).
Attempt accounting uses a $180$‑second timeout per HTTP request and no explicit temperature setting. Retries are allowed up to three attempts with $1$‑ and $2$‑second backoffs for network failures; logical model calls and HTTP attempts are counted separately.
**Figure 8.** Visualization of scene distribution in SpatialScout
**Figure 9.** Visualization of object distribution in SpatialScout
Multi‑view point estimation requires at least two observations per query, a minimum camera separation of $5$ cm, and a positive condition number for the matrix $A_k$; queries failing these checks are rejected. The tool reports the RMS perpendicular line residual and the maximum pairwise ray angle, warning when the angle is below $3^\circ$, the residual exceeds $0.1$ m, or the point lies beyond $r+0.5$ m.
Scoring assigns each prediction to a target via groupwise minimum‑total‑Euclidean‑distance assignment, then computes mean, median, RMSE, and mean IoU on that assignment. Predictions are ordered lexicographically by task‑frame coordinates; targets are ordered by center and side lengths, ensuring reproducible tie‑breaking.
Aggregation uses object‑micro means $s \in D \, M_s / \sum_s M_s$ for submitted objects and $s \in D \, N_s / \sum_s N_s$ for all targets; scene‑macro summaries average per‑scene ratios equally. Empty maps yield zero for IoU and overlap‑rate aggregates, and undefined center errors are excluded from averages.
The omission penalty adds a configurable cost $\lambda$ per missing target; we use $\lambda=5$ m, so each omitted target contributes $5$ m to $L_{\text{scene}}$. For an empty map $L_{\text{scene}}=\lambda$, and for a non‑empty map $L_{\text{scene}}=C\bar{e}+(1-C)\lambda$, where $C$ is coverage and $\bar{e}$ is mean center error.
Object‑scale analysis shows that $6.7\%$ of targets have a side shorter than $1$ cm; at IoU $>0.9$ the allowed displacement is below $\ell/19$, roughly $5.26\%$ of the axis length, corresponding to $0.526$ mm for the $1$ cm minimum side.
Inventory‑only baselines submit $p_i=0$ for every inventory item, while the uniform‑ball policy samples a direction from a normalized Gaussian and a radius $r_U \propto U^{1/3}$. Their scene‑macro full‑scene losses are $3.113$ m and $4.701$ m respectively, providing RGB‑free localization baselines.
Map‑update replay reconstructs the acquired‑image count from the initial capture and all subsequent successful capture events, allowing evaluation of the same accepted maps under different omission penalties $\lambda\in\{1,2,5,10\}$ m to isolate reporting trade‑offs from prediction changes.
Limitations include reliance on accurate camera calibration, sensitivity to very small objects, and the fact that omission penalties are heuristic rather than learned; future work may address these by improving sensor models and learning adaptive penalties.