Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes
Yu Wang, Rui Dai, Wei Chen, Yi Wang, Bo Dang, Kaikui Liu, Xiangxiang Chu, Yansheng Li
Urbex replaces exhaustive city-scale 3D candidate ranking with an agentic, active spatial search policy.
How can we perform city-scale 3D grounding without the computational overhead of precomputing instance proposals for every object in a massive scene?
City-scale 3D grounding is currently bottlenecked by static candidate ranking, which requires precomputing thousands of object proposals and becomes prohibitively slow as scenes grow. Urbex reformulates this as an active spatial search: a Vision-Language Model (VLM) agent interacts with a multi-view environment, using tools to anchor, zoom, and verify regions until it can commit a grounded bounding box. This agentic approach achieves state-of-the-art accuracy on CityRefer while improving inference speed by an order of magnitude compared to ranking-based baselines.
Paper Primer
The core mechanism is a tool-use loop that exposes the massive 3D scene through a compact, coordinate-aligned interface. The agent parses the query, calls a Landmark Query to anchor the search, uses Zoom-in to recover high-resolution local geometry, and renders Oblique Views for 3D verification, all while maintaining a consistent history of spatial observations.
The policy is optimized via reinforcement learning using Group Relative Policy Optimization (GRPO). The reward function combines terminal IoU with dense shaping rewards—such as distance-to-target progress and tool-level feedback—to guide the agent's search trajectory without requiring supervised demonstration data.
Urbex achieves superior grounding accuracy and significantly faster inference than ranking-based methods.
On the CityRefer benchmark, Urbex achieves the best Hit@1 and IoU metrics while reducing inference time to ~3.2 seconds per example. An order-of-magnitude speedup over the strongest ranking baseline (CityAnchor), which requires ~32.1 seconds per example.
The learned search policy generalizes to unseen urban environments.
Zero-shot evaluation on STPLS3D-Refer shows Urbex outperforming all baselines, confirming the agent learns reusable spatial search behaviors rather than dataset-specific ranking rules. Achieves 23.3 Hit@1 on landmark-free expressions in STPLS3D-Refer without further training.
Why is active search more effective than the standard candidate-ranking approach?
Ranking methods are constrained by the quality of precomputed proposals and scale poorly because they must score every candidate in the scene. Urbex avoids this by actively inspecting only the most informative regions, which is both more efficient and less dependent on the completeness of initial instance segmentation.
What is the primary assumption enabling this agentic framework?
Urbex assumes that a city-scale scene can be effectively represented through a small number of coordinate-aligned views (BEV and oblique) that preserve metric grounding, allowing the VLM to map 2D tool interactions back to 3D world coordinates.
Urbex demonstrates that city-scale 3D grounding is better framed as a sequential decision process than a retrieval task. Researchers should shift from optimizing static scoring models to designing agentic policies that can navigate and verify large-scale spatial environments.
Motivation and Problem Framing
City-scale 3D grounding is bottlenecked by exhaustive candidate ranking, prompting an active search reformulation.
Existing city‑scale 3D grounding pipelines first generate thousands of object proposals and then rank them against the referring expression. Scoring every candidate becomes prohibitively expensive as scene size grows, and the final accuracy hinges on proposal quality.
Locate a target object directly within a massive urban point cloud instead of first enumerating all possible objects.
Score each precomputed object proposal against the language description and pick the highest‑scoring one.
**Figure 1.** Overview of Urbex. Unlike regular grounding methods that downsample city-scale point clouds or ranking-based methods that score many precomputed instances, Urbex formulates grounding as agentic spatial search. A VLM agent actively interacts with the scene through tool calls, moving between global, zoomed, and oblique views before committing the final target location.
The primary bottleneck of existing city‑scale grounding is the exhaustive scoring of thousands of candidates.
The Urbex Agentic Framework
Method details the active spatial search mechanism, its tools, and the RL training that drives the VLM policy.
Urbex reframes city‑scale 3D grounding as an active spatial search, replacing exhaustive candidate ranking with a VLM‑driven, tool‑guided exploration of a compact multi‑view space.
Instead of scoring every possible object, the system lets a VLM agent wander through a few strategically chosen views, zooming and rotating only where the query suggests the target might be.
Step 1: Predict coarse box → world‑space region covering $2000\times2000$ pixels.
Step 2: Zoom‑in on that region → high‑resolution crop of $500\times500$ pixels.
Step 3: Oblique Rendering of the $100\times100$ sub‑region provides depth cues.
Step 4: Agent submits final box after confirming alignment across the three views.
The search quickly discards the vast majority of the city, focusing only on a handful of pixels that matter for the query.
How is Active Spatial Search different from static candidate retrieval?
Static retrieval pre‑computes scores for every object in a large set $C$, which scales linearly with city size. Active search lets the VLM generate and evaluate only a few views on‑the‑fly, so compute grows with the number of tool calls rather than the number of objects.
The policy $\pi$$\theta$ is a vision‑language model that treats each tool call as a step in a dialogue, using the full interaction history to keep spatial references consistent.
Why does the policy need the full interaction history instead of just the latest view?
Spatial references are expressed relative to earlier frames (e.g., “zoom further into the box you just saw”). Without the full history the model would lose the coordinate context needed to maintain consistency across views.
The environment offers three lightweight tools—Landmark Query, Zoom‑in, and Oblique Rendering—that together give the agent global anchors, high‑resolution detail, and 3D shape verification.
Why are these three tools sufficient for city‑scale grounding?
Landmark Query provides a global anchor, Zoom‑in supplies fine‑grained texture, and Oblique Rendering adds the missing depth dimension. Together they cover the three information gaps that static 2‑D BEV images cannot resolve.
Initialize with the global BEV image $I_{\text{global}}$ and the natural‑language query $q$.
At step $t$, feed the current observation $o_t$ and history $h_t$ to the VLM policy $\pi$$\theta$.
Sample a thought trace and an action $a_t$; if $a_t$ is a tool call, the environment executes it and returns a new view $I_{v'}$ and transform $T_{v'}$.
Append the tool’s textual response and any new image to $h_{t+1}$, preserving coordinate frames via $T_{v'}$.
Repeat until the policy outputs a final bounding box, which is projected to world coordinates using the stored transforms.
**Figure 2.** Architecture of Urbex: (left) a multi-view environment exposing global BEV, zoomed, and oblique views; (middle) the policy’s reasoning trace and tool calls over observation $o_t$ and interaction history $h_t$; (right) RL under a terminal reward (IoU and center-distance) plus tool-level shaping.
Experimental Setup
Urbex’s experimental setup and benchmark results on CityRefer.
Urbex processes an example in 3.2 seconds, an order of magnitude faster than the nearest baseline.
Table 1 reports inference time: Urbex 3.2 s vs CityAnchor 32.1 s.
The benchmark evaluates how well a model can locate a target object in a city‑scale 3D scene using only a free‑form BEV box.
We evaluate Urbex on three axes: in‑domain performance, cross‑scene generalization, and component sensitivity, using the CityRefer Benchmark and the STPLS3D‑Refer dataset converted to a common BEV‑based interactive environment. All tool‑using models share the same action protocol and evaluation code. Baselines include general‑purpose VLMs (GPT‑5.4, Qwen3.6‑plus), indoor 3D grounding methods (EDA, InstanceRefer), and city‑scale ranking methods (CityRefer, CityAnchor).
**Table.** Comparison of Urbex against various baselines on city-scale 3D grounding tasks, evaluating performance metrics (Hit@k, IoU, F1) and inference time.
Performance and Ablation Studies
Urbex delivers order‑of‑magnitude faster inference and superior accuracy across benchmarks.
Urbex runs inference in 3.2 seconds per example, roughly ten times faster than the strongest baseline CityAnchor (32.1 seconds).
Urbex processes only a small number of informative regions, while CityAnchor scores many candidate instances.
**Figure 3.** Qualitative examples on CityRefer. The visualization shows how Urbex moves from global BEV search to local zoom-in and final grounding in city-scale scenes with dense repeated objects and landmark-based references.
Zero‑shot transfer to STPLS3D‑Refer confirms that Urbex’s learned search policy generalizes beyond CityRefer. Across all reported metrics, Urbex attains the highest scores, demonstrating that the policy captures reusable behaviors such as anchoring, zooming, verification, and stopping rather than overfitting to a specific dataset.
Generalization and Transfer Analysis
We quantify how each tool and reward component shapes Urbex’s search performance.
Recall that Urbex treats city‑scale grounding as an active spatial search, using a VLM agent to zoom into relevant regions rather than processing every object.
With all three tools (Zoom‑in, Landmark Query, Oblique Rendering) Urbex attains 36.9 Hit@3, far above the 10.5 Hit@3 achieved when no tools are used.
Table 3 reports 36.9 Hit@3 and 14.6 2D IoU with all tools enabled, versus 10.5 Hit@3 and 2.1 IoU with none.
RL‑trained Urbex raises Hit@1 to 10.0, compared with 2.7 for the base Qwen model under the same fixed tool order.
Table 4 (policy comparison) records 2.7 Hit@1 for Qwen and 10.0 Hit@1 for Urbex RL.
Landmark‑robustness experiments show that omitting the named landmark in a query causes a performance drop exceeding ten points, whereas a modest 10 m coordinate noise only reduces Hit@1 by 1.2 points.
Requiring the predicted box to have BEV IoU ≥ 0.25 cuts Hit@1 from 23.53 to 18.71, a 4.8‑point decline.
Appendix D.5 reports the drop when the BEV IoU threshold is enforced.
**Figure 4.** Qualitative examples on STPLS3D-Refer. The visualization highlights zero-shot transfer cases where the same active-search policy anchors, zooms, verifies, and grounds targets in scenes with different geometry and object distributions from CityRefer.
Agent Trajectory Analysis
Ablations quantify how tool ordering, stochastic decoding, and landmark reliability affect Urbex performance.
We evaluate three orthogonal axes: stochastic decoding variance, the contribution of the learned tool order, and the dependence on accurate landmarks. Each ablation isolates one component and measures the resulting Hit@1 change.
Urbex’s stochastic decoding yields a modest but consistent gain over the strongest baseline.
Table 8 shows Urbex Hit@1 = 25.62 ± 0.33 % versus CityAnchor = 23.4 %.
**Figure 7. Training dynamics on CityRefer.** (a) Total task reward. (b) Train/validation 2D grounding IoU with periodic validation markers and final-value annotations. (c) Per-episode tool-call frequency, showing convergence from over-calling `zoom_in` and under-using `oblique_render` to one call per tool.
Execute a Landmark Query to retrieve a reference coordinate.
Zoom in on the retrieved region to obtain a high‑resolution top‑down view.
Render an oblique perspective of the zoomed region for 3‑D verification.
Predict the final 2‑D bounding box in the zoomed view.
Constraining the agent to the fixed L‑Z‑O order drops Urbex’s Hit@1 by over ten points.
Table 9 reports free Urbex Hit@1 = 23.5 % versus fixed = 13.1 %.
The figure illustrates a multi-step visual navigation process involving three distinct views: Global BEV, Zoomed BEV (Z1), and Oblique View. The process begins with a "Landmark Query" to identify a location on the global map, followed by a "Zoom In" action to focus on a specific region of interest, and finally transitioning to an oblique perspective to inspect details like a specific car in a parking lot.
Noisy landmark perturbations reduce Urbex’s Hit@1 by several points, confirming landmark precision matters.
Table 10 shows Urbex Clean = 16.3 % versus Noisy 10 m = 12.7 %.
Tables 8–10 collectively demonstrate that (i) stochastic decoding contributes a small but reliable boost, (ii) the learned tool order is essential for peak performance, and (iii) accurate landmarks are a non‑trivial prerequisite for the active spatial search.
Failure Modes and Robustness
Limits of active spatial search and analysis of proposal and matching diagnostics.
Recall that the paper treats grounding as an active spatial search, using a VLM agent to zoom into relevant city regions instead of pre‑processing every object.
We replace ground‑truth candidates with Soft‑Group proposals (recall 35.6 % at $IoU\ge0.25$) and compare the resulting pipeline to the GT‑candidate baseline and to Urbex. The proposal pipeline achieves $Hit@1$ = 6.5 % versus 14.2 % with GT candidates, and its direct $BEV\ IoU$ drops to 4.2 % from Urbex’s 14.6 %.
We re‑score the same predictions under three matching rules: the original $IoU$, $IoU$ with positive‑overlap filtering, and CIoU. $Hit@1$ varies by at most 0.29 points (23.53 % → 23.24 %), showing that identification is stable across reasonable rule changes.
When we additionally require the free‑form box to achieve $IoU_{BEV}\ge0.25$ (or $0.5$) and to rank first, $Hit@1$ drops from 23.53 % to 25.62 % (NO) and 11.46 % (NO) respectively, illustrating the cost of loose extents.
Counting candidate overlaps shows a mean of 6.5 (NO) and 4.6 (ND) at any positive $IoU$, but the median number of overlaps at $IoU\ge0.25$ is one for both splits, indicating that most predictions have only a single strong match.
The predicted‑to‑target area ratio has a median of 1.40 (NO) and 1.56 (ND), with $P_{90}$ values of 9.92 and 7.21 respectively, revealing a tail of oversized predictions that can hurt small targets.
Trajectory diagnostics (Table 16) show that regional‑search success declines as target‑landmark distance grows (e.g., 40.0 % < 50 m vs. 14.3 % ≥ 200 m) and that local‑disambiguation success drops sharply with more same‑class distractors (e.g., 64.7 % with 0 distractors vs. 25.1 % with ≥10 on ND).
Inference latency breakdown (Table 17) shows that cached tools (landmark query < 1 ms, zoom‑in ≈ 70 ms, oblique rendering ≈ 292 ms) total ≈ 0.36 s, yielding a full‑model query time of 3.2 s, whereas CityAnchor’s warm online inference costs 32.1 s per query—a roughly ten‑fold speedup.
**Figure 9.** Aggregate trajectory diagnostics from Table 16. Left: success versus target–landmark distance. Right: success versus same-class distractor count within 50 m. These curves visualize the reported bin rates, not individual failure trajectories; per-bin sample counts and error bars are unavailable.
**Observed failure.** Z1 excludes the target car. The agent retains its proposed box after oblique inspection.
**Figure 10** Qualitative failure trajectories on CityRefer-NO. (a) The first zoom excludes the annotated target. (b) The zoom includes the target, but the prediction selects vehicles in a different parking row. Scene context in (a) is an offline display crop of the high-resolution BEV. Zoom and oblique panels preserve the complete recorded tool images; GT and prediction overlays on BEV views are added for diagnosis. Green denotes GT, red denotes the prediction, and dashed blue denotes the first zoom extent. Both runs retain their proposed box after oblique inspection and obtain zero BEV IoU.
Implementation Details
Implementation specifics for Urbex, covering training setup, hardware, and tool definitions.
Urbex is built on the Qwen3‑VL‑4B‑Instruct model and trained on the CityRefer split using the GRPO reinforcement algorithm with a rollout batch size of 256 and 300 optimization steps per batch.
Training runs on 16 NVIDIA H20 GPUs; inference timing is measured on a single H20 GPU, counting model generation and tool execution but excluding offline preprocessing such as cached renders.
Each trajectory permits up to eight assistant turns—seven tool calls followed by a final answer—while the reward combines IoU ($\lambda_{iou}=1.0$), distance ($\lambda_{dist}=0.3$, $\tau=80 m$), step cost ($\lambda_{step}=0.05$), and tool‑level shaping as defined in Equation 2.
**Table 5.** Definitions, inputs, and outputs of the city-aware tools.
**Figure 5.** Prompt template for tool-using VLM agents, specifying the task, tools, coordinates, action format, and final box format shared by prompted VLM baselines and Urbex.
Dataset Statistics
Dataset and tool details for evaluation, including benchmark construction and baseline settings.
This appendix defines the auxiliary tools used for evaluation and describes the datasets and baselines employed in our experiments.
Given a global BEV view, the agent crops a high‑resolution region and optionally rescales it, exposing detail that the global observation cannot show.
From a cardinal direction (north, south, east, or west) the agent renders a perspective view of a candidate region, revealing 3‑D shape, height, and facade appearance.
A.3 Baseline Settings – Closed‑source VLMs receive the same Urbex tool environment and turn budget but no task‑specific policy optimization; they follow the same action format, coordinate convention, and tool‑output history before submitting a final BEV‑aligned box.
B.1 CityRefer Data Preparation – After removing duplicate (scene, query) rows and low‑quality annotations, the dataset is split into train, validation, ND (unseen descriptions) and NO (held‑out‑scene) partitions, yielding 2,744 training, 549 validation, 1,098 ND, and 1,396 NO queries across twelve scenes.
Small objects are difficult to distinguish in a down‑sampled global BEV, which motivates retaining a high‑resolution BEV for each scene and using Zoom‑in to retrieve detailed local crops during search.
B.2 STPLS3D‑Refer Construction – We build a zero‑shot benchmark by extracting instances from two publicly available drone‑photogrammetry scenes (USC and WMSC), registering them to WGS‑84, and augmenting them with OSM landmarks and VLM‑generated expressions.
Instance extraction clusters semantic masks with DBSCAN (buildings: eps = 2.0 m, `min_samples` = 80, footprint ∈ [4, 5000] m²; vehicles: eps = 0.6 m, `min_samples` = 30, footprint ∈ [1.5, 30] m²), then filters fragmented or noisy clusters, yielding 59 buildings/62 vehicles in USC and 11 buildings/17 vehicles in WMSC.
Geo‑registration aligns each scene to WGS‑84 via a first‑order affine transform fitted to 5–7 ground‑control points, achieving RMS residuals of 0.72 m (USC) and 2.19 m (WMSC); OSM features are queried, projected back, and matched to building instances, producing 278 real landmarks.
Scene conversion exports a global BEV image, view‑to‑world mapping, Zoom‑in crops, optional oblique renderings, and annotations compatible with both candidate‑based (CityAnchor) and candidate‑free (Urbex) pipelines.
Referring‑expression generation renders a centered BEV crop and four oblique side views per target, then prompts the VLM to produce concise CityRefer‑style expressions that include landmark cues; after automatic filtering and manual inspection, 445 high‑confidence landmark‑grounded samples remain.
**Figure 6.** Additional qualitative examples on the CityRefer NO split.