UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

Tianjie Ju, Zheng Wu, Yueqing Sun, Yuhan Cui, Bobo Li, Shengqiong Wu, Pengzhou Cheng, Haodong Zhao, Zongru Wu, Xinbei Ma, Doris Zhang, Kunling Li, Mong-Li Lee, Wynne Hsu, Hao Fei, Qi Gu, Gongshen Liu, Zhuosheng Zhang

URBANGROUND evaluates whether MLLM agents can maintain spatial agency in a real-scale, physically constrained city.

Can current MLLM agents maintain a usable spatial understanding of a city as they move through it, or does their local perception fail once they leave their initial viewpoint?

Multimodal Large Language Models (MLLMs) can recognize urban landmarks in static images, but they struggle to maintain a coherent spatial plan as they move through a city. Once a landmark disappears from view, agents often lose their orientation and fail to reconcile past observations with new surroundings. URBANGROUND is a Unity-based sandbox that streams georegistered 3D data of Hong Kong into a closed-loop environment. It forces agents to navigate using first-person perception and an interactive map while facing physical constraints like terrain, road closures, and moving pedestrians. While current models perform well on local spatial reasoning, their success collapses as navigation horizons extend. Performance drops significantly when agents must maintain a goal across multiple city blocks or adapt to dynamic changes in the environment.

Paper Primer

The framework evaluates spatial agency through a five-level ladder of increasing complexity, ranging from local visual recognition to multi-stop navigation in dynamic conditions. It hinges on a closed-loop interaction model: the agent receives a first-person view, executes a physical action, and receives a new observation determined by the city's geometry and collision constraints.

Local spatial competence does not compose into sustained city-scale navigation.

While models achieve high accuracy on local question-answering tasks, success rates for long-range navigation drop to near zero across most tested architectures. Navigation success falls from ~75% on short-range tasks to ~0–3% on long-range, multi-stop, or constrained routes.

Agents fail to adapt to dynamic environmental changes.

When faced with road closures or moving pedestrians, agents continue to produce locally plausible movements but fail to revise their global spatial plans, leading to persistent collision or goal abandonment. Road-closure success rates are effectively 0% across nearly all models.

Why is a real-scale city environment necessary for this evaluation?

Game-based or bounded indoor environments do not require agents to maintain a spatial frame that remains valid over long distances. URBANGROUND forces agents to reconcile continuous physical movement with a complex, georegistered geographic structure.

Does high performance on local question-answering tasks predict navigation success?

No. The study finds that navigation magnifies performance gaps that are invisible in static question-answering, as small errors in spatial recovery compound over time during closed-loop execution.

Urban agency requires more than atomic recognition; it demands the ability to maintain and revise a persistent spatial state. Future research must focus on how agents can correct accumulated errors during long-horizon exploration.

Introduction: The Challenge of Spatial Agency

We frame the challenge of turning local visual perception into sustained spatial agency in real‑scale cities.

Multimodal large language models (MLLMs) can recognize objects in a single street view, yet urban agency demands that this local evidence stay useful as the agent walks through a city.

Spatial Agency is the capability of an agent to keep a coherent estimate of its position and surroundings after each movement, so that later decisions can rely on earlier observations.

We introduce URBANGROUND, the first sandbox that reproduces Hong Kong’s full 3‑D geography and lets MLLM agents interact from a first‑person view while a map records their trajectory.

Our analysis follows three research questions: (RQ1) can an agent ground a local scene well enough to answer spatial queries after brief observation? (RQ2) does that grounding scale to navigation beyond the visible scene? (RQ3) can the agent adapt when the city layout or pedestrian flow changes?

The image displays the logos and names of six academic and research institutions: Shanghai Jiao Tong University, National University of Singapore, Meituan (LongCat), The Chinese University of Hong Kong, Shanghai University, and the University of Oxford.

The gap between local perception and sustained spatial agency is the key obstacle for current MLLM agents in real‑scale urban environments.

Related Work

We survey how prior work evaluates spatial agency across game and physical environments.

Spatial agency requires an agent to link its current visual observation to a persistent spatial frame that stays valid after the view changes, and evaluation environments are needed to test this transition. Existing literature splits this problem between game‑based and physical‑world settings.

Game environments offer controlled, repeatable settings where multimodal models can be tested on turning visual inputs into decisions, and later benchmarks isolate visual‑spatial reasoning or provide explicit state for verification. These setups expose visual‑control failures while keeping action consequences easy to measure.

Persistent 3D worlds extend evaluation to larger spaces and longer trajectories, revealing that agents remain fragile as interactions continue. Yet their spatial structure is still defined by the game itself, limiting insight into true spatial anchoring.

Physical‑world benchmarks embed perception and action within real scene geometry, with indoor embodied QA and navigation tasks converting visual history into local motion, and open‑vocabulary target search testing semantic destination finding under physical constraints. However, their bounded extents limit how far a local observation can be related to a larger spatial frame.

Urban‑scale evaluations use street‑view, aerial imagery, or recorded trajectories to assess navigation in real cities, but interactions remain mediated by discrete visual views, lacking continuous movement through the same geometry. Interactive urban simulators restore closed‑loop control yet rely on generated environments that do not preserve the full georegistered complexity of an actual metropolis, leaving open whether local multimodal competence can compose into true spatial agency.

The official dataset pages provide the 3D Visualisation Map and the 3D Pedestrian Network for further research.

The URBANGROUND Framework

URBANGROUND provides a real‑scale urban sandbox that lets MLLMs be tested for spatial agency via a closed‑loop interaction.

Evaluating spatial agency requires a testbed where evidence gathered from one viewpoint can be reused after the agent moves. Existing simulators either lack city‑scale fidelity or expose privileged state, making it hard to assess persistence and adaptation.

It turns territory‑scale geospatial data into a physically constrained city that an MLLM can explore, while keeping a hidden interaction state that the model never sees.

How does URBANGROUND differ from other urban simulators that provide map APIs?

URBANGROUND never exposes the hidden state $s_t$ or any route information; the model only receives raw visual observations and a limited action set, so any spatial reasoning must be built from first‑person evidence rather than from privileged map data.

Compute $m_0 = U_0(o_0, g)$ – the model extracts a local map of nearby streets.

Select $a_0 = \pi(m_0, g)$ – the model issues a “move forward 5 m” action.

Simulator updates $s_1 = T(s_0, a_0, \xi_0)$ – the agent’s pose advances along the street, collision is checked.

Render $o_1 = O(s_1)$ – a new first‑person view shows the park entrance now in sight.

Update $m_1 = U(m_0, o_1, a_0, g)$ – the model integrates the new view, preserving the earlier street layout.

Select $a_1 = \pi(m_1, g)$ – the model issues “move forward 2 m” to enter the park.

This trace shows how each turn’s observation is derived from the hidden state, and how the model’s action only indirectly updates that state, forcing the agent to retain useful spatial information across moves.

It defines five task levels that progressively require the agent to retain more spatial state, from local grounding to dynamic adaptation under changing weather and road closures.

Why not just test navigation directly instead of using a ladder?

The ladder isolates incremental challenges—grounding, persistence, implicit instruction, multi‑task planning, and adaptation—so we can pinpoint which capability a model lacks rather than conflating all difficulties into a single navigation benchmark.

Level 1: $t\!=\!0$ – observe the shop façade; answer “what is the shop’s name?” (local grounding).

Level 1: $t\!=\!1$ – move forward 3 m; the shop disappears, but the model must still recall its name.

Level 2: $t\!=\!2$ – receive the instruction “go to the park on the right”; the model must keep the earlier shop observation while planning a route.

Level 2: $t\!=\!3$ – navigate around a corner to reach the park, demonstrating persistence of the initial grounding.

Even a two‑level task forces the model to retain the initial visual evidence after moving, revealing whether the spatial state is truly persistent.

**Figure 1.** URBANGROUND is a real-scale urban sandbox built from territory-wide 3D geospatial data. It supports direct first-person play and programmatic control by MLLM agents. We release the sandbox on the web and as native builds for macOS, Windows, and Linux. It also includes diverse tasks for studying how multimodal agents perceive and act in a real city.

**Figure 3.** The spatial agency evaluation ladder increases the state that must remain usable across action. Level 1 supports RQ1. Levels 2–4 support RQ2. Level 5 and matched visual interventions support RQ3.

Using geo‑registered data ensures that observations and trajectories can be precisely aligned with the real‑world map, making the evaluation faithful to actual city geometry.

Experimental Setup

Describes models, agent protocol, tasks, and metrics used for evaluation.

We evaluate a suite of contemporary MLLMs across a diverse set of urban navigation tasks, using a fixed interaction protocol and standardized metrics.

PNA measures how often the agent’s movement stays on the mapped pedestrian network, reflecting realistic navigation compliance.

Total action time = 10 seconds.

Time on network = 6 seconds.

PNA = 6 / 10 = 0.6 (60 %).

Even with a correct final answer, a low PNA reveals that the agent cut corners off the pedestrian network.

How does PNA differ from raw answer accuracy?

Accuracy only checks whether the final answer matches the ground truth, while PNA evaluates the quality of the entire navigation path by measuring the proportion of time the agent stays on the pedestrian network.

Receive the current first‑person RGB observation, the task instruction, and the text interaction history.

Select one structured action from the predefined physical or map action space.

Execute the action, update the agent’s state, and record the post‑action position.

Repeat until the agent submits an answer, signals completion, or reaches the 100‑step limit.

**Figure 4.** Spatial distribution of experimental tasks across Hong Kong.

Local Spatial Grounding Results

MLLM agents excel at visual recognition but falter on directional grounding and road adherence.

We evaluate three atomic spatial tasks—visual recognition, orientation, and active exploration—using the metrics reported in Table 1.

Agents achieve high accuracy on visual recognition when the instruction and evidence are explicit, but accuracy drops modestly for active exploration and collapses to near‑random for directional grounding.

Newer generations (e.g., GPT‑5.5) improve orientation and exploration performance relative to earlier models, yet all models still exhibit low pedestrian‑network adherence, often shortcutting to faster routes at the expense of road compliance.

**Figure 6.** Example of active exploration by GPT-5.5. The agent is asked to answer which bank is next to Beijing Tong Ren Tang.

Navigation and Goal Maintenance

Evaluating whether models can keep a spatial goal alive over longer routes.

GPT‑5.5 attains the highest short‑range navigation success (75 %) but all models drop to near‑zero on long‑range navigation.

Table 2 shows GPT‑5.5 at 75.0 % for SN and 0.0 % for LN, while other models report ≤2.5 % for LN.

Success is concentrated in the shortest‑distance bins and steadily declines as the interaction horizon grows, a pattern visualised in Figure 7.

**Figure 7.** Navigation success as a function of interaction horizon. The panels report success across four equal-count bins for (a) ShortNav and (b) InstructionNav.

Even when agents do not finish the long‑range task, more than half of the episodes still end closer to the goal than they started, and multi‑stop planners reach roughly a tenth to a fifth of their required waypoints.

**Figure 8.** Long-range navigation progress and multi-stop completion. The left panel reports the proportion of LongNav episodes that end closer to the goal than at initialization. The right panel reports the mean proportion of required destinations reached in multi-stop planning. Error bars show 95% confidence intervals.

Navigation magnifies model differences that appear modest in pure QA. For example, GPT‑5.5 and GPT‑5.4 diverge sharply on short routes, and these gaps widen as the episode length grows.

**Figure 9.** Example of GPT-5.5 traversing a complex interchange using the official footbridge and continuing steadily toward the goal.

Dynamic Environment Robustness

Dynamic‑environment performance of MLLMs across weather, time, and road‑closure scenarios.

We now test whether the capabilities seen in the default city survive under dynamic weather, time, and road‑closure changes.

GPT‑5.5 achieves the highest Cloudy‑condition QA accuracy at 79.1 %, showing strong robustness to illumination changes.

Table 3 reports GPT‑5.5 reaching 79.1 % QA accuracy in Cloudy conditions, outperforming all other models.

**Figure 10.** Example of GPT-5.5 crossing the road toward the goal direction before becoming blocked by a central obstacle and being unable to continue. (a) Step 2, zoom map by 0.25x (b) Step 6, sprint forward 2.0 s (c) Step 12, close map (d) Step 44, sprint forward 2.0 s (e) Step 59, close map (f) Step 73, sprint forward 2.0 s, yaw -1 (g) Step 81, sprint forward 1.5 s, yaw -1 (h) Step 100, close map

**Table 1.** Performance comparison of various models across different environmental conditions (Clear, Dusk, Night, Cloudy, Rain) for Local QA Accuracy and Short Navigation Success.

**Table 4.** Dynamic-environment results, where SR is the goal-reaching rate, PNA is the fraction of action time spent on the pedestrian network, SPR is the safe progress rate, which represents the fraction of road-closure episodes that respect the closure, and PCR is the pedestrian-collision rate.

These results reveal a consistent pattern: models reliably produce locally plausible motions, yet they struggle to revise plans when the environment changes.

Failure Mode Analysis

We revisit the spatial‑state gap and examine how GPT‑5.5’s trajectories fail.

The central premise—that MLLMs must keep a usable spatial state after moving—remains the benchmark for evaluating urban agency. This section probes where GPT‑5.5 falls short of that goal.

Endpoint regression measures how far a model’s final reported position deviates from the best reachable checkpoint it could have claimed, given the same trajectory.

**Figure 11.** GPT-5.5 LongNav runs grouped by their run-ending trajectory pattern. Substantial progress denotes a reduction of at least 20% in horizontal goal distance. The final recorded checkpoint defines the endpoint of each run.

Oracle rescoring dramatically reduces endpoint regression for GPT‑5.5.

Observed closeness to start improves from 57.5 % to 98.7 % when the oracle selects the nearest checkpoint.

The failure‑type breakdown shows that 31.25 % of runs lose substantial progress after initially approaching the goal, while 27.5 % never make such progress at all, underscoring the need for robust global orientation.

Experimental Task Design Details

We detail the five‑level task ladder and how each task is constructed and verified.

Each task instance is manually built in URBANGROUND, then verified by two independent annotators who confirm that the task can be completed from the given start state using the prescribed tools and constraints.

LEVEL 1 probes local spatial understanding: Visual Recognition asks the agent to label visible urban content; Orientation Understanding requires extracting a consistent heading from the scene; Active Exploration Questions demand the agent to move locally to gather missing evidence within a 60‑second window.

LEVEL 2 tests navigation with explicit goals: Short‑Range Goal Navigation moves to a visible target without a map; Long‑Range Goal Navigation reaches an unseen destination using map‑based global routing; Instructional Navigation follows a sequential verbal route; Constrained Navigation adds explicit pedestrian‑area restrictions to the path planning.

LEVEL 3 removes a concrete target: Place‑Type Search gives a category (e.g., “café”) and requires the agent to locate any instance; Implicit Intent Inference provides an outcome description, forcing the agent to infer the specific urban destination before navigating.

LEVEL 4 introduces multi‑task planning: Time‑Window Scheduling supplies several destinations with temporal windows, demanding a feasible schedule; Multi‑Stop Route Planning gives an unordered set of locations, requiring the agent to compute an efficient visitation order.

LEVEL 5 evaluates robustness to environmental change: Dynamic Road‑Closure Replanning notifies the agent of a blocked pedestrian segment mid‑journey, forcing route revision; Navigation Among Pedestrians injects moving avatars, requiring the agent to adapt its motion while preserving progress.

Experimental Prompts

All prompts and interaction protocols used for evaluating agents are detailed here.

This section enumerates every prompt and protocol that the evaluation harness supplies to the models.

Action‑space schemas for both question‑answering and navigation tasks.

Action space (choose exactly one action per exploration turn). Every action uses one flat JSON object. Its required `action` field is one of the literal action names listed below; include only the parameters defined for that action. The descriptions below are schemas, not example actions. First-person mode actions: - move: `action` = "move"; `dir` is one of "forward", "backward", "left", or "right"; `seconds` is a number in [0.05, 2.0]. Optional: `yaw_rate` and `pitch_rate` are numbers in [-180, 180] degrees per second; `jump` is a boolean; `jump_at` is a number in [0, seconds]. - sprint: same fields and ranges as move, with `action` = "sprint". - look: `action` = "look"; `yaw` is a number in [-180, 180] degrees, where positive turns right and negative turns left; `pitch` is a number in [-90, 90] degrees, where positive looks up and negative looks down. - jump: only the `action` field with value "jump". - `open_map`: only the `action` field with value "`open_map`". Map mode actions: - `map_select`: `action` = "`map_select`"; `x` and `y` are normalized screen coordinates in [0, 1], with x increasing left-to-right and y increasing top-to-bottom. - `map_pan`: `action` = "`map_pan`"; `east` and `north` are distances in meters, each in [-2000, 2000]. - `map_zoom`: `action` = "`map_zoom`"; `factor` is a number in [0.25, 4.0]; values below 1 move closer and values above 1 move farther. - `map_orbit`: `action` = "`map_orbit`"; `yaw` is a number in [-180, 180] degrees and `pitch` is a number in [-90, 90] degrees. - `close_map`: only the `action` field with value "`close_map`". Available in both first-person and map mode: - terminate: only the `action` field with value "terminate". Choose it once you have enough visual evidence to answer confidently; exploration ends immediately and you will then be asked for the final multiple-choice answer. Use first-person actions only while viewing the first-person scene and map actions only while the map is visible. Never request move or sprint for longer than 2 seconds. If a larger duration is supplied, the environment clamps it and executes only 2 seconds. Do not infer or emit actions outside this complete action space.

B.1 SHARED AGENT INTERACTION PROTOCOL – for every episode the system message concatenates a task‑specific system prompt with the action‑space prompt and a ReAct output protocol.

Question‑answering ReAct protocol prompt – the agent must loop through Observation, Reason, Action, returning a JSON object with fields ‘observation’, ‘reason’, and ‘action’.

Navigation ReAct protocol prompt – identical structure to the question‑answering loop, but the reasoning focuses on reaching a physical destination.

B.2 RQ1 / LEVEL 1: LOCAL ENVIRONMENT UNDERSTANDING – three task families (VR, OU, AEQ) share a multiple‑choice format with a question and four options.

Visual Recognition (VR) – agents must identify landmarks from a photorealistic Hong Kong scene, gathering visual evidence before answering.

Orientation Understanding (OU) – agents preserve the initial camera orientation as a reference frame and determine the target’s relative direction.

Active Exploration Questions (AEQ) – agents actively explore the local area, using controlled turns and short moves to locate evidence for the answer.

B.3 RQ2 / LEVEL 2: NAVIGATION UNDER EXPLICIT INSTRUCTIONS – four navigation task types (SGN, LGN, IN, CN) each provide a textual goal and require physical arrival.

Short‑range Goal Navigation (SGN) – the destination lies within visual range; agents may use the map but often rely on direct observation.

Long‑range Goal Navigation (LGN) – the goal is far; the map shows start and target but does not compute a route.

Instructional Navigation (IN) – agents follow a sequence of natural‑language turn‑by‑turn directions.

Constrained Navigation (CN) – certain road segments are permanently closed; agents must open the map first to avoid crossing them.

B.4 RQ2 / LEVEL 3: EXPLORATION UNDER IMPLICIT INSTRUCTIONS – two tasks (PTS, III) require agents to infer a place type and locate it.

Place‑type Search (PTS) – the agent must travel to a nearby location of a requested type (e.g., park, toilet).

Implicit Intent Inference (III) – the agent infers the underlying POI category from an everyday goal and then finds it.

B.5 RQ3 / LEVEL 5: DYNAMIC ENVIRONMENT INTERACTION – two tasks (DCR, NP) test replanning when closures appear.

Dynamic Road‑closure Replanning (DCR) – after a system notice a new road closure appears; agents must reopen the map and reroute.

Navigation among Pedestrians (NP) – identical to long‑range navigation but with dense pedestrian traffic.

Comprehensive Results Tables

Model performance on answer accuracy and pedestrian adherence across all metrics.

This section reports the full quantitative outcomes for each evaluated model, organized by the three research questions.

**Table 7.** Matched LongNav endpoint and retrospective-checkpoint results. The oracle row rescores GPT-5.5 at its closest initial or post-movement checkpoint.

**Table 6.** Complete pedestrian-network-adherence results. Values are percentages of executed action duration.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers