GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?

Qifeng Zhang, Kaixiang Huang, Heng Dong, Huang Fang, Junting Chen, Junjie Zhu, Yonghang Chen, Zhiyu Zhang, Wei Li

GST-Bench exposes that current VLMs fail to maintain global spatial awareness across long-horizon video streams.

Can current Vision-Language Models (VLMs) maintain a globally consistent spatial representation over long-horizon egocentric video, or are they limited to local, frame-level perception?

Vision-Language Models (VLMs) struggle to build a coherent mental map of an environment, often failing to track objects or their own position once they move out of the immediate field of view. The authors introduce GST-Bench, a benchmark requiring models to integrate long-horizon egocentric video, reason from novel viewpoints, and align observations with top-down maps. The core move is a simulation-based pipeline that forces cross-frame reasoning by ensuring target objects are invisible from the query viewpoint. Even the strongest proprietary models achieve only 42.68 on the benchmark, trailing the human baseline of 79.08 by a wide margin.

Paper Primer

Current spatial benchmarks rely on single images or short clips, which allow models to solve tasks through local perception rather than true global scene understanding. GST-Bench forces global reasoning by requiring models to map egocentric video to top-down representations and answer queries about objects that are never visible in the current frame.

The method uses a simulation-based pipeline to generate diverse indoor scenes, recording precise 3D coordinates and camera trajectories. By sampling query viewpoints independent of the exploration video and strictly filtering out samples where the target is visible, the benchmark ensures that models must consolidate long-horizon observations to succeed.

Current VLMs exhibit a severe global spatial reasoning gap compared to humans.

Evaluation of 22 state-of-the-art models against human performance on 12 subtasks. The best zero-shot model (Gemini-3-Pro) scores 42.68, while humans score 79.08.

Proprietary and open-source models fail for different reasons.

Controlled local variants show proprietary models succeed at local perception but fail at cross-frame integration, whereas open-source models struggle with both. Proprietary models gain up to 39 points when the task is reduced to local perception, while open-source models show inconsistent or negligible improvement.

Why do embodied-tuned models not outperform general-purpose models on this benchmark?

The authors conjecture that current embodied training recipes prioritize local affordances and short-term spatial relations rather than the long-horizon memory and cross-viewpoint alignment required for global scene consistency.

Can this gap be closed with targeted training?

Yes; fine-tuning Qwen3-VL-8B on the authors' GST-Train dataset improved its score from 25.89 to 53.52, surpassing all zero-shot models evaluated.

Introduction: The Global Spatial Gap

We expose why current VLMs miss global spatial awareness and introduce a benchmark to measure it.

Vision‑Language Models excel at single‑frame perception but stumble when they must stitch together a continuous egocentric video into a unified world map. This inability blocks embodied agents from answering “where am I?” or “where is the target?” after exploring a scene.

It is the ability to maintain a coherent top‑down representation of an environment while observing it from a moving first‑person viewpoint.

To quantify this gap we built GST‑Bench, a VQA suite that forces models to answer questions only solvable by global reasoning. Each task hides the target from the query view and asks the model to map its egocentric video onto a top‑down layout.

**Figure 1** Overview of GST-Bench. We evaluate global spatial awareness along three core competencies — self localization (Where am I?), object localization (Where is the target?), and scene structure understanding (What does the scene look like?). Given an egocentric exploration video and a novel query view, models must localize themselves, reason about targets unseen in the current view and map observations onto a global top-down representation. As shown on the right, even the strongest VLM (Gemini-3-Pro, 42.68) falls far below the human baseline (79.08).

The shift from local frame‑level perception to long‑horizon global spatial awareness is the missing piece for embodied intelligence.

GST-Bench Architecture

We detail the GST‑Bench benchmark, its inputs, core competencies, and twelve evaluation tasks.

GST‑Bench is a suite of twelve tasks that probe whether a Vision‑Language Model can fuse long‑horizon egocentric video into a globally consistent spatial map.

The benchmark stitches together video, top‑down maps, and novel query views so a model must remember and reason about the whole environment, not just single frames.

These three abilities are what an embodied agent must master to navigate and manipulate a real environment.

The video is processed frame‑by‑frame, accumulating a pose‑indexed feature map.

The occupancy map is rasterized to the same resolution as the feature map.

The current view is encoded, and the model searches for the rotation that maximizes cross‑correlation between the view’s feature map and the accumulated map.

The highest‑scoring rotation is 42° clockwise from the map’s north axis.

This concrete pipeline shows that the model must retain a global pose‑indexed memory and perform a global alignment at inference time, rather than relying on local cues.

**Figure 2.** Representative GST-Bench samples from each of the twelve task types, organized by the three core competencies: self localization, object localization, and scene structure understanding.

Automated Data Generation

Automatic simulation creates diverse egocentric videos, annotated views, top‑down maps, and QA pairs for GST‑Bench.

The pipeline automatically turns raw simulated scenes into a rich collection of egocentric videos, annotated views, top‑down maps, and templated QA pairs, guaranteeing that models must build a global spatial representation.

Prepare scenes from diverse simulation assets (BEHAVIOR‑1K, HyperSim, ArtVIP, etc.) and hold out evaluation scenes.

Sample spatially distributed viewpoints, connect them into efficient traversal trajectories, and render egocentric RGB exploration videos.

Render instance and semantic segmentation, overlay red bounding boxes on target objects across all frames to create object‑annotated videos.

Collect independent short trajectories, segment them into brief clips for egocentric‑to‑global alignment tasks.

Render three top‑down images per scene: photo‑realistic, occupancy map, and floor‑plan.

Sample off‑trajectory query viewpoints with independent camera orientations; retain only those with sufficient overlap and with the target object absent.

Combine inputs with task‑specific templates to produce QA pairs; compute answers directly from scene geometry.

Apply automatic filters (invalid projections, ambiguous visibility, low overlap, degenerate choices) and then have annotators verify answerability, discarding failing samples.

Sample a camera pose outside the exploration trajectory (e.g., position (2.1 m, 1.4 m), yaw = 45°) and render the RGB image.

Project the chair’s 3D bounding box into the camera view; visibility test reports “not visible”.

Compute visual overlap between the query view and the exploration video frames; overlap = 0.62 (> 0.5 threshold), so the view is retained.

This step forces the model to rely on memory of the chair’s location from the video rather than direct visual evidence in the query view.

**Figure 3.** Overview of the automatic data generation pipeline. Starting from diverse simulation scenes, the pipeline produces exploration videos, object-annotated videos, novel-viewpoint images, top-down images at three abstraction levels, and template-based QA pairs, followed by automated filtering and human verification.

Evaluation Methodology

We detail the models, zero‑shot protocol, and metrics used to assess global spatial awareness.

To quantify Global Spatial Awareness we evaluate a broad suite of Vision‑Language Models under a uniform zero‑shot protocol and report performance across four GST‑Bench task families.

Select a target model from the 22‑model baseline.

Construct the model‑specific zero‑shot prompt as prescribed by the authors.

Run inference with greedy decoding (no sampling, no temperature).

Collect raw predictions for each GST‑Bench sub‑task.

Compute the task‑specific metric (MRA, angular accuracy, point accuracy, or standard accuracy).

Aggregate the 12 sub‑task scores by arithmetic mean to obtain the final GST‑Bench score.

Compute relative error: $|\hat y - y|/y = |9.2 - 10.0|/10.0 = 0.08$.

Check each threshold: for $\theta=0.50$, condition $0.08 < 1-0.50 = 0.50$ holds (1); for $\theta=0.75$, $0.08 < 0.25$ holds (1); for $\theta=0.90$, $0.08 < 0.10$ holds (1).

Sum the indicator values: $1+1+1 = 3$.

Average over $|C_d|=3$: $\text{MRA}=3/3 = 1.0$ (100 % relative‑accuracy for this example).

This toy calculation shows that a small relative error easily satisfies even the strictest confidence thresholds, illustrating why MRA emphasizes proportional error rather than absolute distance.

Headline Results

GST‑Bench results show VLMs lag far behind humans, with modest gains for proprietary models.

Proprietary VLMs achieve the highest scores on GST‑Bench but remain far below human performance.

Gemini‑3‑Pro scores 42.68 while humans average 79.08 (Table 1).

The evaluation protocol and dataset remain identical across all models, so differences reflect model capabilities rather than experimental variance.

Open‑source VLMs perform near random guessing on most GST‑Bench tasks.

Most open‑source scores lie within three points of the random baseline; four models even fall below it (Table 1).

**Table 1.** Performance of various models on GST-Bench across three core competencies: Object Localization, Self Localization, and Scene Structure Understanding. The Avg. is the arithmetic mean of all 12 subtask scores. The highest, second-highest, and third-highest scores in each column are marked in light red, light orange, and light yellow, respectively, excluding Qwen3-VL-8B (fine-tuned), which is fine-tuned on our proposed training set GST-Train. ED = Ego. Direction, EDist = Ego. Distance, GP = Global Position, Pos = Top-Down Position, Ori = Top-Down Orientation, TDS = Top-Down Selection, Traj = Trajectory Selection. $v / s$ = visual / semantic modality.

Detailed Performance Metrics

We report quantitative gains of our fine‑tuned model over baselines and human performance on GST‑Bench.

Our fine‑tuned model reaches 53.7 % mean accuracy, a 33.5‑point improvement over the baseline.

Baseline mean accuracy is 20.2 %; our model achieves 53.7 % on the same GST‑Bench test set.

All reported numbers share the same training corpus, evaluation protocol, and 20‑sample‑per‑task setting, so differences reflect model architecture and fine‑tuning rather than data or metric variations.

Prior Spatial Benchmarks

We survey prior spatial reasoning benchmarks and highlight gaps that GST‑Bench addresses.

Early work treats spatial understanding as categorical judgments from a single view, e.g., verifying preposition statements. CV‑Bench expands this to relation and counting queries for 2D, while 3DSRBench and Spatial457 add 3D relations, height, orientation, and collision prediction.

To overcome single‑view ambiguity, recent systems accept multiple images. MM‑Spatial and SPAR support single‑ and multi‑view inputs for spatial relationship prediction and metric estimation, BLINK adds classic vision tasks like relative depth, and ViewSpatial‑Bench and MMSI‑Bench require integration across camera‑centric and human‑centric frames.

Video‑based suites push reasoning to room‑scale by aggregating observations over time. VSI‑Bench and its long‑horizon variant VSI‑Super evaluate configurational reasoning and continual counting, while MMSI‑Video‑Bench, OpenEQA, OST‑Bench and STI‑Bench probe episodic memory, planning, and precise spatio‑temporal quantities.

Despite these advances, three gaps remain: (1) many tasks can be solved from a single informative frame, obscuring cross‑frame reasoning; (2) directional reasoning is often coarse categorical rather than precise angular measurement; (3) benchmarks rarely test alignment between egocentric video and an explicit top‑down global representation.

GST‑Bench addresses all three gaps by constructing tasks that require global cross‑frame integration, scoring answers with exact numerical quantities, and providing top‑down images to evaluate the correspondence with egocentric observations.

Egocentric Direction Performance

Key performance shifts on the Egocentric Direction benchmark.

Open‑source Vision‑Language Models close the L‑Video gap to proprietary systems, with InternVL3.5‑8B reaching 33.10 % versus the proprietary peak of 35.40 %.

Table 2 shows InternVL3.5‑8B at 33.10 % and GPT‑5 at 35.40 % on the L‑Video metric.

Egocentric Distance Performance

Proprietary models dominate egocentric distance performance, yet all remain far below human level.

Proprietary VLMs consistently outperform open‑source and embodied‑tuned models on egocentric distance tasks.

Table 3 shows the highest values in every column belong to proprietary systems such as Gemini‑2.5‑Pro and GPT‑5.

**Table (b) Egocentric Distance**

Global vs. Local Reasoning

We test whether failures stem from missing global integration or from basic spatial perception.

The central premise is that VLMs excel at local perception but struggle to fuse information across long‑horizon egocentric video. This section isolates whether the bottleneck lies in global integration or in basic spatial perception.

Three controlled variants systematically remove the need for cross‑frame reasoning while keeping the spatial question identical.

How does the Local‑Video setting differ from the Global setting?

In Local‑Video the current view already contains the target, so the exploration video no longer provides essential information; the model can answer using only the visible frame, whereas Global forces the model to combine video frames because the target is hidden.

Proprietary models dramatically improve when cross‑frame reasoning is removed.

Table 2 shows Gemini‑3‑Pro’s score jumps from 22.11 (Global) to 61.20 (Local‑Image), a gain of +39.09 points; all three top proprietary models gain between +34 and +43 points on Egocentric Distance.

Open‑source models exhibit mixed or limited gains in the local settings.

Table 2 reports InternVL3.5‑2B improves by +7.29 points on EDs Local‑Image, while InternVL3.5‑8B degrades by ‑8.03 points; even the best open‑source score (25.40) remains far below proprietary scores (66.49).

**Figure 4.** Illustration of the three controlled settings used to disentangle global reasoning from local perception. Global (target absent from the current view, requires cross-frame integration), Local-Video (current view contains the target, exploration video redundant), and Local-Image (exploration video removed, reduced to single-image spatial understanding).

Future Directions and Limitations

Targeted training narrows but does not close the global spatial gap.

Current VLMs excel at local perception but struggle to stitch together information across long‑horizon egocentric video, leaving a global spatial awareness gap. GST‑Bench quantifies this gap by testing models on cross‑view alignment and long‑range memory.

Proprietary models generally succeed locally yet fail to integrate globally, whereas open‑source models falter at both stages, with local perception being the more acute limitation. The tables show proprietary scores climbing modestly (e.g., Gemini‑3‑Pro reaching 65.61) while open‑source scores remain low (e.g., Qwen3‑VL‑8B at 25.89).

Fine‑tuning Qwen3‑VL‑8B on GST‑Train, while mixing in general multimodal instruction data, lifts its average GST‑Bench score from 25.89 to 53.52. This surpasses all zero‑shot models evaluated, including the strongest proprietary systems, demonstrating that explicit supervision for global spatial reasoning can dramatically improve long‑horizon spatial memory. Nevertheless, the fine‑tuned model still trails human performance, indicating that targeted supervision narrows but does not eliminate the spatial reasoning challenge.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers