Show, Don’t Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text

Xu Wang, Kaixiang Yao, Miao Pan, Xiaohe Zhou, Xuanyu Liu, Wenqi Zhang, Xuhong Zhang

ProVisE enables metric-compatible spatial evaluation by parsing pixel-space outputs from generative models.

Can we improve spatial reasoning in AI by forcing models to "show" their work as generated images rather than forcing them to serialize spatial concepts into text?

Existing spatial benchmarks force models to output coordinates or text, creating an interface mismatch for image-generation models that naturally express spatial judgments in pixels. ProVisE (Protocolized Visual Evaluation) constrains generative models to produce specific visual formats—like masks or trajectories—and parses them into structured predictions compatible with original benchmark metrics. Experiments show that image-generation models excel when spatial answers can be externalized as pixel-level evidence, while text-output models remain superior at compositional reasoning and state transformation.

Paper Primer

The core challenge is that current spatial benchmarks are designed for text-output models, requiring symbolic answers like bounding boxes or labels. This forces image-generation models to either use an unreliable auxiliary VLM judge or fail to participate in standard evaluations, obscuring their true spatial cognition.

ProVisE solves this by replacing the response interface with a protocol-guided generation task: the model renders a visual answer (e.g., a depth map or point), and a deterministic parser converts that image into the benchmark's required format. An Agentic builder automates this by constructing task-specific contracts, ensuring the evaluation remains metric-compatible without requiring model-specific tuning.

Visual and textual interfaces provide complementary spatial capabilities.

Paired sample-level analysis shows that visual answering solves 30–37% of items missed by text-output models, indicating that visual successes are not merely a subset of textual ones. Visual answering is most competitive on perception tasks like depth estimation and relationship verification, while text models lead by 17.63 points on reasoning-heavy tasks.

Most visual evaluation failures are incorrect spatial predictions rather than communication breakdowns.

Diagnostic attribution shows that 88% of non-correct outcomes yield valid, parseable predictions that simply fail the task's correctness criteria. Only 8.46% of failures are due to protocol noncompliance and 3.45% to parser failure.

Why not just use a VLM to "judge" the generated images?

Judge-based scores are unreliable because they reflect the judge model's own uncertainty rather than the generated answer, and they are difficult to align with the original, objective metrics used for text-output models.

Does the choice of parser change the evaluation results?

Yes; substituting the task-specific parser with a universal VLM parser changes both model scores and rankings, indicating that black-box parsing adds a model-dependent interpretation layer rather than providing a neutral evaluation.

Spatial evaluation should move toward hybrid interfaces: using text for constraint-heavy reasoning and visual generation for externalizing geometry, with deterministic operators to verify the results.

The Spatial Serialization Bottleneck

Spatial reasoning is hampered by forcing visual judgments into textual or coordinate formats.

Spatial intelligence is essential for agents to act in the physical world, yet current benchmarks force models to express continuous spatial judgments as coordinates or text. This answer‑interface mismatch hinders evaluation of image‑generation models, which naturally produce visual answers in pixel space. Consequently, comparing image‑generation models with text‑output VLMs under shared task semantics has been impossible.

**Figure 1.** The answer-interface mismatch in spatial evaluation. A spatial judgment can be expressed directly as a graspable region, whereas existing benchmarks often require an exact coordinate list. This burdens text interfaces with precise serialization and leaves a generated visual answer incompatible with the original evaluator.

The requirement to serialize continuous spatial judgments into text or coordinates obscures whether a model truly understands the space.

Spatial reasoning is currently limited by the requirement to output text.

The ProVisE Pipeline

ProVisE swaps free‑form image output for a fixed visual protocol while keeping the original benchmark metric.

The translation bottleneck forces image‑generation models to emit unrestricted pictures, making it impossible to compare their spatial reasoning directly to text‑based benchmarks.

ProVisE replaces the free‑form image output with a protocol‑driven visual answer, then parses that image back into the exact answer format the benchmark expects, so the original scoring rule stays unchanged.

How does ProVisE differ from simply asking a model to describe the scene in text?

Text answers still require a language model to map pixels to words, which re‑introduces the translation bottleneck. ProVisE forces the model to embed the answer directly in the image layout, so the parser can read it without any linguistic mediation.

A visual protocol is a paired guidance prompt and deterministic parser that together define exactly how an answer must appear in pixel space and how to extract it.

The prompt says “Place a blue square in the cell that matches the direction ‘front‑right’.”

The model outputs an image with the blue square in the top‑right cell.

The parser scans the grid, finds the blue cell at coordinates (row = 0, col = 2), and maps it to the label “front‑right”.

The label is compared against the benchmark’s ground‑truth using the original accuracy metric.

The protocol turns a categorical answer into a spatial layout, making the parsing step a simple geometric lookup rather than a language understanding problem.

An autonomous agent builds a protocol for any previously unsupported benchmark task by normalizing the task into a contract and then picking a construction route (Reuse, Build, or Fallback).

Why doesn’t the agent generate model‑specific code instead of a declarative protocol?

The agent’s output is a static prompt + parser description, not executable code. This design prevents any hidden model‑dependent tricks and ensures every model is evaluated under exactly the same visual constraints.

Select the pre‑computed protocol for the task (routing or agentic construction).

Feed the guidance prompt to the image‑generation model.

Model emits a constrained visual response (e.g., a marked grid or mask).

Run the deterministic parser on the image to obtain a structured prediction.

Map the prediction to the benchmark’s answer space via the metric mapping.

Score the prediction with the original task metric.

**Figure 2.** Overview of ProVisE. Text-output VLMs follow 2a and are scored directly by the original evaluator. Image-generation models follow 2b, where a routed protocol guides generation and parsing into structured answers. Both are scored by the same source-task metric under shared semantics.

**Figure 3** Agentic protocol construction in ProVisE. A benchmark is normalized into task contracts; the Agent constructs a protocol for each task; and the resulting generation–parser pair is automatically validated before being applied unchanged to every target model.

**Figure 4.** Representative protocol-execution cases for ProVisE. Each column shows a benchmark input (top), the protocolized output (bottom), and the recovered structured prediction.

SpatialGen-Bench Design

SpatialGen-Bench provides a hierarchical, multimodal suite for evaluating spatial cognition.

Existing spatial benchmarks force models to translate continuous spatial relationships into text, creating a bottleneck that masks true spatial understanding. SpatialGen‑Bench eliminates this by preserving each task’s native answer format while organizing them hierarchically.

A hierarchical suite that keeps every task’s original answer contract (count, mask, trajectory, etc.) and groups tasks into four capability levels so evaluation can pinpoint where a model’s spatial cognition breaks down.

Assign each task to its corresponding level in the hierarchy.

Record the answer type for each: count → integer, viewpoint → categorical, reasoning → multi‑choice, navigation → coordinate pair.

Sum the samples: 1 + 1 + 1 + 1 = 4 total examples.

Observe that the answer contracts differ, yet all are evaluated under the same shared metrics pipeline.

The hierarchy lets a researcher see exactly which capability (e.g., reasoning) a model fails on, without conflating errors across heterogeneous answer formats.

**Figure 5.** Overview of SpatialGen-Bench. Tasks are organized into perception, understanding, reasoning, and interaction.

How does SpatialGen‑Bench differ from prior benchmarks like BLINK or ViewSpatial‑Bench?

Those older benchmarks expose only a single answer contract (typically multiple‑choice) and lack a hierarchical organization; SpatialGen‑Bench preserves each task’s native contract—counts, masks, trajectories, etc.—and groups them into four capability levels, enabling fine‑grained diagnosis of spatial cognition.

SpatialGen‑Bench covers perception, reasoning, and interaction tasks.

Experimental Results

Key results show text models beat visual ones overall, yet visual answers rescue many text failures.

We evaluate 31 model‑interface systems (20 text, 11 visual) on SpatialGen‑Bench, treating each interface as a separate system. All visual protocols—prompt, parser, invalid‑output rules, and metric mapping—are frozen and shared across models. Missing or unparsable outputs receive a zero score.

Text‑answering models achieve higher overall scores than visual‑answering models on SpatialGen‑Bench.

GPT‑5.4 reaches 61.04 % overall (top text system) while GPT Image 2 reaches 54.49 % overall (top visual system) (Table 2).

Both modalities remain far below human performance.

Human overall is 87.79 %, exceeding the best machine by 26.75 points (Table 2).

**Table 2.** Main results on SpatialGen-Bench (%). Capability scores macro-average their constituent tasks, and Overall macro-averages task scores. Bold marks the best model within each answer interface.

**Figure 6. Where does showing help?** Bars report visual-minus-text score differences under the same source-task metrics; positive values favor visual answering. (a) compares the text- and visual-model means. (b) provides a partial same-family comparison between the two SenseNova-Vision-7B-MoT interfaces on identical items. Both comparisons are descriptive because their model pools or decoding paths differ.

**Table 3.** Paired sample-level outcomes on the shared benchmark items. Entries report counts and benchmark shares in parentheses. Visual rescue is Visual only/(Visual only + Neither): the fraction of text-incorrect items solved by the paired visual interface.

**The Agentic builder transfers beyond SpatialGen-Bench.** The constructed protocols support end-to-end, metric-compatible GPT Image 2 evaluation on all six benchmarks and matched comparison with the text baselines. The results broadly echo the main benchmark: text models lead four benchmark columns, whereas GPT Image 2 leads SAT and RoboAfford, so neither answer interface is uniformly superior. This pattern is supporting context; the primary result is successful adaptation across heterogeneous task contracts. Because each column is a task-balanced pilot, the values are not full-benchmark leaderboard estimates.

**Table.** Parser substitution changes both scores and rankings. ProVisE and Qwen3-VL-8B rank GPT Image 2 first, whereas Qwen2.5-VL-72B ranks JoyAI-Image first and Llama 4 Scout ranks Seedream 4.5 first. Relative to ProVisE, the three universal parsers yield rank correlations of 0.60, 0.31, and 0.60 and mean absolute score changes of 5.87, 5.63, and 5.26 points. Their conditional valid-parse rates remain high (99.54%, 98.49%, and 91.96%), indicating that the shifts arise from semantic interpretation as well as response-format failures. Because parser substitution raises some model scores while lowering others, it is not a uniform calibration offset. The experiment therefore shows that a universal black-box parser adds a model-dependent interpretation layer; it does not rank the universal VLMs or image-generation systems themselves.

**Table 18.** Bootstrap 95% confidence intervals for all model-interface systems. Entries are point estimates [2.5th, 97.5th percentile].

Parser Sensitivity Analysis

Evaluating how depth‑parser design choices affect parsing success and accuracy.

This ablation isolates two design knobs of the Depth parser: the route used to extract the A/B depth evidence, and the spatial kernel that aggregates local grayscale values.

**Table 13.** Task-level universal VLM parser sensitivity. Overall macro-averages task scores. Parse is the fraction of all expected records normalized into the required answer structure; unavailable images and invalid responses remain explicit zero-scored records.

Benchmark‑coordinate grayscale sampling raises the parse rate by over twenty points compared with the prior mixed route.

Parse %: 99.39 (grayscale) vs 78.48 (mixed), a +20.91 % increase.

Using the grayscale‑sampling route also improves downstream accuracy by more than thirteen points.

Accuracy %: 68.79 (grayscale) vs 55.45 (mixed), a +13.34 % gain.

**Table 12.** Depth sampling-kernel sensitivity. Results use the same 330 model-record pairs; 5 × 5 is the protocol default.

Changing the depth‑sampling kernel size alters accuracy by only a marginal amount.

Accuracy ranges from 68.79 % (1 × 1) to 69.39 % (11 × 11), a span of 1.21 % points.

VLM Parser Sensitivity

Table 13 quantifies how each image model’s parser performance changes when components are removed.

Table 13 reports the full task‑level comparison for each image model under the matched availability boundary, showing how parser performance reacts to component ablations.

**Table 13.** Universal VLM Parser Sensitivity Details. Table 13 reports the full task-level comparison for each image model under the matched availability boundary.

Excluding the Depth component does not drive the overall sensitivity result.

Spearman correlations remain $0.71/0.43/0.43$ and mean absolute shifts stay around $6.40/5.59/5.74$ points when Depth is omitted.

Failure Attribution Analysis

We dissect where visual‑answering models fail, attributing errors to generation, protocol, parsing, or prediction.

The failure‑attribution analysis uses only cached outputs and applies a strict label precedence: generation failure overrides protocol noncompliance, which overrides parser failure, which finally yields a correct or incorrect prediction. When the parser runs but finds no recoverable evidence the case is marked protocol noncompliance; an exception or unsuccessful parse is a parser failure. Tables 14 and 15 break down these categories for each model and each SpatialGen‑Bench task, and Figure 7 visualizes the same information at model‑level and task‑level.

The majority of errors (57.7 %) are plain incorrect predictions, while protocol noncompliance and parser failures together account for under 8 % of cases, and generation failures are negligible. These proportions reflect the diagnostic thresholds applied (precision ≥ 0.5 for Affordance, DFD < 0.4 for Trajectory, normalized ≥ 0.999 elsewhere) and do not alter the native benchmark scores.

**Figure 7.** Failure attribution by visual-answering model and task. Bars partition all responses, with correct predictions retained in the denominator. The categories are diagnostic only and do not replace the native metrics in Table 2; task separators denote capability groups.

Parsed Output Sensitivity

Evaluating how scores shift when only parsable outputs are counted.

We recompute the visual‑answering scores after discarding any output that cannot be parsed into a metric‑consumable prediction. This isolates the effect of parse failures on each model’s ranking.

**Table 16.** Visual-answering results after filtering invalid parses. Scorable reports the percentage of outputs for which the assigned parser returns a prediction accepted by the native task metric. All-output is the official fixed-denominator score; Scorable-only excludes parse failures within each task before macro-aggregation.

SenseNova‑Vision‑7B‑MoT climbs from rank 7 to 3 when only parsable outputs are considered.

Table 16 shows the model’s rank improves by four positions after filtering.

OmniGen‑v1 improves its rank from 9 to 6 under the Scorable‑only evaluation.

Table 16 records a three‑position upward shift for this model.

JoyAI‑Image and FLUX.2 9B each drop two ranks (5→7 and 6→8) when invalid parses are excluded.

Both models’ ranks increase by +2 in Table 16 after filtering.

Qualitative Case Studies

ProVisE demonstrates strong qualitative spatial reasoning across multiple visual tasks.

ProVisE evaluates eight distinct qualitative spatial tasks, revealing high visual‑system success rates.

Pass counts span 7–10 of 11 systems per task, with the strongest performance on spatial grounding.

Pass rates range from 7 / 11 to 10 / 11 across the seven tasks, indicating consistently strong performance of visual systems on qualitative spatial reasoning.

**Figure 11** Size. Question: Is the second giraffe from the right the shortest? Options: (A) no; (B) yes. GPT-5.4: A (no). Ground truth: B (yes). GPT Image 2 outlines the queried giraffe and gives an affirmative visual mark; 9 of 11 visual systems pass.

**Figure 12** Spatial Grounding. Question: Locate the object described as a bus in front of others. Return one normalized point inside the target object. Answer space: a normalized point (no discrete options). The ground-truth target is the RefCOCOg segmentation mask of the foreground bus. GPT-5.4 predicts (0.677,0.634), shown in magenta on a copy of the source image; the point falls just outside the target mask. The parser reads (0.603, 0.768) from the GPT Image 2 response, placing the cyan marker inside the mask; 10 of 11 visual systems pass.

**Figure 13** Geometric Feasibility. Question: Is there enough space for another person to sit in the ski lift? Options: (A) no; (B) yes. RoboBrain2.5-8B-NV: A (no). Ground truth: B (yes). GPT-5 Image Mini places a second passenger on the same bench with visible remaining clearance; 8 of 11 visual systems pass.

**Figure 14.** Orientation. Question: From the camera’s perspective, which direction is the cat facing? Options: (A) front; (B) front-right; (C) right; (D) back-right; (E) back; (F) back-left; (G) left; (H) front-left. **GPT-5.4:** C (right). **Ground truth:** B (front-right). Nano Banana 2 fills the bottom-right grid cell, which the deterministic parser maps to front-right; 7 of 11 visual systems pass.

Conclusion and Related Work

We recap ProVisE and SpatialGen‑Bench, then argue for hybrid text‑visual spatial evaluation.

Recall the translation bottleneck: forcing spatial relationships into text hides true spatial understanding. ProVisE evaluates spatial cognition by routing model outputs through protocol‑guided visual parsers, while SpatialGen‑Bench provides a shared benchmark for both text‑ and image‑based answers.

Our experiments reveal a split of strengths: image‑generation models excel when answers map directly to pixels, preserving continuous geometry, whereas text‑answering models dominate tasks requiring constraint composition, counterfactual reasoning, or precise state updates. Most visual failures remain parseable but encode the wrong answer, and both the external pilots and the parser‑sensitivity study show that scores depend on how generated evidence is decoded.

We therefore recommend a hybrid approach: let text or latent representations handle planning and constraints, use visual generation to externalize geometry, and apply deterministic operators to verify results. Selecting the representation per task and reasoning stage enables “show” and “tell” to exchange intermediate states and cross‑check each other.

Dataset Construction Details

Describes how SpatialGen‑Bench was assembled, formatted, and quality‑checked.

SpatialGen‑Bench aggregates public spatial benchmarks across four capability levels, assigning each source task to a capability based on its objective rather than its original benchmark name. For instance, object‑facing direction is classified as perception, while viewpoint selection falls under understanding. Licenses are recorded per source, with “N/A” indicating no explicit license field was found.

**Table 7.** The table describes the fields, their data types, and their meanings for the public release data format.

All samples are stored as a single JSONL row using a unified schema that normalizes field names, media paths, and answer representations while preserving the original task semantics. This format enables consistent loading and scoring across diverse tasks. The schema is deliberately flat to remain portable across JSONL and table‑based hosting.

**Table 7.** Released SpatialGen-Bench JSONL schema. Paths are relative to the dataset root. `mask_path` and evaluator metadata are scoring targets, not model inputs.

Deterministic integrity checks verify unique IDs, valid paths, and answer‑schema conformity for every record, and a manual review confirms that instructions, visual evidence, and scoring contracts remain consistent. Only records passing both automated and human checks are released. This two‑tier validation ensures the split is reliable for evaluation.

Protocol Design Specifications

Defines reusable and Agentic parsers, their thresholds, and construction contracts.

The protocol design defines two families of parsers: reusable SpatialGen‑Bench protocols that rely on fixed HSV thresholds, and Agentic Build/Fallback parsers that may invoke auxiliary VLMs or generated‑only logic. Each parser emits a label—direction, choice, or boolean—and specifies an invalid‑output condition such as missing component or unreadable depth map.

Reusable protocols include Instance Marking (threshold green pixels $H\!=\!35\text{-}90$, $S/V\!\ge\!80$, integer count), Direction Grid (3 × 3 morphology, blue pixels $H\!=\!100\text{-}130$, $S\!\ge\!120$, $V\!\ge\!50$, at least 50 pixels), Fixed‑Slot Coding (magenta pixels $H\!=\!140\text{-}175$, $S/V\!\ge\!80$, ≥30 pixels), Binary Relation Recover (green subject + blue object overlays), Relative Depth (sample $5\times5$ local grayscale means at A/B locations, brighter = closer), Region Masking (grayscale $>127$, CLIP ViT‑B/32 similarity), and Red‑Component Path extraction (nearest novel red component, skeletonize, sample 20 points).

Each reusable parser also declares its output label type—direction, choice, or boolean—and an invalid‑output condition such as missing component, merged markers, unreadable depth map, empty mask, missing candidates, or disconnected path.

Agentic parsers are constructed on demand. A fixed auxiliary VLM reads the numbered hole inside a blue circle and counts yellow‑trajectory holes; a cyan‑pixel rule ($H\!=\!80\text{-}100$, $S/V\!\ge\!70$) selects the most compact component and normalizes its centroid; indexed endpoint parsing fails on missing or ambiguous circles; normalized‑point parsing fails on missing markers, multiple prominent marks, or diffuse components.

Additional Agentic parsers recover magenta/cyan anchors, yellow dimension lines via color differencing, and perform OCR on adjacent scalar‑unit tags; role‑colored subject/reference masks compute horizontal, vertical, dominant‑axis, or distance relations; a constrained parser maps only visible object‑comparison evidence; a generic fallback parser parses the declared answer schema directly from the generated response.

**Table 8.** Parser rules for reusable and Agentic protocols. HSV ranges use OpenCV hue values; fixed auxiliary models and generated-image-only Fallback are named explicitly.

**Build contract.** For each task, the construction Agent receives up to three answer-hidden examples together with the observed input and answer schemas, candidate choices, metric contract, and registered protocol and Parser Ops inventories. It returns a structured configuration that Python compiles into generation guidance, an executable parser, invalid-output rules, and metric mapping. Static checks reject unknown operators, type or metric mismatches, missing media, and answer leakage; ground-truth answers and correctness feedback are withheld from both the initial and revision calls.

The build contract supplies up to three answer‑hidden examples, input/answer schemas, candidate choices, metric contracts, and registered protocol/parser inventories to a construction agent, which emits a Python‑compatible configuration. Static checks reject unknown operators, type or metric mismatches, missing media, and answer leakage; ground‑truth answers are withheld from both initial and revision calls.

The implementation registers 11 reusable high‑level protocols and 41 typed parser ops covering color, geometry, OCR, and CLIP similarity; inputs that cannot be normalized or scored are marked as unsupported and produce no evaluation result.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers