RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment
Qingchuan Ma, Yuexiao Ma, Xiang Meng, Yixuan Zou, Tianyu Xie, Wang Chen, Yuhui Zeng, Yuhang Wu, Xiawu Zheng, Rongrong Ji
RoTA-Bench evaluates whether vision-language models can translate 2D visual observations into precise 3D spatial control.
Can current vision-language models (VLMs) perform precise 3D spatial reasoning by iteratively aligning 2D projections of objects?
Vision-language models often struggle to translate 2D visual cues into the precise 3D spatial reasoning required for physical interaction, such as rotating an object to match a target silhouette. The authors introduce RoTA-Bench, an interactive benchmark where models must iteratively rotate 3D objects to align their 2D projections with a target, using a discrete action space of axes and step sizes. Frontier models fail to bridge the gap, achieving only 17% success compared to 100% for human participants, and often prioritize 2D silhouette overlap at the expense of 3D pose accuracy.
Paper Primer
RoTA-Bench treats spatial reasoning as a closed-loop projection-matching task. The system constructs 3D objects by extruding 2D icons into a spherical volume, then challenges the model to recover a target silhouette by selecting from 24 discrete rotation actions—a process that forces the model to infer 3D geometry from 2D feedback rather than relying on static image recognition.
Frontier models exhibit a massive performance gap in 3D spatial control compared to humans.
In a 100-task test set, the best-performing model (GPT-5.6-Sol) achieved a 17% success rate, while human participants achieved 100% success.
The authors find that models often "cheat" the task by optimizing for 2D silhouette overlap while ignoring the underlying 3D pose. At matched states, models are nearly twice as likely as humans to improve silhouette alignment while simultaneously worsening the actual 3D pose error.
Why is this benchmark interactive rather than static?
Static benchmarks allow models to rely on pre-trained priors or token-space search; RoTA-Bench requires closed-loop feedback, forcing the model to demonstrate genuine 3D understanding by refining its actions based on successive visual observations.
What is the role of the "auxiliary view" in the task?
The auxiliary view provides depth-shaded information to compensate for the inherent ambiguity of binary silhouettes, helping the model resolve depth and occlusion cues that are otherwise missing from a flat 2D projection.
Introduction
We expose the VLM 2D‑to‑3D reasoning gap with RoTA‑Bench.
Vision‑Language Models (VLMs) struggle to translate 2D observations into 3D understanding; RoTA‑Bench forces models to iteratively align 2D projections of 3D objects to a target pose, exposing their inability to perform precise multi‑step geometric reasoning.
VLM spatial reasoning is the ability of a vision‑language model to infer the three‑dimensional layout of a scene from visual input and to use that layout to guide actions.
**Figure 1.** RoTA-Bench overview. The benchmark contains 674 canonical tasks, with 100 test tasks selected by stratified sampling over reference path lengths. In the T-shirt example, the solver chooses discrete rotations from visual observations and receives updated views until projection IoU reaches 0.90 or another stopping rule applies. The right panel shows task success rates and final IoU on the test set; full results are reported in Table 1.
The gap between 2D observation and 3D geometric action remains a major obstacle for current VLMs.
Related Work
Prior multimodal models and benchmarks summarized for context.
Vision-Language Model (VLM) combines an image encoder with a language model to jointly process visual and textual inputs, enabling tasks such as captioning and visual question answering.
Early VLMs emphasized contrastive image–text alignment, later systems added generative and instruction‑following capabilities, and specialized architectures introduced explicit 3D representations, coordinate frames, metric relations, and object orientations to improve spatial understanding.
Static benchmarks assess spatial abilities on fixed inputs—from synthetic 3D scene queries to pose estimation—while interactive benchmarks add closed‑loop, multi‑turn evaluation, though many still limit objects to axis‑aligned grids, restricting true 3D pose reasoning.
RoTA‑Bench offers an interactive evaluation focused on iterative projection alignment, providing a distinct test of geometric reasoning compared with the static and interactive suites described above.
Benchmark Design
We detail the benchmark, its task, construction pipeline, and evaluation protocol.
We now describe the design of RoTA‑Bench, the task formulation, automated construction, and evaluation protocol.
A benchmark that tests whether a vision‑language model can iteratively align 2D projections of a 3D object to a target pose, exposing its ability to perform multi‑step geometric reasoning.
Why not evaluate models on existing 3D CAD datasets?
CAD models contain familiar shapes that VLMs have seen during pre‑training, enabling shortcuts; RoTA‑Bench generates novel geometries from 2D icons, forcing genuine spatial reasoning.
The model repeatedly adjusts the object’s pose based on visual feedback until the projected silhouette matches the target, using projection IoU as a success signal.
How does projection IoU differ from measuring the actual 3D pose error?
IoU only evaluates 2D silhouette overlap, so different 3D orientations can yield the same IoU; pose error measures the angular distance in $SO(3)$ and distinguishes those cases.
A single episode where the target pose is the identity rotation and the initial pose is a random reversible rotation sequence, guaranteeing that the task is always solvable.
Why is the target pose fixed to the identity rotation?
Fixing $R^\star=I_3$ ensures the target silhouette exactly reproduces the original 2D icon, providing a consistent reference across all objects.
**Figure 2.** RoTA-Bench construction and evaluation pipeline. (a) (1) Source 2D icons are (2) filtered, sampled, and normalized, then (3) converted into 3D objects through sphere-bounded extrusion. (4) Reversible initialization generates an initial pose with a feasible solution. (5) Pose-guided beam search and 30$^\circ$-restricted breadth-first search provide verified reference paths, which are used to select 100 test tasks from 674 canonical tasks by stratified sampling over reference path lengths. (6) The action space contains 24 discrete rotations across three axes, two directions, and four angles. (b) In closed-loop evaluation, the solver selects rotations from visual observations and interaction history, then receives updated observations. Success requires projection IoU $\ge 0.90$.
Curate high‑quality 2D vector icons, filter by connectivity and aspect ratio.
Extrude each icon within a spherical envelope of radius $r$, producing a solid $G$.
Generate an initial pose $R_0$ by applying $L$ random $30^\circ$ rotations, forbidding immediate inverses.
Compute reference paths using pose‑guided beam search and $30^\circ$‑restricted breadth‑first search.
Stratify tasks by reference path length and sample 100 test tasks via Hamilton’s method.
Define the action space $A$ with 24 discrete rotations across three axes and four step sizes.
Compute IoU after $R_0$: $s_0 = \text{IoU}(I_{R_0}, I^\star) = 0.45$ (below the 0.90 threshold).
Select action $a_1$ = yaw +10°; update pose $R_1 = Q(a_1)R_0$.
Re‑compute IoU: $s_1 = 0.68$.
Select action $a_2$ = pitch +5°; update pose $R_2 = Q(a_2)R_1$.
Re‑compute IoU: $s_2 = 0.92$ → episode succeeds because $s_2 \ge 0.90$.
This walk‑through shows how a few coarse rotations can cross the IoU threshold, highlighting the benchmark’s sensitivity to action granularity.
Main Results
We report model performance on RoTA‑Bench and analyze failure modes.
We evaluate nine Vision‑Language Models—including GPT‑5.6‑Sol, Gemini 3.6 Flash, and SpatioLM‑Understanding‑InternVL3.5—alongside two agent frameworks, two stochastic baselines, and two greedy baselines, following a common task protocol and decision budget.
GPT‑5.6‑Sol is a Vision‑Language Model that jointly processes images and text; it serves as the primary baseline for our evaluation of interactive 3‑D projection alignment.
How does GPT‑5.6‑Sol differ from the other GPT‑5.6 variants listed?
GPT‑5.6‑Sol is the “Sol” (solution‑oriented) variant that includes a specialized tool‑use module for spatial tasks, whereas “Terra” and “Luna” lack this module and therefore cannot issue explicit rotation commands.
GPT‑5.6‑Sol achieves the highest success rate among the evaluated models.
Table 1 shows a 17.0 % success rate, the best of all VLMs.
**Figure 3.** Mean IoU versus executed rotations for all methods with exported step trajectories on the same 100 tasks, including the 400 Human task runs and both extension runs. Each marker is the mean at one integer step (0–48), joined without smoothing; terminal IoU is retained after termination through step 48. The dashed line marks the success threshold.
There is a substantial performance gap between current models and human baselines on RoTA‑Bench.
Ablations and Analyses
We dissect how input cues and interaction history affect GPT‑5.6‑Sol’s performance.
Recall that RoTA‑Bench forces VLMs to iteratively align 2‑D projections to a target pose, exposing gaps in multi‑step geometric reasoning. This section quantifies how removing visual depth cues or interaction history degrades that ability.
**Table 2.** Input ablation for GPT-5.6-Sol on 30 test tasks, with ablation conditions sorted by Final IoU. Columns report Task Success Rate (Succ.), final projection IoU, Effective Direction Rate (Eff. dir.), percentage distribution across the four legal rotation angles, and Spearman correlation ($\rho$) between pre-action IoU and rotation angle magnitude with 95% confidence intervals.
**Figure 4.** Absolute Spearman correlations between selected rotation angles and pre-action projection IoU or 3D pose error across the 100 test tasks. Bars show $|\rho|$; whiskers show the original 95% bootstrap intervals mapped to absolute values. An interval crossing zero has a lower bound of zero after this transformation. SpatioLM abbreviates SpatioLM-Understanding-InternVL3.5.
**Figure 5.** Context and action outcomes across 211 matched states. Prior rotations and coverage are measured before each matched action. UC, DC, UF, and DF denote the four combinations of IoU increasing or decreasing and pose error decreasing or increasing, respectively.
Human Evaluation
Humans solved all tasks, averaging 0.923 IoU and 16.2 rotations per task.
Human participants achieved perfect task success on the 100‑task test set.
Aggregated results across 400 runs show 100 % success, average IoU 0.9228, and 16.22 rotations per task (Table 5).
The evaluation environment, observable inputs, action space, and success criteria were identical for humans and models. Participants interacted via button presses that map directly to the discrete actions returned by the models in JSON, ensuring a one‑to‑one correspondence between human and model behavior.
**Table 4.** Shared human and model evaluation protocol. The interaction environment, visible observations, action space, and task rules are identical; humans use buttons to submit the same actions that models return in JSON.
The table displays the distribution of reference rotations across "All tasks" and "Test tasks". It includes rows for specific rotation values ranging from less than 4 to 16, with summary rows for "Eligible (≥ 4)" and "Total".
Limitations and Conclusion
We outline current limits of RoTA‑Bench and sketch paths toward richer 3D spatial evaluation.
Recall that RoTA‑Bench probes whether vision‑language models can turn 2D observations into 3D understanding via interactive projection alignment. The current benchmark has two main limitations. First, 3D objects are built by extruding a small collection of 2D icons, which restricts shape diversity and cannot capture complex geometries. Second, each task presents only a single target and uses discrete rotation actions, making the interaction dynamics far simpler than those in continuous physical environments.
Future work can address these constraints. Generative models could synthesize richer 2D icons or directly generate expressive 3D meshes, expanding the variety of objects. The benchmark could also incorporate continuous rotation control, external occlusions, and multi‑object alignment or assembly tasks, raising the difficulty of spatial reasoning. Moreover, automatically generated tasks would enable scalable reinforcement‑learning fine‑tuning to study whether interactive training improves precise 3D control.
In summary, RoTA‑Bench constructs silhouette‑preserving 3D objects with reversible initialization, yielding 674 tasks and a stratified 100‑task test set. Evaluations expose a substantial human–model success gap, show that image‑space and pose‑space progress can diverge, and reveal that fine‑grained control is highly sensitive to interaction history.
AI Use Statement – generative AI tools assisted in writing and literature search, but the evaluated vision‑language models were not used to aid the research process; the authors verified all claims and assume full responsibility for the paper.
Ethics Statement – the human evaluation involved 20 undergraduate participants who gave informed consent and received fair compensation; only task interactions and timestamps were recorded, with no personally identifiable information collected.
Reproducibility Statement – the authors release the task‑construction code, evaluation pipeline, and the complete 674‑task dataset; the appendix details prompting protocols, 3D geometry construction, and validation procedures.
Object Symmetries
How we infer rotational and reflective symmetries of objects from 2D points.
Object symmetries capture the fact that many orientations correspond to the same underlying pose. Detecting these symmetries lets us treat multiple equivalent views as a single state, simplifying alignment in the benchmark.
We search for rotations and reflections that map the normalized 2D point cloud onto itself, then lift the surviving transformations to 3D.
For the point‑matching check we use up to 1,500 evenly indexed points. We compute the median nearest‑neighbor distance $h$ in the full set and set the tolerance $\tau = 3h + 10^{-6}$.
In each direction (original→transformed and transformed→original) we require the 99th‑percentile distance to be $\le \tau$ and the absolute maximum distance to be $\le 2\tau$. Candidates passing this sample test are then verified on the full point set.
Planar symmetries $Q$ induce 3D rotations $S(Q)=\bigl(Q\;0\;0\;\det Q\bigr)$. Rotations preserve depth, while reflections reverse depth but still yield $\det S(Q)=+1$. All such matrices constitute the symmetry group $G$.
Pose error is computed with Equation 6, which finds the smallest rotation distance from the current pose to any equivalent target $R^\star S$. The arccosine input is clipped to $[-1,1]$ and the result reported in degrees. This metric is a diagnostic; success is ultimately decided by IoU.
We fix the target orientation to $R_{\text{target}} = I_3$ so that projecting along the depth axis preserves the icon’s planar details. At other orientations the depth extension can cause overlapping projections, obscuring those details.
Surface coverage is measured by reconstructing the object’s closed triangular mesh with a grid spacing of $0.75$, assigning each triangle its geometric area, and using the centroid as the representative point. A point is visible at step $t$ if its outward normal faces the camera and an unblocked ray reaches the camera.
**Figure 6.** Human evaluation interface with the same prompt content, current observation, historical observations, and discrete action space used by the models. Humans submit actions through buttons that map to the model action tokens and angles.
Icon Processing
Icon processing pipeline extracts, samples, and projects SVGs into deterministic 3‑D renderings.
We rasterize each SVG to a 512 × 512 RGBA canvas and estimate the background colour $c_{bg}$ from the four corners. A pixel $(u, v)$ is kept as foreground only if its opacity $\alpha(u, v)$ exceeds 8 % and its RGB vector $C(u, v)$ differs from $c_{bg}$ by more than 30 in $\ell_1$ distance. This mask isolates the icon content from transparency and background before any geometric processing.
Foreground pixels are scanned row‑wise and mapped to continuous coordinates $\big(24\frac{u-256}{256},\;24\frac{v-256}{256}\big)$, rounded to four decimals. If the total count $n$ exceeds 90 000, we deterministically down‑sample to $m=90\,000$ points by keeping indices $\big\lfloor \frac{k\,(n-1)}{m-1} \big\rfloor$ for $k=0,\dots,m-1$; otherwise all points are retained. Additional occupancy‑grid tests (component ratio, circle occupancy, bounding‑box occupancy, aspect ratio, and minimum foreground count) prune fragmented or degenerate shapes.
Each retained point is scaled by 1.24, then translated so that the minimum enclosing circle becomes the origin. The points are finally fit inside a sphere of radius $r=28$ by clipping their depth to $0.98\,r$.
For each point $(x_i, y_i)$ we generate 31 depth samples. The two endpoints are $z_{i0}=-h(x_i,y_i)$ and $z_{i1}=h(x_i,y_i)$. The remaining 29 interior samples are obtained by $z_{ij} = (2q_{ij} - 1)\,h(x_i, y_i)$, where $q_{ij}\in[0,1)$ is a deterministic function of $(i,j)$ and the task seed. This yields a finite 3‑D thickness while keeping the sampling reproducible.
Rotated points $(x_t, y_t, z_t)$ are projected orthographically to image coordinates $(c_x + \lambda x_t,\; c_y + \lambda y_t)$. Here $(c_x, c_y) = (160, 160)$ is the image centre and $\lambda = 5$ scales the outline to a 320 × 320 canvas. The resulting pixel locations are rounded and a 2‑pixel disk is filled with intensity 255; out‑of‑bounds pixels are discarded.
An auxiliary image encodes depth cues by mapping the same point to $(c_x + 0.9\lambda x_t,\; c_y + 0.9\lambda y_t)$ and assigning intensity $g_t = \min\big(235,\; \max\big(40,\; 110 + 6z_t + 1.8x_t\big)\big)$. The scaling factor 0.9 slightly shrinks the outline, while the linear terms modulate brightness according to depth $z_t$ and horizontal position $x_t$.
For 320 × 320 game‑icon images each point is rasterized with a radius of $\operatorname{round}\big[(320/64) \times 0.36\big]=2$ pixels, filling small gaps without overly expanding the outline. Within each disk the intensity decays by $\big\lfloor 2.2 \sqrt{\Delta u^{2} + \Delta v^{2}} \big\rfloor$, and overlapping disks keep the maximum value. Both the binary mask and the auxiliary image undergo one morphological closing with a $3\times3$ elliptical kernel, and the auxiliary image receives two additional $3\times3$ Gaussian passes ($\sigma=0$) to smooth residual artefacts.
The final observation concatenates the target projection, the current auxiliary image, and the current projection into a $972\times320$ panel, separated by two 6‑pixel strips of intensity 12. This fixed layout guarantees that every method receives identical visual information for evaluation.
Search Reference
Describes how the Search reference is built using inverse initialization and constrained beam/BFS searches.
The inverse initialization sequence yields a feasible solution independent of search. To verify that transformations and projection scoring reproduce successful alignment, we replay candidate paths using two constrained searches: a pose‑guided Beam‑256 search and a fixed‑angle breadth‑first search limited to 30° rotations.
At each depth, Beam‑256 evaluates six directional residuals, selects the closest allowed angle for each, and retains up to 256 candidates ranked by pose distance and orientation diversity. The fixed‑angle search expands all six actions per state and stops when the replayed projection meets the IoU success threshold, selecting the shorter verified candidate as the Search reference.
The table lists various methods and their performance metrics, including the number of successes out of 100 tasks, the success percentage, and the 95% confidence interval.
**Table 7.** Task counts in the 674-task dataset and 100-task test set by Search-reference length. Tasks with at least four reference rotation steps are eligible for test selection.
Dataset and Additional Results
We detail the test set construction and present additional performance tables.
The canonical collection comprises 674 tasks, each identified by a unique source‑icon. Of these, 642 tasks contain reference paths with at least four rotations and are therefore eligible for test selection.
We allocate 100 test places proportionally across reference‑length groups using the Hamilton largest‑remainder method, then rank tasks deterministically by a SHA‑256 hash of the selection seed and task identifier.
**Table 8.** Context before matched actions and their outcomes across 211 participant–task–state pairs. Prior rotations count completed rotations before each action. Prior coverage is cumulative direct surface coverage up to and including the current observation. All values are averaged over pairs. UC improves IoU and reduces pose error; DC reduces pose error only; UF improves IoU only; DF improves neither.
The table compares the performance of "Human" and "GPT-5.6-Sol" across two metrics: "Effective Direction Rate (%)" and "Angle Gap (°)", with a final row showing the difference ("Human minus Sol").
Additional Evidence
Additional evidence supporting the three main findings with detailed statistical analyses.
K.1 reports success counts and $95\%$ Wilson confidence intervals for the nine models and two agents on the 100‑task test set (Table 11). The intervals capture uncertainty under a binomial task model. Although GPT‑5.6‑Sol shows the highest raw success rate, the overlapping intervals mean we cannot claim a statistically reliable superiority without paired analysis.
**Table 11.** Success counts and 95% Wilson confidence intervals for models and agents on the 100 test tasks. Rows follow the order in Table 1.
K.2 audits response formats and episode outcomes (Table 12). “Stagnation” marks termination after repeated rotations without IoU improvement, “STOP” indicates an explicit early stop, “Limit” hits the step cap, and “Repeat” measures consecutive identical rotation pairs. Ten of eleven methods return all actions as valid JSON within two retries, while GPT‑6‑Astra reaches $90.86\%$ validity on the first try and $98.45\%$ within two retries.
**Table 13.** Distribution of successfully returned actions by the number of retries needed to obtain valid JSON, over saved trajectories. Only responses that contain output tokens count as valid returns; transmission failures (no returned tokens) are excluded. “retries$\le k$” is the cumulative share of actions produced within $k$ retries (i.e. on the $k+1$-th content-bearing response).
K.3 describes the statistical resampling protocol. We repeatedly sample tasks with replacement (10 000 resamples) and keep all steps from each sampled task, preserving the four human runs per task. Paired comparisons use the same sampled tasks for both methods, and $95\%$ confidence intervals are taken from the $2.5^{\text{th}}$ and $97.5^{\text{th}}$ percentiles.
K.4 reports direction and angle diagnostics (Table 14). Humans exceed Sol by $25.16$ percentage points in effective direction rate and have a $3.50^{\circ}$ smaller Angle Gap, with both differences statistically significant.
**Table 14.** Action metrics on 100 tasks, with paired 95% bootstrap intervals. Angle Gap uses effective directions only. Differences are Human minus Sol; direction differences are in percentage points.
K.5 evaluates input ablations on a fixed set of 30 tasks (Table 16). Removing history (C) eliminates fine‑grained $1^{\circ}$ adjustments, while keeping the full input (A) yields the highest success rate (5/30, $16.67\%$). Table 15 shows that the cost of the full configuration is modest compared with larger models.
**Table 16.** Full input minus each ablation on the same 30 tasks, with paired 95% bootstrap intervals. Success differences are in percentage points.
K.6 compares associations between rotation angles and two error metrics (IoU‑angle and Pose‑angle). The difference $\Delta\rho = |\rho(\theta,\text{IoU})| - |\rho(\theta,\text{Pose})|$ is positive for Sol (0.169) and negative for Human (−0.055), with $95\%$ intervals excluding zero, confirming opposite tendencies.
K.7 performs oracle action decomposition (Table 17). Replacing Sol’s direction with the optimal one reduces successor pose error by $15.61^{\circ}$, while correcting the angle alone yields a $5.70^{\circ}$ improvement; jointly correcting both achieves the largest gain ($19.99^{\circ}$).
K.8 contrasts two matched 30‑task sets (Table 18). The Poly Haven mesh set yields higher mean final IoU (0.7179 vs 0.5656) and a slightly higher success rate (20 % vs 16.7 %). This suggests that using more realistic meshes reduces reliance on memorized 2D icon priors.
**Table 18.** GPT-5.6-Sol on two 30-task sets with exactly matched reference-path length distributions. The icon-derived set is the original ablation subset; the comparison set uses 30 distinct Poly Haven CC0 meshes based on real-world object categories.