Spatial Reasoning with a General Coding Agent

Rui Gong, Ruoyu Wang, Yalun Dai, Hao Li, Sidi Yang, Qihao Zhao, Hongyu Li, Zeyang Guan, Fangzhou Hong, Zhaoxi Chen, Kim-Hui Yap, Ziwei Liu

S-Agent 2.0 closes the spatial reasoning loop by integrating a perception toolkit into a general coding agent.

How can we enable general-purpose coding agents to perform reliable 3D spatial reasoning by providing them with a command-line perception toolkit that closes the evidence-gathering loop?

Agentic vision-language models often fail at spatial reasoning because they treat perception and reasoning as separate, disconnected channels. Even when tools provide correct geometric data, the model frequently misinterprets the readings or fails to ground them in the scene. S-Agent 2.0 solves this by using a general-purpose coding agent as the controller, adding a command-line perception toolkit that maintains a persistent scene memory. The toolkit automatically interprets measurements into plain spatial language, ensuring the model's reasoning remains grounded in the tools' geometry. On the MMSI-Bench and ReVSI benchmarks, this approach outperforms specialized spatial agents, achieving significant accuracy gains across both open and closed-source backbones.

Paper Primer

The core mechanism is an evidence loop that forces the model to interleave its own perception with tool-based measurements. The toolkit acts as a persistent registry: it fuses frames into a metric, gravity-aligned scene memory where every object is named and addressable, allowing the agent to query spatial relations as a lookup rather than re-perceiving them.

The interface is designed to cut along the line of competence: the model handles semantic tasks like identifying objects and phrasing prompts, while the toolkit handles the geometric heavy lifting. By returning readings already interpreted in the spatial language of the question, the toolkit eliminates the "translation gap" where models previously failed by misreading coordinate frames or rotation signs.

S-Agent 2.0 significantly outperforms specialized agentic baselines on spatial reasoning benchmarks.

On the ReVSI benchmark, S-Agent 2.0 (Gemini 3.8 Flash) achieves 90.5% accuracy compared to 79.7% for the specialized SpatialClaw baseline.

The toolkit's performance gains are robust across different model backbones and coding agent harnesses.

The method maintains substantial gains even as the backbone model becomes more capable, whereas specialized baselines see their tool-use advantage shrink significantly. Maintains a +9.6 point gain on ReVSI with Gemini 3.8 Flash, while SpatialClaw's gain drops to +2.7.

Why use a general coding agent instead of building a custom spatial controller?

General coding agents already provide the necessary scaffolding—shell access, file systems, and a mature inspect-and-revise loop—that allows the model to cross-check its own perception against tool outputs before committing to an answer.

What is the primary failure mode this system addresses?

It addresses the "evidence gap" where models collect correct geometric data but fail to use it, either by misinterpreting the reference frame or by ignoring the tool readings in favor of their own flawed visual intuition.

Introduction

We expose the missing evidence loop that hampers spatial reasoning and introduce S‑Agent 2.0 to close it.

The current gap lies between the raw measurements provided by 3D perception tools and the model’s final answer: tools often return correct evidence, yet the Vision-Language Model (VLM) fails to translate it into the right conclusion. This unfilled evidence loop leaves spatial reasoning brittle, especially when the model must reason about object positions, relations, and motions.

Understanding where objects are in three dimensions, how they relate, and how they move through space.

Two failure modes dominate: the model misreads a correctly acquired measurement—e.g., flipping a rotation sign or using the wrong coordinate frame—and the evidence never reaches the answer because the reading is stored but not consulted during reasoning. Closing the loop therefore requires three steps: acquisition that integrates with the VLM’s perception, a persistent scene memory that keeps evidence accessible, and interpretation that expresses readings in plain spatial language. S‑Agent 2.0 embodies this principle by equipping a general coding agent with a command‑line perception toolkit to build and query such a memory.

The key shift is moving from monolithic end‑to‑end spatial training to an agentic tool‑use paradigm that explicitly gathers, stores, and reasons over evidence.

S-Agent 2.0 Architecture

S-Agent 2.0 couples a coding agent with a perception toolkit to acquire, store, and query scene evidence.

The agent must continuously gather visual evidence, verify it, and reuse it for answering spatial queries. Our design solves this by letting a coding agent write structured observations into a persistent memory and later retrieve them in plain spatial language.

It is a spatial‑reasoning agent built by attaching a command‑line perception toolkit to a general‑purpose coding agent, so the agent can write observations into a persistent scene memory and later query them in plain spatial language.

The agent registers the cube as object A with position (1, 0, 0) in the scene memory.

The toolkit returns the sphere’s position (0, 1, 0) and writes it as object B in the same memory.

A later reasoning step queries “where is object B?” and retrieves (0, 1, 0) without re‑running any perception.

The answer‑ready reading is formatted as “The blue sphere is 1 meter north of the origin.”

This walk‑through shows how acquisition and persistence let the agent answer spatial questions after a single perception pass, avoiding redundant visual processing.

How does the perception toolkit differ from simply feeding raw image features to the coding agent?

The toolkit does not hand the agent a dense feature map; instead it extracts discrete measurements, writes them into a named scene memory, and returns a concise spatial description. This structure enables later steps to query specific objects by name rather than re‑processing the entire image.

**Figure 1.** S-Agent 2.0 is a spatial reasoning agent built on a general coding agent. A command-line perception toolkit lets the agent close the evidence loop, acquiring scene evidence, remembering it in a persistent scene memory, and interpreting every reading in plain spatial language. S-Agent 2.0 delivers strong performance: it leads MMSI-Bench and ReVSI and stays ahead of the specialized agentic system SpatialClaw on every benchmark and backbone we test.

The Evidence Loop

We expose the three core requirements that close the evidence loop and show how the coding agent fulfills them.

The diagnosis of §1 reveals three unmet requirements for a closed evidence loop: the model must acquire perception jointly with its own outputs, retain evidence across steps, and interpret measurements in the spatial language of the query. Existing systems separate model perception from tool measurements and leave memory and interpretation to the model itself. S‑Agent 2.0 inverts this allocation, letting the CLI toolkit join the two perceptions and carry memory and interpretation.

The loop repeatedly decides which evidence $E$ a question needs, acquires it via the perception toolkit, checks the tool output, stores the result, and finally answers once all required evidence is available.

Step 1 (perception): run the segmentation tool on $I_1$, producing masks and assigning names “cube” and “sphere”.

Step 2 (tool response): the tool returns $oper_c$ = {“cube”: $(1,2)$, “sphere”: $(4,2)$} which is added to the context.

Step 3 (general coding): a Python script reads the stored coordinates from disk as $ogen$ and computes $x_{\text{sphere}} - x_{\text{cube}} = 3$.

Step 4 (decision): since the difference is positive, the answer “yes, the cube is left of the sphere” is emitted.

The loop shows how perception output is immediately usable by a later code action without re‑encoding the raw image, and how the persistent memory $S$ guarantees the coordinates remain accessible across steps.

Determine required evidence $E$ from the question $q$.

Invoke a perception tool from $A_{\text{perc}}$ to acquire $E$, receiving full output $m$ and response $oper_c$.

Store $m$ (and raw frames $I$) in persistent scene memory $S$.

Read $m$ via a general‑coding action $a\in A_{\text{gen}}$, producing $ogen$ that joins the reasoning context.

Update the policy $\pi$ with the new context and repeat until the answer can be generated.

**Figure 2.** Overview of the framework. The agent keeps a coding agent’s full generic freedom over the workspace (viewing images, reading and writing files, writing and running code) and composes perception through the same shell; the toolkit registers everything it perceives into the scene memory and returns interpreted, answer-ready readings.

**Figure 3.** An example question used throughout this section. Left to right: the question, the agent’s command sequence colour-coded by who performs the perception, the outputs those calls produce and the scene memory they write, and the readouts that answer the question.

Persistent Scene Memory

Persistently storing geometry lets the model query exact spatial facts without re‑perceiving.

Without a persistent store, every query forces the model to re‑run perception, which is noisy and costly. The core trick is to write geometry once into a scene memory and read it back on demand.

A persistent 3D store that keeps each object’s position, extent, and 6‑DoF pose so later queries can retrieve exact geometry without re‑running perception.

add‑instance c₁ → writes $\pi$₁ and E₁ into P and I, registers name “c₁”.

add‑instance c₂ → writes $\pi$₂ and E₂, registers name “c₂”.

`R_distance`(c₁, c₂) computes Euclidean distance ‖$\pi$₁‑$\pi$₂‖ = 0.3 m.

`R_size`(c₁) returns volume |E₁| = 0.002 m³; `R_size`(c₂) returns 0.0012 m³.

drop c₁ removes its entries; subsequent reads on c₁ fail, preventing stale geometry from contaminating later queries.

The example shows how a single write per object enables many downstream measurements (distance, size) without additional perception passes.

recon → fuses all frames into a gravity‑aligned scene, registers each camera pose in S.

segment → takes a language prompt from the model, runs a segmentation tool, and writes the resulting mask and extent Eᵢ into I.

add‑instance → the model names a new entity (e.g., “observer”), supplies an initial pose $\pi$ and optional extent E, which are stored in P and I.

drop → removes a previously stored instance, cleaning up erroneous entries before any readout.

R_$\tau$ → any readout function (size, distance, motion) queries the current S to compute a numeric answer on demand.

The figure illustrates a sequential process flow involving variables $q$, $\mathcal{E}$, $x$, $m$, $o^{perc}$, and an "answer". Solid arrows indicate direct transitions labeled with $\pi$ or $S$. Dashed curved arrows represent feedback loops labeled with $\pi$, connecting $o^{perc}$ back to $\mathcal{E}$ and $o^{perc}$ back to $x$.

How does this scene memory differ from a traditional scene‑graph representation?

A scene graph stores objects as nodes with typed edges, but it still requires the model to recompute geometry for each query. Our scene memory stores raw 3‑D geometry (position, extent, pose) directly, enabling closed‑form numeric readouts without any graph traversal or additional perception.

Benchmark Performance

Spatial reasoning needs active evidence gathering and S-Agent 2.0 provides a persistent 3D scene memory.

The central premise is that spatial reasoning requires an agent to actively gather visual evidence; S‑Agent 2.0 supplies a persistent 3‑D scene memory that can be queried throughout a task.

S‑Agent 2.0 (Gemini‑3.8‑Flash) attains the highest average accuracy on MMSI‑Bench, 38.3 %, beating the next‑best open‑weight model by 2.1 points.

Table 1 reports an average of 38.3 % for S‑Agent 2.0 (Gemini‑3.8‑Flash) versus 36.2 % for the strongest open‑weight baseline.

MMSI‑Bench measures how well a model can answer diverse spatial‑reasoning questions (camera, object, and region queries) across multiple categories.

ReVSI evaluates a model’s ability to infer visual‑spatial relationships from rendered scenes, focusing on cross‑object geometry and route planning.

**Table 1.** MMSI-Bench results by question category (C/O/R denote camera/object/region).

**Table 6.** Ablation study of interface components.

The image contains two tables. The left table compares the performance of "SpatialClaw" and "general coding agent" across "Qwen3.8-27B" and "Gemini-3.8-Flash" models, with and without tools, using "ReVSI" and "MMSI" metrics. The right table shows an ablation study for "S-Agent 2.0" variants, comparing "MMSI" and "ReVSI" scores when removing specific components like "scene memory," "interpretation readout," or "model perception interface."

S‑Agent 2.0 consistently outperforms baselines across all categories.

Ablation Studies

Ablation studies quantify the impact of each component and the benefit of the full toolkit.

We evaluate three ablation dimensions: swapping the coding‑agent harness, comparing a full toolkit against none, and removing individual interface components. Each experiment isolates the contribution of a design choice while keeping all other factors constant.

Replacing S‑Agent 2.0’s coding‑agent harness with Codex or Claude Code degrades performance across all benchmarks.

Table 4 (not shown) reports lower scores for the alternative agents, confirming that S‑Agent 2.0 works under every harness.

Equipping the agent with the full toolkit improves ReVSI by +34.3 % for the strongest backbone.

Table 5 lists a +34.3 % gain for S‑Agent 2.0 (Gemini‑3.8‑Flash) over the previous best method.

Removing the model perception interface cuts accuracy by −8.8 % on MMSI‑Bench and −5.9 % on ReVSI.

Table 6 shows the largest drop when the perception interface is omitted.

Removing the scene memory reduces accuracy by −3.5 % on MMSI‑Bench and −2.5 % on ReVSI.

Table 6 records the intermediate drop after the perception interface removal.

Removing the interpreted readout lowers accuracy by only −1.3 % on MMSI‑Bench and −1.2 % on ReVSI.

Table 6 reports the smallest degradation when the readout is omitted.

**Table 5.** Tools against no tools on ReVSI and MMSI-Bench, for two backbones and two harnesses.

**Table 6.** Ablations on interface design. Qwen3.8-27B backbone.

Qualitative Comparison

Ablation results reveal which components drive S‑Agent 2.0’s spatial reasoning gains.

We isolate each architectural piece of S‑Agent 2.0 and measure how its removal harms performance, exposing the role of the evidence‑loop, scene memory, and perception interface.

SpatialClaw runs the perception tools to produce masks and positions, then hands those raw numbers to the model, which must infer the spatial relation itself.

How does SpatialClaw differ from S‑Agent 2.0?

SpatialClaw stops after producing raw masks and positions; S‑Agent 2.0 inserts the question as an oriented instance, cross‑checks the perception against the persistent scene memory, and only then derives the relation, eliminating the reversal error.

**Figure 4.** Qualitative analysis. Left: two items where SpatialClaw calls the right tools and gets the right perception, and still answers wrong, with S-Agent 2.0 correct on both (gemini-3.8-flash on both sides). Top right: S-Agent 2.0 cross-checking its own perception against the scene memory, dropping the wrong masks before grounding the observer and reading the answer. Bottom right: the two remaining grounding failure classes, a wrongly imagined viewpoint and a quantity read in the wrong sense. Zoom in for better readability.

Removing the persistent scene memory reduces MMSI by 3.5 points.

MMSI drops from 66.5 to 63.0 when the scene‑memory component is omitted (Table 6).

Omitting the interpretation readout lowers MMSI by 1.3 points.

MMSI falls from 66.5 to 65.2 without the readout (Table 6).

Removing the model‑perception interface causes the largest drop, decreasing MMSI by 8.8 points.

MMSI falls from 66.5 to 57.7 when the perception interface is removed (Table 6).

Related Work

Positions prior vision‑language and tool‑augmented agents within the spatial reasoning landscape.

Spatial competence in vision‑language models remains limited, prompting three research strands: distilling 3D annotations into instruction data (e.g., SpatialVLM, SpatialRGPT, Cambrian‑S, SenseNova‑SI), geometry‑aware backbone redesigns (Spatial‑MLLM, VST), and training‑signal reshaping via reinforcement or curricula (SpaceR, ViLaSR, MindCube, SpatialLadder).

Tool‑augmented visual agents interleave reasoning with external calls, following the ReAct template; on the visual side, systems such as VisProg and ViperGPT generate full programs, while MM‑ReAct, HuggingGPT, and OctoTools dispatch turn‑by‑turn tool requests, and PyVision writes tooling on the fly. CodeAct shows that executable code outperforms JSON or free‑text formats for general‑purpose agents.

Recent work brings tool use to spatial queries: VADAR synthesizes a Pythonic API, pySpatial composes reconstruction and novel‑view synthesis, GCA splits question formulation from tool execution, RieMind queries a 3D scene graph, SpaceTools fine‑tunes a VLM with supervised and reinforcement phases, and Think3D reasons over rendered point‑cloud views. Closest to our approach, S‑Agent 2.0 schedules multi‑view 3D operations and SpatialClaw drives a VLM through a hand‑crafted plan‑act‑reflect loop, both relying on a persistent scene memory. General‑purpose coding agents such as SWE‑agent and OpenHands already treat image viewing and output revision as ordinary steps, a habit that S‑Agent 2.0 leverages for spatial reasoning.

Toolkit Usage Patterns

We quantify toolkit command usage across benchmarks and compare readout‑enhanced SpatialClaw to S‑Agent 2.0.

Table 8 reports how often each toolkit command is invoked during full‑benchmark runs of S‑Agent 2.0, while Table 9 isolates the impact of our readout functions when transplanted into SpatialClaw.

**Table 8.** Toolkit usage per episode. Calls per episode and, in parentheses, the share of episodes that issue the command at least once. Image views count the agent's own view calls on benchmark frames, segmentation overlays and top-down maps. MMSI-Bench and ReVSI, 1000 items each.

**Table 9.** SpatialClaw with our readouts transplanted into its kernel, against SpatialClaw and S-Agent 2.0. Qwen3.8-27B backbone.

Extended Benchmark Comparisons

Comparing S‑Agent 2.0 and SpatialClaw on three ReVSI tasks.

S‑Agent 2.0 answers correctly on 2 of 3 ReVSI tasks, while SpatialClaw answers none correctly.

`ReVSI_5112` (correct), `ReVSI_6126` (correct), `ReVSI_4065` (incorrect for S‑Agent 2.0); SpatialClaw is incorrect on all three.

Failure Modes and Caveats

Additional Comparisons II highlights where S‑Agent 2.0 diverges from SpatialClaw across motion and object‑placement benchmarks.

Recall that spatial reasoning requires an agent to actively gather visual evidence; S‑Agent 2.0 supplies a persistent 3D scene memory accessed via a command‑line toolkit.

MMSI‑Bench 677 asks which way the camera moves; the ground‑truth answer is “Backward”. SpatialClaw selects “Forward” while S‑Agent 2.0 correctly chooses “Backward”.

MMSI‑Bench 363 evaluates egocentric motion; the correct label is “Front left”. SpatialClaw answers “Forward”, whereas S‑Agent 2.0 picks the correct “Front left”.

MMSI‑Bench 223 queries the cardinal location of a light switch relative to a bedside table; the answer should be “East”. SpatialClaw mistakenly reports “West”, while S‑Agent 2.0 returns the correct “East”.

**Figure.** SpatialClaw Reconstruct (32 frames) + vlm.locate + SAM3; 10 steps. S-Agent 2.0 15 perception calls.

```python $ python3 /app/perception.py segment --prompt "ceiling-mounted projector" --prompt "sink" \ --prompt "chair" --prompt "door" --prompt "pillar" ... 106 instances registered: chair, chair_2, ..., chair_28, door ... door_7, pillar ... pillar_4, sink $ python3 /app/perception.py distance --a "ceiling-mounted projector" --b "sink" | "distance": 5.61 $ python3 -c '... for every sink/chair/door/pillar instance in list_instance: perception.call("distance", {"a": "ceiling-mounted projector", "b": name})' chair_5 : 2.13 m chair_11 : 2.44 m chair_18 : 2.52 m pillar_3 : 3.16 m pillar_2 : 3.45 m door_6 : 5.22 m sink : 5.61 m (28 chairs, 7 doors, 4 pillars, 1 sink ranked; the six nearest and the sink shown) $ echo -n "B" > /app/answer.txt ```

Ground truth C. SpatialClaw answered B (Forward); S-Agent 2.0 answered C (Backward).

```python # step 0 recon = tools.Reconstruct.Reconstruct(InputImages) pose0, pose1 = recon.extrinsics[fi0], recon.extrinsics[fi1] translation_world = pose1[:3, 3] - pose0[:3, 3] d_fwd = np.dot(translation_world, pose0[:3, 2]) # camera forward column Translation in world: [-0.40627634 -0.34786453 11.62174324] d_right (camera local X): -0.4066 m d_down (camera local Y): 0.3478 m d_fwd (camera local Z): -11.6217 m # step 1, vlm.ask(InputImages, "... in which direction is the camera/vehicle moving ...") - The road markings (such as the directional arrows and lane divider dashes) move downward toward the bottom of the frame and become larger, indicating the camera is approaching them. Therefore, the camera/vehicle is moving **Forward**. # step 2, model text: Look at 'pose0[:3, 2]' which is approximately [0, 0, -1]. That means in world coordinates, the camera's forward direction is '-Z'! Wait! If translation in world is '+11.62' along '+Z', does that mean pos1 has Z = +11.62 while pos0 has Z = 0? # step 3, model text: In Frame 0, there is a forward arrow at (484, 575) right in front of the car's hood. In Frame 1, the arrow has disappeared under the bottom of the camera's field of view! This clearly indicates the camera is moving forward down the road. [ReturnAnswer] Answer submitted: B ``` ```bash $ python3 /app/perception.py recon $ python3 /app/perception.py motion --of camera:f0 --of camera:f1 { "kind": "camera pair", "from": "camera:f0", "to": "camera:f1", "turn": "none", "conclusion": "from camera:f0 to camera:f1 the camera moved 10.85 m BACKWARD, 0.28 m to its LEFT, 0.74 m DOWN; kept its heading (turn < 3 deg)", "note": "relative pose between the two camera instants: movement is in the FIRST camera's gravity-aligned ground frame ... Copy the conclusion -- do not re-derive motion from the pictures" } $ echo -n "C" > /app/answer.txt ```

**Figure.** A two-panel perspective view showing a mechanic's workspace.

```bash $ python3 /app/perception.py recon $ python3 /app/perception.py motion --of camera:f0 --of camera:f1 "conclusion": "from camera:f0 to camera:f1 the camera moved 0.74 m FORWARD, 0.32 m to its LEFT, 0.67 m UP; turned 8 deg CLOCKWISE (to its right); tilted its view 20 deg DOWN; its sideways lean changed by -5 deg ..." $ python3 /app/perception.py side --view 0 --of camera:f1 "forward_m": 0.74, "right_m": -0.32, "up_m": 0.67, "direction": "front-left", "conclusion": "camera:f1 is 0.74 m in FRONT of camera:f0, 0.32 m to its LEFT and 0.67 m ABOVE it -- overall to its FRONT-LEFT", "note": "the observer is a CAMERA, so the frame is its gravity-aligned ground heading ... camera tilt cannot skew these. direction is the combined eight-way word ..." $ echo -n "C" > /app/answer.txt ```

**Figure.** Comparison of SpatialClaw and S-Agent 2.0 workflows. The left side shows the SpatialClaw process involving 8 frames, VLM localization, and SAM3 segmentation, culminating in a cardinal coordinate plot. The right side shows the S-Agent 2.0 process involving 9 perception calls, including segmentation of a television and light switch, and a bird's-eye view (BEV) map with a compass rose.

Toolkit Command Reference

Long-video and multi-view perception. Two families of methods supply the visual evidence these questions need. Frame‑compression methods sample, retrieve, or summarize a limited set of frames for a long‑context VLM, as in Video‑LLaMA [48], LongVA [50], and T⋆ [47], which keeps the context tractable at the risk of dropping the frames a question turns on. Reconstruction‑first methods build an explicit 3D representation with multi‑view geometry or feed‑forward models such as VGGT [33] and Depth Anything 3 [20], which grounds the geometry well while paying the reconstruction cost on every question. Both families hand their output to the model as fixed context, leaving association across views and metric comparison to implicit reasoning. S‑Agent 2.0 uses reconstruction and promptable segmentation backends of this kind as tool perception (§2.3), with the agent deciding per question which of them to call and what to ground; what they measure and what the model registers itself land in one scene memory, and every reading comes back interpreted.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers