HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents
Hongcheng Gao, Alex Zhou, Jay Zhu, Zelin Zheng, Shijia Ge, Xiping Li, Di Wu, Yazhe Wang, Jianshu Zeng, Xuan Shangguan, Yongli Wang, Yexing Wan, Lingyu He, Zhiqi Jia, Sihang Wu, Xiao He
HexaShell enables VLM-based coding agents to solve robot manipulation tasks by evolving task-specific geometric tools.
How far can a VLM-based coding agent go in robot manipulation when restricted to RGB input and a thin harness, and what specific failure modes define its limits?
Vision-language model (VLM) agents struggle to control robots because they lack precise depth perception and cannot reliably translate pixels into the centimeter-accurate coordinates required for physical manipulation. HexaShell keeps the robot harness thin, allowing the agent to write, test, and refine small Python tools that use multi-view triangulation to measure 3D positions from RGB images. On the RoboDojo benchmark, this approach achieves a 39.7% success rate, setting a new state of the art by outperforming learned policies that rely on large-scale demonstration data.
Paper Primer
The system functions as a self-improving loop: an acting agent attempts a task, and if it fails, an optimizer agent reads the logs, diagnoses the geometric or planning error, and edits the tool code. This evolution relies on triangulation—using multiple camera views to find where rays meet—which provides significantly higher accuracy than monocular depth networks.
Triangulation-based tools outperform monocular depth estimation in robot manipulation.
On paired layouts, the triangulation method succeeded in 57 cases where the depth-network-based tools failed, losing only 23. 8.7 percentage points higher success rate on the RoboDojo leaderboard.
The system's performance is limited by the physical contact required, not by the agent's reasoning capacity.
Success rates vary drastically by task type: pushing reaches 92.5%, while tight insertion—requiring millimeter precision—remains at 8.3%. A clear performance boundary exists between simple geometric tasks and high-precision contact tasks.
Why does the system use a "thin harness" instead of a more complex, end-to-end learned policy?
The authors aim to isolate the agent's strategy from the robot's motion control. By keeping the harness fixed and simple, the agent's learned knowledge is explicitly stored in reusable, testable code tools rather than hidden in opaque neural weights.
What is the primary cost of replacing depth networks with triangulation?
Active perception: the robot must often move its wrist cameras to obtain the second view required for triangulation. This consumes action steps from the task's limited budget, though the authors find this trade-off is net-positive for success.
For robot coding agents, the bottleneck is not reasoning but the conversion of visual input into precise 3D geometry. Future work in this domain should focus on high-precision contact dynamics rather than further scaling VLM reasoning.
Introduction and Motivation
HexaShell lets a VLM coding agent iteratively fix robot failures using RGB‑only tools.
Vision‑language‑model (VLM) agents now match learned policies on manipulation leaderboards, yet they still stumble when precise 3‑D geometry is required. Without depth sensors they must infer positions from RGB pixels, and monocular depth networks still err by several centimeters on our tasks. This perception gap limits how far a coding agent can drive a robot directly.
Like a programmer who reads code, runs a test, sees a failure, and edits the source until it passes, the VLM coding agent reads the robot’s state, generates a textual tool, executes it, observes the outcome, and can be revised by a second agent.
**Figure 1.** HexaShell turns robot failures into reusable tools. A coding agent operates a robot with two arms through a thin harness and sees RGB images only. After a failure, an optimizer agent diagnoses the episode and writes or fixes tools that use only what the robot observes. Both agents are Codex with GPT-6-Astra at medium reasoning effort. Bottom: where tools help (rigid, planar, tool use, clutter) and where they fall short (cloth).
The key shift is from a computer‑only coding agent that writes scripts for a simulated robot to a robot‑control agent that directly commands a physical robot while learning reusable tools.
The HexaShell Architecture
HexaShell iteratively builds RGB‑only tools to diagnose failures, eliminating depth sensors and learned policies.
Without depth sensors the robot cannot directly observe object geometry, forcing costly trial‑and‑error. HexaShell solves this by turning each failure into a reusable RGB‑only measurement tool, keeping the acting agent blind to the simulator.
HexaShell is a three‑part system that (1) executes motion commands in a sandboxed harness, (2) measures 3‑D positions from plain RGB images, and (3) iteratively creates new tools from failed episodes, all without any depth sensor.
How does HexaShell differ from an end‑to‑end learned manipulation policy?
A learned policy embeds perception, planning, and control in a single neural network trained on many demonstrations. HexaShell separates these concerns: the VLM only generates commands, the harness executes them, and explicit tools perform geometry; the system therefore adapts by editing tools rather than retraining the whole policy.
These are neural policies trained on large demonstration datasets that map sensor inputs directly to robot actions, often relying on depth images or end‑to‑end supervision.
Why can HexaShell succeed where learned policies struggle, even though both use the same VLM?
HexaShell keeps the VLM’s reasoning static and instead augments it with explicit tools that can be edited on the fly. The learned policy, by contrast, must re‑learn its perception‑control mapping for every new visual condition.
The container receives a CLI command (e.g.,
The egress proxy forwards the command to the server, which validates the syntax and adds the model credential.
The server interpolates a straight Cartesian path at $2\,$cm spacing and solves inverse kinematics per waypoint with
A trapezoidal velocity profile respects the robot’s joint limits; the timed path is sampled at $25\,$Hz and sent via
The server returns a JSON record with the planning verdict, the final pose, and the remaining step budget.
Why does the harness “never decide” and how does that help the optimizer?
The harness only executes commands it receives; it never generates new commands. This guarantees that any failure originates from the tool or the command itself, making the optimizer’s diagnosis unambiguous and allowing it to focus on fixing the tool rather than the execution engine.
Triangulation finds a 3‑D point by intersecting two rays that originate from two calibrated RGB cameras viewing the same pixel.
Ray $r_h$ from the head camera is computed using its intrinsics and $(u_1,v_1)$.
Ray $r_w$ from the wrist camera is computed similarly using $(u_2,v_2)$ and $P_w$.
The closest points on $r_h$ and $r_w$ are $p_h$ and $p_w$; their midpoint $p = (p_h + p_w)/2$ is taken as the estimate.
In this toy instance the distance between $p_h$ and $p_w$ is $0.8\,$mm, yielding a sub‑millimeter error.
Triangulation converts two independent 2‑D observations into a precise 3‑D measurement without any depth sensor, but the quality hinges on accurate camera pose and pixel localization.
How is triangulation different from a learned depth‑estimation network?
Triangulation uses explicit geometry: each pixel defines a ray, and the intersection of two rays yields a point. A depth network predicts a scalar depth per pixel from learned patterns; it does not guarantee geometric consistency across views and can hallucinate depth where geometry is ambiguous.
The optimizer reads a failed episode’s command log and camera frames, writes a new tool (or patches an existing one) that can measure the missing quantity, verifies the tool passes a delivery check, and retries the same layout.
Why does editing a tool cost less than retraining a policy after a failure?
Editing a tool modifies only a few lines of deterministic code and requires a single verification run, whereas retraining a policy demands collecting new data, running many gradient steps, and potentially re‑tuning hyperparameters. The optimizer therefore achieves rapid adaptation with far lower compute.
**Figure 3. HexaShell.** Top, one episode: 1 a coding agent in a sandbox issues robo commands; the server validates each one, turns it into joint motion and returns feedback and RGB observations. Bottom, between episodes: 2 the episode log keeps the commands, the camera frames and the agent’s transcript; 3 after a failure the optimizer agent diagnoses the episode, 4 writes or fixes a tool that must pass a delivery check, and 5 the tool is loaded and the same layout is tried again.
Performance and Benchmarking
Key performance numbers show HexaShell dominates the RoboDojo leaderboard.
HexaShell outperforms learned policies on the RoboDojo leaderboard.
Figure 2 shows HexaShell achieving 39.7 % success, surpassing VPP2 (32.3 %) and PhysicalRSI (31.4 %).
In clean versus cluttered layouts HexaShell retains 95 % of its success (35.0 % → 33.3 %), while the no‑tools baseline drops to 87 %. Learned policies vary widely (1–68 %). VPP2 falls from 46.4 % to 12.6 % and PhysicalRSI from 25.7 % to 5.6 % when clutter is introduced.
Triangulation beats the depth network by 8.7 points on the leaderboard metric, converting 57 depth‑network failures into successes while only incurring 23 new failures (McNemar p < 0.001). The depth network’s median pixel error is 41 mm at the head camera and about 100 mm at the wrists, exceeding the ~10 mm tolerance needed for precise grasps.
Viewing moves constitute 3.6 % of all steps with triangulation (0.7 % with depth). Episodes containing a viewing move use a median of 4.9 % of the step limit, yet successful episodes still consume only 70 % of the limit (vs 63 % for depth). The cost concentrates on tasks requiring precise measurement, such as the egg‑carton and coin‑slot tasks.
The RGB runs required 769 optimizer rounds (median 17 per task), each taking about 6.3 minutes. The final code base contains 327 task commands spread over ~78 000 lines of Python. A second camera was invoked for 42 commands, totaling 998 calls across 25 tasks.
A public leaderboard of 54 robot manipulation variants that measures both success rate and progress score using only RGB perception.
How does the RoboDojo Benchmark differ from typical robot manipulation testbeds?
It evaluates agents on a wide suite of tasks without depth sensors, using RGB‑only perception and a leaderboard that combines success rate with a progress score, whereas many benchmarks focus on single tasks or depend on depth data.
**Figure 2.** RoboDojo comparison on the leaderboard’s metrics: success rate (a) and progress score (b). Overall is the leaderboard’s score, the mean of its five task types (Generalization: mean of its standard and cluttered variants). HexaShell uses GPT-6-Astra at medium reasoning effort, RGB images only and frozen tools, tested on ten new layouts of each task, which the tools were not built on (Sec. 4.1). GPT-6-Astra is the same model without tools [69]. Systems are ordered and shaded by Overall success rate.
**Table 1.** Comparison of Triangulation and Depth network methods across various performance metrics.
**Figure 11.** Success rate on every task variant: HexaShell against VPP2 and PhysicalRSI. Variants are sorted by HexaShell's success rate; rand: cluttered layouts. HexaShell: RGB images only, frozen retest on ten new layouts, 10 to 19 of seed 0; VPP2 and PhysicalRSI: leaderboard rollouts, 50 episodes per seed.
Capability Mapping and Limits
HexaShell lets a VLM agent iteratively diagnose failures and use RGB tools, avoiding depth sensors.
A Capability Map tags each task with two orthogonal labels: the physical interaction required and the decision type needed.
How does the Capability Map differ from a simple task taxonomy?
Unlike a flat taxonomy that lists only one attribute, the Capability Map combines a physical interaction axis with a decision‑type axis, so it captures both *what* the robot must manipulate and *what* reasoning is required for success.
Tools boost pushing success by 33 points over the baseline without tools.
Table 5 shows pushing success 92.5 % with tools versus 59.5 % without, a 33‑point gain.
Tools improve rigid manipulation by 13 points.
Table 5 reports rigid manipulation success 50.0 % with tools versus 37.0 % without, a 13‑point increase.
Tools raise memory‑task success by 26 points.
Table 5 indicates memory tasks achieve 65.0 % with tools compared to 39.0 % without, a 26‑point improvement.
RGB tools underperform on tight insertion, dropping 21.8 points compared to rendered depth.
Table 3 shows insertion success 8.3 % with RGB tools versus 30.1 % with rendered depth, a 21.8‑point loss.
For cloth folding, using tools reduces success by 22 points.
Table 3 reports cloth‑folding success 72 % without tools versus 50 % with tools, a 22‑point drop.
**Figure 12.** Progress score per task variant on the development layouts, HexaShell with triangulation against (a) the depth network version and (b) the same model without tools. Points above the diagonal favour triangulation. Colour and shape give the physical class (Sec. 5.1): rigid and push (R, Pu), insertion (In), and articulated, deformable or moving (Ar, Df, Mv). Means over the 54 variants: 40.8, 33.0 and 29.8.
**Table 5.** Results on all 42 RoboDojo tasks. Phys. and Dec.: the physical interaction and the decision a task requires (Sec. 5.1): R rigid pick, place or stack; Pu push, slide or press; In tight insertion; Ar articulated object or tool use; Df deformable, fluid or granular; Mv moving objects; G geometry only; S symbols, language or categories; M memory. RGB: frozen retests on new ten layouts, 10 to 19 of seed 0 (Generalization: standard / cluttered 5 to 9); swap blocks with the benchmark’s fix of its button reset and tools developed again under it. Depth net.: frozen retests on the ten seed-0 development layouts (Generalization: five standard / five cluttered). $^\S$The optimizer read a note on what the checker requires (Sec. 3.4). Rendered: success in % of the earlier tools with rendered depth; $^*\$50 episodes on each of seeds 0–2 (25 per variant for Generalization tasks), otherwise the development layouts; $^\dagger$tools from a run with checker notes, $^\ddagger$tools rebuilt by a team member (Sec. D). VPP2, PhysicalRSI: leaderboard success in %.
Among the 268 failed retest episodes, 57 % exhaust the budget, 30 % are declared successes that the checker disagrees with, and 13 % give up early. Failure patterns differ by class: memory tasks often produce declared‑success mismatches, while moving‑object tasks tend to give up once the target passes.
Case Study Overview
Limits of HexaShell case studies: optimizer round costs and tool edit constraints.
The case studies trace a single layout through one loop iteration: a failed attempt, an optimizer round that edits the tool, and the subsequent attempt that succeeds.
Across 769 RGB optimizer rounds, a failure‑driven round lasts a median of 6.3 minutes (90 % under 11 minutes) and runs about 16 shell commands, editing tools by adding a median of 118 lines and removing six. The round reads a median of 0.75 million tokens (mostly cached) and writes 5.7 thousand tokens, totaling 487 million reads and 3.8 million writes overall.
Table 4 lists seven successful turns after edits, covering each physical class from Table 3. In five cases the optimizer succeeded without any visual input, relying solely on logs and tool feedback; only the “swap T” case examined a frame from the failed episode.
**Figure 4.** Push T, layout 2: a height that the tool refused. Left wrist camera. (a) and (c) show almost the same placement; only the measured height of the block's top differs.
Detailed Failure Analysis
Analyzes failure cases and tool fixes across RoboDojo tasks.
Push T requires the block to land within 7 mm and 7° of the pad. The stereo height tool mis‑estimated the plane because the head camera’s coarse pixels yielded a smaller projected area than the wrist camera, causing a 15 mm error and repeated failures. Replacing the pixel‑count metric with a sum‑of‑footprints test eliminated the discrepancy, and all 73 tests passed.
Swap T swaps two T‑shaped blocks, each of which must end within 2 cm of the other’s start and 3° of its heading. The original colour‑mask tool mixed bright top faces with darker side faces, corrupting height and rotation estimates; fixing the mask to keep only the bright top face and matching hues across cameras resolved the error, and all 80 tests passed.
Play Xylophone has the robot strike eight keys with a held mallet. Estimating the mallet head from a single camera produced a 22.9 mm error, causing missed strikes. The tool was changed to triangulate the head using two views, reducing the error to < 2 mm and passing all 35 tests.
Match and pick from conveyor requires the robot to remember an object and retrieve it later on a moving belt. The original timed‑pick schedule was shorter than the arm’s motion time, so the tool aborted before reaching the object. Adding a pre‑move cost estimate and delaying the closing time allowed the pick to succeed, and all 20 tests passed.
Play tic‑tac‑toe fills a board while alternating with a scripted opponent. The transfer tool rotated the empty gripper at home before moving, wasting ≈ 500 steps in waits. Merging the rotation into the travel path saved ≈ 30 steps per transfer, reducing total steps to ≈ 1028 of 1100 and passing all 192 tests.
Fold clothes folds a shirt’s sleeves; the `cloth_view` tool must select a grasp point sufficiently inside the fabric edge. The original check accepted points with only 1 pixel of cloth, leading to edge grasps and failed folds. Requiring a 12 mm margin on all sides and searching on a 2 mm grid fixed the issue, and all 37 tests passed.
Insert tubes places three tubes upright in a rack, a task where RGB tools perform worst. After lifting a tube, the clearance fallback treated ≈ 30 k voxels along monocular rays as hard obstacles, causing the planner to abort. Adding a re‑check of the original path after a failed fallback allowed successful insertion, and all 183 tests passed.
The optimizer acts like a programmer reading bug reports: it inspects step counts and tool error messages, makes localized code edits, and validates with new tests. Across the seven detailed cases, fixes address measurement errors (Push T, Swap T, Play Xylophone), timing mismatches (conveyor schedule, tic‑tac‑toe route), grasp positioning (Fold clothes), and premature aborts (Insert tubes). These repairs improve success rates but still leave tight insertion as the dominant limitation.
**Figure 5.** Swap T, layout 0: two faces of one colour. (a, c) left wrist camera; (b, d) head camera, cropped. (a) Round 2: the dark side faces share the hue of the top faces (b) Round 2: both blocks within 5 mm, headings about 3° off; check fails (c) Round 3: only the bright top faces are kept; outline fit 0.4 mm (d) Round 3: the tool measures the height itself; success at step 285 of 400
**Figure 6.** Play Xylophone, layout 1: one view against two. (a) head camera, cropped; (b–d) right wrist camera.
**Figure 7.** Match and pick from conveyor, layout 4: time, not position. (a, c) right wrist camera; (b) head camera, cropped. (c) is the last frame of the video, since the episode ended during the lift.
**Figure 8.** Play tic tac toe, layout 0: the step budget. (a, b) head camera, cropped, before the last ring; (c) right wrist camera, last frame of the video.
**Figure 9.** Fold clothes, cluttered layout 1: where to grasp a sleeve. Head camera, cropped, after the sleeve folds (a, c) and after the repair grasp of round 13 (b) and the crossed sleeves of round 14 (d).
**Figure 10.** Insert tubes, layout 4: a stop after the lift. (a, b, d) head camera, cropped; (c) left wrist camera. (a) Round 18: after the lift the tool stops; recovery costs 108 steps (b) Round 18: a later tube lies across the rack; the agent quits (c) Round 19: the recheck passes; the tube ends 1.5 mm from the hole centre (d) Round 19: all three three tubes stand in the rack at step 398 of 500
Appendix: Supplementary Data
Appendix details per‑task results, tool logs, and interface specifications for the RGB runs.
This appendix records the raw per‑task outcomes, the concrete RGB tool logs, and the exact interface contract exposed to the acting agent.
**Table 7.** Run with rendered depth and checker notes: what a note changes. Retests on the ten development layouts (standard / random for Generalization tasks) and success in % over seeds 0–2 (fold clothes, in parentheses: the tools of the first round, which were kept).
**Table.** Task performance across two rounds, showing success rates and the specific notes provided for each task.
**Table 2.** What second views cost, 420 retest episodes each on the development layouts. A viewing move is a base motion followed, before any other motion, by a measurement that reads a wrist camera, or one of three tools whose only job is to look. Viewing inside larger tools is not counted, so the first three rows are lower bounds.
**Table 4.** Seven turns of the loop. Each row is one layout: the attempts that failed on it before the success (Fails), what the optimizer found after the last failure, what it changed, the size of the edit in lines added and removed (tests included) and the length of the round in minutes, the action steps of the next attempt, which succeeded, and the frozen retest of the same layout, with the task's retest over its ten development layouts.
Appendix: Task Performance Tables
Raw numeric tables summarizing tool usage and performance metrics.
Appendix: RGB Tool Details
Appendix provides detailed tool usage and the effect of rendered‑depth notes on task success.
Table 6 lists, for every task, the number of task‑specific commands, the lines of Python code required in the enabled tools, the optimizer rounds performed, the proportion of steps spent on viewing moves, and the final retest success rate (averaged over seeds 0–2).
Table 7 reports how adding rendered‑depth information and checker notes changes the success percentages for each task; the “% What the note supplied” column quantifies the improvement contributed by the note.