Show-Harness: Just a VLM Agent Can Play Robots
Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, Mike Zheng Shou
Show-Harness uses a discrete semantic action interface to let foundation VLMs control robots without opaque motor-regression.
How can we bridge the gap between high-level reasoning in foundation VLMs and low-level robot control without training massive end-to-end action models?
Foundation vision–language models (VLMs) possess broad world knowledge, but forcing them to regress continuous, embodiment-specific motor signals collapses this intelligence into an opaque, brittle mapping. Show-Harness replaces this with a discrete semantic interface: the VLM emits high-level action units (e.g., "move left," "grasp") that a deterministic interpreter grounds into precise robot motion. This approach enables frontier VLMs to act zero-shot and allows small models to be fine-tuned with minimal data, consistently outperforming standard vision-language-action (VLA) paradigms across tasks and embodiments.
Paper Primer
Show-Harness acts as a mediator that keeps the VLM in the loop for fine-grained physical decisions. The system is like a remote-controlled drone operator: the VLM provides the high-level "fly forward" command, while the onboard flight controller handles the low-level stabilization and motor adjustments required to execute that specific move.
Show-Harness enables zero-shot robot control using frontier VLMs and efficient fine-tuning for smaller models.
Across ten diverse pick-and-place tasks, the zero-shot agent (Gemini-3.1 Pro) and fine-tuned agent (Qwen3.5-2B) significantly outperformed VLA baselines like $\pi\$0.5 and GR00T, particularly on held-out objects and cross-environment shifts. The fine-tuned 2B-parameter model achieved 89% average success across tasks, compared to 40% for representative VLA baselines.
The authors also introduced GUMI, a GUI-based interface that allows humans and agents to collect demonstrations using the same semantic action units. This unifies data collection across embodiments without requiring specialized teleoperation hardware.
Why is this interface approach better than simply training a VLM to output motor commands directly?
Direct motor regression forces the VLM to learn embodiment-specific physics, which discards its general semantic knowledge. Show-Harness keeps the VLM's decision-making in a semantic space it already understands, delegating the physical realization to a deterministic interpreter.
Does this system require a new model for every different robot arm?
No. The VLM-facing interface remains constant; adapting to a new robot only requires swapping the embodiment-specific interpreter that translates semantic units into that robot's native control language.
By decoupling semantic intent from physical execution, Show-Harness provides a scalable path to leverage foundation models for robotics without the need for costly, embodiment-specific pretraining.
Introduction
We expose the gap between VLM intelligence and robot control and introduce Show‑Harness to bridge it.
Foundation vision–language models (VLMs) encode rich knowledge about objects, relations, and goals, yet this knowledge does not readily translate into robot behavior. Existing Vision–Language–Action (VLA) approaches collapse that semantic richness into opaque continuous‑action regressors, and hierarchical pipelines still depend on downstream controllers, leaving a gap between VLM reasoning and direct physical execution.
Show‑Harness is a modular perception‑reasoning‑action loop that lets a foundation VLM issue discrete semantic actions, which embodiment‑specific interpreters turn into concrete robot motions.
**Figure 1.** Show-Harness unlocks generalizable embodied manipulation with foundation VLMs across diverse tasks, scenes, and robot embodiments, enabling direct zero-shot deployment with frontier models and capable, efficient control via lightweight adaptation of compact open-source models.
**Figure 2.** Show-Harness connects foundation VLMs to diverse robot embodiments through one semantic action interface, enabling zero-shot control with frontier models and lightweight adaptation of small open models.
Show‑Harness bridges VLM reasoning to robot control by providing a compact, semantically meaningful action interface.
Related Work
We situate Show-Harness among prior robot foundation models and agentic architectures.
Related work spans two main strands: foundation models that directly generate robot actions, and agentic systems that keep a pretrained vision‑language model (VLM) intact while delegating low‑level control to external modules.
2.1 Foundation Models for Robot Manipulation
Vision‑Language‑Action (VLA) models attach a learned action generator to a pretrained vision‑language model (VLM) backbone, enabling direct prediction of robot motor commands from visual and textual inputs.
Low‑level VLA approaches vary: some regress continuous action chunks, others use diffusion or flow matching, while a few discretize motor commands into tokens or learn latent codes from video‑action pairs.
These methods often couple action prediction with future visual dynamics, scaling across heterogeneous tasks but sacrificing interpretability and requiring fresh robot trajectories for each new task family.
Foundation models as intermediate decision makers keep the VLM above low‑level control, emitting subgoals, keypoints, affordance maps, or spatial constraints that downstream controllers realize.
While this preserves semantic richness, it offloads physics to separate pipelines, demanding engineered grounding and long‑horizon planning, feedback, and failure recovery.
2.2 Agentic Robot Systems
Agentic architectures retain the VLM intact and wrap it in a harness that translates its decisions into robot behavior, closing the perception–decision–execution loop.
Design choices differ in how decisions are executed: composing programs over APIs, selecting predefined skills, steering VLA policies with language subgoals, or delegating to symbolic planners.
These systems also provide task context, persistent memory, feedback‑driven monitoring, and failure detection to enable long‑horizon interaction.
Interfaces for embodied execution are central: VLMs act through primitives exposed by the surrounding system, and the level of abstraction strongly influences reliability.
Show‑Harness distinguishes itself by exposing fine‑grained semantic action units whose physical realization is deterministic and transparent, allowing the VLM to remain directly responsible for detailed physical decisions.
In contrast, prior work typically exposes whole skills or opaque controllers as primitives, relegating the “how” to an external executor.
Finally, the discrete semantic action space aligns with broader interface principles seen in digital agents, but Show‑Harness uniquely grounds these actions into real‑world robot motion, enabling strong sim‑to‑real transfer.
The Show-Harness Architecture
Show‑Harness wraps a VLM in a perception–reasoning–action loop that outputs fine‑grained semantic actions.
Show‑Harness translates foundation VLM intelligence into robot control by embedding the model in a closed‑loop perception–reasoning–action cycle.
The loop repeatedly feeds the VLM fresh perception, lets it reason about the task, and asks it to pick a semantic action that is then grounded into robot motion.
Capture multi‑view image $I_t$ and proprioceptive state $p_t$ → form observation $o_t$.
Update interaction history $h_t$ with the previous action $a_{t-1}$.
Run reasoning plugins $P$ on $(\ell, o_t, h_t)$ to obtain context $c_t$.
VLM predicts semantic action $a_t = \pi(c_t)$ from the action space $A$.
Interpreter $g_E$ grounds $a_t$ into robot command $u_t$ and executes it.
Observe the resulting state, loop back to step 1.
A tiny, robot‑agnostic vocabulary of symbolic action units that the VLM can emit, each of which maps to a concrete motion primitive.
Semantic unit $a_t$ = `MV_FWD` → translation vector $d_a = (1,0,0)$, rotation vector $r_a = 0$.
Apply equation (3): $x_{t+1} = x_t + 0.1 \cdot R_E (1,0,0) = x_t + (0.1,0,0)$.
Orientation unchanged: $Q_{t+1} = Q_t$ because $r_a = 0$.
The interpreter sends a Cartesian setpoint 10 cm forward to the robot controller.
Incremental units let the VLM observe the effect of a 10 cm move before deciding the next step, enabling closed‑loop correction.
How does this Semantic Action Interface differ from the action spaces used in typical VLA models?
VLA action spaces usually expose low‑level joint angles or velocities, tying the model to a specific robot morphology. The Semantic Action Interface instead offers a compact set of high‑level symbols (e.g., “move forward one step”), keeping the VLM agnostic to embodiment and allowing the same interface to be reused across many robots via a simple interpreter.
**Figure 3.** The Show-Harness architecture. A modular perceive–reason–act loop connects foundation VLMs to robot control through a shared semantic action interface.
**Table 1.** Harness plugins organized along the perceive–reason–act loop.
Demonstration and Interface
We detail the shared GUMI interface, hardware, tasks, agents, baselines, and data collection used in experiments.
Existing teleoperation pipelines rely on bespoke hardware or simulation‑only controls, making large‑scale data collection cumbersome. A shared GUI that works for both humans and agents solves this bottleneck by exposing the same discrete semantic actions to all participants.
GUMI is a GUI where each discrete semantic action appears as a labeled button or keystroke, letting a human keyboard or an autonomous agent click the same control.
The user presses the “UP” button; GUMI sends a 2 cm upward command to the robot and records ($o_0$, “UP”).
The robot moves upward, the camera view updates, and the new observation $o_1$ is captured.
The user then clicks “GRASP”; GUMI issues the gripper‑close command, records ($o_1$, “GRASP”), and stores the resulting joint trajectory.
This toy example shows how a high‑level semantic command automatically yields both a discrete label and the full low‑level motion trace needed for policy learning.
How does GUMI differ from a typical teleoperation GUI?
Typical GUIs expose raw joint or velocity controls, requiring the operator to know robot kinematics. GUMI instead presents abstract semantic actions that are already grounded by the robot’s interpreter, so the same button works on any platform that implements the same action set.
Our experiments run on two platforms: a 7‑DoF Franka Research 3 arm with an exocentric Intel RealSense D435 and a wrist‑mounted D405, and a bimanual AgileX rig with two 6‑DoF arms observed by three Orbbec Dabai DC1 cameras (one shared egocentric view and two wrist views).
We define ten real‑robot manipulation tasks by pairing five objects (block, banana, tennis ball, teddy bear, chess piece) with two target receptacles (plate, bowl). Success rate and average steps per episode are the metrics; each task is run for 10 trials with random object placements, and episodes are capped at 50 steps.
For VLM agents we use two modes. In zero‑shot (ZS) mode we query the frontier Gemini‑3.1 Pro VLM with three reasoning budgets (low, medium, high) and adopt the medium setting by default. The Adaptive Step plugin chooses a 2 cm fine step when the target appears in the wrist view and a 4 cm coarse step otherwise; the action history retains the five most recent actions. In fine‑tuning (FT) mode we adapt Qwen3.5‑2B with rank‑64 LoRA adapters, freezing the vision encoder and multimodal projector.
We compare against three families of baselines: (1) pure VLA models such as $\pi$0.5 and GR00T that map observations directly to low‑level commands; (2) VLA‑centric agents that add an outer planning layer (Harness VLA, Goal VLA); and (3) code‑as‑policy agents that translate high‑level reasoning into executable programs (CaP‑X, RATS).
Using GUMI we collected 164 real‑robot episodes (7.8 K decision steps): 101 episodes on the Franka arm (5.0 K steps) and 63 on the single‑arm AgileX (2.8 K steps). For sim‑to‑real transfer we gathered 230 simulated episodes (13.5 K steps) across ManiSkill (100 episodes) and RoboLab (130 episodes), covering two simulators and multiple objects.
**Figure 4.** The GUMI interface. GUMI is a GUI-based manipulation interface that enables humans and frontier agents to autonomously collect demonstrations through the same semantic controls.
**Figure 5.** Real-robot rigs.
Generalization Performance
Show‑Harness generalizes across tasks, environments, and robot embodiments, achieving a 52.5% average success rate.
Show‑Harness attains a 52.5% average success rate across task, environment, and embodiment generalization, outperforming the strongest VLA baseline by +30 percentage points.
Table 2 shows Show‑Harness (ZS/FT) averaging 52.5% while the best VLA models ($\pi$₀.₅, GR00T) achieve roughly 34% across the same three generalization categories.
**Figure 12.** Qualitative comparison with $\pi_{0.5}$. Show-Harness completes all three examples, while $\pi_{0.5}$ exhibits manipulation failures (e.g., gripper–object collisions and unstable grasps) highlighted by the red boxes.
Physical and Semantic Adaptability
Show‑Harness dramatically expands robot adaptability across precision and reasoning tasks.
Show‑Harness improves zero‑shot fine‑grained control success by 22 % over the $π_{0.5}$ baseline.
Zero‑shot performance rises from 60 % to 82 % when the interpreter step size is halved, while $π_{0.5}$ attains only 18 % under the same conditions.
Beyond precision, Show‑Harness enables new action compositions that halve the number of execution steps, maintains robustness when extrapolating rotations to unseen 90° orientations (70 % zero‑shot), and tolerates expanded workspaces with only mild performance loss, unlike $π_{0.5}$ which degrades sharply.
Joint multi‑arm coordination further boosts success on collaborative tasks and eliminates collisions, while reasoning‑intensive tasks benefit from situated planning (85 % zero‑shot) and instruction‑guided few‑shot (70 %). In‑context video demonstrations enable perfect zero‑shot execution (20/20) and 95 % few‑shot success when paired with a task outline.
**Figure 6.** Capability analysis of Show-Harness across physical (hand icon) and semantic (brain icon) adaptability.
Ablation Studies
We isolate each component to see how its removal changes robot success.
We systematically remove each component of the Show‑Harness pipeline and measure the impact on success across the five Plate tasks.
Increasing thinking effort inflates wall‑clock time without proportional success gain.
GPT‑5.6‑sol incurs a 3.4× runtime increase when thinking effort is raised.
VLMs follow instructions reliably.
Over 98% of model responses produce valid action units.
Removing subtask planning drops overall success.
Success falls to 60% when planning is disabled.
Disabling action chunking retains high success.
Success remains at 96% while model calls increase.
Enforcing chunking throughout harms performance.
Success declines to 74% under forced chunking.
Adaptive step control balances speed and precision.
Agents achieve 96% success with an average of 30 steps per episode.
Visual Prompt dramatically improves handle‑aware grasping.
Success rises from 40% to 85% when the plugin is enabled.
Failure recovery is essential for robust grasping.
Removing the recovery module reduces success to 72%.
Arbitrary symbols without conventions rarely succeed.
Only 1 out of 20 episodes (5%) succeed under variant D.
Mapping inference under variant D is largely inaccurate.
Only 23.3% of inferred action mappings are correct.
Backbone scaling shows that a 2 B‑parameter model already attains strong performance, while larger models mainly help fine‑grained tasks such as stacking and peg insertion.
Multi‑View Guidance and proprioceptive cues both provide critical information for precise manipulation, especially when visual ambiguity arises.
**Figure 7.** Effect of frontier VLM choice and thinking effort.
**Figure 8.** (a) Effect of fine-tuned backbone scaling over the five Plate tasks; bubble area is proportional to parameter count. (b) Three Qwen3.5 capacities across fine-grained tasks.
**Figure 9.** Plugin ablations on the real Franka arm using Gemini-3.1 Pro as a zero-shot agent.
**Figure 10.** (a) Ablation on action-space representation. (b) Action mappings inferred under setting (D) over 20 episodes, where the model probes unknown symbols and infers their effects from observation changes.
**Figure 11.** Qualitative real-world demonstrations of Show-Harness across diverse tasks and conditions.
Implementation Details
Appendix provides fine‑tuning details and the demonstration corpus.
Fine‑tuning the VLM is lightweight: we train for 40 epochs on 7.9 K single‑arm samples with a learning rate of $1 \times 10^{-4}$, cosine schedule, warmup ratio $0.1$, bf16 precision, $256 \times 256$ views, and an effective batch size of $32$.
The process fits on 24 GB‑class GPUs and fine‑tuning the Qwen3.5‑2B model completes in under 2 hours on a single H200.
**Table 3.** Demonstration corpus collected through GUMI.