OpenVLHarness: Real-World Visual Agentic Tool-Use with Multimodal Memory

Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang

A multimodal reasoning harness that enables MLLMs to use tools and retain persistent, addressable evidence across reasoning steps.

How can we improve the reliability of multimodal agents in real-world tasks by providing them with a structured capability layer and persistent multimodal memory?

Multimodal Large Language Models (MLLMs) struggle with real-world tasks requiring precise measurements or external knowledge, as simply increasing inference effort yields diminishing returns. OpenVLHarness introduces a capability layer for consistent tool interaction and a persistent multimodal memory that retains artifacts like images and structured data as addressable handles. This approach allows models to iteratively acquire, inspect, and compute over evidence, improving average performance by 9.7–12.6 points across 22 benchmarks compared to no-tool baselines.

Paper Primer

The system functions as an orchestrator that delegates tasks to specialized backends while maintaining a shared workspace. The capability layer acts as a translator: it converts heterogeneous tool outputs—such as raw segmentation masks or dense numerical arrays—into model-readable observations that combine visual overlays with concise, structured descriptions.

Persistent multimodal memory is the system's "scratchpad": it registers artifacts under unique handles, allowing the model to reference specific images or data bindings in subsequent turns without re-serializing large payloads. A Python Coding Agent leverages these handles to perform computations, such as calculating depth within a specific segmented region, by executing code directly over the stored data.

OpenVLHarness consistently outperforms base MLLMs and existing training-free tool-use baselines across diverse visual reasoning tasks.

Evaluated on 22 benchmarks using eight backbones, including Qwen3-VL-8B and GPT-6 Astra. Average score improvements of 9.7–12.6 percentage points over no-tool baselines.

Why is a "persistent memory" necessary if the model can already see the tool outputs?

Raw tool outputs like segmentation masks or depth arrays are often too large or complex to serialize into the model's context window repeatedly. Memory provides addressable handles, allowing the model to reference specific artifacts for later computation without needing to re-process or re-describe the underlying data.

Does this approach require fine-tuning the underlying MLLM?

No, OpenVLHarness is a training-free harness. It operates by augmenting the orchestrator's prompt and tool-use workflow, allowing it to be applied to proprietary and open-source backbones without updating their weights.

OpenVLHarness demonstrates that evidence acquisition and persistent memory are more effective for complex visual reasoning than simply scaling the inference budget of a base model.

ABSTRACT

Multimodal models lack reliable tool‑use and persistent memory for real‑world tasks.

Current multimodal large language models (MLLMs) excel at visual reasoning but falter when a task needs evidence that cannot be derived from internal reasoning alone; simply allocating more inference steps yields only modest performance improvements, exposing a gap in reliable tool‑use and the absence of a persistent multimodal memory to store and reuse external observations.

The OpenVLHarness Framework

Introducing OpenVLHarness to bridge tool unreliability and memory gaps in multimodal reasoning.

Multimodal Large Language Models (MLLMs) excel at reasoning over images and text, yet many real‑world visual tasks require evidence that cannot be inferred from the input alone. Scaling the reasoning budget yields only marginal gains, suggesting that better access to task‑relevant multimodal evidence is essential.

OpenVLHarness is a framework that lets an MLLM orchestrator invoke external tools and keep their results for later use, enabling reliable multimodal reasoning.

The capability layer gives each tool a uniform interface and turns its raw output into visual or textual evidence the model can read.

Multimodal memory stores images, structured data, and web references under handles so later steps can retrieve and compute on them.

The harness decouples reasoning from tool execution via a structured capability layer.

Diagnosing Agent Failures

Analyzes failure patterns of Qwen3‑VL‑8B and outlines diagnostic remedy channels.

We run the Kimi K3 critic on 9,893 failed Qwen3‑VL‑8B trajectories, labeling each with a primary failure cause and a suggested fix. The critic’s labels agree with human annotations on 92 % of a 100‑sample audit, confirming reliable diagnosis. Analysis reveals domain‑specific patterns that guide targeted harness specialization.

Pinpoint why a multimodal model’s tool‑use trajectory broke and map the cause to a concrete remedy.

**Figure 4.** **Failure diagnosis and harness specialization with Qwen3-VL-8B.** We critique failure trajectories using Kimi K3. (a) Distribution of failure causes across four domains. (b) Proposed remedies: grouped into creating task-specific tools (A–C), updating harness text (Y), and no proposed remedy (Z). (c) Four datasets selected for contrasting remedy profiles: Y (orange) dominates ERQA and RealWorldQA, while A–C (blue) dominates SparBench and FREAK-VQA. (d) Gains from harness search and tool generation over the unoptimized harness. Harness search is more effective on datasets dominated by text-level changes, while tool generation yields larger gains when task-specific tools are preferred.

Specializing the Harness

Specialized workflows adapt the harness without retraining the model.

Adapting a generic harness to a new domain often fails because the default prompts and tools do not address domain‑specific errors.

Instead of retraining the model, we specialize the harness by either searching for better prompt/tool configurations (harness search) or by generating new task‑specific tools (tool generation), each driven by a coding agent.

How does harness search differ from ordinary prompt‑tuning?

Prompt‑tuning only rewrites the prompt string; harness search simultaneously rewrites the prompt, the tool description, and the evidence‑rendering template, and it evaluates each combination automatically, which yields richer adaptations than changing text alone.

Pair (P₁, T₁) yields an average score of 0.62.

Pair (P₁, T₂) yields 0.58.

Pair (P₂, T₁) yields 0.55.

Pair (P₂, T₂) yields 0.60.

The agent selects (P₁, T₁) as the best candidate because it has the highest score.

Even with only two prompts and two tool descriptions, exhaustive evaluation quickly reveals a non‑obvious combination that outperforms each component in isolation.

Related Work

Survey of prior visual tool‑use approaches and memory mechanisms.

This section surveys the landscape of visual tool‑use research, highlighting how earlier systems handle tool composition, memory, and evidence management.

Baseline systems either pre‑plan a fixed sequence of vision tool calls or interleave reasoning with tool feedback, but they typically lack adaptability to changing tool outputs.

These works treat visual tool‑use as a program‑synthesis problem: a natural‑language question is compiled into a fixed sequence of vision‑tool invocations that produce the final answer.

These systems interleave reasoning steps with visual feedback, allowing the model to adjust its plan after observing tool outputs.

These approaches embed visual tool interfaces into pipelines tailored to a particular application domain (e.g., medical imaging, robotics).

A training‑free tool‑use framework that manually designs tool interfaces and execution strategies for visual reasoning.

Another training‑free system that defines a fixed set of visual tool APIs and composes them via a rule‑based planner.

Techniques that separate compact model‑visible state from richer external stores, allowing agents to retrieve detailed information on demand.

Introduces referenceable variables that store tool evidence, making intermediate results explicitly addressable.

Maintains intermediate results as variables within an executable interpreter, allowing procedural composition of tool calls.

Uses symbolic references to intermediate tool outputs that are resolved during downstream calls, integrating tool evidence into language model reasoning.

Compiles language model plans into executable code, using symbolic references to capture tool outputs for later stages.

Tool-Use Behavior Analysis

OpenVLHarness boosts accuracy by up to 10.1 points while keeping tool‑call costs manageable.

OpenVLHarness improves accuracy by up to +10.1 points over the base models.

Table 5 shows $\Delta$ = +10.07 for Qwen3‑VL‑8B, the largest gain among the five models.

**Figure 5.** Tool-call volume and engagement by domain. (a) Tool calls per question, averaged over all questions in the domain. (b) Fraction of questions answered with no tool call. Engagement varies with model scale within a family and not across families, and call volume does not order the models by accuracy.

Tool engagement scales with model complexity, not family.

Extended Results and Ablations

This section presents additional backbones, ablation details, training baselines, grounded perception analysis, extended visual search results, and a reasoning‑effort study.

We first evaluate OpenVLHarness on three additional backbones—GPT‑5, GPT‑6 Sol, and Kimi K3—then detail the ablation ladder that isolates each harness component, compare against training‑based baselines, analyze performance across object counts, extend visual‑search results, and finally examine how reasoning effort affects gains.

OpenVLHarness raises the overall average from 52.7 to 64.3 when applied to GPT‑6 Sol.

Table 7 reports a 52.7 baseline average and a 64.3 harness average across 22 datasets.

**Table 1.** Performance comparison across various datasets using different models (Base, Codex, Visual Sketchpad, OpenVLHarness) and complexity levels (low, med, high, xhigh, max).

OpenVLHarness achieves a 33.45 overall average, surpassing Pixel‑Reasoner (28.02) and AdaReasoner (25.29) on Qwen2.5‑VL‑7B‑Instruct.

Table 9 lists the averages for each training‑based baseline and the harness.

In the 21–50 object bucket for FSC‑147, OpenVLHarness boosts exact‑match accuracy from 5.0 % to 41.3 % on the 8B model, a +36.3 % absolute gain.

Table 10 lists the bucket‑wise accuracies for FSC‑147.

For FSCD‑LVIS, OpenVLHarness raises box‑level F1 from 8.5 % to 73.8 % in the 21–50 bucket on the 8B model, a +65.3 % absolute increase.

Table 10 lists the FSCD‑LVIS F1 scores.

OpenVLHarness raises LVIS AP from 31.24 to 65.51 on the 8B model, a +34.27 % absolute improvement.

Table 11 reports the LVIS AP values for each model.

OpenVLHarness improves GPT‑5’s average from 31.58 to 41.80 across 11 benchmarks, a +10.22 % absolute gain.

Table 12 lists the per‑benchmark averages for GPT‑5.

At medium reasoning effort, OpenVLHarness scores 56.52, exceeding every other method by at least 7.68 points.

Table 13 reports domain averages for all methods at each effort level.

The ablation study in Table 3 isolates each harness component by adding one mechanism at a time, allowing the marginal contribution of raw tool access, the capability layer, textual memory, and multimodal memory to be measured.

**Table 3.** Ablation study of OpenVLHarness across model scales. We progressively add raw tool access, capability layer, textual memory, and multimodal memory to the no-tool baseline. Results are reported across four domains and overall for Qwen3-VL 8B and 32B models, averaged over three runs; standard deviations are provided in Table 27. Bold marks the best result in each column.

**Table.** Performance comparison of Qwen3-VL-8B and Qwen3-VL-32B models across different domains and datasets using Base, Codex, and OpenVLHarness evaluation methods.

The reasoning‑effort study in Tables 14–16 evaluates how increasing token budgets affects each method, measuring cost, token usage, and per‑benchmark scores.

Implementation Details

This section documents the harness of Section 2: the nine capabilities of Section 2.1, how their outputs are kept in multimodal memory, and the agent’s system prompt.

Capabilities and backends. Following the generalized capability abstraction of Section 2.1, each capability exposes a fixed interface to the agent: typed arguments in, and a natural‑language message plus memory artifacts out. The expert model or service behind the interface is a configurable backend, and switching it changes neither the agent’s prompt, the tool schema, nor the memory interface. Table 18 lists the capabilities and their backends; all experiments in this paper use the default backend of each capability, shown in bold.

Interface Backends Capability

Search - Text Search: text query → summarized results with cited URLs - Image Search: image → visual matches (entities, URLs)

Serper; Gemini with Google Search Serper (Google Lens); Gemini with Google Search Serper with an LLM extractor

Webpage Visit: URLs and goal → goal‑relevant evidence

Perception - Zoom‑in - Visual Grounding - OCR - Depth Estimation - Camera Trajectory

image and box → cropped image image and text query → boxes, masks, annotated image image → transcribed text PaddleOCR‑VL; DeepSeek‑OCR image → depth map Depth Anything 3; VGGT images → camera poses and trajectory plot Depth Anything 3; VGGT built‑in SAM 3; Grounding DINO

Compositional computation - Python Coding Agent: objective → program output and new bindings single‑shot code generator; full coding agent

Table 18: The nine capabilities, their interfaces, and their interchangeable backends. Bold marks the backend used in all experiments.

C.1 SEARCH TOOLS

By default, all three search tools route queries through the Serper API; Text Search and Image Search can instead be served by Gemini (Comanici et al., 2025) with Google Search grounding, a costlier but stronger route. The search tools post‑process the raw payload with a Qwen3‑VL‑8B‑Instruct model as the VLM summarizer. The summarizer runs at decoding temperature 0 with a 4096‑token output cap, and each call retrieves the top 10 raw results before reranking and summarization.

Text Search. Search the web with intelligent query routing and VLM summarization. Given a text query, the tool automatically classifies the query intent and routes it to the most appropriate Serper endpoint, which has general web, news, academic papers, images, maps, local places, shopping, or patents types, and returns a coherent summary of the top‑10 raw results with inline citations [refs] followed by a citations block of source titles and URLs. This format supports multi‑round search: the agent can read the summary for quick understanding and then visit specific cited URLs through Webpage Visit. The summarizer prompt is shown below.

Text Search summarizer prompt (abridged)

You are a research summarizer. Given raw web search results and a user query, do exactly three things: 1. RERANK the numbered organic results by how directly each helps answer the query. 2. KEEP at most K most useful results; emit their original 1‑based ranks as kept indices. If none help, emit kept indices: []. 3. SUMMARIZE the query‑relevant facts from the FULL set of raw results into a concise 3–5 sentence prose paragraph; include named entities, numbers, and dates verbatim. Knowledge‑graph, answer‑box, and top‑stories blocks are ineligible for kept indices but facts should still surface in summary.

Output STRICT JSON with {kept indices, summary, no results justification, the suggested queries, suggested next actions}; when no result is useful, populate the justification and propose alternative queries or downstream tool calls.

Image Search. Identify the objects, landmarks, artwork, text, products, plants, animals, and other entities depicted in an image, using Serper’s reverse‑image‑search (Google Lens) endpoint. Returns a list of visual matches with titles, source URLs, links, and thumbnails. The agent invokes the tool by passing the runtime identifier of the target image. The tool performs reverse search (image → information) on a single image per call; finding images from a text description is instead handled by Text Search with an image‑search intent. A separate multimodal summarizer prompt (below) reranks the candidates against the query image and produces the same JSON envelope as Text Search.

Image Search summarizer prompt (abridged)

You are a visual‑match summarizer. Given raw reverse‑image‑search results (and, in the multimodal variant, the query image plus each candidate’s thumbnail), do exactly three things: 1. RERANK the numbered matches by how likely each identifies the same subject as the query image. 2. KEEP at most K most useful matches as kept indices. 3. SUMMARIZE what the FULL set of raw matches indicates about the query image’s likely subject in a concise 3–5 sentence prose paragraph; capture entity names, dates, attributes verbatim.

Output STRICT JSON with {kept indices, summary, no results justification, suggested next actions}; when no match is useful, justify per‑match and propose follow‑up actions such as cropping with Visual Grounding before re‑searching, or switching to Text Search.

Webpage Visit. Visit one or more webpages, convert them to plain text via Serper APIs, and extract goal‑relevant content using a local Qwen3‑VL‑8B‑Instruct model. This tool returns per‑URL evidence and a summary. The extraction prompt is shown below.

Webpage Visit extraction prompt

Please process the following webpage content and user goal to extract relevant information: Webpage Content: {webpage content} User Goal: {goal} Task Guidelines 1. Content scanning for rationale: locate the specific sections directly related to the user’s goal. 2. Key extraction for evidence: extract the most relevant information; preserve the full original context whenever possible. 3. Summary output: organize the result into a concise paragraph with logical flow that connects the evidence back to the goal.

Output: a single JSON object with keys rational, evidence, and summary (each a string; empty string if no content is available). No code fences, no Markdown, no extra keys.

C.2 PERCEPTION TOOLS

Zoom‑in. Crop a region of an image given a normalized bounding box on the [0, 1000] scale plus padding, and return the cropped image. Used for closer visual inspection after the agent has already localized a region of interest.

Visual Grounding. Locate and highlight regions of an image that match a text query, backed by SAM 3 (Carion et al., 2025) by default or by Grounding DINO (Liu et al., 2024). It returns an annotated image with matched regions highlighted and numbered, along with bounding boxes (and segmentation masks when the backend provides them).

OCR. Extract readable text from an image, backed by PaddleOCR‑VL (PaddlePaddle Team, 2025) by default or by DeepSeek‑OCR (Wei et al., 2025a). Returns the transcribed text as a single string. Used when the task is about words, numbers, or printed labels in the image; not for general object localization or segmentation.

Depth Estimation. Estimate a per‑pixel depth map for an input image, backed by Depth Anything 3 (Lin et al., 2025) or VGGT (Wang et al., 2025b). Returns a depth image (closer surfaces brighter, farther darker). Used when the task requires reasoning about relative distance, foreground‑background separation, or 3D scene structure cues that flat 2D crops cannot resolve.

Camera Trajectory. Given an ordered sequence of at least two images of the same scene, estimate each camera’s pose relative to the first image using Depth Anything 3 (Lin et al., 2025) or VGGT (Wang et al., 2025b), and translate the result into a form the model can reason over directly. Rather than returning raw geometry alone, the tool reports, for every image, a plain‑language summary of translation and rotation with respect to the reference frame (e.g. “salient direction: backward (‑5.43); detailed: left (‑0.20), down (+0.28), backward (‑5.43)”), the accompanying scalars (camera‑center offset, forward direction, yaw, pitch), an explicit statement of the coordinate convention (+X right, +Y down, +Z forward, with positive yaw turning right), and a bird’s‑eye‑view rendering of the camera positions on the reference frame’s X–Z plane. Per‑image intrinsics and world‑to‑camera extrinsics are returned as well for downstream computation.

Translations below one meter and rotations below ten degrees are reported as non‑salient. Used when a question depends on how the viewpoint moved or rotated across frames, or on multi‑view scene structure that no single image conveys; the estimate is metric only up to the model’s calibration and requires sufficient scene overlap between the reference image and the other views.

C.3 PYTHON CODING AGENT

Python Coding Agent. Run Python over the agent’s local computation state and the outputs of previously called tools (bounding boxes, OCR strings, segmentation masks, and so on), returning both standard output and the resulting local variables. The agent describes a computation goal in plain language, and by default a single‑shot code‑generation session of the same backbone model as the orchestrator writes the snippet using prior tool outputs as named variables before it is executed; the same interface can instead be backed by a full coding agent that inspects and revises its own code before returning. Used for aggregating numbers extracted by OCR, filtering detected regions by bounding‑box area, or any cross‑tool numerical reasoning the agent cannot complete in its own response.

C.4 MULTIMODAL MEMORY ARTIFACTS

Table 19 summarizes how each tool’s output is retained in the explicit multimodal memory of Section 2.2 and the handle by which the agent re‑addresses it on later turns.

Artifact Source tool Stored representation in memory Reusable handle

Image artifacts (returned to the model as a native image attachment + an ID) Input image Cropped region Registered verbatim

Cropped image, deduplicated by content Zoom‑in Grounded / mask overlay Colorized depth map

Visual Grounding Depth Estimation Camera‑trajectory plot Camera

Retrieved thumbnails Image / Text Search Annotated image with numbered regions Rendered depth visualization

Bird’s‑eye‑view render of the estimated camera path Image bytes fetched lazily on first reuse Structured data bindings (kept out of the transcript; exposed as typed, named variables) Pixel/normalized boxes, confidences, labels, boxes, scores, labels, image size, per‑region masks + scores + counts

Segmentation masks input image n tool generated image n tool generated image n ... (repeated for each generated image)

named binding (e.g. grounding query) named binding Visual Grounding Visual Grounding Depth Estimation Camera Trajectory OCR

Metric depth array Camera poses OCR results Per‑pixel depth (float array) + shape + colormap Per‑image intrinsic and extrinsic matrices Recognized text, per‑token boxes, confidences named binding

Web references (URLs replaced by short tokens in retrieved text) Text / Image Search‑result pages Search URL + domain + provenance webpage url n

Code outputs (returned as text; emitted images are registered as image artifacts) Python Coding Agent text + image handle(s)

Table 19: Tool‑generated artifacts retained in persistent multimodal memory, grouped by how they are represented and re‑addressed. Heavy payloads (image bytes, numeric arrays) are stored once and replaced in the transcript by a compact handle; the orchestrator reuses an artifact on a later turn by citing its handle, which the harness resolves back to the underlying data.

Environment Update message (example)

Environment Update (not a new question): Currently available image IDs for tool calls: - input image 1: user input - tool generated image 1: tool output: Visual Grounding Tool - tool generated image 2: tool output: Visual Grounding Tool When a tool needs image input, pass one of these IDs via the *image id* field. Do NOT invent image file names or paths.

Available data bindings (auto‑injected into Python Coding Agent Tool): - segment finger food: dict (keys: pred boxes, pred scores, pred masks, num masks, orig img h, orig img w) - segment small pastry with filling: dict (keys: pred boxes, pred scores, pred masks, num masks, orig img h, orig img w)

C.5 SYSTEM PROMPT

System Prompt

You are a visual reasoning agent that answers questions about images using a structured Perception‑Reasoning‑Action framework. Your goal is to provide accurate, well‑reasoned answers. You have access to various tools that compensate for known limitations of vision‑language models. Use native tool‑calling for tools. Do not write textual Action/Action Input to call tools. At most one native tool call is allowed per assistant message/turn. Use the following format: Question: the input question you must answer Observation: your direct perception of the input or the result of the action you took Thought: you should always think about what to do Action: the action to take, should be one of the provided tools, or “Final Answer” if you have a final answer and do not need to call any more tools. Action Input: the input to the action, or the final answer if Action is “Final Answer”

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers