An Exam for Active Observers

Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger

Frontier MLLMs fail to perform active visual observation, a gap that persists despite increased reasoning effort or tool use.

Do current frontier MLLMs possess the "active vision" capability to solve tasks requiring iterative, gaze-like visual exploration, or do they fail when a single glance is insufficient?

Multimodal large language models (MLLMs) treat images as static, one-pass inputs, yet many real-world tasks require iterative, hypothesis-driven inspection—fixating, predicting, and returning to the pixels. ActiveVision is a benchmark of 17 tasks designed to force this iterative "active observation" by creating visual configurations that exceed the information capacity of any single language description. Frontier models collapse on these tasks, solving only 3.5–10.6% of items compared to the 96.1% average achieved by human participants.

Paper Primer

The benchmark isolates three elemental visual operations: distributed scanning of local signals, sequential traversal along connected structures, and fine-grained comparison across regions. Each task is generated from a procedural scaffold and rendered photorealistically, ensuring that models cannot rely on cartoon-like simplicity or linguistic shortcuts to solve the problem.

Frontier MLLMs lack robust active visual observation.

Across 85 items, the best-performing model (GPT-5.5) solved only 10.6% of tasks, while Claude Fable 5 solved 3.5%. Humans outperformed the best model by a factor of approximately nine, averaging 96.1% accuracy.

Increasing reasoning effort—scaling the model's internal deliberation steps—yields negligible gains, suggesting the failure is a perceptual bottleneck rather than a lack of reasoning capacity. Even when models are given autonomous coding tools to segment or measure the image, they remain unreliable because they lack the active perception required to verify if their own code has failed.

Why can't models simply use code to solve these visual tasks?

While code helps with tasks like template matching, it fails on realistic imagery where textures, shadows, and noise cause algorithms to misidentify objects. Because the models lack active perception, they cannot audit these tool failures and often submit incorrect answers based on flawed code outputs.

Is this benchmark just testing how well models can "see" or how well they can "reason"?

It tests the perception-reasoning loop. The tasks are designed so that the visual state is too complex to be captured in a single linguistic summary, forcing the model to return to the image repeatedly as its hypothesis evolves—the core definition of active observation.

ActiveVision establishes that current MLLM architectures are passive perceivers; future progress requires training objectives and architectures that explicitly support iterative, hypothesis-driven visual inspection.

Introduction: The Active Vision Gap

Active observation is missing from current multimodal models, creating a gap between static training and iterative visual reasoning.

Human vision continuously redirects gaze based on intermediate hypotheses, forming a closed‑loop of active observation. Decades of psychophysics argue that this iterative looking is essential for many tasks, yet current multimodal large language models (MLLMs) are evaluated only on static, single‑glance benchmarks.

Active observation means repeatedly returning to visual input, forming a hypothesis, checking it against the image, and updating the hypothesis—much like a person scans, predicts, and re‑examines a scene until the answer is certain.

The model receives the image, encodes it once into a fixed token sequence, and directly generates the answer “0”.

No further visual queries are issued, so the model never inspects the individual regions.

The predicted answer (0) is compared to the ground truth (4), yielding a failure.

This failure illustrates the gap: without an active‑observation loop the model cannot perform the exhaustive scanning required to count separated regions.

**Figure 1.** Seeing is not always a single-glance task. Many visual problems require an observer to coordinate attention and memory across an image: scanning exhaustively for distributed evidence, following connected structures without losing track, and comparing fine-grained attributes across distant regions. ActiveVision turns these abilities into a controlled test of whether MLLMs can keep visual evidence in the reasoning loop.

The core problem is the disconnect between static MLLM training and the iterative nature of human vision.

Related Work

We situate ActiveVision among prior active‑vision theory, perception benchmarks, and application‑driven datasets.

ActiveVision builds on three research strands: the cognitive‑science claim that perception is intrinsically active, evidence that frontier MLLMs lack basic visual perception, and benchmarks that stress vision rather than language.

Eye‑movement studies showed the same painting yields different scanpaths under different task prompts, establishing that gaze is allocated by the task rather than the image alone; subsequent computer‑vision work formalized that active sensor control turns ill‑posed inverse problems (shape from shading, structure from motion, optical flow) into well‑posed ones.

Animate vision argued for computational economy via gaze‑targeted local processing, and the sensorimotor account pushed the strongest version: seeing is mastering how visual input changes under one’s own movements, a view supported by everyday eye‑tracking studies.

ActiveVision operationalizes that prediction for multimodal large language models, providing an explicit active‑observation loop that frontier MLLMs currently lack.

Early vision‑centric benchmarks (MMVP, MMStar, CV‑Bench) stripped away linguistic shortcuts so models had to read the image; these suites are now largely saturated, showing that MLLMs can handle isolated perception when the task is purely visual.

Human‑model‑gap benchmarks (BLINK, BlindTest, ArtQA, BabyVision, ChildBench, KidGym) expose the remaining perceptual gap by posing tasks that even a sighted person never misses, yet they do not isolate which capability is missing.

Application‑driven suites (OSWorld, BenchCAD) combine coding, reasoning, and perception, making failure attribution ambiguous; ActiveVision isolates the active‑observation capability that those tasks implicitly require.

The ActiveVision Benchmark

ActiveVision defines 17 multi‑glance visual tasks and a photorealistic evaluation pipeline.

To probe the limits of frontier MLLMs we built ActiveVision, a benchmark that forces models to revisit the image repeatedly rather than rely on a single glance.

ActiveVision assembles a suite of 17 visual puzzles that can only be solved by iteratively extracting information from the image.

How does ActiveVision differ from conventional static‑image benchmarks?

Standard benchmarks present a single image and expect a one‑shot answer; ActiveVision deliberately embeds information that cannot be compressed into a brief textual summary, so solving a task requires repeatedly consulting the visual input.

**Figure 2.** Overview of all 17 tasks in ActiveVision, grouped into three task families that probe distinct dimensions of active observation. For each task, we show one photorealistic instance produced by the rendering pipeline described in §3.3, together with a brief description of the required operation.

Deterministically generate a procedural scaffold (e.g., a set of random loops) using a Python script.

Attach the exact ground‑truth answer (count, path order, attribute value) to the scaffold.

Compose a task‑specific GPT‑image‑2 prompt that maps the scaffold primitives to a real‑world scene.

Render the prompt with GPT‑image‑2, producing a photorealistic image while preserving geometry and counts.

The Python generator samples 10 random center points and 8 random control offsets per loop, producing a vector of 80 control coordinates.

Ground‑truth count “10” is stored alongside the scaffold.

A GPT‑image‑2 prompt “render yellow ropes tangled on a dark tabletop, preserving the exact number of loops” is sent to the model.

The rendered image shows ten visually distinct rope loops; no loop is merged or omitted.

The evaluation system presents the image and the question “How many loops are present?” to the MLLM.

This concrete run shows how arbitrary positions, shapes, and traces combine to create a visual state that cannot be summarized in a short sentence, forcing the model to count directly from the image.

**Figure 3.** The four-stage generation pipeline, illustrated on *Tangled Loop Counting*. A deterministic Python generator (1) emits the geometric scaffold (2): random smooth closed curves with full ground truth. A task-specific GPT-image-2 prompt (3) re-renders the scaffold into a photorealistic image (4) without altering positions, counts, or topology. The benchmark question and ground-truth answer apply unchanged to both renderings; only the photorealistic image is served to the model at evaluation time.

Table 1 enumerates all 17 generators with one‑line descriptions; we sample five seeds per generator (N = 5) to obtain a fixed 85‑item evaluation split. Models receive the rendered image and the exact question, and we score exact‑match accuracy after normalizing case and punctuation.

Benchmark Performance Results

Frontier models fall far short of human performance on ActiveVision.

Frontier MLLMs achieve at most 10.6 % exact‑match accuracy on ActiveVision, far below human performance.

GPT‑5.5 solves 9 of 85 items (10.6 %) while the human baseline averages 81.7/85 (96.1 %).

Human participants solve almost every task on ActiveVision, providing an upper bound for visual reasoning performance.

All models were evaluated on the same 15‑task ActiveVision suite, using each model’s highest‑effort configuration and the identical exact‑match scoring protocol. The six frontier models exhibit only weakly overlapping success sets, with no single item solved by every model.

**Table 2.** Per-task exact-match accuracy on ActiveVision. Model entries report correct responses out of five. Human entries report the mean across three participants on the same scale (e.g., 14/15 is reported as 4.7/5). The Overall row reports scores out of 85; individual human accuracies are 97.6%, 96.5%, and 94.1%.

**Figure a.** Per-agent accuracy: even the strongest agent (Fable 5, 50.6%) stays far below the human band.

Failure Analysis: Reasoning Effort

Analyzing why frontier MLLMs still falter despite more reasoning steps.

The paper’s core claim is that frontier MLLMs process an image in a single glance and lack active observation. This section revisits that premise before dissecting where the models break down.

Reasoning effort quantifies how many internal “think‑then‑act” cycles a model performs before emitting an answer.

How is “Reasoning Effort” measured here compared to a simple step count?

It is measured by the maximum allowed chain‑of‑thought depth: a model may generate up to N intermediate reasoning statements before producing the final answer, where N is the tier’s ceiling (e.g., 30 for xhigh).

Increasing reasoning effort yields diminishing returns: scaling from none to xhigh raises cost roughly 100× but improves accuracy only from 2.4 % to 10.6 % (≈8 % accuracy gain).

Figure 4 plots each model’s accuracy against its per‑item API cost, showing the shallow slope despite large cost increases.

**Figure 4.** Accuracy versus API cost per item (log scale, public list prices) on the 85-item ActiveVision split. Points are complete pure-CoT runs, with marker size encoding the reasoning-effort tier; lines connect runs of the same model. The dashed line and band are the human mean and range (N=3).

Counting tasks expose a conservative bias: when scenes become crowded, models leave an increasing share of objects uncounted, as if they only glimpse the image once.

**Figure 5:** Predicted versus ground-truth counts across eight counting tasks at each model's highest reasoning effort. Solid lines are least-squares fits; gray dashed lines show the identity $y = x$. Slopes below one reveal increasing undercounting as the true count grows.

Ordered‑walk tracing fails immediately: the share of walks whose first k steps are all correct collapses after the first two moves, and no walk is ever completed.

**Figure 6.** Prefix survival on three ordered-walk tasks (over 18 pure-CoT runs): the share of walks whose first $k$ steps are all correct. Dashed line: shortest walk length ($k=6$); no walk is completed exactly.

On difference‑detection tasks, models overwhelmingly answer “none,” yielding very high miss rates while keeping false‑alarm rates low—a classic response bias toward safety.

**Table 3.** Per-run miss and false-alarm rates on the difference tasks. Highlighted runs answer “none” on nearly every item, missing all real differences.

Can Tool Use Substitute for Vision?

Tool‑use agents replace direct sight with code but still lag humans.

Frontier MLLMs process an image in a single glance and cannot iteratively steer attention; this section asks whether a coding tool can stand in for that missing active observation.

Instead of the model inspecting pixels itself, it writes a short program that manipulates the image data (e.g., thresholding, flood‑fill) and returns a numeric answer, swapping perception for computation.

How does Tool Use Substitution differ from simply giving the model a larger vision backbone?

A bigger backbone expands the model’s internal visual encoder but still relies on the model’s own perception. Tool Use Substitution hands the image to an external algorithm; the model never “sees” the pixels, it only consumes the algorithm’s output.

**Figure (b).** Per-task accuracy (correct out of 5): attribute-transfer tasks largely yield to code, the traversal family resists every agent except Fable 5, and Tangled Loop Counting resists all three agents completely.

Tool‑using agents close a large portion of the human gap but still fall far short on complex visual reasoning.

Claude Code (Fable 5) solves 43/85 items (50.6 %), Codex (GPT‑5.5) 32/85 (37.6 %), Opus 4.8 21/85 (24.7 %); the best tool‑free model reaches only 10.6 %.

**Figure 2.** Failure 2. Weak visual perception prevents the agent from correcting its tool-call errors.

Discussion and Limitations

Frontier MLLMs see only a single glance and cannot perform active observation.

Table 4 shows that each agent item costs between $2.74 and \$7.63 of compute and takes 12–15 minutes, roughly 25 times longer than the half‑minute a human needs, while the best agent solves only about half the benchmark and humans answer 96.1 % correctly without any tools.

The provided image contains a table comparing performance metrics across different tracks (Codex GPT-5.5, Claude Code Opus 4.8, Claude Code Fable 5, and Human). The columns include Accuracy (%), Time/item (min), API cost/item ($), Tool calls per item, Output tokens per item, and Agent turns per item.

Tool‑use runs reveal why active perception matters: classic CV primitives such as findContours, Canny, and template matching, which work on clean inputs, fragment or merge objects on the benchmark’s photorealistic images, leading scripts to return confident but wrong answers.

A human can glance back, notice that two masks belong to the same shape or that a dashed route continues beyond the tracer’s stop, whereas the script lacks this iterative verification and thus depends on active perception for robustness.

The agentic ablation treats visual problem solving as a sequence of checkable operations: the agent writes extraction code for noisy imagery, runs it, and then performs a lightweight visual check, offloading most reasoning to code.

Even with this offloading, the strongest agent remains far less accurate and efficient than an unaided human; attribute‑transfer tasks are solved, traversal tasks are near zero, and all agents fail the Tangled Loop Counting track.

Our benchmark uses synthetic, photorealistically re‑rendered images rather than natural photographs, trading realism for exact ground truth and controlled task structure; consequently, external validity hinges on the elemental visual operations—scanning, tracing, comparing—rather than on the renderings themselves.

Tasks are deliberately constructed so that no short language description contains the answer, a property that may erode as models improve at describing images and will need tightening in future benchmark versions.

ActiveVision quantifies a wide human‑machine perception gap and shows why it matters: exhaustive scanning underlies radiology, cell counting, inventory inspection, and aerial search; sequential tracing is essential for connectomics and vessel mapping; fine‑grained comparison drives latent‑print examination, pathology, and industrial quality control.

Eye‑tracking studies across these professions reveal that expertise is largely about knowing where and how to look; a model that cannot count, trace, or compare reliably in a controlled image has no path to reliability in these settings.

Closing this gap is therefore a prerequisite for building reliable Multimodal Large Language Models and the multimodal agents that depend on them.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers