Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

Matteo Santelmo, Xiuying Wei, Israa Fakih, Felix Bauer, Juan Garcia Giraldo, Chengkun Li, Etienne Bamas, Emmanuel Abbé

A diagnostic benchmark of 235 human-trivial tasks that expose persistent reasoning and perception blind spots in frontier AI models.

How do current multimodal models perform on tasks that are trivial for humans but systematically ignored by standard benchmarks?

Modern AI models often saturate standard benchmarks while failing at tasks humans find trivial, such as counting objects in an image or generating text with an exact character count. The authors introduce Blind-Spots-Bench, a curated dataset of 235 problems collected from graduate students, paired with a structured taxonomy and an automated grading pipeline to systematically identify these residual failure modes. Analysis reveals that closed-source frontier models outperform open-weight alternatives by approximately 10%, though no single model dominates across all task categories.

Paper Primer

The benchmark addresses the "saturation" problem where models perform well on broad, aggregate metrics but remain brittle on precise, multi-step, or spatial reasoning tasks. By focusing on naturally occurring failures rather than synthetic perturbations, the authors create a diagnostic stress test that maps model performance to specific sub-tasks like perceptual counting, character-level manipulation, and spatial reasoning.

The evaluation pipeline uses a two-step process: a solver model generates an answer, and a secondary "grader" model (Gemini-3-Flash) verifies the output against a structured reference solution. This framework enables fine-grained analysis across three main categories—Object-centric, Abstract reasoning, and Language and knowledge—allowing researchers to pinpoint exactly where a model's reasoning breaks down.

Closed-source frontier models maintain a consistent performance lead over open-weight models.

Across the benchmark, the best-performing closed-source systems achieve roughly 10% higher accuracy than the highest-scoring open-weight models, even when those models show comparable performance on established benchmarks.

Model scaling and tool use do not guarantee improved performance on these blind spots.

Larger models within the same family sometimes underperform smaller variants, and enabling Python code execution yields mixed results, occasionally decreasing accuracy due to input handling errors.

Why is this benchmark necessary if we already have broad reasoning benchmarks like GPQA or MMMU?

Existing benchmarks aggregate scores across diverse tasks, which masks residual weaknesses in simple but precise skills. Blind-Spots-Bench decomposes reasoning into fine-grained sub-tasks to reveal specific failure modes that aggregate metrics fail to capture.

Does the use of an AI grader introduce bias toward specific model providers?

The authors found no substantial evidence of pro-Google grading bias when using Gemini-3-Flash. While false positives were slightly more likely for non-Google models, the agreement between human and AI grading remained high (96.6% for text, 90.9% for image tasks).

The benchmark highlights that "general intelligence" scores (like the AAII index) correlate with performance on these blind spots, suggesting that these tasks measure fundamental reasoning robustness rather than just adversarial edge cases.

Researchers should use this taxonomy to diagnose specific reasoning bottlenecks rather than relying on aggregate scores, as model robustness on these "trivial" tasks does not scale linearly with model size or cost.

Introduction: The Blind Spot Problem

We expose systematic blind spots where high‑scoring models still fail trivial human tasks.

Despite impressive scores on standard suites, modern AI systems still stumble on tasks that any human can solve in seconds, such as producing a string of exact length or drawing a dog with five legs.

This mismatch suggests that existing evaluations under‑measure systematic reasoning blind spots, motivating a dedicated stress test.

It is a curated collection of problems that are trivial for people but consistently trip current AI models, exposing where their reasoning collapses.

**Figure 1.** **Left:** Accuracy on blind-spots-bench vs. Artificial Analysis Intelligence Index score for text-only problems. **Right:** Performance of four VLM models on several sub-tasks.

The gap between aggregate benchmark scores and performance on human‑trivial tasks reveals a fundamental disconnect in current model evaluation.

Benchmark Design and Taxonomy

How the Blind‑Spots‑Bench was assembled, annotated, and vetted.

The benchmark was built from graduate‑course submissions, then cleaned, annotated, and rigorously reviewed to isolate human‑trivial yet model‑failing tasks.

Collect raw questions from 57 students (≈5 per student) → 287 initial items.

Filter duplicates and overly hard or easy items; standardize phrasing.

Annotate each entry with a reference solution, correctness criteria, and observed failure modes.

Label the question format (text‑only, image‑gen, multi‑to‑text) and assign a task‑category/sub‑task from the Capability Taxonomy.

Run a three‑stage manual review: verification, difficulty thresholding, and ambiguity refinement.

Release the final 235‑sample Blind‑Spots‑Bench on Hugging Face.

Duplicate detection finds no identical entry, so the item passes.

Annotators write the exact answer “19 airplanes” and note that models often miscount by ±2.

Format is tagged as multi‑to‑text because the input includes an image and the output is a textual count.

The item is classified under the Object‑centric category, sub‑task “Perceptual counting”.

During review, the item survives the difficulty filter because all human annotators answer correctly within seconds.

This concrete walk‑through shows how a single raw submission becomes a fully annotated benchmark entry.

The taxonomy breaks reasoning problems into three high‑level families—Object‑centric, Abstract reasoning, and Language & knowledge—each further split into concrete sub‑tasks.

How does this Capability Taxonomy differ from the “checklist” taxonomy used in prior NLP work?

Checklist defines coarse capability buckets (e.g., “coreference”, “negation”) without linking them to multimodal sub‑tasks. Our taxonomy explicitly separates visual, abstract, and language families and adds fine‑grained sub‑tasks (spatial reasoning, generative counting) that are essential for vision‑language evaluation.

Spatial reasoning appears 53 times—the most frequent sub‑task—highlighting the persistent difficulty VLMs have with geometric relations.

**Figure 2.** Representative examples from the dataset. While these tasks are generally easy for human, we find that they remain challenging for frontier models. Each example is annotated with its solution (an abbreviated version here), question format, task and sub-task categories, illustrating the diversity of skills evaluated in the benchmark. Full taxonomy and additional examples in Appendix A.

**Figure 3.** Left: Question format composition. Right: Task category composition. Some questions (about 15) involve multiple subtask categories; for these cases, we count one occurrence for each applicable subtask. The bar chart reports the total number of occurrences for each fine-grained task category, grouped into three major categories.

Evaluation Pipeline

How the blind‑spots benchmark is run and measured across many AI systems.

The experiments automate blind‑spots evaluation for a wide spectrum of AI systems, keeping the procedure identical across all model families.

The pipeline first asks a solver model a benchmark question, then hands the raw answer to a separate grader model that decides correct versus incorrect.

How does this two‑stage pipeline differ from a conventional single‑model benchmark?

Standard benchmarks let the model produce a final answer that is judged directly, often with fuzzy human scoring. Here the grader is an independent AI that applies a deterministic binary rule, eliminating subjectivity and enabling large‑scale automation.

The solver model is invoked with a zero‑shot prompt containing only the question.

The model returns the string “12”.

The grader constructs a prompt that embeds the question, the solver’s answer, and the ground‑truth solution, then executes `int("12") == 12`.

The execution succeeds, so the grader outputs the binary verdict “correct”.

This concrete run shows how code execution turns a textual answer into an exact logical check, removing ambiguity that would arise from purely lexical comparison.

Grader Validation samples over 100 solver responses (balanced correct/incorrect) and annotates them manually; the AI grader matches humans 96.6 % of the time on text tasks and 90.9 % on image‑generation tasks.

To probe provider bias, the same validation set is run on Google‑origin models (Gemini, Gemma) and on non‑Google models; false‑positive rates are slightly higher for the latter, but overall agreement remains comparable.

Select a solver model from the curated family list.

Enable thinking mode (if supported) and set maximum output tokens to $32{,}768$.

Run the solver on each blind‑spots‑bench question; retry up to five times on failure.

Feed the question, solver output, and reference solution to the grader (gemini‑3‑flash) with code execution enabled.

Record the binary verdict; for textual tasks repeat the whole process $k=4$ times and average.

Aggregate per‑model accuracy and report alongside model size and modality.

Evaluation loop for a single solver model.

Performance Analysis

Experimental results quantify model performance, cost, and tool‑use trade‑offs.

Closed‑source models achieve higher blind‑spots‑bench accuracy than open‑weight models even when matched for general capability.

Gemini‑3.1‑pro (closed‑source) reaches 66.9 % multimodal accuracy, while the best open‑weight model (Kimi‑K2.5) attains 42.3 %.

**Figure 4.** Accuracy on blind-spots-bench vs. average cost for 100 samples. Colors distinguish model families; models of the same version but different sizes are connected.

**Table 1.** LLM and VLM model results by evaluation subset. Accuracy and output length are reported as mean$_{\pm\text{stderr}}$. Evaluation cost in reported in USD-per-100-samples.

Enabling code execution (tool‑use) improves accuracy for several models but has mixed effects on token consumption. Gemini‑3.1‑pro gains +2.78 % accuracy while reducing output tokens by 199, whereas GPT‑5.4 loses 5.32 % accuracy and cuts tokens by 451.

Closed‑source models outperform open‑weight models on these tasks, even when controlling for general capability.

Task Examples

Concrete dataset examples illustrate the taxonomy and common failure modes.

This appendix showcases representative prompts from the Blind‑Spots‑Bench and walks through the expected correct behavior.

A word search puzzle grid containing various highlighted words.

Scanning each row, the solver extracts “WORD” from the top row.

Continuing left‑to‑right, “SEARCH” appears on the second row, “JOURNEY” on the fourth, and so on.

The final answer list is the ten words above; any deviation is marked wrong.

Success requires exact attribute binding—matching the exact highlighted word strings without adding or omitting entries.

**Solution:** The correct answer is a shortest directed path from A to B together with its total weight. An output is correct if and only if (1) every step follows an existing directed edge in the graph (respecting arrow direction) and (2) the total weight equals the minimum over all valid directed A→B paths.

Starting at $A$, the only outgoing edge is to $F$ (weight 2).

From $F$ the only forward edge is to $I$ (weight 3), accumulating 5.

Continuing $Iarrow D$ (+5), $Darrow E$ (+1), $Earrow C$ (+2), $Carrow G$ (+2), $Garrow B$ (+2) yields total weight 17.

No alternative directed route yields a smaller sum, so this is the shortest path.

Respecting edge direction is essential; ignoring it creates infeasible paths that the model must not propose.

The line’s perpendicular distance to the square’s opposite vertices implies the square’s side length $s$ satisfies $s = 5 \text{cm} + 7 \text{cm} = 12 \text{cm}$.

Area $= s^2 = 12^2 = 144\;\text{cm}^2$.

A correct answer must both render the geometry accurately and report $144\;\text{cm}^2$.

Failure can occur in two ways: the generated image misplaces $B$ or the computed area is omitted, exposing gaps in attribute binding and geometric reasoning.

Section A.1 analyses the taxonomy of task types that expose blind spots in modern AI agents.

Object‑centric tasks often trip models on attribute and pattern recognition because they default to memorized statistical patterns rather than grounding visual content.

Abstract‑reasoning tasks reveal a cascade of errors: each additional inference step compounds the model’s greedy local decisions, leading to incorrect final answers.

Language‑and‑knowledge tasks suffer from tokenization limits and spurious correlations, causing failures in exact character‑level control and robustness to irrelevant input changes.

Grader Validation

Even top‑performing models miss simple human‑trivial tasks, exposing systematic blind spots.

We validate the automatic grader by comparing its correctness judgments against human annotations on the Blind‑Spots‑Bench, reporting agreement metrics for both text and image outputs.

**Table 5.** Pipeline Validation Results: agreement scores between human-annotated correctness scores and values obtained with the automatic evaluation pipeline.

Grader Implementation

Exact prompts used to grade text‑ and image‑generation answers.

This appendix lists the full grading prompts that the automatic grader receives for both textual and visual outputs. Each prompt specifies the required format of the grader’s response.

Prompt for evaluating text‑generation answers

Prompt for evaluating image‑generation answers

Taxonomy Details

Provides the capability taxonomy and validation metrics for the evaluation pipeline.

Table 4 enumerates the capability dimensions used throughout the Blind‑Spots‑Bench, grouping concrete sub‑tasks under four high‑level categories.

Tasks that require perceiving, reasoning about, or generating visual objects and their attributes.

Tasks that demand a directed chain of logical inferences.

Tasks that involve non‑visual symbolic manipulation, geometry, or constraint satisfaction.

Tasks that test textual robustness, fine‑grained character handling, and factual/world knowledge.

Table 5 reports agreement between the automatic evaluation pipeline and human annotators for the two output modalities.

Detailed Performance Metrics

Full tables of grader validation, model metadata, and per‑task accuracies.

This appendix reproduces the quantitative tables referenced in the main analysis, allowing readers to inspect the raw numbers behind the aggregated scores.

Table 7 lists each Solver Model’s data‑type, GPU count, output‑price per 1 M tokens, and, when available, parameter count and release date.

Table 8 reports per‑subtask accuracy for every model; GEN models dominate the “3‑1” subtask with perfect scores, while LM and VLM families show more variance.

Table 9 aggregates performance across three high‑level task groups. VLMs such as gemini‑3.1‑pro‑preview achieve the highest scores on Object‑centric, Abstract Reasoning, and Language‑and‑Knowledge categories.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers