SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

Abhigya Verma, Khyati Mahajan, Amit Kumar Saha, Shruthan Radhakrishna, Sagar Davasam, Vikas Yadav, Sai Rajeswar Mudumba

A synthetic benchmark for long-context visual document understanding that isolates failure modes in frontier vision-language models.

How do current Vision Language Models perform on long-context, multi-page documents that require complex reasoning across charts and text?

Frontier vision-language models (VLMs) perform well on isolated document tasks, but real-world documents interleave text, charts, and complex layouts across dozens of pages, making it impossible to diagnose why a model fails. SYNTHDOCBENCH uses a fully synthetic, combinatorial generation pipeline to create documents where length, layout, modality, and question difficulty are varied as independent, controlled axes. The benchmark reveals that even top-tier models suffer from sharp performance degradation as evidence complexity increases, systematic positional bias, and a collapse in precise chart-reading accuracy in long-context settings.

Paper Primer

The benchmark employs a dual-layer generation strategy: every document is rendered as a visual PDF while simultaneously maintaining a hidden, structured metadata manifest. This allows for deterministic ground-truth derivation without human annotation, enabling the systematic isolation of reasoning bottlenecks that are otherwise confounded in real-world document corpora.

Frontier VLMs exhibit a systematic "middle-document" performance collapse.

Across eight models, the middle third of a document is the hardest to process, with five models showing a negative Early-to-Late performance trend and a steepest decline of 8.3 percentage points.

Visual perception, rather than reasoning, is the primary bottleneck for complex document tasks.

An OCR-only baseline achieves 0.798 accuracy on complex multi-hop questions, while vision-only models achieve only 0.360, confirming that complex evidence is largely text-recoverable but visually opaque.

Why is a synthetic benchmark necessary for this domain?

Real-world documents contain co-varying factors like layout density and answer depth that cannot be disentangled; synthetic control allows researchers to attribute model failures to specific, isolated causes.

How does this benchmark differ from existing long-context evaluations?

Existing benchmarks prioritize breadth of coverage over diagnostic decomposition, whereas SYNTHDOCBENCH explicitly varies document length, page depth, modality, and question type as independent axes to probe specific reasoning primitives.

Introduction and Motivation

We expose why existing visual‑document benchmarks miss key failure modes in long‑context VLMs.

Visual language models excel on single‑page VQA and chart‑QA benchmarks, yet real documents intertwine text, tables, and charts across dozens of pages, making it impossible to pinpoint why a model fails.

SynthDocBench deliberately isolates each difficulty—length, layout, modality, and question complexity—so we can attribute a model’s error to a single cause instead of a tangled bundle.

Compute ACC on early pages → 85 %.

Compute ACC on middle pages → 76.7 % (8.3 pp lower than early).

Compute ACC on late pages → 82 %.

Observe a U‑shaped accuracy curve, with the middle segment hardest for five of six models.

This positional sensitivity shows that models struggle to maintain context continuity across the document’s central region, a failure invisible on single‑page benchmarks.

**Figure 1.** Landscape of benchmarks Top: benchmark comparison by average document length (pages) and average textual context (tokens). SynthDocBench occupies a unique region with both long multi-page context and high textual density. Bottom: comparison with existing chart and document VQA benchmarks. Prior benchmarks typically isolate either charts or long documents, whereas SynthDocBench is designed to study their intersection under long contexts.

The gap between single‑page VQA performance and real‑world multi‑page document complexity remains unmeasured by existing benchmarks.

Related Work

We position SynthDocBench among prior chart and long‑document benchmarks.

Chart understanding has been studied largely in isolation, with benchmarks such as ChartQA, ChartQAPro, ChartGalaxy, VisuLogic, ChartMuseum, and MultiChartQA advancing chart‑oriented VQA but never embedding charts in full‑document contexts.

Long‑context multimodal benchmarks have progressed from simple retrieval tasks to full‑document QA (DocVQA, MP‑DocVQA, SlideVQA) and then to extensive token‑level challenges (MMLongBench‑Doc, MMLongBench, LongDocURL, M‑LongDoc), yet they still lack a diagnostic focus on chart and figure reasoning across many pages. Vision‑Language Models such as Qwen3‑VL and InternVL3.5 now saturate single‑page QA, while performance on long‑document suites remains far below, highlighting the need for a benchmark like SYNTHDOCBENCH that varies document length, modality composition, and question type in a controlled synthetic setting.

Benchmark Generation Pipeline

Describes how synthetic documents, QA pairs, and evaluation are constructed.

The benchmark is built in three tightly coupled stages: synthetic document generation, deterministic QA creation, and vision‑only evaluation on rendered pages. This design isolates reasoning performance without relying on noisy human annotations.

The pipeline turns a topic seed into a fully styled visual report and a machine‑readable QA manifest, guaranteeing that every question has a known ground truth.

Content generation extracts three paragraphs from a climate wiki page and creates a structured outline with a “Results” table.

Design samples the “Editorial” archetype, placing the chart on page 3 and a pull‑quote on page 2.

Visualization synthesis renders the CO₂ line chart (D3.js) and stores $V_{1}$ = {x‑axis: year, y‑axis: ppm, points: [(1990, 354), …, (2020, 415)]}.

Validation recomputes the chart’s max value (415 ppm) and inserts it into the manifest $M$; Playwright renders the final PDF.

This concrete run shows how every visual element is paired with exact numeric metadata, enabling the benchmark to generate provably correct answers.

Generate a synthetic visual report $D$ and its QA manifest $M$ from a topic seed.

Parse $D$, recover evidence from $M$, and synthesize difficulty‑controlled QA pairs.

Render each page of $D$ to an image and evaluate VLMs under the Vision‑Only Protocol.

**Figure 2.** Synthetic visual document generation pipeline. From a topic seed, the pipeline generates grounded report content, applies document-level visual styling, synthesizes visualizations, performs metadata and QA validation, and assembles the final HTML/PDF reports with a machine-readable QA manifest.

The QA generation stage parses the rendered report into text, table, and visualization channels, recovers chart semantics directly from $V_k$, and composes multi‑hop questions that span 2–4 evidence units. Three families are produced: chart‑reading, cross‑modal, and complex multi‑hop, each labeled with difficulty levels L1–L5 (see Table 13).

**Table 1.** Per-document statistics across all 200 reports.

The full benchmark contains 200 reports and 1 788 questions (597 chart‑reading, 597 complex multi‑hop, 594 cross‑modal). Documents average 51.1 pages and 16.7 charts, placing the suite firmly in the long‑document regime.

**Figure 4.** Document composition statistics across 200 reports. (a) Document property distributions (pages, words, charts). (b) Layout archetypes

**Table 13.** Difficulty taxonomy for SYNTHDOCBENCH

Evaluation Protocol

We detail the vision‑only pipeline, scoring rubric, and experimental configuration.

We evaluate VLMs under a strict vision‑only protocol using the pipeline shown in Figure 8.

Models receive only the rendered page‑image sequence $I$ and never see HTML, embedded metadata, or extracted text.

Why does prohibiting HTML access matter for testing visual reasoning?

It forces the model to rely solely on pixel information; success therefore reflects genuine visual understanding rather than a text‑lookup shortcut.

Each model answer $\hat{a}$ is scored by a separate judge model that maps the pair $(\hat{a},a^{*})$ to a 0‑10 rubric, and accuracy counts answers with score ≥ $\tau=6$.

What ensures the judge scores are comparable across models?

The judge operates deterministically (temperature 0) with a fixed system prompt, and its high correlation with other judges shows the scores are stable and not model‑specific.

**Figure 8.** Evaluation pipeline. Rendered PDFs are converted to page images at 144 DPI, grouped into concatenated 5-page strips, and supplied directly to candidate models at temperature 0. Candidate answers are then scored against deterministic reference answers by GPT-5 acting as the judge model $\mathcal{J}$.

Rasterize each PDF at 144 DPI, cap at 120 pages, and split into single‑column 5‑page vertical strips (max 7,900 px, ≤ 4 MB).

Feed each strip to the candidate VLM with temperature 0 using a fixed system prompt; obtain answer $\hat{a}=f_{\theta}(I,q)$.

Pass the pair $(\hat{a},a^{*})$ to GPT‑5 (the judge) which returns a JSON score in $[0,10]$.

Discard any response that fails to parse (score −1) from all aggregates.

Compute the mean judge score and ACC = fraction of valid answers with score ≥ $\tau=6$.

Stratify the metrics by question family (Chart, Complex, Cross‑Modal) and difficulty level L1–L5.

The strip is fed to a VLM, which outputs “The bar chart shows a rising trend.”

The judge scores this answer 8.5.

Since 8.5 ≥ 6, the response counts as correct for ACC.

This single query contributes 1 / 1 = 100 % to the ACC numerator for its subset.

This toy run demonstrates how a single end‑to‑end trial flows through rasterization, model inference, judging, and metric aggregation.

We evaluate eight VLMs on SYNTHDOCBENCH; detailed ACC and mean‑score numbers appear in Table 2, while judge‑agreement statistics are reported in Table 9.

Results and Analysis

Key performance shifts and persistent failure modes across models.

Cross‑modal accuracy trails chart‑reading accuracy by 13–16 pp across all eight models.

Figure 5 shows vision‑only ACC = 0.46 vs OCR‑augmented = 0.30 on chart questions, while OCR‑augmented leads on complex questions (0.80 vs 0.36), confirming a systematic gap.

These numbers are derived from the same SynthDocBench evaluation pipeline, using the same judge (GPT‑5) and identical question sets for every model.

**Figure 6.** Hard failures: 109 questions where all six models score $\le 3$, broken down by error category and question subset (Ch. = chart-reading; Cx. = complex; XM. = cross-modal).

**Figure 5.** GPT-4o vision vs. OCR+GPT-4o (text-only) ACC by subset. Vision dominates on Chart; OCR dominates on Complex.

Hard failures are questions where every evaluated model scores ≤ 3, exposing systematic weaknesses that aggregate metrics hide.

Why does “hard failure” focus on the ≤ 3 score threshold instead of a higher cutoff?

Scores ≤ 3 indicate that judges consider the answer essentially incorrect; using a stricter threshold (e.g., ≤ 2) would yield too few examples, while a looser one (≤ 5) would mix borderline cases with genuinely correct answers, diluting the diagnostic power.

Modality alignment is the primary bottleneck for current VLMs on complex documents.

Evaluation Pipeline Details

Describes how SynthDocBench scores VLMs using a vision‑only feed and LLM judges.

The pipeline first converts each synthetic PDF to a 144 DPI image, stacks five pages vertically, and passes the strip to the candidate Vision‑Language Model with temperature 0, i.e., without OCR or metadata.

**Table.** Gemini-as-judge independently replicates GPT-5's ranking with a maximum ACC deviation of 3.5 points.

Model outputs are then judged by GPT‑5 (judge J) using the rubric, producing a numeric score for each of the 1,788 questions.

**Table 7.** Performance of evaluated VLMs on established visual document understanding benchmarks. — = not publicly reported at time of writing.

The table lists six AI models (Gemini-3.1-Pro, GPT-4o, Claude-Sonnet-4.5, Qwen3-VL-235B, InternVL3-78B, and Qwen2.5-VL-7B) and their associated values across five categories labeled L1 through L5.

Comparing SynthDocBench ACC to existing benchmarks yields a moderate Spearman correlation of 0.657, indicating that high scores on legacy datasets do not guarantee success on the controlled suite.

The table provides a list of symbols, their types, and their definitions used in the document generation process. The symbols include $\tau$, $\mathcal{A}$, $a \sim P_{\mathcal{A}}(\tau)$, $\mathcal{D}$, $\mathcal{V}_k$, $\mathcal{T}_k$, $\mathcal{M}$, $q$, $a^*$, $\hat{a}$, $f_{\theta}$, $\mathcal{I}$, $\ell$, and $\mathcal{J}$.

**Table 11.** Bootstrap 95% CI half-widths ($\pm$) for ACC at $\tau=6$ stratified by difficulty level, computed over 2,000 resamples (seed 42).

**Table 12.** ACC ($\tau=6$) by fine-grained question category (numerical companion to Figure 7). Best per row in bold.

These controlled failure modes mirror those seen on real‑document benchmarks: models that falter on chart‑reading or cross‑modal integration in SynthDocBench exhibit analogous weaknesses on MMLongBench‑Doc’s figure‑heavy and multi‑page questions.

Question Taxonomy and Implementation

Implementation details enumerate the taxonomy, layout, metadata, and inference settings supporting reproducibility.

The benchmark groups every question into one of nine fine‑grained categories, each tied to a reasoning capability and a difficulty level $ℓ$.

Table 14 lists the nine categories with concise definitions; it is the reference for the category‑level breakdown shown in Table 12.

Table 13 provides the difficulty taxonomy ($L1$–$L5$), indicating how many evidence hops a question requires regardless of its category.

Table 15 is the full symbol glossary used throughout the benchmark pipeline, covering generation ($τ$), layout archetypes ($A$), and model interfaces ($f_θ$, $I$, $J$).

Table 16 enumerates the six layout archetypes (Magazine, Dashboard, Academic, Editorial, Infographic, Brutalist) that diversify document grammar and visual placement.

Table 17 details the visualization metadata schema (fields such as vizId, chartType, axes, data) that guarantees deterministic QA generation from rendered charts.

Table 18 describes the serialized QA record format, including question text, answer, difficulty $ℓ$, and the category label used for analysis.

Table 20 specifies the fixed inference hyperparameters (resolution 144 DPI, max 120 pages, ACC threshold 6) and the deterministic judge prompt used for scoring.

**Figure 9.** Distribution of judge scores (0–10) per model.

**Figure 11.** Gemini-3.1-Pro accuracy (ACC, $\tau=6$) per topic domain, grouped by question type. Black diamonds = overall accuracy per domain. Dashed line at 0.5. Based on the 57 domain-annotated reports (513 questions).

**Figure 12.** ACC ($\tau=6$) by question subset. Cross-modal questions are consistently hardest across all models, confirming modality alignment as the primary bottleneck.

Qualitative Examples

Qualitative examples expose model errors across chart, cross‑modal, and multi‑hop tasks.

Only two model‑question pairs are fully correct across the four qualitative examples.

Gemini‑3.1‑Pro and Claude‑Sonnet‑4.5 answer Example 1 correctly; every other model misses at least one required statistic.

All models were evaluated with the Vision‑Only Protocol, measuring ACC per question.

**Figure 14.** OpenAI AI Text Classifier Performance on AI Text. The bar shows 26% True Positive (Identified) and 74% False Negative (Missed).

Gemini‑3.1‑Pro reads the gray segment and outputs “74 % False Negative”.

Claude‑Sonnet‑4.5 does the same, yielding the correct answer.

GPT‑4o, Qwen models, and InternVL3 read the blue segment and report 26 % (or other hallucinated values), which is incorrect.

Correct models identify the gray segment; the majority misinterpret the blue segment as the answer.

The ground‑truth drop is 120 → 40 files, a reduction of 80 files (66.7 %).

All six evaluated models produce various incorrect counts (e.g., 200 → 50, 175 → 50, 250 → 150) that do not match the chart.

Claude and InternVL3 refuse to answer despite the visible data.

Models hallucinate plausible‑looking numbers instead of reading the actual bar heights.

**Figure 1.** Distribution of AI-Generated Code in Enterprise Files. The 40–50% bin has 120 files; the 50–60% bin has 40 files.

The correct answer is “Dev 6, Control = 180 min → Treatment = 85 min”.

All six evaluated models return “Not found”, failing to locate the label despite the chart’s clarity.

This failure highlights an inability to map visual labels to textual queries.

Gemini‑3.1‑Pro reports correct Senate and House numbers but misstates all base values for Ireland and Argentina.

Qwen3‑VL‑235B gets the child‑marriage drop and US delta right but swaps Ireland’s and Argentina’s starting scores.

GPT‑4o correctly notes the Congress doubling and US 50‑point drop but gives wrong percentages for the other countries.

InternVL3 captures the House increase and US drop but omits numeric values for Ireland and Argentina.

Claude‑Sonnet‑4.5 and other models provide no answer.

Even when models retrieve some numbers, they frequently mix up base values across related charts.

Conclusion

Conclusion: SYNTHDOCBENCH exposes key weaknesses and outlines next steps

We introduced SYNTHDOCBENCH to probe VLMs’ ability to locate multiple facts across long‑context documents and reason over them. Across six frontier models we observed three failure modes: sharp drops as evidence complexity and reasoning depth increase, a pronounced positional sensitivity where the middle third of a document is hardest, and a collapse of precise chart‑reading accuracy in long contexts.

Future work will broaden the benchmark to richer multimodal settings, adding tables, forms, mixed‑layout reports, longer contexts, and more complex tasks such as multi‑hop aggregation and cross‑document grounding.

We release all evaluation code at https://github.com/ServiceNow/SynthDocBench and the dataset at https://huggingface.co/datasets/ServiceNow‑AI/SynthDocBench, and we disclose all hyperparameters throughout the paper and appendix.

All synthetic documents are generated programmatically from public topic seeds; no human subjects or personal data are involved, and we manually screened outputs for hate speech or harmful content.

Model evaluations use commercial APIs (Gemini, GPT‑4o, Claude) under their terms of service; no model was fine‑tuned on SYNTHDOCBENCH data, and the GPT‑5 judge scoring rubric is fully disclosed for replication.

Large‑scale evaluation required roughly 150 GPU‑hours for rendering and 200 API‑hours for inference, a cost we report transparently to inform future work and encourage efficient experimental design.

To guard against Goodhart’s Law we randomize 40 % of layouts, breaking spurious topic‑layout correlations, and we invite the community to extend the benchmark rather than over‑fit to its current instantiation.

SYNTHDOCBENCH is intended as a research diagnostic for long‑context visual document understanding, not for deployment in high‑stakes decision‑making systems.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers