Hyperbrowsecomp: A Multilingual and Multi-Modal Stress Test for Web-Browsing Agents

Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri, Seung Hun Eddie Han, Ryandito Diandaru, Qinrong Cui, Jan Christian Blaise Cruz, Badrinath Chandana, Peerawat Chomphooyod, Ahmed Attia, Jonibek Mansurov, Emilio Villa-Cueva, Canh Duong Nguyen, Imran Turganov, Minghao Wu, Peerat Limkonchotiwat, Irina Nikishina

A multilingual, multimodal benchmark of 423 human-authored questions designed to stress-test web-browsing agents.

How well do current web-browsing agents perform on a multilingual, multimodal stress test designed to require locating obscure, verifiable information?

Existing web-browsing benchmarks are often English-centric, rely on static corpora, or focus on simple trivia that models can answer via internal knowledge. This limits their ability to evaluate how agents perform on the open, dynamic web where evidence is often implicit, multilingual, and non-textual. Hyperbrowsecomp introduces a benchmark of 423 manually authored questions across 13 languages, requiring agents to navigate heterogeneous sources like videos, maps, and scanned documents. The authors filter out questions answerable by parametric knowledge alone to ensure the benchmark measures genuine evidence discovery. The strongest evaluated model achieves only 31.68% accuracy, with over half of the questions remaining unsolved by any of the five frontier models tested.

Paper Primer

The benchmark forces agents to perform "interpretive search": agents must identify what and where to search, aggregate candidate answers, and perform multi-hop retrieval across diverse media. The core move is the use of human-authored, language-specific questions that require navigating the open web rather than a fixed dataset, ensuring the benchmark captures cultural context and language-specific information ecosystems.

Current frontier models struggle significantly with open-web evidence discovery.

Across 423 questions, 57.68% received no correct answer from any of the five models evaluated using native search. The highest-performing model (Gemini 3.7 Flash) reached only 31.68% accuracy.

The authors find that agent performance is highly sensitive to the retrieval harness. For instance, replacing a model's native search with a standardized interface (Exa) significantly degraded performance for Gemini models, suggesting that proprietary agents are tightly co-adapted to their native search ecosystems.

Why is this benchmark needed if other browsing benchmarks already exist?

Existing benchmarks often isolate single axes of difficulty, such as English-only text or static corpora. Hyperbrowsecomp combines natively authored multilingual questions with multimodal evidence (video, maps, audio) to simulate the complexity of real-world, open-web information seeking.

How did the authors ensure the questions actually require web search?

They audited all submissions using seven frontier models without internet access. Any question answered correctly by two or more models—or by one model if the answer was highly specific—was excluded to prevent reliance on parametric knowledge.

Researchers should treat the retrieval harness as a first-class component of agent evaluation, as performance is not portable across different search integrations.

Benchmark Overview

Introducing Hyperbrowsecomp, a multilingual, multimodal benchmark for multi-step, verifiable web retrieval.

Hyperbrowsecomp is a stress test for web‑browsing agents that pushes them to locate obscure, publicly verifiable information across languages and modalities.

It presents a single, hard question whose answer can only be obtained by chaining together multiple evidence sources on the open web.

**Figure 1.** Example question and evidence route from HYPERBROWSECOMP. Solving it requires locating a specific moment in a video, grounding the scene to a real-world location, and retrieving information from a financial report.

The German query asks which continuing operation of a specific bank reported the second‑highest operating profit in 2010, requiring the agent to find a YouTube video, locate a timestamp, ground the scene to a real‑world bank, and read its remuneration report.

The benchmark’s focus on multi‑step, verifiable retrieval forces agents to perform genuine web‑search reasoning rather than relying on memorized facts.

Benchmark Design Principles

How the Hyperbrowsecomp benchmark was built, from question authoring to source vetting.

**Figure 2.** Distributions of (a) languages, (b) modalities, and (c) primary domains in the 423 HYPER-BROWSECOMP questions. Labels report counts and percentages of all questions. Modalities may overlap; each question has one language and one primary domain.

Native‑speaker annotators receive a verified answer and its supporting evidence.

They craft a question that forces a non‑trivial search to recover that answer.

Annotators record the canonical answer, acceptable aliases, source URL, exact evidence location (e.g., page number, video timestamp), language, and modalities.

They write a concise solution trace describing a plausible search path.

All entries are reviewed against a detailed guideline covering difficulty, uniqueness, source quality, temporal scope, and multimodal requirements.

Entries that violate the source‑policy (paywalls, private accounts, fragile social‑media posts) are discarded.

Annotator writes the question in French, ensuring the wording does not reveal “Io” directly.

They locate the Wikipedia article, find the target paragraph, and note the character offsets as the exact evidence location.

They list “Io” and its alias “Jupiter I” as acceptable answers.

Solution trace: search “satellite de Jupiter 1610” → open Wikipedia → scroll to paragraph → extract “Io”.

This concrete example shows how a multilingual, text‑only item is still treated like any multimodal entry: the annotator must pinpoint the exact evidence fragment, which later agents must discover without prior knowledge of the page structure.

Motivation and Contributions

We present the HYPERBROWSECOMP benchmark and evaluate multilingual LLMs with three search methods.

HYPERBROWSECOMP is a multimodal search benchmark that stresses agents by requiring retrieval of obscure, publicly verifiable information from the open web.

All questions are manually authored by native speakers, ensuring they are grounded in language‑specific web environments and that the benchmark spans 13 languages with 423 items.

Many items demand reasoning over heterogeneous sources and extraction of decisive evidence from non‑textual content, deliberately making the tasks extremely difficult to promote a long‑lasting benchmark lifespan.

We compare HYPERBROWSECOMP to existing benchmarks such as BrowseComp‑Plus, BrowseComp‑ZH, K‑BrowseComp, XBCP, MM‑BrowseComp, and BrowseComp‑V3.

Our contributions are twofold: (1) the release of HYPERBROWSECOMP itself, and (2) an empirical study of five frontier LLMs using three search integration methods—built‑in, Exa, and Open Web Learning (OWL)—to assess accuracy, cost, and failure modes.

Contextualizing the Benchmark

We situate HYPERBROWSECOMP among prior benchmarks, highlighting its multilingual and multimodal scope.

Recent work has produced a spectrum of benchmarks that probe knowledge, multilinguality, and multimodal search. Some focus on expert‑level factual recall, others on open‑web retrieval, while a few combine both challenges.

These benchmarks target questions that lie beyond closed‑domain expertise, requiring agents to locate obscure, verifiable information on the open web.

**Table 1.** Comparison with other browsing and search benchmarks.

Evaluation Setup and Validation

We detail how questions are validated, how difficulty is audited, and the benchmark evaluation protocol.

To build a reliable benchmark we first validate each question, then audit difficulty, and finally run a controlled evaluation across multiple agents.

The harness is the orchestrated collection of tools (search API, web fetcher, screenshot engine) that an agent uses to retrieve information from the web.

How does this Search Harness differ from a simple web‑search API?

A plain API only returns raw text snippets, whereas the harness also coordinates multimodal tools (e.g., screenshot capture, OCR) and enforces a common interaction protocol across agents, making the evaluation comparable across heterogeneous models.

Validation ensures each benchmark question is precise, answerable, and uniquely tied to verifiable evidence.

The candidate set of books is filtered to those with any cat illustration.

Applying the fingerprint removes all but one book that has precisely 13 cat illustrations.

The remaining book uniquely satisfies the question, guaranteeing that the retrieved answer originates from the intended source.

Fingerprints do not aid the search engine; they merely guarantee that any correct answer must come from the uniquely identified source.

Why aren’t the added fingerprints useful for the search process?

Because they describe properties (e.g., exact illustration count) that are not searchable via text queries, they serve only to eliminate ambiguous candidates after retrieval, ensuring the answer’s provenance.

Select five models (Gemini 3.1 Pro Preview, Gemini 3.7 Flash, GPT‑5.6 Terra, GPT‑5.6 Luna, GPT‑5.6 Sol).

Run each model under three harnesses: built‑in search, Exa, and OWL.

For built‑in and Exa, employ the ReAct paradigm: alternate between model generation and tool calls for up to 25 research steps.

For OWL, activate the multi‑agent workflow with task manager, coordinator, researcher, and synthesizer roles.

Collect the full response for each question‑harness‑model triple.

Feed the question, response, and reference answer to an LLM judge that scores binary correctness.

Repeat the judging with three different LLM judges to measure variance (max 0.80 pp).

**Table.** Performance and resource use by language and modality.

Headline Performance Results

Benchmark scores and resource consumption for each model across settings.

Gemini 3.7 Flash achieves the highest accuracy on the benchmark

31.68 % accuracy reported in Table 2

Gemini 3.1 Pro Preview follows with 25.53 % accuracy, and the larger ChatGPT variants (GPT‑5.6 Sol, Terra, Luna) show progressively lower scores, confirming the expected size‑performance trend.

**Table 2.** Performance and total resource use across retained questions. Tokens are rounded to 0.1 million; time is cumulative end-to-end sample time, rounded to 0.1 hour.

There is substantial headroom for improvement across all models.

Harness Sensitivity Analysis

Human judges reveal how retrieval harnesses impact both accuracy and cost.

The benchmark’s core premise is that a browsing agent’s success depends on the retrieval harness it runs on; changing the harness can dramatically shift both performance and cost.

Harness sensitivity measures how much an agent’s accuracy and resource consumption vary when the underlying search‑harness is swapped.

How does “harness sensitivity” differ from simply reporting accuracy?

Accuracy alone ignores the retrieval pipeline; harness sensitivity adds the dimension of how the same model’s results and resource usage change when the search component is replaced, revealing hidden dependencies on the search backend.

To contextualize the model's accuracy and cost, we evaluated human performance on 30 randomly sampled questions, with ten each in Indonesian, Thai, and Vietnamese. We recruited native speakers who were permitted to browse the web but prohibited from using LLM assistance when answering our questions. Participants were allowed to give up after one hour. They submitted answers for 26 questions and gave up on four. Across all 30 attempts, including give-ups, the mean recorded elapsed time was 89.5 minutes and the median was 54.6 minutes.

Analyzing Question Difficulty

Analyzes which question attributes drive shared failures across models.

Across the five native‑search model runs, 244 of 423 questions (57.68%) receive no correct answer, while 179 are answered correctly by at least one model, of which 49 succeed uniquely.

**Figure 3.** Question difficulty across five models with built-in internet search. (a) Counts of questions answered correctly by different numbers of models; cumulative success groups overlap. (b) Selected examples of characteristics associated with shared failures. Percentages and fractions indicate questions answered incorrectly by all five models within each category. The dashed line marks the overall shared-failure rate (57.68%); the patterns are descriptive, not causal.

**Figure 7.** Shared failures across all 13 languages, sorted by shared-failure rate. The dashed line marks the overall rate (244/423; 57.68%). The languages contain different question sets, so these rates do not isolate language ability.

**Figure 8.** Shared failures across 12 domains (sample sizes differ), ordered by shared-failure rate. The dashed line marks the overall rate (244/423; 57.68%). These associations are descriptive, not causal.

Limitations and Ethical Considerations

We discuss the benchmark’s coverage gaps, variability, computational constraints, and reproducibility safeguards.

Hyperbrowsecomp is deliberately a stress test, so its queries are far from the distribution of everyday web searches. Consequently, the benchmark does not model typical user behavior.

Evaluating on the live web adds unavoidable variability: search rankings shift, page contents evolve, geographic availability differs, and tool implementations change over time.

We mitigate this source of noise by preserving every interaction trace, enabling later re‑analysis despite future changes.

The benchmark’s language set does not span all linguistic communities, and the amount and quality of searchable information vary widely across the selected languages.

Searching for obscure facts can raise privacy concerns; we therefore exclude any question that targets sensitive personal information, private individuals, or resources that require private accounts.

Running a single benchmark instance can consume days of compute and hundreds of millions of tokens, which forced us to restrict the primary study to five representative frontier models.

Some models also posed API‑access hurdles, further limiting the set of systems we could evaluate.

Our evaluation covers only two retrieval settings—provider‑native search and Exa—plus an OWL‑based harness, mirroring prior work but leaving many harness designs unexplored.

Future studies could keep the model fixed while varying the search tool, or broaden the suite of harnesses to assess progress in tool integration under comparable resource budgets.

To support reproducibility, we document the experimental setup, data sources, evaluation procedures, and implementation details, with full prompts, tool schemas, model identifiers, harness parameters, retry logs, grading results, and cost figures in Appendices A–C.

Because live webpages, rankings, and provider tools evolve, exact reruns may differ; the preserved outputs and traces enable auditing of reported results.

We used generative AI only for manuscript editing—improving grammar, clarity, and readability—while the core research ideas and theoretical contributions remain entirely human‑generated.

Evaluation Details

Details of the evaluation prompts, agent configurations, and grading procedures used in the benchmark.

We recruited the highest‑scoring undergraduate from the Computer Science program (Thai) and the Business program (Indonesian, Vietnamese), each having passed a comprehensive QA/NLI exam with >90% accuracy.

Annotators used a platform that enforced a one‑hour limit per question; if the limit was exceeded they could abandon the item.

Internet search was permitted, but any assistance from AI models was explicitly disallowed.

The evaluation harness follows the ReAct paradigm and is powered by the open‑source Inspect AI framework, which supplies tool schemas, model‑graded scoring, and the standard ReAct agent loop.

Three retrieval configurations are provided: “no‑tools” (web search disabled), “built‑in‑search” (uses the provider’s native web search), and “Exa” (combines web search with subsequent web fetch).

The OWL setting bypasses Inspect AI and runs a CAMEL‑style workforce comprising a task manager, coordinator, web specialist, multimodal specialist, and synthesis specialist, all sharing the same primary model.

OWL’s budget constraints are a 2400‑second wall‑clock limit, a cap of 50 external tool calls, a per‑call round limit of 12, and a forced stop 120 seconds before the deadline.

Both the ReAct and OWL agents must output answers in the fixed three‑field format: Explanation, Exact Answer, and Confidence (0–100%).

The agent system prompt instructs the model to act as a web‑research agent, to rely on tools rather than memory, and to respond with the three required fields once the answer is known.

Retrieval instructions differ per setting: “no‑tools” disables web search and fetch; “built‑in‑search” enables `web_search` but disables `web_fetch`; “Exa” enables both `web_search` and `web_fetch`, with explicit prohibitions on mentioning unavailable tools.

Inspect AI augments these prompts with its standard ReAct instructions, which guide the model to reason about actions, parallelize independent tool calls, and invoke a submit tool after producing a final answer.

OWL’s workforce prompts assign decomposition and delegation to the task manager and coordinator, web and visual browsing to the respective specialists, and evidence reconciliation to the synthesis specialist, while forbidding claims of media inspection unless a tool call succeeded.

The question prompt requires the same three‑field response format, ensuring consistency across all evaluation conditions.

If an agent reaches the research‑step limit without submitting, a step‑limit finalization prompt is appended, demanding a best‑effort answer using only the information already gathered.

The grading prompt asks a judge to extract the final answer, compare it to the reference answer, and report correctness and confidence, all in a prescribed textual layout.

Both the built‑in‑search and Exa ReAct configurations run within a unified agent loop capped at 25 research steps; exceeding this cap triggers the step‑limit finalization prompt described earlier.

Search Harness Configurations

This section details the search harness configurations and tool settings used in the experiments.

For the Exa harness, queries are sent to the Exa endpoint and the service returns page text and highlighted passages, with a default cap of five results per query. The Exa fetch tool retrieves page content but truncates it to 20 000 characters, and both tools abort after 60 seconds of execution.

The table lists four settings and their corresponding available tools: - **No tools**: None - **Built-in search**: Provider-specific `web_search` - **Exa**: Exa-backed `web_search` and `web_fetch` - **OWL**: DuckDuckGo and Wikipedia search, screenshot-based visual browsing, and image, video, rendered-PDF, and audio analysis

**Table 5.** Tools exposed under each retrieval setting.

The scoring phase re‑uses the same model as the answering run but starts from a fresh context containing only the original question, the model’s full response, and all accepted reference answers; it emits a binary correct/incorrect decision, treating any unparsable output as incorrect.

Judge Sensitivity Details

Analyzes how grader choice and reporting metrics affect benchmark results.

Token usage counts all tokens consumed by a model: for ReAct it sums input and output tokens across answering, tool‑using, finalization, and grading calls; for OWL it aggregates tokens from every workforce role and perception call, reported separately from the Inspect grading call.

Agent turns measure the number of model‑generated steps: ReAct counts model turns excluding the grading step, while OWL counts internal model calls across all workforce roles.

Execution time records the end‑to‑end elapsed time for each sample, including answer generation and grading. Sample‑level errors tally execution or grading failures, distinguishing those that persist after retries.

For every model and retrieval setting the results report five counts: attempted samples, successfully scored samples, unscored samples, retried samples, and terminally failed samples. This prevents failure‑rate differences from being hidden by accuracy computed only on successfully scored samples.

Judge sensitivity analysis re‑scores each saved response with three fixed graders accessed via OpenRouter—Gemini 3.7 Flash, GLM‑4.7, and Kimi K2 Thinking—using the same binary grading prompt and reference answers as the original self‑judge.

Overall, high agreement rates (> 98.7%) and minimal accuracy shifts (≤ 0.80 percentage points) across evaluators confirm that self-judging introduces negligible bias for short, unambiguous targets demonstrating that aggregate benchmark rankings remain robust regardless of the chosen evaluator.

Data Collection Statistics

Details on how the benchmark questions were filtered, annotated, and validated.

Across evaluators we observed agreement rates above 98.7 % and accuracy shifts no larger than 0.80 percentage points, confirming that self‑judging introduces negligible bias for short, unambiguous targets.

No‑internet screening (Table 7) examined how many of seven models could answer each question without web access; of the 500 submitted questions, 121 were answered correctly by at least one model, 54 by two or more, and only a single question by all models.

All 54 questions that at least two models answered correctly were removed. For the remaining single‑model correct questions we evaluated answer specificity with an LLM classifier and manual review, ultimately discarding 23 more and retaining 423 questions for the benchmark.

The retained set was later annotated for domain (12 possible categories plus “Other”) and modality (seven evidence‑source labels plus an “Arithmetic” task label) using a gpt‑5.6‑sol1 model; the modality step also recorded whether non‑text interpretation was required, not required, or uncertain.

This figure displays the failure rates of different modalities in answering questions, measured as the percentage of questions with no recorded correct answer. The modalities are listed on the left, with corresponding failure percentages and raw counts on the right. A vertical dashed line indicates the overall average failure rate.

Domain Classification Criteria

Briefly notes the inability to classify answer specificity due to missing data.

The section flags the evaluation as uncertain because the required question‑answer pair is absent, making it impossible to judge whether an answer is specific or generic.

Performance Tables

Compact tables detail accuracy and token usage across models, languages, and modalities.

Table 7 reports how many questions are excluded and retained for each model when a minimum of $k$ models must answer correctly without internet access; the rows span thresholds from $k=1$ up to $k=11$.

**Table 8.** Accuracy summary across all evaluated models and setups by language. Values are percentages; $n$ gives the number of retained questions in each row.

The table presents performance metrics across various languages for different models categorized under "Built-in", "Exa", and "OWL".

**Table 10.** Total token use (millions) across all evaluated models and setups by language. Values are rounded to 0.1; $n$ gives the number of retained questions in each row.

Additional Performance Data

Additional tables detail sample times and token usage across languages, modalities, and setups.

**Table 11.** Total cumulative end-to-end sample time (hours) across all evaluated models and setups by language. Values are rounded to 0.1; $n$ gives the number of retained questions in each row.

**Table.** Modality performance breakdown across different models (Built-in, Exa, and OWL).

**Table 13.** Total cumulative end-to-end sample time (hours) across all evaluated models and setups by modality. Values are rounded to 0.1; $n$ gives the number of retained questions in each row.

Extended Difficulty Analysis

Analyzes which factors make Hyperbrowsecomp questions hard and highlights key problematic terms.

We extend the difficulty analysis to the full language, domain, modality, and wording breakdowns of the 423 Hyperbrowsecomp questions, using the five provider‑native‑search model runs while excluding the Exa runs and the seven‑model no‑internet audit. A confirmed shared failure is defined as five recorded incorrect answers.

Category‑level analysis (Figures 7‑9) reports shared‑failure statistics across languages, primary domains, and modality/task categories. For wording analysis we compare English translations of the 244 shared failures with the 179 questions that have at least one correct answer, applying lowercasing, tokenization, stop‑word removal, and filtering terms that appear in 8–80 % of the questions. TF‑IDF weights use raw counts and smoothed IDF, followed by per‑question L2 normalization, and we rank the 264 eligible terms by the mean TF‑IDF difference between the zero‑success and solved groups.

Figure 10 displays the 100 terms with the highest positive TF‑IDF differences; the dots indicate the proportion of questions containing each term that have no recorded correct answer (reference rate 57.68 %). The top‑ranked terms are “issue” (19/25; 76 %), “video” (35/47; 74.5 %), and “page” (30/39; 76.9 %). These exploratory associations do not prove that the words cause difficulty.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers