Deep Search in Real-World Web Environments
Xiaomin Yu, Hongliang Chen, Zhe Huang, Shilin Yan, Shao-Rong Su, Xuqiao Ma, Feiyang Wang, Jiayu Chen, Jiwei Fang, Yu Yan, Yini Huang, Chen Chen, Ruoxi Cheng, Yuzhi He, Yuhui Zhang, Xiaoxing Hu, Chen Liu, Jianbo Lin, Yijiang Li, Xiaobin Hu, Menglong Li, Zhuoran Lu, Weishi Wang, Zhongqi Yue, Chengwei Qin
OpenDS-World evaluates AI agents on multi-step, real-world web research requiring diverse modalities and interactive navigation.
How do current frontier AI agents perform when tasked with end-to-end research in complex, real-world Web environments compared to static retrieval?
Current deep search benchmarks fail to capture the complexity of the real-world web, where information is scattered across heterogeneous formats and requires interactive navigation rather than simple retrieval. The authors introduce OpenDS-World, a benchmark of 200 tasks that force agents to autonomously discover and connect evidence across diverse modalities, file formats, and interactive web interfaces. Frontier models struggle significantly, with the best-performing model achieving only 39.0% accuracy, while human participants reach 21.1% in independent attempts.
Paper Primer
The OpenDS task requires agents to start from a video anchor and perform multi-step research to answer a factual question. The core move is the construction of interdependent evidence chains: each task is designed so that no single clue is sufficient, forcing the agent to recursively search, process heterogeneous files (e.g., Parquet, PDF, HTML), and manipulate interactive interfaces to reach a unique terminal answer.
Current frontier models lack the robustness required for autonomous deep search in real-world environments.
Evaluation of 12 models on 200 tasks under a unified tool environment. The top-performing model (GPT-6 Astra) achieves 39.0% accuracy, while 61% of tasks remain unsolved by the best model.
Increased resource consumption does not correlate with higher task success.
Comparison of tool calls and wall-clock time against accuracy across models. Models like DeepSeek-V4-Pro spend 2.39x the time of GPT-6 Astra but achieve only 12.5% accuracy, indicating inefficient search strategies.
Why is this benchmark necessary if existing web-agent benchmarks already exist?
Existing benchmarks often rely on static retrieval or simplified environments; OpenDS-World specifically targets the "real-world" complexity of interactive access (e.g., virtual tours, parameter updates) and heterogeneous data formats that require multi-step, autonomous evidence integration.
What does the human performance data reveal about the difficulty of these tasks?
Human solvers achieved only 21.1% accuracy in independent attempts, and 42.5% of tasks were not solved by any human participant, confirming that the benchmark is highly challenging even for humans using standard web browsers.
Researchers should shift focus from simple retrieval accuracy to long-horizon execution management and tool coordination, as current agents frequently exhaust budgets or fail to recover from tool-use errors.
The Challenge of Real-World Deep Search
OpenDS task measures agents’ multi‑step deep search beyond static retrieval.
Current evaluation pipelines rely on static retrieval, which assumes a single document answer and ignores the need for agents to navigate, interact with, and combine evidence from many web sources. This omission leaves three critical gaps: (1) diverse modalities such as video and audio, (2) heterogeneous file formats like PDF and Parquet, and (3) interactive access steps that require browsing or tool use. The OpenDS task is introduced to expose these gaps and measure agents’ ability to perform genuine deep search.
Deep Search is an autonomous research capability where an AI agent discovers, extracts, and stitches together evidence from multiple, heterogeneous web sources to answer a factual query.
**Figure 2.** An example of deep search in OpenDS-World. The example connects audiovisual identification, virtual-tour navigation, document inspection, and structured-data retrieval across multiple Web sources.
Static retrieval benchmarks fail to capture the coordination, modality diversity, and interactive steps needed for real‑world research.
Designing the OpenDS-World Benchmark
We build OpenDS-World, a benchmark of 200 tasks with multimodal, multi-format, interactive evidence chains.
OpenDS-World is a curated set of 200 real‑world Web research tasks that require agents to locate, retrieve, and combine evidence across many media types, file formats, and interactive interfaces.
How does OpenDS-World differ from a standard retrieval benchmark that only supplies a text passage?
Standard benchmarks give a static passage and ask a question; OpenDS-World forces the agent to discover the passage itself, navigating across modalities, formats, and interactive steps before any text is even presented.
An evidence‑chain is an ordered sequence of entities, each supported by at least one Web source, that collectively justify the final answer.
Why is an evidence‑chain more demanding than a simple citation list?
A citation list can be verified by checking each reference in isolation; an evidence‑chain requires the agent to reason about the semantic relations between consecutive nodes, often needing to transform or combine information from different modalities before the next step can be generated.
Stage 1 – Select a publicly accessible video and identify a verifiable anchor entity that maps to a unique Wikidata entry.
Stage 2 – Iteratively expand the evidence path: generate a search query from the current node, retrieve candidate Web sources via Google Search, select a source that is semantically related, extract a new entity, and verify it against additional independent sources.
Stage 3 – Invert the completed evidence path into a textual question whose constraints encode the intermediate nodes without revealing them directly.
Human verification – Ten independent reviewers re‑run the full path, checking both evidence validity and task validity, discarding any task that fails these checks.
Final assembly – The remaining 200 tasks form the OpenDS‑World benchmark, each with a reference answer and a fully annotated evidence chain.
Stage 2 query: “Starry Night provenance” → Google returns a museum catalog page (HTML) and a high‑resolution image (WebP).
Select the catalog page, extract the acquisition year (1889) as the next node; verify the year on a PDF exhibition brochure.
Generate a new query “1889 Impressionist exhibitions” → retrieve a timeline widget (interactive JavaScript) requiring a date‑range parameter.
Set the parameter to 1889, obtain a list of exhibitions; pick the “Paris 1889 World Fair” entry (HTML) as the terminal node.
Invert the chain: ask “Which painting, created before 1890, was displayed at the 1889 Paris World Fair?” – the answer is uniquely “The Starry Night”.
This toy run shows how a single task weaves together video, HTML, PDF, and an interactive widget, illustrating the three core attributes in action.
**Figure 3.** Three complementary properties of a real-world Web source. An Our World in Data page illustrates how a single source combines (A) diverse modalities, including text and visual charts; (B) heterogeneous formats, including HTML, CSV, JSON, and downloadable ZIP bundles; and (C) interactive access, including region selection, temporal indexing, and declarative API queries. Obtaining relevant evidence requires navigating these connected representations and source states.
**Figure 5.** Task diversity and statistics of OpenDS-World. Each annotated evidence-chain entry is characterized by its modality, resource format, and access type, collectively covering four modalities, 42 formats, and seven access types. Top: representative Web sources illustrating these three dimensions. Bottom: (a) modality coverage across 200 tasks, (b) format coverage across 200 tasks, and (c) task coverage of interactive access types. Access-type codes in (c) are defined in Table 3. Parentheses report counts; Other formats comprise 108 entries spanning 35 types. Modality, format and access categories can overlap within a task.
**Figure 10.** **Task composition and annotated evidence structure of OpenDS-World.** (a) Distribution of 200 tasks across eight domains. (b) Distribution of anchor-video durations across five temporal intervals. (c) Number of annotated source entries per task, with six or more entries grouped into the final bin. Percentages are calculated over all 200 tasks, with task counts reported in parentheses. Evidence-chain statistics exclude the separately supplied anchor video.
The benchmark’s strength lies in its systematic coverage of four modalities, 42 formats, and seven access types, forcing agents to master truly multimodal, multi‑format, interactive deep search.
Model Performance and Search Costs
Shows the experimental setup, tool‑use budget, and main performance results on OpenDS‑World.
GPT‑6 Astra achieves the highest overall accuracy on OpenDS‑World.
Table 1 reports 39.0 % accuracy over 200 tasks, a 1.0 % margin over the runner‑up.
A hard cap on how many tool calls a model may issue, forcing it to prioritize the most promising actions.
Step 1: Use a Search call to locate a relevant article (cost 2, remaining budget 3).
Step 2: Use a GUI call to click the first link (cost 1, remaining budget 2).
Step 3: Use another GUI call to scroll to the answer section (cost 1, remaining budget 1).
Step 4: No further calls are needed; the model generates the final answer.
This example shows that with a tight budget the model must sequence high‑impact actions rather than issuing many low‑value calls.
How does the Tool‑Use Budget differ from simply limiting runtime?
Runtime limits only restrict wall‑clock time, whereas the Tool‑Use Budget caps the number of interactions with external tools. A model could spend a long time on a single call, but the budget forces it to think about *which* calls to make, not just *how long* to run.
**Figure 1.** Model and human performance on OpenDS-World.
**Figure 6.** Search cost across models and task difficulties. (a) Mean tool calls and elapsed time per target task. (b) Distribution of task-level elapsed time, with a logarithmic scale above one minute. (c) Mean tool calls per target task within each difficulty level. Mean costs are computed from recorded trajectories using the full target-task count as the denominator.
Frontier models still exhibit large performance gaps across difficulty levels, with accuracy falling from ~60 % on easy tasks to < 20 % on hard ones.
Failure Modes and Budget Exhaustion
OpenDS‑World quantifies deep‑search performance under realistic budget constraints.
In the error analysis, GPT‑6 Astra, GPT‑5.6 Sol, and Claude Opus 5.5 naturally terminate on 69.0 %, 65.0 %, and 60.5 % of the 200 tasks, respectively, while Qwen 3.8‑Flash terminates naturally on only 6.5 % and hits the tool‑call limit on 80.0 % of tasks.
Despite a high natural termination rate, GPT‑6 Astra achieves only 39.0 % answer accuracy. Tool failures are disproportionately costly: they represent 2.83 %–9.42 % of recorded calls but consume 19.27 %–61.90 % of tool‑execution time; for GPT‑6 Astra the figures are 6.34 % of calls and 42.57 % of time. After a failure, most models switch tools (Claude Opus 5.5 83 %, Gemini 3.8 Flash 81 %, GPT‑5.6 Sol 79 %), whereas Kimi K3 and GLM 5.3 retry the same tool with modified arguments (≈38 % and 37 %). These observations highlight three core challenges: budget exhaustion, incorrect answers at termination, and expensive tool failures.
Budget analysis (Figure 9) shows that the share of tasks hitting at least one limit ranges from 16.5 % (Claude Opus 5.5) to 81.5 % (Qwen 3.8‑Flash). Call‑only limits dominate for Gemini 3.8 Flash (37 %) and Qwen 3.8‑Flash (80 %), while time‑only limits exceed call‑only rates for Claude Fable 5.1 (28 %), Kimi K3 (29.5 %), and GLM 5.3 (41 %). Simultaneous exhaustion of both budgets occurs in at most 1 % of tasks per model. Notably, GPT‑6 Astra and Claude Opus 5.5 experience little budget exhaustion yet their accuracies stay below 40 %, underscoring the need for better source selection, evidence integration, and stopping decisions.
Budget exhaustion dominates failures for several models, with up to 81.5 % of tasks hitting a limit.
Qwen 3.8‑Flash reaches a limit on 81.5 % of tasks; DeepSeek‑V4.1‑Flash 78.0 %; GLM 5.3 73.0 %; DeepSeek‑V4‑Pro 72.5 %.
**Figure 9.** Budget exhaustion under the one-hour time limit and the 100-call tool-use limit.
**Figure 8.** Search termination, tool failures, and recovery behavior. (a) Termination outcomes across 200 target tasks. Other includes research terminations caused by incomplete outputs, context overflow, or execution errors. (b) Failed calls as a percentage of all recorded calls and their share of total recorded tool execution time. (c) The next decision following a failed tool call: retrying with the same arguments, modifying arguments, switching tools, or making no subsequent decision.
Research and Finalization Procedures
Additional prompts, evaluation details, and extra experimental analyses.
The research stage gathers evidence using tools, while the finalization stage produces the answer solely from that evidence, without any tool calls.
**Table 5.** Default configuration parameters for the research and finalization stages.
**Table.** Performance comparison of various AI models across different file formats (HTML, MP4, PDF, MP3, JPG, CSV, ZIP, Other), showing accuracy percentages and the number of tasks completed.
Final Answer Generation runs after the research phase. The harness switches to a separate finalization stage that can issue at most two model requests within a 5 000‑token budget, with 4 000 tokens allocated to the first request. Only non‑empty, complete responses without tool calls are accepted.
Search Prompts provide a unified system prompt that tells agents which tools they may use (web, browser, download, YouTube) and instructs them to conduct autonomous research, inspect media, verify evidence, and avoid fabricating facts. The prompt leaves the choice of search strategy, source selection, and tool‑call ordering entirely to the model.
The Final‑Answer Prompt is issued once research ends. It forces the model to answer only from the evidence already collected, forbidding any further tool calls or invented facts. The answer must be a short English paragraph (1–3 sentences, ≤80 words) and may follow any explicit format requested in the question.
LLM‑as‑Judge evaluation uses Qwen3.8‑27B as a strict answer judge. The judge compares a candidate answer to a reference answer, requiring semantic equivalence and completeness, and returns a JSON object indicating correctness and confidence. The judge does not access external tools or solve the question independently.
Domain Performance Analysis shows that models vary widely across task domains. GPT‑6 Astra excels in Society (51.5 %) and Nature (50.0 %), while Claude Opus 5.5 shines in Nature (50.0 %) and Science (45.5 %). Gemini 3.8 Flash leads in History (36.8 %) and shows the highest accuracy in Culture and Economy (44.4 %).
Evidence Attributes Performance Analysis breaks down accuracy by modality, data format, and access requirement. GPT‑6 Astra is strongest on Text (38.69 %) and Image (48.31 %); Gemini 3.8 Flash leads on Video (43.66 %) and ties for Audio (50.00 %). Across data formats, GPT‑6 Astra dominates in six categories, notably CSV (60 %).
Video Duration Analysis groups tasks by video length and finds variation in model accuracy, but the differences do not establish a causal link between duration and difficulty.
Natural Long‑Horizon Tasks in OpenDS‑World require agents to iteratively discover, access, and verify evidence across multiple web sources. Even tasks with only three annotated sources involve median tool‑call counts around 74, showing that compact evidence chains can demand extensive exploration.
OpenDS‑World serves as a realistic testbed for Recursive Self‑Improvement. Agents must navigate a dynamic web, handle heterogeneous interfaces, and cope with tool‑budget exhaustion (15.5 %–80 % of trajectories). Failed tool calls consume up to 61.9 % of execution latency, offering concrete targets for improving recovery and resource allocation.