DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment

Xinyu Geng, Xuanhua He, Sixiang Chen, Yanjing Xiao, Fan Zhang, Shijue Huang, Haitao Mi, Zhenwen Liang, Tianqing Fang, Yi R. Fung

DeepSearch-World provides a deterministic, verifiable environment for training search agents via iterative self-distillation.

How can we improve tool-use agents by self-distilling verified trajectories instead of relying on sparse reinforcement learning rewards or fixed supervised datasets?

Training tool-use agents to improve autonomously is difficult because sparse rewards and unreliable teacher signals often lead to performance plateaus. DeepSearch-Evolve addresses this by using a deterministic, verifiable Wikipedia-based environment to generate high-quality, process-supervised trajectories that the agent uses to iteratively refine its own tool-use behavior. This framework allows a 9B-parameter model to achieve competitive performance against larger proprietary systems without relying on external synthetic data or stronger teacher models.

Paper Primer

Existing self-evolving pipelines for tool-use agents struggle with static imitation signals and sparse outcome rewards that fail to pinpoint whether a failure occurred during query formulation, tool selection, or evidence synthesis. DeepSearch-World solves this by providing a deterministic, offline Wikipedia environment where intermediate tool-use progress is objectively verifiable through entity-level matching.

DeepSearch-Evolve is an iterative self-distillation framework: it generates scaffolded trajectories using a teacher agent that tracks progress and recovers from failures, then converts these into standard ReAct-format supervision to train the student. The core move is replacing opaque, sparse rewards with dense, verifiable process signals derived from the environment's ground-truth entity sets.

DeepSearch-World-9B achieves competitive performance against proprietary and open-source agents without distillation from stronger models.

Performance on benchmarks including BrowseComp (31.2%), GAIA (61.5%), and HotpotQA (93.4%). Outperforms the Qwen3.5-9B-Instruct backbone by +23.8 on BrowseComp and +48.1 on HotpotQA.

The agent's tool-use efficiency improves significantly over training rounds: while the baseline model terminates after ~4.7 steps, the evolved agent sustains ~18.0 steps, demonstrating a shift from premature answering to sustained multi-step evidence gathering.

Why is a deterministic environment necessary for this self-distillation approach?

Deterministic environments allow for objective, order-free verification of intermediate tool-use progress, providing the dense, reliable supervision signals required for on-policy self-distillation that live web environments cannot guarantee.

How does this approach differ from standard reinforcement learning (RL) for agents?

Unlike RL methods that rely on sparse, trajectory-level rewards, this framework uses scaffolded process supervision to explicitly train the agent on planning, error recovery, and evidence extraction steps, effectively turning verified experience into supervised learning targets.

The Challenge of Self-Evolving Search Agents

We expose why existing tool‑use training stalls and propose a deterministic self‑distillation pipeline.

Supervised Fine‑Tuning (SFT) – training on fixed, human‑provided input‑output pairs – quickly saturates because the teacher data cannot exceed the backbone’s capability. Reinforcement Learning (RL) – learning from scalar rewards based on task outcomes – suffers from sparsity, giving little guidance on where failures occur. Consequently, long‑horizon tool‑use agents lack reliable, process‑level supervision.

Existing training pipelines either cling to static imitation signals (SFT) or rely on sparse reward feedback (RL), both of which leave long‑horizon agents without detailed guidance on intermediate tool actions.

**Figure 1.** Conceptual comparison of self-evolving paradigms for tool-use agents. SFT imitates positive traces, reward-based RL learns from sparse outcomes, whereas our DeepSearch-Evolve distills verified tool-use behaviors in a deterministic environment DeepSearch-World, providing controllable process-level supervision for planning, memory, error correction, rollback, and query rewriting.

The key advance is replacing sparse RL rewards with verified trajectory distillation, turning noisy outcomes into dense, trustworthy supervision.

Tool-Use Agents and Environments

We situate our work among tool-use agents and self-improving methods, highlighting deterministic environments.

Tool‑augmented LLM agents extend capabilities via external search, API calls, and environment interaction. Prior work explores large‑scale tool‑use training, web navigation agents, and trajectory‑level tuning. Recent studies improve reproducibility by using virtual tool environments such as API caching, programmatic simulators, or LLM‑based environment modeling.

A tool‑use agent is an LLM that can invoke external functions (search, APIs, or simulated tools) to gather information or act on the world, thereby overcoming the model’s internal knowledge limits.

Our approach replaces the loosely defined virtual environments with a deterministic offline environment built from Wikipedia snapshots. This environment guarantees reproducible observations and eliminates hallucinated results, while keeping serving costs low because all interactions are pre‑computed.

Self‑evolving agents iteratively refine their behavior using trajectories they generate themselves and verification signals. GRPO‑style group RL dominates current post‑training pipelines but suffers from sparse rewards and weak verifiability, leading to unstable optimization. OPSD attempts to mitigate this by distilling on‑policy generations, yet the teacher policy often fails to provide a reliable, stationary target distribution.

We introduce an evolving SFT method that trains agents on verified high‑quality trajectories within the deterministic environment. The verifiable tool environment supplies exact observations, enabling stable self‑distillation without the noisy feedback that hampers prior RL‑based approaches.

The DeepSearch-Evolve Framework

We detail the deterministic environment, scaffold teacher, and self‑evolving loop that enable self‑distillation.

Fixed teacher data bottlenecks tool‑use learning, motivating a self‑distillation loop that generates its own supervision.

DeepSearch-World is a deterministic, offline Wikipedia sandbox that lets agents query and read without any external network variability.

How does DeepSearch-World differ from a typical live web environment?

Live web search returns nondeterministic results that depend on indexing updates and ranking changes. DeepSearch-World fixes the index and the document set, so the same query always yields identical snippets and articles, which is essential for reliable self‑distillation.

DeepSearch-Evolve closes the loop by repeatedly turning the current model into a teacher that generates verified trajectories, then distilling those trajectories into a stronger student.

Is DeepSearch-Evolve just another form of self‑play?

No. Self‑play typically generates synthetic trajectories without external verification. DeepSearch-Evolve explicitly verifies each tool call against the deterministic environment, discarding unverified or low‑quality steps before distillation.

Self‑distillation treats the model’s own high‑quality outputs as teacher signals, turning its own successes into training data.

How does this self‑distillation differ from classic knowledge distillation?

Classic distillation transfers knowledge from a large pretrained teacher to a smaller student, often using soft logits. Here the “teacher” is the same model at a previous iteration, and the supervision consists of full tool‑use trajectories verified against a deterministic environment.

Rejection sampling discards generated trajectories that fail verification, keeping only those that satisfy strict correctness criteria.

Why not simply keep all trajectories and weight them by confidence?

Weighting by confidence still mixes incorrect reasoning into the training signal, which can reinforce hallucinations. Rejection sampling guarantees that every retained example is fully verified, providing a clean supervision signal.

Beyond binary acceptance, quality filtering removes redundant evidence, weak goal alignment, and inconsistent reasoning from accepted trajectories.

Is this filtering similar to standard data cleaning?

It is more fine‑grained: instead of discarding whole examples, it operates at the level of individual tool‑use steps, ensuring that each retained step is both correct and essential for the final answer.

Plan: initialize progress state $s_0$ with empty

Act: for up to $T_{\text{max}}$ steps, select a tool call $a_t$, receive observation $o_t$, and update $s_{t+1}=U(s_t, a_t, o_t, r_t)$ where $r_t$ is the environment‑grounded reflection.

End: when evidence is sufficient or the step budget is exhausted, emit a concise answer derived from the verified working memory.

Step 1 (Plan): $s_0$ is created with

Step 2 (Act): the teacher issues

Step 3 (Act): a failed

Step 4 (Act): the corrected query succeeds, adding $e_2$ to

Step 5 (End): with both entities verified, the teacher outputs the final answer “$e_1$ and $e_2$”.

Even in this tiny scenario, the reflection mechanism turns a failed tool call into a concrete query revision, guaranteeing that every step moves toward completing $T$.

Self‑evolving training loop

**Figure 2.** Overview of the DeepSearch-World and DeepSearch-Evolve. DeepSearch-World provides a deterministic and verifiable tool environment, while DeepSearch-Evolve performs self-distillation through a self-evolving loop of trajectory generation, rejection sampling, importance-sampling-based data mixing, and SFT training.

**Figure 3.** Three-phase scaffolded teacher rollout. The teacher proceeds through the initial Plan, iterative Act, and End period.

**Figure 4.** Example scaffold trajectory generated by the teacher agent. The teacher decomposes a fuzzy multi-hop question, alternates between search and visit tools, updates working memory with verified evidence, uses grounded reflection to recover from failed searches, and produces a final answer supported by verified observations.

Experimental Results and Ablations

Key performance gains and the impact of quality filtering are presented.

Recall that DeepSearch‑Evolve self‑distills verified trajectories in a deterministic environment, avoiding reliance on fixed teacher data or sparse RL signals.

DeepSearch‑World‑9B outperforms Qwen3.5‑9B‑Instruct on every benchmark, achieving up to +48.1 points on HotpotQA.

Gains of +23.8 (BrowseComp), +22.9 (BrowseComp‑ZH), +9.0 (HLE), +37.6 (GAIA), +29.0 (xbench), and +48.1 (HotpotQA) over the backbone.

**Table 1.** Main results on deep search and related reasoning benchmarks. All scores are reported as percentages. A dash indicates that the corresponding result is unavailable or not applicable.

**Figure 5.** Data scale effect in evolving SFT. Compared with 100K total training data, 420K QA lead to a higher target-validation plateau, lower residual format errors, and more stable gains in tool success and entity hit ratio.

**Figure 6.** Tool-use behavior and advanced capability on DEEPSEARCH-VAL. Compared with Qwen3.5-9B-Instruct, DeepSearch-World sustains longer interactions, performs more search and visit calls, and achieves a substantially higher advanced capability score.

**Table 2.** Ablation of rejection sampling and trajectory quality filtering. RS only selects answer-correct trajectories, QF only applies the trajectory quality filter, and w/o both removes both. Results are in percentages averaged over three runs.

**Table 3.** Ablation on trajectory-to-training-data conversion strategies. Results are in percentages averaged over three runs.

Performance gains across benchmarks hinge on high‑quality, self‑distilled trajectories; quality filtering is essential.

Limitations and Future Directions

We outline the current limits of DeepSearch‑World and discuss ethical implications.

Our current environment, DeepSearch‑World, is built on a static Wikipedia dump, which inherently limits coverage and domain diversity. Extending verifiable environments to broader knowledge sources could improve generalization, but such extensions are not yet implemented.

The update rule depends on evolving supervised fine‑tuning (SFT); alternative RL‑style or OPSD‑based updates might increase flexibility, yet how to inject higher‑level capabilities such as planning, error recovery, and tool‑use strategies into RL training remains underexplored.

Because DeepSearch‑World draws exclusively from Wikipedia, it cannot represent domains outside the encyclopedia’s scope, restricting the agent’s ability to generalize to real‑world tasks that require broader knowledge.

Ethically, the work uses publicly available Wikipedia content licensed under CC‑BY‑SA and does not involve private or personally identifiable information. We will release DeepSearch‑World and the constructed dataset under a compatible license, with attribution to Wikipedia contributors.

The environment is fully offline and deterministic, reducing risks from uncontrolled web access during training. Nevertheless, the dataset may contain publicly available names and biographical facts, and Wikipedia’s societal biases can be inherited by trained models. Future work should explore more diverse data sources and stronger safety alignment for tool‑use agents.

Qualitative Case Study

A ReAct‑style case study uncovers the revived opera L’Oracolo by Franco Leoni.

This section walks through a ReAct‑style reasoning trace that discovers a long‑dormant opera production. The process illustrates how systematic search and verification steps can answer a historically layered query.

Search query “defunct performing organization founded surplus profits visiting opera company” → result: Glasgow Grand Opera Society (founded 1906).

Search query “Glasgow Grand Opera Society first independent production” → result: first independent production in 1911 was Gounod’s Faust.

Search query “Gounod Faust original Paris premiere venue” → result: premiered at Théâtre Lyrique, Paris, 1859.

Search query “Glasgow Grand Opera Society interwar conductor British premiere Mozart Idomeneo” → result: Erik Chisholm (composer‑organist) conducted the British premiere of Mozart’s Idomeneo in 1934.

Search query “Glasgow Grand Opera Society moved to New Athenaeum Theatre 1993 opera first UK production 80 years” → result: productions moved to the New Athenaeum Theatre (Royal Conservatoire of Scotland); in 1994 the society staged Franco Leoni’s L’Oracolo, the first UK production in ~80 years.

Search query “L’Oracolo opera Franco Leoni 1905” → result: L’Oracolo is a one‑act opera composed by Franco Leoni, premiered 1905.

The trace shows that a disciplined ReAct workflow can resolve a multi‑clue historical puzzle without external knowledge beyond targeted web searches.

The final answer is *L’Oracolo* by Franco Leoni, confirming the case study’s hypothesis that ReAct‑style reasoning can retrieve obscure cultural facts.

This paper’s text was edited with large‑language‑model assistance for grammar and LaTeX formatting only; no prompts encouraging positive bias were used.

Teacher Rollout Implementation Details

Implementation and experimental details for reproducibility.

During teacher rollout the agent may interact with the virtual environment for up to $T_{max}$ steps; a sliding‑window context retains the most recent $w$ steps in full detail while earlier steps are collapsed into a summarized progress state.

Tool use follows a text‑based function‑calling protocol: each call is emitted as plain text surrounded by delimiter tokens that specify the function name, JSON‑encoded arguments, the tool result, and the returned observation.

The delimiters are chosen to be unlikely in natural language and are parsed with regular expressions, making the rollout code backend‑agnostic and compatible with models lacking native function‑calling APIs.

When observations are too long to fit the context window, the environment either returns a condensed representation or applies deterministic truncation as described in Appendix A.2.

Benchmark evaluation spans six search‑intensive suites: BrowseComp (1,266 English browsing questions), BrowseComp‑ZH (289 Chinese multi‑hop questions), HLE (2,158 English multiple‑choice academic questions), GAIA‑Text (103 text‑only queries), xBench‑DeepSearch (100 Chinese deep‑search items), and Search‑QA (a mix of single‑ and multi‑hop QA datasets).

Baseline comparisons are organized into two settings. Direct Reasoning models answer in a single pass without external retrieval, including GPT‑5, Claude‑4‑Sonnet, Claude‑3‑7‑Sonnet, OpenAI Deep Research, OpenAI‑o3, and the Qwen3.5‑9B‑Instruct backbone.

Open Source Agents encompass recent search and deep‑research agents of comparable scale, such as R1‑Searcher‑7B, Search‑R1‑7B, ZeroSearch‑7B, ASearcher‑7B, DeepResearcher‑7B, PokeeResearch‑7B, WebSailor‑7B, WebExplorer‑8B, Marco‑DR‑8B, MiroThinker‑v1.0‑8B, and DeepDive‑9B.

The self‑evolving training loop runs for $R=15$ rounds; each round generates a batch of $B=10{,}000$ questions, requires at least $C_{min}=4{,}000$ correct trajectories to trigger training, and targets $N_{target}=4{,}000$ trajectories after importance‑sampling mixing.

Exponential decay across rounds uses factor $\gamma=0.5$, and training employs Llama Factory with one epoch per round, learning rate $5\times10^{-6}$, cosine schedule, $10\%$ warmup, DeepSpeed ZeRO‑2, BFloat16 precision, and a maximum sequence length of $32{,}768$ tokens.

The prompt template is the stripped version “`qwen3_5_nothink`”, ensuring that no extraneous thinking instructions interfere with tool‑use behavior.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers