Multi-Turn Agentic Scientific Literature Search via Workflow Induction

Jisen Li, Bingxuan Li, Nanyi Jiang, Xuying Ning, Xiyao Wang, Yifan Shen, Heng Wang, Yuqing Jian, Xiaoxia Wu, Ben Athiwaratkun, Pan Lu, Jiaxuan You, Bingxin Zhao

PAPERPILOT treats scientific literature search as an interactive, editable workflow induction problem rather than a static query.

How can search agents move beyond single-query retrieval to iteratively refine scientific literature search through multi-turn workflow induction?

Scientific literature search agents often fail because they treat user intent as a static query, forcing the system to guess preferences without a way to refine the underlying retrieval strategy. PAPERPILOT frames search as workflow induction: the agent constructs an executable directed acyclic graph (DAG) of search operators that can be explicitly edited and refined through multi-turn user feedback. This approach significantly improves retrieval quality and stability, with the trained PAPERPILOT-9B model increasing Hit@5 from 58.0 to 77.0 and eliminating workflow execution errors.

Paper Primer

PAPERPILOT replaces fixed retrieval pipelines with a library of typed paper-search operators—such as citation expansion, filtering, and reranking—that the agent composes into a DAG. The core move is treating user feedback not as additional query text, but as a set of instructions to modify the workflow structure and parameters, effectively acting like a programmable search engine.

Explicit workflow induction significantly outperforms static retrieval pipelines in multi-turn scientific search.

In multi-turn interaction, PAPERPILOT-9B improves Hit@5 from 58.0 to 77.0 and MRR from 47.5 to 59.4 compared to the base Qwen3.5-9B toolset agent.

Workflow-induction training improves the stability of agentic search under multi-turn refinement.

The model reduces workflow execution errors from 9.5% to 0% compared to the untrained base model.

Why is workflow induction superior to simply using a more powerful language model for search?

Fixed pipelines or implicit reasoning make it difficult for users to steer the search process. By using an explicit DAG, the agent allows for precise, inspectable refinements—such as changing a filter or citation direction—that a black-box model cannot reliably execute.

What is the scope of this system, and where might it struggle?

The system is currently optimized for computer science literature and relies on a predefined symbolic operator library. It may not generalize to specialized domains requiring search behaviors outside this library or to open-ended, real-world search scenarios not covered by the benchmark.

The Challenge of Scientific Search

We frame scientific literature search as an iterative, preference‑dependent workflow induction problem.

Scientific literature search is inherently iterative: a single query rarely captures a researcher’s full intent, which evolves as preferences become clearer through interaction. Current search agents follow fixed pipelines or rely solely on language‑model reasoning, making their strategies opaque and hard to steer.

When a researcher issues an initial query, the system must guess latent preferences—such as recency, methodological focus, or citation direction—without explicit guidance.

**Figure 1.** Overview of PAPERPILOT. Compared with single-turn scientific literature search, PAPERPILOT uses multi-turn feedback to clarify user intent and refine the search workflow before producing final results.

Static queries fail in scientific research because they cannot capture evolving, preference‑dependent intents, leading to irrelevant or incomplete retrieval.

Inducing Search Workflows

PAPERPILOT builds adaptive DAG workflows to steer multi‑turn scientific paper search.

The fixed pipeline used by prior systems cannot adapt to diverse user intents, leading to irrelevant or incomplete results.

Instead of a one‑size‑fits‑all retrieval chain, the agent builds a custom search workflow on the fly, picking only the operators that matter for the current query.

Node $v_1$ = (`kw_extract`, {keywords = {“attention”, “transformer”}}) produces a set of keyword strings.

Node $v_2$ = (`cite_expand`, {depth = 1}) consumes the keyword set and returns papers citing any keyword match.

Node $v_3$ = (rank, {metric = “LLM‑score”}) takes the cited papers, scores them, and outputs a ranked list of three papers.

This example shows how the DAG wires the output of one operator directly into the input of the next, eliminating any unnecessary intermediate steps.

How does this differ from a static retrieval pipeline?

A static pipeline runs the same sequence of operators with fixed parameters for every query. Workflow induction builds a DAG that is tailored to the current query and can change its structure or parameters after each user interaction.

The DAG is a symbolic representation of the search process: nodes are concrete operator calls, edges carry intermediate results, and the overall graph encodes the execution order.

Is the DAG just another name for a search tree?

No. A search tree branches on decisions, whereas a DAG explicitly models data flow between operators; multiple branches can converge on the same intermediate result, which a tree cannot represent.

Observe current state $s_t$ (anchor paper, query, candidate set, interaction history).

Induce a DAG $G_t = \text{Induce}(q, p_0, P_t, H_t)$ by selecting operators and setting parameters.

Execute $G_t$ node‑by‑node to obtain new candidates $P_{t+1}$ and visible response $y_t$.

Present $y_t$ (ranked list, evidence, or clarification) to the user.

Receive user feedback $f_t$ and update the interaction history $H_{t+1}$.

Refine the workflow $G_{t+1} = \text{Refine}(G_t, f_t, H_{t+1})$ for the next turn.

Turn‑level loop for inducing, executing, and refining a DAG workflow.

Training teaches the agent to generate valid DAGs and to prefer those that best satisfy user intent.

**Figure 2.** Overview of PAPERPILOT. Given an anchor paper and a user search intent, the agent induces a DAG-structured search workflow from a predefined toolset, executes the workflow over the literature corpus, and refines the workflow through multi-turn user interaction.

**Figure 11.** Workflow refinement example

**Figure 12.** An example PaperPilot search session

Empirical Performance and Human Studies

Multi-turn interaction with PAPERPILOT‑TOOLSET reaches 84.0 Hit@5, the highest score.

Multi‑turn interaction with PAPERPILOT‑TOOLSET achieves the highest Hit@5, improving by 12.0 points over the best single‑turn baseline.

GPT‑5.4 with Web Search and PAPERPILOT‑TOOLSET reaches 84.0 Hit@5 versus 72.0 Hit@5 in single‑turn.

The agent iteratively asks clarification questions, incorporates user feedback into a symbolic DAG, and re‑executes retrieval, refining the search workflow over several turns.

How does the Multi‑turn Protocol differ from a single‑turn search?

In single‑turn the agent receives only the original query and returns a static result list; in Multi‑turn it repeatedly queries the user, updates the symbolic DAG with concrete edits, and re‑runs retrieval, allowing the workflow to evolve toward the user’s intent.

Multi-turn citation search powered by a symbolic DAG agent.

**Figure.** A user interface prompt asking for clarification on which branch of the Transformer architecture to prioritize for a follow-up search. The options provided are: "NLP Encoders & Pre-training", "Generative Language Models", "Vision & Multimodal", "Efficiency & Long-Sequence", "All branches equally", and a "Stop & finalize now" button.

**Figure 3.** Distribution of agent actions across turns.

**Figure 4.** Effect of multi-turn refinement on retrieval quality across representative systems.

**Table 4.** Human-study results averaged over five sessions per participant. SR = Success Rate (%); T1D = Top-1 Distance (lower is better); QSS = Question Satisfaction Score (0–5); NTS = Number of Turns Until Satisfaction (lower is better). Best values are in bold.

Multi‑turn refinement consistently outperforms single‑turn baselines.

Workflow Refinement and Ablations

We examine how workflow‑induction components affect performance and outline remaining constraints.

Scientific literature search proceeds iteratively; PAPERPILOT improves it by inducing and refining symbolic DAG‑structured workflows through multi‑turn interaction.

We evaluate two workflow‑level tasks: (1) workflow induction, where the model generates a complete executable DAG from the full interaction context, and (2) step‑wise workflow refinement, where the model updates an existing DAG given the latest feedback.

Workflow‑induction training raises whole‑workflow TF‑IDF cosine similarity from 0.0470 to 0.3619.

PAPERPILOT‑9B vs. base Qwen3.5‑9B on the generation task.

Signature‑level similarity improves from 0.0281 to 0.2238 thanks to workflow‑induction.

Same comparison as above.

PAPERPILOT‑9B achieves a relative efficiency score of 3.26 while costing only \$0.00108 per case.

Cost‑efficiency (CE) and Rel. Eff. metrics in Table 3.

For workflow editing, the final full‑workflow cosine similarity reaches 0.9469.

PAPERPILOT‑9B after step‑wise refinement.

Query‑level similarity after editing peaks at 0.8777.

Same editing evaluation.

We vary the first‑stage candidate pool size K from 8 to 20 to test whether larger pools boost retrieval quality.

Recall@50 declines from 0.411 at K=8 to 0.374 at K=20.

Retrieval metrics across candidate‑pool sizes.

Hit@5 drops from 0.705 at K=10 to 0.660 at K=20.

Same sensitivity sweep.

In a human study, participants interacted with each system over five sessions, measuring success rate, top‑1 distance, question satisfaction, and turns until satisfaction.

PAPERPILOT attains a success rate of 74.7 %.

Human‑study results, Table 4.

Its mean top‑1 distance is 2.4, indicating closer ranking to user preference.

Same human‑study evaluation.

Question satisfaction scores reach 4.2 out of 5.

Human‑study evaluation.

Limitations include reliance on a fixed symbolic operator library, potential bias from teacher‑model generated supervision data, a benchmark confined to computer‑science papers, and dependence on an LLM‑based user simulator for scalability.

**Table 3.** Workflow-level evaluation on generation and editing tasks. **CE** denotes successful workflow-generation cases per dollar, and **Rel. Eff.** denotes the log-scale normalized CE score.

**Figure 6.** Workflow-generation similarity between Qwen3.5-9B and PAPERPILOT-9B.

Tooling and Sensitivity Analysis

Appendix A details the operator library and formal task definition for the paper‑search agent.

This appendix enumerates the concrete operators that PAPERPILOT can invoke and formalizes the multi‑turn search problem as a decision process.

The library groups low‑level paper‑search primitives (e.g., keyword lookup, citation expansion, scoring) into reusable operators that the agent composes into a workflow.

**Table 5.** Toolset.

The sensitivity analyses reveal a trade‑off: expanding the keyword pool increases coverage but also introduces distractors, leading to non‑monotonic performance trends.

Experimental Configuration

Evaluation setup, baselines, metrics, and analysis details.

We use Qwen3.5‑397B‑A17B as the fixed simulator for all multi‑turn experiments, guaranteeing that every agent receives identical user‑model feedback while the simulator alone accesses the hidden gold papers.

In the single‑turn setting each case is evaluated independently: the agent retrieves metadata from Semantic Scholar, runs one retrieval pass, and returns the top‑50 papers.

In the multi‑turn setting a clarification loop precedes retrieval; at each turn the clarifier LLM asks one question with 3–5 answer options, the simulator answers, and the pair is fed through a strict leakage‑control pipeline before being appended to the interaction history.

C.4 lists the baselines against which PAPERPILOT variants are compared.

Web‑search uses GPT‑5.4 with a standard web‑search tool, representing a strong general‑purpose agent that can issue natural‑language queries but does not employ the PAPERPILOT toolset.

DeepResearch is evaluated as a one‑shot commercial scientific search system; its API only supports single‑turn requests and lacks a controllable workflow‑refinement interface.

PAPERPILOT‑WORKFLOW runs Qwen3.5‑9B, Qwen3.5‑397B, and GPT‑5.4 with a deterministic pipeline of sourcing, scoring, and reranking, isolating the benefit of tool use from workflow induction.

PAPERPILOT‑TOOLSET enables the same models to generate a DAG‑structured workflow using the full symbolic toolset, allowing configuration of operators and parameters.

PAPERPILOT‑9B is the trained model that learns to generate and refine DAG‑structured workflows conditioned on anchor papers, queries, interaction history, and simulated feedback.

C.5 defines the retrieval and workflow evaluation metrics used throughout the experiments.

Hit@K measures whether at least one gold paper appears in the top‑K results (reported for K = 5, 10, 15).

Recall@K quantifies the fraction of gold papers recovered within the top‑K list (main table reports Recall@50).

Mean Reciprocal Rank (MRR) captures how early the first gold paper appears, averaging the reciprocal of its rank across cases.

nDCG@K evaluates rank‑sensitive quality by normalizing discounted cumulative gain against an ideal ranking.

Workflow‑generation metrics include TF‑IDF cosine similarity over the whole workflow, query fields, filter predicates, and function signatures, as well as edit‑distance statistics for add/modify/remove operations.

Cost is computed from agent‑side token usage only; cost‑effectiveness (CE) reports successful cases per dollar, and Relative Efficiency (Rel. Eff.) provides a log‑scaled normalized CE score.

C.6 details the training regimen for the models.

Supervised fine‑tuning runs for 3 epochs with learning rate $2\times10^{-4}$, sequence length 14,336, and LoRA adaptation on attention and MLP projections.

Preference optimization continues from the SFT checkpoint, using an IPO‑style DPO objective with $\beta=0.2$, training for 3 epochs at learning rate $3\times10^{-5}$, sequence length 16,384, and gradient clipping norm 1.0.

C.7 evaluates generated DAGs against reference workflows using both graph‑level and text‑level metrics.

Graph‑level metrics compute Jaccard similarity over node identifiers and operator types, and report edit‑distance counts for added, removed, and modified nodes.

Text‑level metrics serialize workflows and compute TF‑IDF cosine similarity for the whole workflow, query‑related fields, filter predicates, and function signatures.

Refinement metrics assess whether each edit moves the current workflow closer to the reference next‑state workflow by measuring changes in cosine and Jaccard similarity.

C.8 describes the human study used to gauge interactive search performance.

Four dimensions are recorded: Success Rate (SR %); Top‑1 Distance (T1D) measuring rank distance between the user‑preferred paper and the system’s top result; Question Satisfaction Score (QSS, 0–5); and Turns Until Satisfaction (NTS) counting conversational turns to a satisfactory outcome.

D presents a sensitivity analysis on the size of the first‑stage candidate pool, showing that larger pools do not monotonically improve retrieval quality and can reduce recall‑oriented metrics.

The table reports Hit@1, Hit@5, Recall@15, Recall@50, nDCG@15, MRR, and pool size for K values ranging from 8 to 20, illustrating the trade‑off between candidate breadth and distractor noise.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers