AREX: Towards a Recursively Self-Improving Agent for Deep Research

Shuqi Lu, Chaofan Li, Kun Luo, Zhang Zhang, Hui Wang, Hongwang Xiao, Zheng Liu, Lei Xiong, Jiahao Wang, Sen Wang, Xiyan Jiang, Wanli Li, Yuyang Hu, Hongjin Qian, Bingyu Yan, Ziyi Xia, Yingxia Shao, Kang Liu, Zhicheng Dou, Di He, Chaozhuo Li, Qiwei Ye, Zhongyuan Wang

AREX uses recursive verification and autonomous context updates to transform long-horizon research into a series of targeted, constraint-aware tasks.

How can we improve deep research agents by using a recursive self-improvement loop that verifies and refines candidate answers?

Deep research agents often struggle with long-horizon tasks because they accumulate noisy interaction histories and fail to systematically resolve conflicting evidence or unmet constraints. AREX addresses this by alternating between an inner research loop that gathers evidence and an outer loop that audits the provisional answer constraint-by-constraint, recursively refining the research objective until confidence thresholds are met. To maintain focus, the agent autonomously compresses its interaction history into a compact state that preserves verified findings and unresolved constraints, preventing the model from revisiting invalidated paths or losing track of its research plan.

Paper Primer

The core mechanism is a bi-level recursive loop: the inner loop executes search and tool use, while the outer loop evaluates the resulting answer's confidence. If confidence is low, the outer loop identifies unresolved issues and reformulates the research objective for a new round, effectively treating verification as an active control signal rather than a final filter.

To prevent context bloat, AREX uses an autonomous context-update tool that acts like a project manager: it reads the messy, accumulated research log and distills it into a clean "improvement state" containing only verified evidence, rejected candidates, and the next actionable steps.

AREX significantly outperforms comparable-scale baselines across diverse search and reasoning benchmarks.

AREX-Base (122B MoE) consistently improves over the Qwen3.5-397B backbone and outperforms specialized research agents like MiroThinker-H1 and DeepSeek-V4-Pro on benchmarks including DeepSearchQA and WideSearch-en. The full AREX system achieves a 22.9-point accuracy gain on BrowseComp compared to a baseline without recursive self-improvement or context updating.

Why is this approach more effective than simply giving the model a larger context window?

Larger context windows do not solve the problem of "stale" information; models often struggle to distinguish between verified evidence and obsolete plans within a raw, uncompressed history. AREX’s autonomous updates explicitly filter out noise and organize the state around unresolved constraints, ensuring the model focuses on what remains to be proven.

What is the primary scope of this method?

AREX is designed for multi-constraint deep research tasks where verification can be decomposed into simpler, checkable conditions. It is less applicable to tasks where the "correct" answer is subjective or cannot be verified against external evidence.

Researchers should shift from viewing long-horizon agentic tasks as a single, continuous search trajectory toward a recursive process where intermediate verification drives the next research objective.

Motivation and Problem Framing

We expose the discovery‑verification cost gap that hampers deep research agents and motivate AREX.

Deep research agents must locate evidence and satisfy multiple coupled constraints, yet naïve longer search does not guarantee progress. The core obstacle is a discovery–verification asymmetry: finding a jointly satisfying answer is expensive, while checking a candidate can be broken into cheap, constraint‑wise tests. This gap motivates a new recursive self‑improvement paradigm.

Deep research agents waste effort because they treat discovery and verification as a single monolithic step, ignoring the fact that verification is far cheaper and can guide subsequent discovery.

The dominant bottleneck in deep research is the cost asymmetry between expensive discovery and cheap verification.

The AREX Framework

Recursive Self-Improvement repeatedly refines answers via inner and outer loops.

Long‑horizon research demands many tool calls, which quickly exhaust context windows. AREX tackles this by recursively alternating a focused inner loop with a concise outer loop that decides whether to accept, refine, or restart the answer.

The system runs an inner research loop that gathers evidence and proposes a provisional answer, then an outer loop that checks a confidence score and either accepts the answer, refines the trajectory, or restarts from the original query.

Step 1: Derive objective $q^{(1)}$ = “Find capital of France”; trajectory $\mathbf{h}^{(1)}_0$ is empty.

Step 2: Policy proposes action $a^{(1)}_1$ = “search('capital of France')”; the environment returns observation $o^{(1)}_1$ = “Paris (Wiki)”. The analysis $m^{(1)}_1$ records this finding and the trajectory becomes $\mathbf{h}^{(1)}_1$.

Step 3: Invoke

Step 4: Call

Step 5: Outer loop compares $s^{(1)}=85$ to threshold $\tau=80$ and accepts the answer.

This toy run shows how a single search result can be compacted, scored, and accepted without ever revisiting the full interaction history.

**Figure 2.** AREX's recursive self-improvement framework. The inner research loop maintains the research state and externalizes a provisional answer with supporting evidence and confidence, while the outer self-improvement loop accepts, refines, or restarts based on confidence and trajectory assessment.

How does the outer loop differ from a simple confidence‑threshold check?

Beyond checking whether $s^{(k)}\ge\tau$, the outer loop also runs a recoverability assessment $G_{\theta}$ that decides whether the existing trajectory contains useful information to preserve (Refine) or is too noisy and must be discarded (Restart). This extra step lets the system reuse partial progress instead of always starting over.

Constructing Research Trajectories

We build a specialized dataset that forces agents to perform multi‑step, evidence‑grounded research.

AREX needs training data that go beyond single‑turn QA; otherwise it cannot learn the recursive research behavior the paper targets.

The pipeline creates two complementary resources: (1) a set of hard research tasks that require multi‑hop evidence gathering, and (2) high‑quality teacher trajectories that demonstrate how to solve those tasks step‑by‑step.

Generate a latent answer $y$ and a set of constraints $C(y)$ describing verifiable research objectives.

Transform $C(y)$ into an indirect query $x = f(y, C')$ that forces multi‑hop search.

Collect teacher trajectories $\tau$ for each $x$ using a strong model in the same tool environment as AREX.

Apply three quality‑control filters: iterative investigation, tool‑interaction validity, and evidence‑grounded answer reconstruction.

Discard trajectories with confidence score $s_{\text{conf}}$ below the threshold $\tau_{\text{conf}}$.

Produce the final task set $D_{\text{task}}$ and trajectory set $D_{\text{traj}}$ for training.

Form the constraint set $C(y)=\{c_1,c_2\}$.

Transform $C(y)$ into an indirect query $x$: “Which European city is home to the Eiffel Tower and lies on the Seine River?”

Run a teacher model on $x$, which first searches a web page about landmarks, then retrieves a page about French rivers, and finally combines the evidence to output “Paris”.

Check that the trajectory includes both tool calls and that the final answer can be reconstructed from the retrieved pages.

Compute the confidence score $s_{\text{conf}}$; if it exceeds $\tau_{\text{conf}}$, keep the trajectory.

This toy example shows how abstract constraints force the model to perform multi‑hop retrieval and synthesis rather than memorizing a single fact.

How does this task synthesis differ from ordinary QA dataset creation?

Standard QA datasets usually provide a direct question that can be answered by a single fact lookup. Here we start from a hidden answer, generate multiple verifiable constraints, and then rewrite them into an indirect query that requires the model to gather evidence from several sources and verify each constraint before arriving at the answer.

The pipeline yields multi‑step investigation trajectories that teach AREX to iteratively gather evidence, maintain research state, and produce evidence‑grounded answers.

Agentic Mid-training

Training pipeline combines staged capability building with step‑aware reinforcement learning.

The training pipeline first builds heterogeneous research capabilities in stages, then refines them with a step‑aware reinforcement‑learning objective that emphasizes decisive decisions.

We first teach the model basic browsing and evidence‑gathering, then strengthen its long‑form reasoning, finally replay only the most informative intermediate decisions to avoid interference.

Stage 1‑step 1: model issues query “who discovered X?” → receives web result “Alice”.

Stage 1‑step 2: reads snippet “Alice discovered X in 1990.” → stores evidence.

Stage 1‑step 3: extracts fact “Alice discovered X”.

Stage 2‑step 1: hypothesizes “Alice is the answer.” → matches stored evidence.

Stage 2‑step 2: verifies hypothesis by checking consistency with extracted fact → succeeds.

Key‑step consolidation: only step 3 (evidence extraction) and Stage 2‑step 2 (verification) are marked as key steps for replay.

Focusing loss on the evidence‑extraction and verification steps yields stronger signals for the most decision‑critical moments while keeping earlier tool‑use behavior intact.

How does this multi‑stage approach differ from simply mixing all data in one large training set?

Mixing all data treats browsing and reasoning uniformly, which leads to interference: the model can forget tool‑use when over‑exposed to reasoning examples. By separating stages and then replaying only the high‑value key steps, we preserve early capabilities and amplify learning on the most informative decisions.

Instead of assigning the same credit to every token in a trajectory, we compute a step‑level advantage and give extra bonus to annotated key steps, so the model learns to prioritize decisive actions.

Why not apply the key‑step bonus to every step instead of only the annotated ones?

Only a small fraction of steps are truly decision‑critical; rewarding all steps would dilute the signal and encourage the model to over‑fit routine actions. The annotations isolate the high‑impact moments, allowing the bonus to sharpen learning where it matters most.

Together, the staged mid‑training and step‑aware RL form the AREX recipe: heterogeneous capability acquisition followed by fine‑grained credit assignment that emphasizes the most informative research decisions.

Experimental Results and Ablations

AREX attains top accuracy on deep‑search benchmarks while remaining efficient.

AREX reaches 82.5 % accuracy on the BrowseComp benchmark when both ACU and the outer self‑improvement loop are enabled.

Table 3 shows the full system (ACU + outer loop) achieving 82.5 % versus 59.6 % without either component.

Recall that AREX decomposes research into an inner discovery loop and an outer self‑improvement loop, using ACU to refresh its context. We evaluate the system on six benchmarks that span deep research (BrowseComp, DeepSearchQA), agentic task completion (GAIA, xbench‑2510), broad‑coverage retrieval (WideSearch‑en), and tool‑augmented reasoning (HLE).

**Figure.** AREX benchmark results

When accounting for model scale, AREX‑Base (≈10 B active parameters) outperforms the much larger Qwen3.5‑397B family and rivals leading closed‑source systems. It also beats the strongest open‑source contender, MiroThinker‑H1, on DeepSearchQA and text‑only HLE.

The outer self‑improvement loop adds another ~10 points: without ACU the outer loop raises accuracy from 59.6 % to 69.8 %; with ACU it lifts performance from 71.4 % to 82.5 %.

**Figure 4.** Average step loss after full-trajectory mid-training.

Table 2 reveals that the dominant trigger for `update_context` is “revise search strategy” (66.9 % of calls), and the most frequently preserved content are “next‑step plan” (96.4 %) and “unresolved constraints” (95.5 %). This pattern shows ACU acting as a proactive consolidation step rather than a hard‑limit safeguard.

AREX’s combination of ACU‑driven context refreshes and recursive self‑improvement yields large accuracy gains on deep‑search benchmarks, confirming the effectiveness of the discovery‑verification asymmetry.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers