VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

Seohyun Lee, Seoung Choi, Dohwan Ko, Jongha Kim, Hyunwoo J. Kim

VideoSearch-R1 uses iterative latent-space query refinement and reinforcement learning to bridge the gap between coarse video retrieval and fine-grained temporal grounding.

How can we improve video retrieval and reasoning by iteratively refining queries using learned soft tokens instead of static text?

Existing video retrieval pipelines treat search as a one-shot preprocessing step, meaning that if the initial retrieval is slightly off-target, the subsequent fine-grained temporal grounding task is doomed to fail. VideoSearch-R1 replaces this static approach with an iterative agentic loop that verifies retrieval relevance and uses Soft Query Refinement (SQR) to adjust query embeddings in continuous latent space. This framework achieves state-of-the-art performance on Video Corpus Moment Retrieval (VCMR) benchmarks, significantly improving both retrieval accuracy and temporal localization precision.

Paper Primer

VideoSearch-R1 treats video retrieval as a multi-turn interaction rather than a fixed input. At each turn, the model evaluates whether the retrieved video matches the query; if it detects a mismatch, it generates a small set of soft query tokens to refine the search representation.

Soft Query Refinement (SQR) is the core mechanism: instead of rewriting queries as text, the model generates continuous latent embeddings that are appended to the original query. This is like a search engine that doesn't just re-read your keywords, but instead adjusts the "vibe" of your search vector based on the specific visual errors it just encountered.

VideoSearch-R1 achieves state-of-the-art performance on VCMR benchmarks, including ActivityNet-FIG, Charades-FIG, and DiDeMo-FIG.

The model outperforms both zero-shot and fine-tuned baselines in retrieval accuracy (VR) and temporal grounding (VCMR) across all three datasets. On the DiDeMo-FIG dataset, the model improves the 0.3/R@1 metric by 9.7 points compared to the fine-tuned baseline.

SQR is more efficient and precise than explicit text-level query refinement.

SQR requires only 8 latent tokens to outperform hard query refinement (HQR), which averages 26.8 tokens and often introduces semantic noise that confuses the search engine. SQR improves R@1 in video retrieval by 7.2 points, nearly double the 3.7-point gain achieved by HQR.

Why is this iterative approach necessary compared to simply using a more powerful retrieval model?

Even powerful models struggle with the "decoupled" nature of existing pipelines, where retrieval and reasoning operate independently. VideoSearch-R1 uses reinforcement learning to jointly optimize these stages, allowing the model to self-correct retrieval errors based on the specific requirements of the downstream temporal grounding task.

What is the scope of this framework — does it work for any video task?

The framework is specifically designed for Video Corpus Moment Retrieval (VCMR), which requires both corpus-level retrieval and fine-grained temporal localization. It relies on a predefined maximum number of turns (T=2) to balance computational efficiency with accuracy.

For researchers building video agents, this paper demonstrates that continuous latent-space refinement is a more efficient way to steer retrieval than explicit text rewriting. The shift toward joint RL-based optimization of retrieval and reasoning is the new standard for high-precision video understanding.

Introduction and Motivation

VideoSearch‑R1 tackles the gap between coarse video retrieval and fine‑grained reasoning.

Large video corpora demand systems that first locate a relevant video (inter‑video reasoning) and then perform precise, query‑conditioned operations inside that video (intra‑video reasoning). Existing pipelines treat retrieval as a one‑shot preprocessing step; when the initial retrieval is wrong, there is no mechanism to correct it, and downstream fine‑grained reasoning inevitably fails.

The core problem is the disconnect between coarse video retrieval and the fine‑grained reasoning that follows it.

VideoSearch‑R1 embodies this loop: it queries a video search engine, checks whether the returned video matches the user’s intent, and, if not, refines the query in a continuous latent space using Soft Query Refinement (SQR). The refined query guides the next retrieval step, and the process repeats until a suitable video is identified, after which the system performs intra‑video reasoning such as temporal grounding. The entire loop is trained with Group Relative Policy Optimization (GRPO) to maximize task‑level rewards from both retrieval and downstream reasoning.

**Fig. 1:** An illustrative example of VIDEOSEARCH-R1. As an agentic AI system, VIDEOSEARCH-R1 enables multi-turn interaction through iterative video retrieval and reasoning, leveraging an external video search engine. This pipeline unifies corpus-level inter-video reasoning (e.g., video retrieval) with intra-video reasoning (e.g., temporal grounding) grounded in the retrieved video.

The primary gap this work addresses is the mismatch between coarse inter‑video retrieval and the fine‑grained intra‑video reasoning required for real‑world video tasks.

Related Work and Baselines

Key related work and the hard query refinement mechanism.

Related work spans three strands: video retrieval systems that rely on dual‑encoder similarity, multi‑turn reasoning frameworks that iteratively refine decisions, and soft‑reasoning approaches that update latent states instead of explicit text.

Encodes images and text into a shared embedding space; retrieval is performed by cosine similarity of the embeddings.

Fine‑grained vision‑language models compute cross‑modal attention between a query and candidate videos, improving top‑K reranking quality.

Agents interleave tool calls (e.g., retrieval) and language model inference, refining their internal state across turns to solve complex tasks.

A reinforcement‑learning algorithm that shapes the policy gradient to align intermediate decisions with downstream objectives.

Represents the final hidden state of an LLM as a compact latent token, enabling downstream modules to consume a single vector instead of a full text sequence.

HQR rewrites the user’s textual query into a new discrete token sequence, explicitly changing the words that the retrieval engine sees.

How does Hard Query Refinement differ from Soft Query Refinement?

HQR produces an explicit textual rewrite—new discrete tokens that replace the original query—whereas Soft Query Refinement (SQR) appends continuous latent embeddings to the query vector without altering the visible text. The hard edit can trigger lexical effects in the retriever, while the soft edit only shifts the embedding space.

The VideoSearch-R1 Mechanism

We detail Soft Query Refinement and the iterative retrieval‑reasoning loop for training and inference.

Static text queries often retrieve irrelevant videos, forcing the system to repeat costly searches.

Instead of rewriting the whole query, the model appends a few learned soft tokens that subtly shift the query embedding toward the target video—like adding a few brush strokes to a sketch to refine details without redrawing the entire picture.

Initialize two soft token vectors: $s_{1}=[0.1,\,0.0]$, $s_{2}=[0.0,\,0.1]$.

Concatenate: $q_{1}= [q;\,s_{1};\,s_{2}] = [0.5,\,-0.2,\,0.1,\,0.0,\,0.0,\,0.1]$.

Feed $q_{1}$ to the search engine; retrieval score rises from $0.30$ to $0.45$ (illustrative).

InfoNCE loss back‑propagates, updating $s_{1},s_{2}$ toward vectors that better align with the target video.

After a few updates the soft tokens become $s_{1}=[0.2,\,-0.1]$, $s_{2}=[0.1,\,0.2]$, yielding a refined query that retrieves the correct video.

Only a handful of soft vectors are needed to steer the query embedding, making refinement cheap and fully differentiable.

A single retrieval often fails; the model therefore enters an iterative loop that alternates verification and soft‑query refinement until a match is found or the turn budget $T$ is exhausted.

The system treats each turn as a hypothesis‑testing cycle: retrieve a video, verify relevance, and if wrong, subtly adjust the query with soft tokens before trying again.

How is this multi‑turn loop different from standard single‑turn video retrieval?

Standard retrieval issues one query and accepts the top result, whereas the loop explicitly verifies the match, feeds back a learned corrective signal (soft tokens), and repeats the search. The extra verification and refinement steps allow the system to recover from initial mismatches.

Retrieve top‑1 video using the current query $q_{t}$.

Run the verification module to obtain reasoning trace $r_{t}$ and decision $y_{\text{ret}}$.

If $y_{\text{ret}}$ is “not match”, generate $N$ soft tokens $q^{\text{soft}}_{t}$ and construct $q_{t+1}=[q_{1}\,\|\,q^{\text{soft}}_{t}]$.

Feed $q_{t+1}$ back to the search engine for the next turn.

If $y_{\text{ret}}$ is “match”, invoke the temporal grounding module to predict $y_{\text{time}}$ (start/end timestamps) and terminate.

Stop automatically after $T$ turns if no match is found.

Turn 1: retrieve video A; verification says “not match”; generate soft token $s_{1}=[0.2]$; refined query $q_{2}=[q_{1}\,\|\,s_{1}]$.

Turn 2: retrieve video B; verification says “not match”; generate soft token $s_{2}=[0.1]$; refined query $q_{3}=[q_{1}\,\|\,s_{1}\,\|\,s_{2}]$.

Turn 3: retrieve video C; verification says “match”; temporal grounding predicts start = 12 s, end = 18 s.

Each soft token accumulates a corrective signal, so after a few turns the query drifts toward the correct video without ever discarding the original semantics.

Training proceeds in two stages: first we warm‑start the model with supervised fine‑tuning (SFT) to learn basic reasoning and soft‑token generation, then we refine the whole loop with GRPO reinforcement learning.

Stage 1 (SFT): initialize from Qwen3‑VL‑2B‑Instruct and train on supervised data using verification loss $L_{\text{verif}}$, temporal loss $L_{\text{time}}$, and the InfoNCE retrieval loss $\mathcal{L}_{\text{ret}}$ for soft tokens.

Stage 2 (GRPO): treat the entire iterative loop as a policy, sample trajectories, compute rewards $R_{\text{format}}$, $R_{\text{verif}}$, $R_{\text{ret}}$, and $R_{\text{time}}$, and update parameters with the GRPO algorithm.

**Fig. 2:** Comparison between hard query refinement and our Soft Query Refinement (SQR). SQR generates soft query tokens to perform fine-grained adjustments to the original query representation. In SQR, the soft query tokens are trained using the InfoNCE objective $\mathcal{L}_{ret}$, which provides richer discriminative supervision than the standard next-token prediction used in hard query refinement.

**Fig. 3:** Iterative video retrieval and reasoning of VIDEOSEARCH-R1. Given an initial query $q_1$, VIDEOSEARCH-R1 retrieves the top-1 video from a corpus via a video search engine and performs verification, producing a reasoning trace $r_t$ and a matching decision $y_t^{ret}$. If $y_t^{ret} = \text{'not match'}$, the model performs SQR by generating soft query tokens $q_t^{soft} \in \mathbb{R}^{N \times D}$ to construct a refined query $q_{t+1} = [q_1 || q_t^{soft}]$. If matched, the model conducts temporal grounding to predict the start and end timestamps $y^{time}$.

Experimental Framework

Key results and the experimental design that enable them.

VideoSearch‑R1 attains $33.3\%$ 0.3 IoU R@1 on Charades‑FIG, the highest among all reported baselines.

Charades‑FIG VCMR results show VideoSearch‑R1 surpassing CONQUER (12.2 %), SQuiDNet (22.0 %), and Qwen3‑VL‑2B (FT) (17.2 %).

The experimental pipeline follows two training stages and evaluates on three VCMR benchmarks, each measuring joint retrieval and temporal grounding.

Given a textual query, the system must first pick the correct video from a large corpus and then pinpoint the exact start‑and‑end timestamps of the described moment inside that video.

How does VCMR differ from standard video retrieval?

Standard retrieval stops after selecting a video. VCMR continues by requiring the model to output exact start and end timestamps, so a method must reason about both “which video?” and “where in the video?” simultaneously.

**Table 1.** Performance comparison on Charades-FIG, DiDeMo-FIG, and ActivityNet-FIG datasets for VCMR, VER, and VR tasks.

**Table.** Ablation study on the impact of different rewards on VCMR, VER, and VR performance.

Charades‑FIG, DiDeMo‑FIG, and ActivityNet‑FIG each pair a large video corpus with dense temporal annotations, providing a rigorous joint test of retrieval and moment grounding that aligns with VideoSearch‑R1’s design goals.

Performance Evaluation

VideoSearch‑R1’s iterative refinement yields large gains across retrieval and reasoning metrics.

VideoSearch‑R1 raises R@1 by 6.0 on ActivityNet‑FIG compared with the strongest baseline.

Table 2 shows the baseline Qwen3‑VL‑2B (FT) achieving 28.4 R@1, while VideoSearch‑R1 reaches 34.4 R@1.

**Table 4.** Ablation studies of reward design on DiDeMo-FIG.

**Fig. 4:** Effect of the number of soft tokens. R@1 is computed over samples with refined queries.

**Fig. 5: Effect of multi-turn inference.** The performance on VCMR saturates at $T = 3$.

**Table 3.** Effect of scaling Stage 1 training data.

Ablation and Sensitivity Analysis

Ablation analysis shows soft query tokens boost retrieval and outperform hard textual refinement.

We first examine how adding soft query tokens affects retrieval quality, then compare SQR against HQR and study multi‑turn inference.

**Fig. 6:** Changes in the retrieved video as the number of soft tokens increases. The rank of the ground-truth video gradually improves as soft tokens are appended, capturing increasingly fine-grained semantics.

Performance saturates after two inference turns, with no further gain at turn 3.

R@1 improves from turn 1 to turn 2 but remains unchanged at turn 3.

**Table 5.** Quantitative results of hard query refinement (HQR) and our soft query refinement (SQR) on ActivityNet-FIG

**Fig. 7.** Qualitative comparison between SQR and HQR.

In a multi‑turn example, the initial query “man in a dark gray t‑shirt applying lotion to a black shoe indoors” retrieves a visually similar but semantically mismatched video at turn 1; the model’s reasoning trace flags missing cues, generates a <REFINE> token, and after SQR refinement at turn 2 the ground‑truth video rises to rank 1 with accurate temporal localization (IoU 0.89).

Conclusion

We wrap up VideoSearch‑R1’s contributions and acknowledge supporting grants.

VideoSearch‑R1 is an agentic framework that repeatedly retrieves candidate videos, checks their semantic fit to the user’s intent, refines the query, and reasons over the retrieved content, all within a multi‑turn loop.

Soft Query Refinement (SQR) optimizes the query in latent space, avoiding verbose textual rewrites and achieving fine‑grained adjustments with far fewer generated tokens; reinforcement learning trains the whole pipeline to set $a$ new VCMR benchmark.

This work was supported by the InnoCORE program (30 % funding), the ETRI grant (30 % funding), and the IITP grant (40 % funding) from the Korean government.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers