Video-DeepResearch: Towards the Next-Generation Multimodal

Zhen Fang, Yu Zeng, Wenxuan Huang, Yiming Zhao, Shiting Huang, Tianfei Ren, Qi Lu, Qingnan Ren, Qisheng Su, Lionel Z. Wang, Qingyu Yin, Shuang Chen, Zehui Chen, Lin Chen, Zhenfei Yin, Yao Hu, Shaohui Lin, Wanli Ouyang, Shaosheng Cao, Feng Zhao

Video-DeepResearch enables autonomous agents to perform multi-step web exploration grounded in continuous video streams.

How can we extend multimodal agents from static images to continuous video streams for deep research tasks?

Current multimodal agents struggle to move beyond static images, often defaulting to textual search while ignoring visual evidence in video streams. This "modality bias" leads agents to rely on internal memory rather than active, verifiable exploration. Video-DeepResearch (Video-DR) solves this by decoupling perception from exploration, forcing the agent to perform exhaustive visual grounding before accessing web tools. The resulting 35B-parameter model achieves a state-of-the-art 64.0% accuracy on complex multi-hop tasks, outperforming proprietary models like Claude-4.5-Sonnet.

Paper Primer

The framework employs a two-stage training recipe: Supervised Fine-Tuning (SFT) on 7,000 curated trajectories, followed by Group Relative Policy Optimization (GRPO) to incentivize autonomous exploration. The core mechanism is a stage-wise tool unlocking strategy: the agent is initially restricted to visual tools (keyframe selection and entity cropping) to ensure it extracts sufficient visual context before it is permitted to trigger web-based text search.

Video-DeepResearch-35B-A3B establishes a new state-of-the-art in video-grounded deep research.

Performance on the VIDEODR-BENCH benchmark. 64.0% average accuracy, surpassing Claude-4.5-Sonnet (59.0%) by 5.0 percentage points.

Training methodology significantly outweighs raw parameter count in agentic performance.

Comparison between the 30B-A3B model and the 397B-parameter baseline. The 30B model achieves higher tool usage diversity and competitive accuracy (59.3%) compared to the 397B baseline, which relies heavily on text-only search.

Why is "parametric knowledge leakage" a critical failure mode for these agents?

Models often answer questions correctly using their internal training memory rather than by interacting with the environment, which masks their inability to perform genuine, verifiable multi-step research.

What distinguishes VIDEODR-BENCH from existing video benchmarks?

It consists of 200 multi-hop VQA instances that are provably dependent on external tools; any question answerable via internal memory alone is filtered out during the human-AI collaborative annotation process.

Effective video-based deep research requires a shift from passive inference to active, decoupled spatiotemporal grounding. Future agent development should prioritize training curricula that enforce tool-use diversity over simply scaling model parameters.

Introduction to Video-DeepResearch

We expose the core challenge of extending research agents from static images to continuous video streams.

Current multimodal agents excel on static images but falter when confronted with continuous video streams that require dense spatiotemporal grounding and open‑web exploration. Two critical bottlenecks emerge: (1) modality bias, where agents sidestep visual tools in favor of textual search, and (2) parametric knowledge leakage, where models lean on internal memorization rather than genuine tool‑augmented execution. These gaps prevent agents from reliably extracting and reasoning over evolving visual content across time.

Video‑DeepResearch reframes video‑based research as a sequence of key‑entity trajectories, forcing agents to continuously ground visual content while intermittently consulting the web for external knowledge.

The shift from static‑image research to continuous video streams forces agents to intertwine dense visual grounding with open‑world knowledge retrieval.

Formulating Video Research

We turn video research into a stepwise grounding pipeline built on keyframe selection and crop‑based search.

Directly applying image‑centric agents to video fails because the agent never isolates the moments that actually answer the query, leading to almost no visual tool usage.

Think of a video as a storyboard; the agent walks through it panel by panel, first picking the most informative panel, then zooming on the object of interest before asking the web for the answer.

Why not simply sample frames at a fixed interval instead of using `Select_Keyframe`?

`Select_Keyframe` is a learned decision that conditions on the query and past observations, so it chooses the frame most likely to contain the answer. Fixed‑interval sampling ignores the query and often wastes steps on irrelevant frames.

`Select_Keyframe` picks $t=3$, the frame where the car is first fully visible.

`Crop_Search` receives $v_3$ and a box $B=(120, 45, 80, 40)$ around the car, producing $c_{\text{vis}}$.

The cropped image is sent to an image‑search API, returning candidate images labeled “red sedan”.

`Text_Search` uses “red sedan” together with the original query to retrieve a web paragraph confirming the car’s color.

Agent assembles the final answer: “The car is red.”

The two‑step visual grounding isolates the exact moment and object, turning a vague video stream into a precise image query that downstream text tools can handle.

**Figure 1.** Overview of the VIDEO-DEEPRESEARCH pipeline.

Empirically, existing agents invoke visual tools on only 0.10 of tasks while relying on 1.27 text searches per task, and GPT‑5 reaches 57 % accuracy without any tool calls, confirming that current evaluations reward memorization over visual grounding.

The agent’s toolbox is a small menu: pick a frame, crop an entity, then fall back to ordinary web operations.

How does `Crop_Search` differ from simply feeding the whole keyframe to the image‑search engine?

`Crop_Search` isolates the target entity with a bounding box, producing a focused visual query that eliminates background clutter. Sending the full keyframe forces the search engine to consider many irrelevant objects, reducing precision.

Initial Empirical Findings

Empirical results reveal tool usage bias and knowledge leakage across three video agents.

Open‑source models invoke visual tools at a rate of only 0.10 per task, while relying on text tools about twelve times more often.

Table 1 shows Qwen3.5‑397B‑A17B uses 0.10 visual ops and 1.27 text ops per task.

Table 1 summarizes the accuracy and average tool invocation counts for each model. The two findings above highlight a strong modality bias toward text tools and a dangerous reliance on parametric knowledge.

Naïve image‑based agents fail on video tasks because they avoid visual tools and can cheat by memorizing answers.

The Video-DeepResearch Framework

We describe the Data Synthesis Pipeline that turns raw videos into grounded VQA trajectories.

The core obstacle is that no public dataset provides video‑QA pairs with explicit visual evidence, which blocks agents from learning reliable grounding and web‑search behaviors.

The pipeline converts raw, heterogeneous videos into high‑quality VQA instances that are explicitly tied to visual evidence, enabling agents to practice grounding and web‑search in a closed loop.

For each keyframe we run a pretrained object detector and obtain 1 entity per frame (e.g., “bee” in frame 1, “flower” in frame 2).

We crop each entity (size ≈ 64 × 64 px) and issue a visual‑search query; the search returns the top‑1 matching image.

Using the “single‑entity” template we generate a question like “What is the insect shown?” and fill the answer with the detector label (“bee”).

Parametric‑leakage filtering checks that the answer appears verbatim in the retrieved image’s caption; if not, the pair is discarded.

The remaining 3 pairs (out of 4) are stored together with their cropped entity, the retrieved image, and the originating keyframe index.

This toy run shows how each pipeline stage reduces noise: filtering cuts videos, detection isolates entities, search grounds them visually, and leakage filtering guarantees that the final QA is truly answerable from the evidence.

How does this Data Synthesis Pipeline differ from conventional VQA dataset creation?

Typical VQA datasets pair a static image with a question and answer, assuming the image alone provides sufficient context. Our pipeline starts from video, explicitly extracts entities across time, and attaches the exact keyframe and retrieved image as evidence, so the agent can later verify a web‑search result against a concrete visual cue.

**Figure 3.** Overview of VIDEO-DEEPRESEARCH. **Phase I:** Raw videos from diverse sources are filtered via rule-based and agent-based stages. **Phase II:** Keyframes are selected, entities are cropped for visual search, and VQA pairs are synthesized through single- and multi-entity patterns with parametric-leakage filtering. **Phase III:** Trajectories are constructed via a decoupled perception-exploration pipeline: the agent first grounds entities across frames using `Select_Keyframe` and `Crop_Search`, then the action space expands to Search and Visit for web exploration; only correct trajectories survive reject sampling.

**Table 6.** Tools available to the agent. `select_crop_search` is the only tool exposed during the exploration phase; search and visit are added in the answering phase.

Data Synthesis and Curation

We force agents to ground visual content before any web search by decoupling perception from language tools.

Agents trained on Video‑DR routinely skip visual tools, answering questions by pure text search. This modality bias leaves the visual grounding sub‑task under‑exploited and harms downstream reasoning.

The agent is first locked into a vision‑only toolbox, forced to harvest entities across multiple keyframes before any web query is permitted.

Keyframe 1: crop entity A (a baseball cap) and entity B (a microphone).

Keyframe 2: crop entity C (a script) and entity D (a director’s chair).

Keyframe 3: crop entity E (a poster) and entity F (a stage light).

After six crops the horizon “≥ 5 distinct entities” is satisfied, so the toolbox unlocks.

The agent now issues a

Because the answer can only be produced after the unlock, the agent learns to treat each visual crop as essential evidence rather than an optional shortcut.

How does this two‑stage unlocking differ from a standard end‑to‑end VQA pipeline that simply queries the web after a single visual glance?

In a standard pipeline the visual module may stop after one frame and immediately fire a web search, allowing the model to “cheat” by relying on textual cues. Our method forces a prolonged visual collection phase and rejects any trajectory that succeeds without it, so the final answer is guaranteed to be grounded in multiple, cross‑frame visual observations.

Generate candidate keyframes via CLIP inter‑frame similarity; select the best $vt$ with Qwen‑3.5‑397B‑A17B.

Predict bounding boxes $B$ for each entity $e$ in $vt$; crop the regions.

Run visual search on each crop; verify semantic alignment with a secondary Qwen‑3.5‑35B‑A3B model.

Assemble the tuple ⟨$vt$, $B$, entity name, search summary⟩.

Apply Phase I filtering (rule‑based coarse, agent‑based fine) to discard low‑quality videos.

Execute Phase II visual search and VQA generation (single‑entity & multi‑entity patterns), penalizing superficial attribute questions.

Perform four tool‑free rollouts per VQA; discard any that are answered correctly without tools.

Enter Phase III trajectory generation: run the stage‑wise tool unlocking described in the ConceptCard, reject any trajectory that answers early, and keep only the successful ones.

By the end of this pipeline we obtain roughly 30 k VQA pairs and 7 k verified execution trajectories, each explicitly engineered to require visual grounding before web exploration.

Training the Agent

We train Video‑DeepResearch in two stages: supervised fine‑tuning then policy optimization.

Pure supervised fine‑tuning leaves the agent stuck at a static performance ceiling, while naïve reinforcement learning adds prohibitive variance. The two‑stage recipe first grounds the model in the decoupled perception‑exploration syntax, then lets it explore autonomously to break that ceiling.

Stage 1 (SFT) gives the model a solid “cold start” on the target reasoning format; Stage 2 (GRPO) lets the same model practice on its own rollouts, receiving rewards that push it beyond what imitation alone can achieve.

How does this two‑stage pipeline differ from the usual “fine‑tune then RL‑fine‑tune” approach?

In the standard pipeline the RL stage often uses a separate value head and treats each trajectory independently. Here the SFT stage already embeds the multi‑modal reasoning syntax, and GRPO computes advantages by ranking rollouts within the same batch, eliminating the extra value network and keeping the policy update tightly coupled to the SFT‑learned representation.

Stage 1: Compute $L_{\text{SFT}}$ on the two examples. Suppose the model assigns probabilities 0.6 and 0.7 to the correct tokens, yielding a loss of $-(\log0.6+\log0.7)/2 \approx 0.66$.

Stage 2: Sample two rollouts per example. Assume one rollout per example receives reward $r=1$, the other $r=0$.

Compute group‑relative advantage $\hat{A}$: for each example the rewarded rollout gets advantage $+1$, the unrewarded gets $0$.

Apply GRPO update: the ratio $\pi_\theta/\pi_{\text{old}}$ for the rewarded rollouts is 1.2, clipped to $[0.8,1.2]$, so the weighted term is $1.2 \times 1 = 1.2$; unrewarded terms contribute $0$.

Average over the batch and subtract a small KL penalty (e.g., $0.05$), yielding a net policy improvement.

The second stage can raise the expected reward even when the first stage already fits the data, because GRPO leverages relative performance within each batch rather than absolute likelihood.

VideoDR-Bench Construction

We introduce VIDEODR‑BENCH, a 100‑pair VQA benchmark spanning six video domains.

VIDEODR‑BENCH, built on the Video‑DR premise of grounding video streams, contains 100 human‑annotated VQA pairs across six video domains.

The dataset comprises exactly 100 instances, each pairing a short video clip with a multi‑hop VQA requiring visual grounding and web‑search reasoning.

**Figure 2.** Overview of VIDEODR-BENCH. The benchmark spans six video domains: Knowledge (29.5%), Entertainment (22.0%), Daily Life (18.5%), Game & Sports (14.5%), News (12.0%), and Others (3.5%). Every instance requires joint visual grounding and multi-hop external knowledge reasoning.

It is a curated collection of short video clips paired with multi‑hop VQA that forces agents to ground visual content and consult the open web for answers.

How does VideoDR‑Bench differ from conventional VQA datasets?

Typical VQA corpora stop at a single image‑question pair, assuming all knowledge is internal to the model. VideoDR‑Bench forces agents to (1) ground a video frame in visual entities, (2) retrieve up‑to‑date web evidence, and (3) reason across multiple hops, making external‑tool usage essential.

The benchmark’s six‑domain coverage ensures that agents are evaluated on a truly diverse set of real‑world video scenarios.

Main Evaluation Results

Video‑DeepResearch‑35B‑A3B sets a new 64.0 % average accuracy, outpacing the prior best by 5 pp.

Video‑DeepResearch‑35B‑A3B achieves a 64.0 % overall average accuracy on Video‑DR, surpassing the previous best closed‑source model Claude‑4.5‑Sonnet by 5.0 percentage points.

Table 3 reports 64.0 % for our model versus 59.0 % for Claude‑4.5‑Sonnet.

Direct execution answers queries using only the model’s internal knowledge, while Agentic execution lets the model call external visual or textual tools during inference to retrieve or compute additional information.

How does Agentic execution differ from the Direct baseline in practice?

In Direct mode the model must answer solely from its pretrained parameters; in Agentic mode it can issue tool calls—such as a visual search or a web query—and incorporate the returned results, effectively extending its knowledge beyond the static weights.

Across the six Video‑DR benchmark categories, our 35B model leads in Knowledge (66.1 %) and Entertainment (61.4 %), while also improving Daily Life (56.8 %) and News (41.7 %). The 30B variant reaches 59.3 % overall, matching Claude‑4.5‑Sonnet and surpassing GPT‑5 (52.5 %).

The larger 35B model gains a larger margin (+21.2 %) over its base than the 30B model (+18.8 %), indicating that our training pipeline scales favorably with capacity. However, the modest boost on the News category (8.4 % vs 16.6 % for Daily) suggests brittleness on temporally dynamic content that warrants further investigation.

Profiling Tool Usage

Our model dramatically increases tool usage, achieving the highest visual and text calls on VideoDR.

Our 30B Video‑DeepResearch model attains the highest text tool usage on VideoDR, with 4.24 calls per task.

Table 4 shows 4.24 calls, exceeding the next best 3.81 by Qwen3.5‑35B.

An agent decides which external tools (search, vision, language) to invoke during a task, using its internal state to plan multi‑step interactions.

VideoDR‑Bench forces models to perform many more visual and textual operations than the original VideoDR set. For example, GPT‑5’s visual calls rise from 0.00 to 0.31 and text calls from 0.12 to 1.43, eliminating parametric knowledge leakage.

Baseline Qwen3.5‑397B makes only 0.10 visual calls while relying heavily on text tools (1.27). In contrast, our 30B model makes 2.33 visual and 4.24 text calls, reshaping the exploration strategy.

Our trajectory pipeline embeds active spatiotemporal perception, and mixed‑text training yields a balanced multimodal search that overcomes the modality bias of baselines.

Despite having far fewer parameters than the 397B baseline, our 30B model uses more tools, showing that training methodology outweighs raw scale for agentic capability.

The strong link between diverse tool usage and task performance confirms that overcoming modality bias is essential for effective Video‑DR agents.

Ablation Study

Each training component adds a measurable boost, confirming its necessity.

We isolate the impact of each training phase by progressively adding them to the baseline model. The numbers in Table 5 quantify how visual grounding, textual deep‑research data, and reinforcement learning each contribute to the final accuracy.

Adding the 7 K synthesized video‑DR trajectories (Video‑Centric SFT) raises average accuracy by +12.5 % over the baseline.

Baseline avg 40.5 % → 53.0 % after 7K‑SFT (Table 5).

Incorporating 7 K text‑only QA instances (+7k‑text‑SFT) yields an additional +3.8 % average improvement.

Avg 53.0 % (7K‑SFT) → 56.8 % with +7k‑text‑SFT (Table 5).

Applying reinforcement‑learning optimization (+2K‑RL, GRPO) adds a final +2.5 % average lift.

Avg 56.8 % (7K‑SFT+7k‑text‑SFT) → 59.3 % with 2K‑RL (Table 5).

Discussion and Related Work

We reflect on how Video‑DeepResearch’s curriculum, not scale, drives emergent video intelligence.

Video‑DeepResearch treats video research as a sequence of key‑entity trajectories, letting agents ground visual content continuously and launch targeted web searches to answer complex queries.

Our experiments overturn the belief that sheer model size yields emergent video capabilities. The 397 B‑parameter Qwen3.5‑397B‑A13B matches the performance of our 30 B model only because the latter follows the decoupled perception‑exploration curriculum we propose.

Active grounding proves that true video understanding requires the agent to act on visual input, not merely recall facts. GPT‑5’s competitive scores with zero tool calls expose a knowledge‑leakage loophole, whereas our pipeline forces exhaustive visual grounding before any web retrieval.

The rise in visual tool usage—from 0.10 to 2.33 operations per query—signals a shift from passive recall to an epistemic strategy of verification, confirming that the agent learns when to perceive and when to retrieve.

Training methodology can outweigh raw scale by nearly an order of magnitude: our 30 B model exhibits richer tool diversity than the 397 B baseline, indicating that modality bias stems from data distribution rather than architecture.

Achieving true multimodal parity therefore demands aligned data and curricula, not just larger parameter counts.

Early video‑LLMs sampled frames uniformly and performed single‑turn inference, which limited their ability to query dynamically and made them prone to error accumulation and hallucinations.

Recent agentic frameworks add interactive perception tools but remain confined to closed‑world video contexts, leaving knowledge‑intensive tasks that require external evidence unsolved.

VIDEO‑DEEPRESEARCH bridges this gap by coupling internal video grounding with an iterative open‑web exploration loop, turning video understanding into a robust, multi‑source reasoning process.

While autonomous deep‑research agents have progressed from text‑only to image‑centric Vision‑DR systems, they still rely on static‑image techniques—reverse image search, GRPO optimization, entity‑level cropping—that sidestep continuous video dynamics.

Video‑DR demands decoupling dense spatiotemporal dynamics and performing multi‑step verification across noisy frames, a substantially harder challenge than static image reasoning.

Existing benchmarks focus on static multimodal factuality, external knowledge grounding, or image‑based search, and thus cannot adequately evaluate video‑stream reasoning.

Moreover, current Video‑DR benchmarks depend on labor‑intensive manual annotation, creating a scalability bottleneck.

We address this by introducing a highly scalable human‑AI collaborative annotation framework that enables robust evaluation of Video‑DR systems at scale.

Conclusion

We wrap up contributions, list authors, and discuss current limitations.

VIDEO‑DEEPRESEARCH unifies data synthesis, agent training, and evaluation, delivering 30 K video‑grounded QA pairs and 7 K curated trajectories. Our 35B‑A3B model reaches 64.0 % accuracy, surpassing prior SOTA, while the 30B‑A3B variant hits 59.3 %, comparable to Claude‑4.5‑Sonnet. We also release VIDEO‑DR‑BENCH, a 200‑instance multi‑hop VQA suite that provably requires both visual search and external knowledge.

The author team comprises Zhen Fang, Yu Zeng, Wenxuan Huang, Yiming Zhao, Shiting Huang, Tianfei Ren, Qi Lu, Qingnan Ren, Qisheng Su, Lionel Z. Wang, Qingyu Yin, Shuang Chen, Zehui Chen, Lin Chen, Zhenfei Yin, Yao Hu, Shaohui Lin, Wanli Ouyang, Shaosheng Cao, and Feng Zhao, spanning institutions such as USTC, Xiaohongshu Inc., CUHK, The Hong Kong Polytechnic University, ZJU, UCLA, Oxford, ECNU, and THU. The first four authors contributed equally, Yu Zeng and Wenxuan Huang led the project, and correspondence should be addressed to Wenxuan Huang, Shaosheng Cao, and Feng Zhao.

While pioneering a comprehensive pipeline for video‑deep research, the approach demands considerable GPU resources due to large‑scale model deployment and dynamic web‑search operations. Moreover, the benchmark’s high‑quality evaluation relies on meticulous human annotation, limiting rapid scalability. Future work will explore more computationally efficient pipelines, lightweight architectures, and automated LLM‑based metrics to reduce human dependence.

Training Details

Implementation specifics, hyperparameters, and annotation procedures for reproducible training.

We trained the model using supervised fine‑tuning (SFT) on a high‑performance cluster of four nodes, each with eight 80 GB GPUs, for a total of thirty‑two GPUs, leveraging the Megatron‑LM framework.

To handle the ultra‑long 80 000‑token context without data packing we employed a mixed‑parallelism strategy: Tensor Parallelism size 4, Context Parallelism size 2, Expert Parallelism size 8, and enabled Sequence Parallelism.

The training ran for three epochs with a micro‑batch of one and a global batch of sixty‑four; the learning‑rate followed a linear warmup over the first five percent of steps to a peak of $1\times10^{-5}$ and then decayed to $5\times10^{-7}$.

For the Mixture‑of‑Experts architecture we added an auxiliary load‑balancing loss of $1\times10^{-6}$ and set the expert capacity factor to 2.0 to avoid token dropping.

Advanced optimizations such as permute‑operation fusion, Grouped GEMM, and overlapping shared‑expert computation with communication further reduced compute overhead.

Memory pressure was mitigated with FlashAttention, PyTorch’s `expandable_segments`=True, and full activation checkpointing at every layer, while system‑level tweaks included fused cross‑entropy loss, 32 OpenMP threads, 128 preprocessing processes, and eight DataLoader workers.

Model checkpoints were serialized in Safetensors format every 500 steps, deliberately excluding optimizer and RNG states to keep storage overhead low.

Keyframe extraction uses CLIP‑ViT‑L/14@336px to compute inter‑frame similarity; frames with similarity > 0.8 or monochromatic content are dropped, and at most twenty keyframes are retained per video, a rule applied consistently across data synthesis and evaluation.

The VIDEOHUNT benchmark was built over three weeks by eight annotators experienced in multimodal LLM data labeling; annotators completed a structured training session and passed a qualification test before contributing.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers