TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

Yuhan Zhu, Changlian Ma, Xiangyu Zeng, Xinhao Li, Zhiqiu Zhang, Songze Li, Jun Zhang, Tianxiang Jiang, Yuandong Yang, Ziang Yan, Zikang Wang, Xinyu Chen, Haoran Chen, Shaowei Zhang, Limin Wang

TimeLens2 enables generalist video temporal grounding by treating evidence as verifiable interval sets.

How can we train generalist multimodal LLMs to accurately ground temporal events in long videos without relying on brittle, sparse, or manually annotated interval labels?

Video multimodal models can describe video content, but they struggle to identify the specific timestamps that support their claims. This makes long-video search unreliable, as models often miss repeated evidence or produce imprecise boundaries. TimeLens2 introduces a staged pipeline that constructs reliable supervision through consensus-based localization and optimizes predictions using a temporal Wasserstein reward. This reward measures the distance between predicted and target interval sets, providing dense feedback even when predictions do not overlap with the ground truth. Across seven benchmarks, TimeLens2-4B outperforms significantly larger open-source models, improving over its backbone by 13.0 mIoU points.

Paper Primer

The core challenge is a structural mismatch: temporal evidence is a set of intervals, but standard training relies on brittle global annotations and reinforcement learning rewards that fail to distinguish near-misses from distant errors. TimeLens2 treats evidence as a first-class interval set throughout the entire lifecycle, from data construction to geometric optimization.

The method hinges on two moves: a staged curation pipeline (TimeLens2-93K) that uses hierarchical captions to propose, verify, and refine evidence intervals, and a temporal Wasserstein reward that computes the exact 1D distance between uniform distributions over merged interval supports. This reward acts like a distance-aware guide: it ranks predictions by how much "transport" is needed to align them with the target, ensuring the model learns from near-misses rather than receiving a flat zero-reward signal.

TimeLens2-4B achieves state-of-the-art performance on temporal grounding benchmarks, surpassing models with up to 397B parameters.

Comparison across seven benchmarks including Charades, ActivityNet, and Ego4D-NLQ. Average mIoU improvement of 13.0 points over the Qwen3-VL-4B backbone.

The temporal Wasserstein reward significantly improves optimization stability for group-relative policy optimization (GRPO).

Analysis of rollout groups during training. Rescues 75.8% of all-zero-tIoU rollout groups that would otherwise provide no learning signal.

Why is the temporal Wasserstein reward superior to standard temporal Intersection-over-Union (tIoU) for training?

tIoU is a binary-like metric that assigns zero reward to any prediction not overlapping the target, leaving the model unable to distinguish between a near-miss and a completely wrong guess. The Wasserstein reward provides a continuous, geometry-aware signal that ranks predictions based on their distance to the target, effectively turning silent training groups into active learning signals.

Does this model require specific query formats to function?

No. The model is trained with high instruction and response-format diversity, sampling from 27 request types and 28 response specifications. This ensures the model learns to search for evidence rather than simply imitating a specific query template or output syntax.

The paper demonstrates that temporal grounding is not merely a scaling problem, but an evidence-verification problem; the performance gains are driven by the quality of the curated 93K-instance corpus rather than raw model size.

Researchers should shift from one-pass annotation to staged, consensus-based verification pipelines and replace sparse overlap-based rewards with geometry-aware metrics like Wasserstein distance to enable robust temporal grounding.

Introduction to Temporal Grounding

We expose why current video MLLMs fail to locate evidence and outline TimeLens2’s set‑valued grounding solution.

Current video multimodal large language models (MLLMs) can narrate what happens but they rarely tell *when* the supporting evidence appears, leaving users to scan entire timelines manually. This gap stems from two structural mismatches: supervision treats temporal evidence as a single global label despite its inherently set‑valued nature, and reward functions such as temporal IoU give a zero score to any disjoint prediction, making a near‑miss indistinguishable from a completely wrong answer — like a test that marks any answer not exactly matching the key as zero, regardless of how close it is.

Temporal grounding asks a model to return the exact time intervals in a video that support a given query, potentially producing multiple disjoint spans.

TimeLens2 resolves these mismatches by treating evidence as a first‑class interval set throughout both supervision and optimization, and by introducing a Wasserstein‑based reward that accounts for temporal distance rather than binary overlap.

**Figure 1.** TimeLens2 as a generalist video temporal-grounding model. Top: mIoU across seven benchmarks covering diverse evaluation dimensions. Bottom: Representative queries from each benchmark.

**Figure 6.** Additional qualitative results on all seven benchmarks. Each row pairs five frames near the target evidence with the query, the temporal reasoning challenge, and full-video timelines showing the ground truth and predictions from four models. The TimeLens2 variants consistently recover the intended evidence, whereas the baselines select distractors, miss events, overextend boundaries, or fragment their predictions.

The shift from video description to temporal localization turns vague narration into verifiable evidence, enabling reliable search and reasoning over long videos.

Related Work

We situate TimeLens2 within prior video grounding work and highlight its novel data and reward.

Our contributions are threefold: (1) TimeLens2‑93K, a staged evidence‑verification pipeline that yields reliable single‑ and multi‑span supervision for long videos; (2) a matching‑free temporal Wasserstein reward that provides graded, fragmentation‑invariant feedback; and (3) compact 2B, 4B, and 8B generalist models evaluated on seven diverse grounding benchmarks.

The seven grounding dimensions span short indoor actions, event‑level scenes, highlight moments, long‑video multi‑span queries, user‑style long‑video searches, interrogative long‑video questions, and first‑person egocentric queries.

Temporal grounding has progressed from dedicated localization architectures that match proposals or regress boundaries to generative multimodal large language models (MLLMs) that express temporal evidence via language.

Early benchmarks paired short videos with single moments; later datasets expanded difficulty along axes such as subtitle‑aware retrieval, repeated evidence, long‑form video, and egocentric search, yet supervision quality has lagged behind scale.

Optimization methods have traditionally relied on overlap‑based rewards that only activate after prediction and target intersect, while newer approaches (GRPO/RLVR, MUSEG, NWD) explore richer geometry but still struggle with fragmentation and unequal cardinality.

TimeLens2-93K Data Pipeline

We build TimeLens2-93K by combining caption-driven proposals with dual‑agent verification and refinement.

Long videos require full‑video context for evidence search, yet sparse annotations and many distractors make precise labeling unreliable.

We first generate coarse, caption‑driven proposals from the whole video, then verify and sharpen them locally with cheap, focused models, keeping the overall cost scalable.

How does this pipeline differ from naïve full‑video annotation?

Instead of running a heavy model over the entire video for every query, we first narrow the search to a few caption‑derived clips and then apply lightweight, high‑precision models only on those short segments, dramatically reducing compute while preserving label quality.

**Figure 2.** TimeLens2-93K construction pipeline and corpus profile. Steps 1-3 construct caption-derived queries and coarse proposals; Steps 4-6 relocalize proposals with independent agents, verify interval sets, and refine their boundaries. The right column summarizes the retained corpus.

Two independent models act like separate detectives; when both independently locate the same interval, we trust the result.

Why use two models instead of a single stronger one?

Each model makes different mistakes; requiring agreement filters out model‑specific errors without the computational cost of a larger single model.

Compute IoUset = |[10,13] ∩ [11,13]| / |[10,13] ∪ [11,13]| = 2 / 3 ≈ 0.67.

Since IoUset < 0.9, the pair is rejected as inconsistent.

If Agent B had returned {[10 s, 13 s]}, IoUset would be 1.0 and the interval would be kept.

High IoUset ensures that both agents locate essentially the same temporal segment, preventing spurious matches caused by one model’s bias.

We accept an interval only if both agents agree temporally and the visual content matches the query semantically.

What if the intervals have high IoUset but low embedding similarity?

The instance is discarded because, despite temporal agreement, the visual content does not correspond to the query, preventing semantically incorrect labels from entering the corpus.

After high‑confidence intervals are identified, we fine‑tune their start and end points within a narrow window, like trimming a video clip to its exact cut.

Why restrict refinement to a ±3‑second window?

Most annotation errors lie within a few seconds of the true transition; a small window captures the correct boundary while avoiding expensive full‑video processing.

The TimeLens2 Model and Reward

We split learning into evidence search and interval calibration using a Wasserstein‑based reward.

Sparse, noisy interval labels give weak supervision, and traditional tIoU rewards treat near‑misses as total failures. To obtain a robust generalist video model we split learning into a long‑context supervised phase and a reinforcement‑learning phase that directly optimizes a geometry‑aware reward.

Long‑context supervised grounding treats temporal grounding as a search problem: the model must locate sparse evidence in full‑video context before it can estimate precise boundaries.

Instruction and response‑format diversity forces the model to be invariant to how a grounding request is phrased or how timestamps are serialized, preventing it from over‑fitting to a single prompt style.

Rollout‑guided hard‑sample mining uses off‑policy completions to estimate per‑example tIoU; low‑scoring examples receive higher sampling weight, turning the model’s own failure distribution into an adaptive curriculum.

A multimodal large language model that consumes video frames together with a textual query and emits free‑form text that can include one or many temporal intervals.

How does a Generalist Video MLLM differ from a plain text‑only LLM?

Besides the usual language modeling head, it incorporates a vision encoder and cross‑modal attention layers, enabling it to attend to visual evidence when generating interval tokens.

Instead of a binary overlap score, RTW measures how much temporal “mass” must be shifted to align a predicted interval set with the ground‑truth support, rewarding predictions that are temporally closer even when they do not intersect.

Why does RTW give a non‑zero signal when tIoU is zero?

tIoU only looks at overlap, so disjoint intervals score zero. RTW first spreads each set into a uniform mass; even disjoint masses have a finite transport cost, so the exponential yields a graded similarity that reflects how far apart the intervals are.

Merge $Y$ → $[10,15]\cup[25,30]$, length $|\text{merge}(Y)|=10$.

Merge $\hat Y$ → $[12,14]\cup[28,32]$, length $|\text{merge}(\hat Y)|=6$.

Uniform densities: $\mu_Y(t)=1/10$ on its support, $\mu_{\hat Y}(t)=1/6$ on its support.

Compute 1‑Wasserstein distance $W(\hat Y,Y)=\int|F_{\hat Y}(t)-F_Y(t)|dt = 4.0$ (area between CDFs).

RTW $=\exp(-4.0/(10+\epsilon))\approx\exp(-0.4)=0.67$.

Even though none of the predicted intervals overlap the ground truth, RTW assigns a moderate score because the predictions lie close to the true spans, unlike tIoU which would give zero.

**Figure 3.** Overlap plateaus and matching artifacts in temporal interval-set rewards. (a) tIoU ties the near miss A and distant error B at zero, whereas $R_{TW}$ ranks them by temporal proximity. (b) tIoU favors the single-moment prediction A, while $R_{TW}$ favors B, which also recovers part of the second target moment. MUSEG addresses these failures with NGIoU under one-to-one matching [25], but introduces new artifacts: (c) fragmentation forces P2 into a false cross-event match; (d) identical merged support receives different $R_{NGIoU}$ scores solely because the prediction is partitioned differently.

**Figure 5.** Temporal mass models. The same intervals [[10, 15], [25, 30], [32, 38]] are converted into four probability distributions for the Wasserstein reward.

Experimental Results

TimeLens2 delivers strong, consistent gains across all temporal grounding benchmarks.

The paper’s core claim is that sparse, noisy supervision hampers temporal grounding, and the Wasserstein Temporal Reward ($RTW$) mitigates this by rewarding proximity rather than binary overlap.

TimeLens2‑4B improves mean IoU by up to +19.6 points over the strongest open‑source baseline on the hardest VUE‑TR benchmark.

Table 1 shows a 19.6‑point mIoU gain versus Qwen3‑VL‑4B on VUE‑TR, and similar gains on VUE‑TR‑V2 and MomentSeeker.

**Table.** Performance comparison of various models across multiple benchmarks (Charades, ActivityNet, QVHighlights, VUE-TR, VUE-TR-V2, MomentSeeker, and Ego4D-NLQ) using R1@0.5 and mIoU metrics.

**Figure 4.** Qualitative comparison on temporal grounding. Each panel shows five target frames, the query, and a full-video ruler from 0 to the video duration; colored spans follow the shared legend. Dense labels summarize many short windows after merging small display gaps.

Across diverse benchmarks, TimeLens2’s $RTW$‑guided training yields uniformly higher recall and IoU, confirming that dense, distance‑aware rewards improve temporal grounding at any model scale.

Ablation Study

We isolate the contribution of each data component by ablating the TimeLens2‑93K corpus.

To understand which part of the training data drives performance, we ablate the TimeLens2‑93K corpus and fine‑tune Qwen3‑VL‑4B on several mixtures.

Training with TimeLens2‑93K improves the question‑form MomentSeeker benchmark from 15.3 % to 25.8 % mIoU.

MomentSeeker mIoU rises to 25.8 % when the model is fine‑tuned on the full TimeLens2‑93K mixture.

TimeLens‑100K, despite having a similar number of QA turns, attains only 39.4 % average mIoU, showing that mere scale does not explain the gains observed with TimeLens2‑93K.

**Table.** Scale alone does not explain the gain: TimeLens-100K has a comparable number of QA turns but reaches only 39.4 average mIoU, with substantially weaker long-video and QVHighlights results. Adding it to TimeLens2-93K yields a smaller improvement from 45.8 to 46.0, mainly on the TimeLens-Bench-style datasets; Ego4D-NLQ then raises the average to 46.4 and strengthens egocentric grounding. TimeLens2-93K therefore provides the transferable search capability, while TimeLens-100K and Ego4D-NLQ add complementary benchmark and viewpoint coverage.

Full Results

Comprehensive benchmark scores for all evaluated models across four metrics.

The following table reports each model’s performance on the four evaluation metrics.

Additional Qualitative Examples

TimeLens2’s qualitative gains across temporal‑grounding benchmarks.

TimeLens2 reaches 59.3 % R1@0.3 on CharadesTL, a 12‑point absolute improvement over the strongest prior baseline.

Prior work reported at most 47.3 % R1@0.3 on the same benchmark.

Limitations

Temporal grounding performance degrades on short and noisy intervals.

The distribution of RTW scores (e.g., 3.2, 5.6, 9.5, …, 62.9) exhibits a heavy tail: many clips achieve modest similarity while a few attain very high values, indicating sensitivity to clip length and annotation noise. Consequently, the system struggles with brief events (<5 s) and with densely overlapping intervals, limiting its reliability for fine‑grained temporal grounding.

Future Directions

Evaluation setup, data overlap audit, and avenues for extending temporal grounding.

We evaluate temporal grounding on seven benchmarks using VLMEvalKit, adapting it for the TimeLens re‑annotations of Charades‑STA, ActivityNet Captions, QVHighlights, the vision‑query subsets of VUE‑TR/VUE‑TR‑V2, the text‑only MomentSeeker subset, and Ego4D‑NLQ‑v2. All models receive only visual frames; audio and subtitles are omitted. The per‑benchmark input configuration (Table 12) fixes frame‑rate, pixel‑area bounds, and total pixel budget, while any model‑specific deviations are listed in Table 13.

To ensure a clean split between training data and evaluation, we canonicalized YouTube IDs across all seven subsets and found no overlap. This audit removed prefixes, normalized clip names, and searched metadata fields for hidden identifiers, confirming that the training source set is disjoint from every benchmark.

TimeLens2‑93K was built with cost‑effective open‑weight models, which limits annotation richness and temporal precision; replacing them with stronger proprietary multimodal models (e.g., Gemini 3.1 Pro) could substantially improve data quality. Beyond better annotations, the framework naturally extends to long‑context spatiotemporal grounding, enabling models to locate events, track entities over extended periods, and support embodied agents that must reason over persistent visual memory. We hope this work inspires larger, richer spatiotemporal datasets and models that turn long‑video perception into reliable, grounded action.

Discussion

Discussion interprets ablations, highlighting label curation, context length, instruction diversity, and reward design.

Progressive label curation yields a clear quality‑over‑quantity trend: temporal consensus, semantic verification, and boundary refinement each raise average mIoU while dramatically reducing the number of labels.

Long‑context training improves average mIoU from 44.4 % at 16 K to 46.4 % at 100 K, but gains are uneven—some datasets peak early while others keep rising, suggesting a context‑threshold effect.

Diversity in both grounding requests and timestamp renderings boosts performance; response‑format diversity alone adds 0.5 % mIoU, request diversity 0.1 %, and their combination yields a synergistic 0.9 % gain.

Among temporal‑reward variants, uniform support consistently outperforms endpoint atoms, center Gaussians, and boundary Gaussians, because it preserves both the location and duration of evidence intervals.

Replacing RTW with NGIoU drops average mIoU to 47.1 % from 47.7 %; the gap widens on multi‑interval benchmarks, highlighting RTW’s robustness to fragmented predictions.

Zero‑overlap diagnostic ablations reveal that RTW’s distance‑aware signal, not merely dense credit, drives the observed performance gains.

Conclusion

RTW restores credit for near‑misses and reshapes reward signals, boosting video MLLM performance.

Table 8 reveals a clear distance effect: adding $R_{\text{TW}}$ recovers positive overlap for 21.9 % of near misses, 15.8 % of mid‑distance misses, and only 5.7 % of far misses; the corresponding mIoU drops from 7.5 % to 4.0 % to 1.4 %.

**Table 8.** Zero-overlap misses under $R_{tIoU}$ and after adding $R_{TW}$.

GRPO’s group‑relative learning hinges on breaking reward ties; without $R_{\text{TW}}$, non‑overlapping predictions receive uniform zero reward, yielding no gradient after mean‑centering.

**Table 9.** Impact of adding $R_{TW}$ on reward distribution metrics.

Beyond tie‑breaking, $R_{\text{TW}}$ expands within‑group variance (4× increase) and raises the mean absolute advantage, directly amplifying the policy update.

TimeLens2‑93K couples this dense reward with a verification pipeline—evidence proposal, independent localization, consensus, semantic verification, and boundary refinement—turning evidence into a traceable interval set.

With that combination, compact 2 B, 4 B, and 8 B video MLLMs surpass much larger open‑source models on seven benchmarks, proving that reliable, auditable temporal grounding can be a native capability of generalist video MLLMs.

Supplementary Overview

Supplementary overview details extra experiments, data, and evaluation protocols.

The supplementary material expands the paper with full results, qualitative examples, and detailed evaluation protocols.

A.1 introduces a factorized diversity scheme for instruction phrasing and response formatting, preventing the model from conflating localization with surface‑form generation.

**Table.** Composition of the 28 response specifications

This combinatorial factorization drives the controlled 2 × 2 analysis reported in Table 5, ensuring that temporal semantics are learned independently of any particular phrasing or serialization.

Instruction Diversity

Defines the response format for temporal grounding queries and reports full benchmark results.

The grounding request expects a textual query (e.g., “When does q happen?”) and requires the model to return one or more time intervals that satisfy the query.

Response specifications are defined by a strict syntax: each interval is expressed as a start‑time and end‑time pair, and multiple intervals can be concatenated with commas, conjunctions, or wrapped in a bracketed array.

Timestamp encoding accepts three canonical forms—decimal seconds (12.0 s), minute‑second (00:12), and hour‑minute‑second (00:00:12)—all of which are normalized internally to seconds.

When several disjoint events are relevant, the specification may list them as “12.0 s to 18.5 s, and 31.0 s to 36.0 s” or as a JSON‑like array [[12.0, 18.5], [31.0, 36.0]]. Both render to the same internal representation.

**Table 11.** Full temporal grounding results across IoU thresholds.

Specifically, the TimeLens2‑2B model attains average scores of 57.9 % at R1@0.3, 47.0 % at R1@0.5, and 32.0 % at R1@0.7, illustrating the benefit of the scalable verification pipeline even at modest model size.

Evaluation Protocol

Describes how model performance is measured across datasets using RTW and related metrics.

We evaluate on three disjoint splits: a training set used to fit the model, a validation set for hyper‑parameter tuning, and a held‑out test set for final reporting. Each split contains a diverse collection of video clips with annotated temporal intervals.

The core metric is the Wasserstein Temporal Reward ($RTW$), which penalizes temporal distance rather than treating all non‑overlapping predictions as equally wrong. In addition we report conventional temporal‑grounding scores (e.g., IoU‑based recall) to facilitate comparison with prior work.

For each video the verification pipeline computes a scalar $RTW$ score; the numbers listed in the two blocks above are these per‑video scores for the validation and test splits respectively. Missing entries (shown as “–”) indicate videos for which no ground‑truth interval was available and are excluded from the average.

Train-Test Audit

Audit of temporal grounding scores highlights distribution gaps between training and test splits.

The audit lists per‑category temporal grounding scores for the training and test splits. Training scores range from 8.2 to 74.6, while test scores range from 46.2 to 84.1, with several test entries missing.

Input Configurations

Input configurations produce a wide range of performance scores.

Prompt Templates

Performance scores for each prompt template.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers