OmniCompass: a Holistic Benchmark for Omni-Modal Video Grounding and Understanding

Xuyang Liu, Shihao Wang, Xinyu Wei, Minghan Li, Zeqing Wang, Zuhao Yang, Zichen Wen, Lei Zhang

OmniCompass is a holistic benchmark for evaluating MLLMs on video grounding and understanding across visual and audio-visual tracks.

How do current Multimodal Large Language Models (MLLMs) perform on holistic video grounding and understanding tasks when evaluated with both text-only and text-plus-image reference cues?

Existing video benchmarks are fragmented, often testing only visual or audio-visual evidence, or failing to evaluate the same target under both text and image-based specifications. This makes it difficult to diagnose whether models truly understand events or are merely guessing based on inconsistent cues. OmniCompass introduces a unified evaluation framework that pairs text-only and text-plus-image target specifications across two tracks: OmniCompass-VL (visual) and OmniCompass-AV (audio-visual). It uses a Cross-Specification Consistency (CS) score to measure both task accuracy and prediction stability across these paired references. Evaluations of 50 models show that proprietary models significantly outperform open-source counterparts, particularly in audio-visual event binding, while even leading models struggle with precise spatiotemporal grounding.

Paper Primer

The benchmark addresses the "fragmentation" of current evaluation methods by standardizing tasks across four dimensions: target specification (text vs. text+image), evidence type (visual vs. audio-visual), task coverage (temporal grounding, spatial grounding, and QA), and video duration.

Proprietary models maintain a substantial performance lead over open-source models, especially in audio-visual integration.

On the AV track, the top proprietary model (Gemini-3.8-Flash) achieves a CSAvg of 57.4%, while the top open-source model (MiMo-V2.5) reaches 24.6%. A 32.8% performance gap in audio-visual understanding.

Models exhibit high QA accuracy but poor grounding performance, indicating a disconnect between semantic understanding and spatiotemporal localization.

While top models exceed 69% in CSQA, the best CSTG (temporal grounding) and CSSG (spatial grounding) scores remain below 28% and 35%, respectively. Grounding performance lags behind QA by over 30 percentage points.

Why is the Cross-Specification Consistency (CS) score necessary?

Standard accuracy metrics can mask model instability; CS scores require models to be both correct and consistent when the same target is specified via different modalities (text vs. text+image), revealing if a model is truly identifying the target or relying on superficial cues.

What is the role of timestamps in the evaluation?

The authors observe that providing frame-level timestamps consistently improves performance across all tracks and models, suggesting that explicit temporal grounding helps models better align sampled frames with the video's timeline.

Researchers should prioritize improving cross-modal alignment and audio-aware grounding, as current models struggle to bind audio cues to visual events despite high semantic QA performance.

Abstract

We introduce OmniCompass, a unified benchmark for multimodal video grounding and QA.

OmniCompass is a unified benchmark that evaluates multimodal large language models on video grounding and question answering across visual and audio modalities.

It defines two tracks—OmniCompass‑VL for vision‑language models and OmniCompass‑AV for omni‑modal models—each comprising temporal grounding, spatial grounding, and QA tasks.

Table 1 compares OmniCompass to prior video benchmarks (Charades‑STA, TVQA, TVQA+, VidSTG, NExT‑QA, HC‑STVG, QVHighlights, EgoSchema, TempCompass, MVBench, MLVU, Video‑MME, LongVideoBench, CG‑Bench, V‑STaR, IV‑Bench, AVUT, Daily‑Omni, OmniVideoBench, WorldSense, JointAVBench, Video‑MME‑v2), highlighting that earlier works use only text or limited modalities, whereas OmniCompass provides both text‑only and text‑plus‑image target specifications and includes audio‑visual evidence.

Introduction

We expose the missing unified evaluation of video understanding for multimodal LLMs.

Current video understanding benchmarks are split across separate tasks and modalities, making it unclear how well MLLMs truly understand video content. This fragmentation prevents systematic comparison and hides performance gaps that could guide future research.

OmniCompass is a single, human‑annotated video dataset that evaluates MLLMs on temporal grounding, spatial grounding, and question answering, using either pure text or text‑plus‑image reference specifications.

Benchmark Design and Scope

We detail the OmniCompass benchmark, its tasks, and the experimental protocol for evaluating MLLMs.

Multimodal large language models (MLLMs) now process text, images, video, and audio jointly. To assess their omni‑modal video understanding we need a benchmark that unifies target specification, evidence modality, task coverage, and video duration.

The model receives a target description and must output the start‑and‑end timestamps of the video segment where that target appears.

How does Temporal Grounding differ from standard video classification?

Classification predicts a single label for the whole clip, whereas TG predicts a precise time span where the queried target occurs, demanding fine‑grained temporal alignment.

Given a target description, the model must locate the region (bounding box or mask) in the video frame where the target appears.

Why can’t we treat Spatial Grounding as a pixel‑wise classification problem?

SG requires predicting a compact region that may span multiple frames, not a per‑pixel label; the evaluation focuses on region overlap, not dense segmentation.

The model receives a question about the video and must output a textual answer (multiple‑choice or free‑form).

Is Video QA just a language‑only task?

No—answers must be grounded in visual (or audio‑visual) evidence; a model that ignores the video cannot reliably answer.

Each instance is presented twice: once with a pure textual query and once with the same query plus a reference image of the target.

Why evaluate the same target with both Text and Text+Image?

It isolates whether a model relies on language alone or can exploit visual cues, revealing cross‑modal robustness.

**Figure 2. Overview of OmniCompass.** The benchmark spans four dimensions: target specification, video evidence, task coverage, and video duration. Paired Text/Text+Image specifications are evaluated on the same targets. OmniCompass-VL contains 820 videos and 2,460 instances, while the 240-video OmniCompass-AV subset contains 720 instances with native audio. Both tracks cover temporal grounding, spatial grounding, and question answering.

Existing video benchmarks suffer three key limitations: (i) they cover either visual or audio‑visual evidence but not both; (ii) they fix target specification to a single modality, usually text; and (iii) they focus on either grounding or QA, rarely both together.

Select a set of 50 MLLMs (proprietary and open‑source) to evaluate.

Run each model on the OmniCompass‑VL track (visual evidence only) for TG, SG, and QA tasks.

Run each model on the OmniCompass‑AV track (audio‑visual evidence) for the same three tasks.

For each model, compute the Cross‑Specification Consistency Score (CS) by comparing predictions between the Text and Text+Image settings.

Aggregate per‑model CS averages (CSAvg) across tasks and tracks, and report the best‑performing models.

Compute the overlap of the two intervals: intersection = 00:13–00:15 (2 seconds), union = 00:12–00:16 (4 seconds).

Intersection‑over‑Union (IoU) = 2 / 4 = 0.5.

CS for this instance is 0.5; averaging over all instances yields the model’s CSAvg.

Even a small temporal shift reduces CS, highlighting the need for precise grounding consistency across specifications.

Related Work

Survey of prior video models and benchmarks, highlighting OmniCompass’s unified design.

Recent multimodal large language models (MLLMs) such as LLaVA‑OneVision, LLaVA‑Video, and Qwen3‑Omni extend language reasoning to video by integrating visual and audio encoders, while newer systems like TimeLens add temporal grounding via interleaved timestamps. Parallelly, video‑understanding benchmarks vary along target specification, evidence modality, task coverage, and video duration, but most address only a subset of these dimensions.

**Table 1.** Comparison of existing video and audio-visual benchmarks with our proposed OmniCompass-VL and OmniCompass-AV.

Benchmark Construction

Describes how OmniCompass videos and annotations are collected, filtered, and verified.

The benchmark is built through a three‑stage pipeline that turns raw video candidates into the final OmniCompass dataset.

The pipeline first gathers a large pool of videos, then annotates tasks on a curated subset, and finally verifies all annotations before release.

How does this pipeline differ from a typical video‑dataset collection process?

Typical pipelines stop after filtering; here we add a profiling step with Qwen3‑VL‑8B‑Instruct, enforce four cue types, rank by four quality criteria, and perform multi‑round cross‑verification, which together ensure high‑quality temporal and spatial grounding.

Check duration: $45$ s ≥ $30$ s → passes the length filter.

Verify cue set: missing attribute and scene cues → fails the four‑cue requirement.

Result: video is excluded from the eligible pool, contributing to the reduction from $8{,}489$ candidates to $6{,}946$ eligible videos.

The four‑cue rule removes videos that would otherwise lack enough semantic information for reliable grounding or QA.

Collect $8{,}489$ candidate videos from nine public datasets and $240$ self‑collected videos, covering seven domains.

Profile each candidate with Qwen3‑VL‑8B‑Instruct to extract entity, action, attribute, and scene cues.

Filter out videos shorter than $30$ seconds or missing any cue, leaving $6{,}946$ eligible videos.

Rank videos by reference necessity, grounding suitability, temporal‑spatial utility, and evidence diversity; shortlist respecting domain, duration, source, and subcategory constraints.

Manual review retains $580$ high‑quality videos; combine with $240$ self‑collected videos to obtain $820$ videos for the VL track (the same $240$ also form the AV track).

Train four annotators (two‑day lead‑expert training) on the four annotation principles.

Annotate temporal grounding, spatial grounding, and QA instances under paired Text and Text+Image settings.

Run a pilot on $30$ VL and $30$ AV videos spanning all domains and durations, producing $180$ task instances and exposing guideline inconsistencies.

Refine guidelines and independently construct the remaining instances, consuming approximately $960$ annotator‑hours.

Cross‑verify all annotations by four annotators; resolve discrepancies and perform final expert acceptance, requiring about $320$ annotator‑hours and $500$ lead‑expert‑hours.

**Figure 3.** Data construction pipeline. OmniCompass requires about 1,800 human-hours for rigorous annotation, verification, and revision. Stage 1: We screen 8,489 videos from nine public datasets, retain 6,946 after duration and referential-cue filtering, and select 580 to complement 240 self-collected videos. Stage 2: Four trained annotators independently construct paired Text and Text+Image TG, SG, and QA instances. AV cases include native audio. Stage 3: Annotators cross-verify and revise cases before final acceptance and correction by the lead expert, yielding 820 videos across seven domains, paired settings, two tracks, three tasks, and three duration groups.

Main Results

VL Results show proprietary models outperform open‑source ones, with a notable performance gap.

OmniCompass provides a unified, human‑annotated benchmark for video understanding, evaluating MLLMs on temporal grounding, spatial grounding, and QA. This section reports how models perform on the VL track of that benchmark.

Proprietary MLLMs achieve higher CSAvg than open‑source models on the OmniCompass‑VL benchmark.

Gemini‑3.8‑Flash scores 42.8% versus Kimi‑K3’s 39.6%.

The VL track measures how well a model can understand video content using only visual frames and textual cues.

The AV track extends the VL evaluation by adding synchronized audio, testing multimodal reasoning.

Proprietary models retain a performance edge over open‑source alternatives on OmniCompass‑VL.

**Figure 1.** Overall model performance on OmniCompass-VL. Gemini-3.8-Flash and Kimi-K3 achieve the highest scores among proprietary and open-source models, respectively, at 42.8% and 39.6%.

Performance Analysis

Ablation studies reveal timestamps and multimodal inputs boost performance across tasks.

Grounding and fine‑grained perception lag behind reasoning: most models exceed 50% CSQA, but the best CSTG and CSSG are only 27.1% and 34.9%.

Visual references help proprietary models: Text+Image improves average performance for 11 of 36 models, especially the eight proprietary ones, while only three open‑source models benefit.

Larger models do not dominate every task: Qwen3.5‑397B outperforms the 122B model on CSAvg (37.2% vs 33.5%) but trails on CSTG (23.1% vs 24.3%).

Cross‑reference consistency is limited: CSTG, CSSG, and CSQA scores are on average 4.3–6.4% below the best single‑setting scores, showing strong single‑setting results do not guarantee consistency.

We next evaluate the impact of frame‑level timestamps on both VL and AV tracks.

We also compare three input modalities—Text‑only, Vision, and Vision+Audio—using matched eligible pairs.

**Figure 5.** Fine-grained QA performance across tracks. We report category-wise CS^QA scores in percent.

**Figure 6.** Impact of frame-level timestamps. Columns report CS^TG, CS^SG, and CS^QA in percent. Dashed links connect scores without and with timestamps. Axis ranges vary by panel.

**Figure 7.** Impact of input modalities on OmniCompass-AV. Bars report CS in percent for Text-only, Vision, and Vision+Audio on matched eligible pairs within each model and task.

Extended Results

Long videos reduce temporal grounding performance, with the biggest drop of 18.3%.

All ten models lose CS on long videos in Temporal Grounding, with Seed‑2.1‑Pro‑260915 dropping from 33.4% to 15.1% and Gemini‑3.8‑Flash from 33.8% to 21.0%.

Figure 8 visualizes the decline across short, medium, and long video categories.

All models were evaluated under identical settings: the same OmniCompass‑VL benchmark, the same three tasks, and the same Compass Score (CS) metric.

**Figure A1.** Performance across video durations in OmniCompass-VL. Bars show task-specific CS in percent for short videos below 90 seconds, medium videos from 90 to below 300 seconds, and long videos of at least 300 seconds. Each model uses its own eligible pairs.

Sensitivity Analysis

Additional ablation analyses of OmniCompass performance across granularity, domains, frame scaling, and reference specs.

Section A.2 shows that all ten VL models improve monotonically in temporal grounding (TG) as the relative target duration grows, while AV models peak in the highest duration‑ratio quartile.

Gemini‑3.8‑Flash rises from 7.9 % to 43.1 % on VL and from 27.1 % to 56.2 % on AV between the extreme quartile groups; it scores 65.5 % in the third AV quartile and 53.6 % in the fourth.

Spatial trends differ: every VL model peaks in the largest‑box group, whereas every AV model peaks in the third quartile, reflecting the distinct granularity of spatial targets.

Quartile boundaries are computed separately for each track; groups are not equal in size, and IoU depends on target extent, so the curves capture associations rather than a pure granularity effect.

Section A.3 reports that all ten VL models achieve their lowest spatial‑grounding (SG) and question‑answering (QA) scores in the Traffic domain (7.8 % SG, 37.3 % QA). Variety has the lowest TG mean (8.1 %) but the highest SG mean (26.6 %).

These domain rankings reflect differences in video duration, target size, and QA content, and apply to the shared eligible subsets rather than every individual video.

Section B.1 examines frame‑scaling. On OmniCompass‑VL, increasing the frame budget from 8 to 128 frames raises TG CS by 22.2 % (Gemini‑3.8‑Flash) and 19.6 % (Qwen3.5‑397B‑A17B), SG by 22.8 % and 16.9 %, and QA by 20.0 % and 18.6 % respectively.

On OmniCompass‑AV, SG improves by 18.3 % (Gemini‑3.8‑Flash) and 22.3 % (Qwen3.8‑Omni‑Flash), QA improves by 9.9 % and 13.4 %, while TG remains essentially unchanged, likely because audio already supplies temporal cues.

Section B.2 studies reference specification. Adding image cues (Text+Image) improves TG for seven of ten models and SG for nine, but reduces QA accuracy for eight models in action‑conditioned questions.

Gemini‑3.8‑Flash gains 10.4 % in action‑conditioned SG, whereas Qwen3.8‑Max loses 7.6 % under the same condition, illustrating that visual cues can both help and hurt depending on the task.

Section B.3 measures cross‑specification consistency. TG and SG CS average 5.0 % and 5.7 % below the lower single‑setting mean, while for QA between 10.5 % and 25.1 % of eligible pairs are correct under exactly one setting.

Kimi‑K3 attains nearly identical QA accuracies (73.0 % Text, 72.9 % Text+Image) but only 67.6 % of pairs are correct under both settings, showing that aggregate scores can mask divergent instance‑level behavior.

Section B.4 provides qualitative case studies. In the first VL example, Seed‑2.1‑Pro predicts the correct answer under both specifications, while Kimi‑K3 and GPT‑6‑Astra change predictions when the image cue is added. In the AV example, Qwen3‑Omni‑30B‑A3B switches from C to B when the image is present, whereas Gemini‑3.8‑Flash and video‑SALMONN‑2+‑72B remain consistent.

**Figure A2.** Grounding across target granularity. Curves report $CS^{TG}$ and $CS^{SG}$ in percent across quartiles of target-to-video duration and mean endpoint box-to-frame area. Quartiles are track-specific; models share eligible pairs within each track and task.

**Figure A3.** Performance across video domains in OmniCompass-VL. Bars report task-specific CS in percent for ten models. The shared eligible subsets contain 592 TG, 673 SG, and 689 QA pairs across the seven domains; comparisons use the same instances within each task.

**Figure B1.** Impact of frame scaling. Each panel reports task-level paired CS (%) under 8, 16, 32, 64, and 128 uniformly sampled frames. Rows correspond to the VL and AV tracks, columns correspond to TG, SG, and QA, and curves distinguish the evaluated models.

**Figure B2.** Effects of reference specification in OmniCompass-VL. Cells show Text+Image minus Text in percent for TG `m_tIoU`, SG `m_sIoU`, and QA accuracy on shared per-task subsets. Green denotes increases and blue denotes decreases; color does not encode significance.

**Figure B3.** Consistency under paired reference specifications. Grounding compares CS with the lower setting-specific mean on identical eligible pairs. QA bars partition pairs by correctness in the two settings; blue denotes CS^QA. The right column reports correctness under exactly one setting. Bold indicates category maxima, not uniformly better performance.

**Figure B4.** Qualitative Analysis on our OmniCompass-VL and OmniCompass-AV. Different models predictions on six cases in our two evaluation tracks.

Data Sources

Details of video source selection, candidate scoring, and QA taxonomy for OmniCompass construction.

OmniCompass‑VL aggregates 580 public‑source videos and 240 self‑collected clips, yielding an 820‑video pool. The public portion draws from nine datasets (see Table C1), while the self‑collected set comprises 70 movie, 70 TV‑series, 50 variety‑show, and 50 news clips. Both VL and AV tracks share these videos, but their temporal‑grounding (TG), spatial‑grounding (SG), and question‑answering (QA) annotations may target different events within the same footage.

To populate the candidate pool we profiled 8,489 domain‑aligned public videos using Qwen3‑VL‑8B‑Instruct. Deterministic filtering removed 1,454 videos shorter than 30 seconds, leaving 7,035; a further 89 lacked at least one of the four cue types (entity, action, attribute, scene), resulting in 6,946 eligible candidates. These counts reflect the raw candidate set before any manual task construction.

The scoring stage combines four heuristic components: $R$ (reference necessity), $G$ (grounding suitability), $T$ (temporal‑spatial utility), and $E$ (evidence diversity). $R$ measures how essential a reference image or text is for disambiguating the task; $G$ gauges whether the video content can be reliably grounded to that reference; $T$ captures the usefulness of temporal and spatial cues; $E$ reflects the variety of evidential cues present. Table C2 lists how each component is quantified on a 0‑to‑10 scale.

QA items are organized into a taxonomy that separates perception‑driven questions from reasoning‑driven ones. Perception covers direct recognition (Fact Recognition, Action Recognition, Attribute Perception, Visual Counting, Spatial Perception), while reasoning requires temporal, causal, or comparative inference. Table C3 enumerates the eight QA types used in OmniCompass‑VL, with the AV track adapting the set to incorporate audio‑visual evidence.

**Figure C1.** Seven high-level domains of videos in OmniCompass.

Quality Assurance

Details the video profiling pipeline, cue distribution, and quality scoring for OmniCompass.

The profiling process extracts reference images from each video, categorizing them as entity, action, attribute, or scene cues; entity cues dominate with 1,960 instances (79.7 % of the total). Annotation guidelines enforce strict separation between the video content and the reference cues to prevent semantic leakage, and reference images are sourced either as crops from the video itself or as external Internet images. Quality is quantified by four components—Reference Necessity ($R$), Grounding Suitability ($G$), Temporal‑Spatial Utility ($T$), and Evidence Diversity ($E$)—which are combined into a final score ($S$) using a weighted sum.

**Table C2.** Mapping from structured video profiling outputs to task-oriented scoring signals. We summarize how the structured video profiling outputs produced by Qwen3-VL-8B-Instruct are converted into intermediate statistics and then used to compute the four score components.

Evaluation Protocols

Details the verification protocol, human effort, and QA taxonomy for OmniCompass.

The verification protocol checks five core aspects—identity, temporal‑event completeness, endpoint box alignment, QA answer uniqueness, and evidence sufficiency. For audio‑visual (AV) tasks it adds two extra checks: speech/sound dependence and audio‑visual correspondence. After a correction is made, the task is re‑verified and finally approved by a lead expert.

Human‑effort records indicate a total of 1,784 human‑hours (223 eight‑hour person‑days) were spent on construction and verification. This comprises 1,280 annotator‑hours and 504 lead‑expert‑hours, covering training, task construction, cross‑verification, correction, adjudication, and final auditing (see Table D3).

**Figure C3.** QA taxonomy in our OmniCompass-VL and OmniCompass-AV.

**Table C4.** QA taxonomy and examples in OmniCompass-AV. We group the five QA types into Perception and Reasoning. Examples show only the question text from the Text target-specification setting; answer options are omitted.

**Table 1.** Categorization of QA tasks into Perception and Reasoning, including task definitions and example questions.

Taxonomy Details

Appendix F details the reference taxonomy, verification pipeline, and construction effort for OmniCompass‑VL.

Table C5 categorises reference images into four types—Entity, Action, Attribute, and Scene—each supplying a distinct cue for matching the target across video views.

**Table C5.** Functional taxonomy of reference information in OmniCompass-VL. Each type states what the reference image contributes to target identification and gives representative examples.

Table D1 documents a three‑stage verification pipeline for the VL subset: 820 videos enter annotator cross‑checking, 466 pass unchanged (56.8 %), 354 are sent for revision, and all 820 are finally accepted after lead‑expert review.

**Table D1.** Multi-stage quality verification for OmniCompass-VL. All counts use the final 820-video release set. A video passes only when its paired Text and Text+Image specifications and its TG, SG, and QA annotations satisfy the verification criteria. Representative findings are non-exclusive.

Table D2 applies the same three‑stage process to the AV subset (240 videos). After annotator verification, 151 videos (62.9 %) require no repair; 89 undergo revision for 105 task‑level issues before all 240 are accepted.

Table D3 aggregates the human labor required to build OmniCompass: 1,784 hours in total, with the guideline‑training stage consuming 896 hours (112 person‑days) and the final audit adding 128 hours (16 person‑days).

**Table D3.** Estimated human effort for OmniCompass construction and quality assurance. Hours are reconstructed from dated project records using an eight-hour workday. Automated processing, inference, and waiting time are excluded.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers