VideoJev: System One Model for Video Understanding and Interaction
Yuhan Zhu, Xiangyu Zeng, Qingyi Si, Chenxu Yang, Yifei Li, Luozheng Qin, Junhao Zhou, Dingyu Yao, Jun Zhang, Chuanyu Qin, Zhiqiu Zhang, Yuandong Yang, Yuqi Xu, Tianyu Zhang, Hongcheng Guo, Jiaqi Wang, Nan Duan, Limin Wang
VideoJev unifies temporal grounding and video QA by replacing autoregressive generation with direct candidate scoring.
How can we unify video temporal grounding and question answering into a single, efficient "System One" model that avoids redundant visual processing?
Visual agents often waste compute by generating long-form text to answer simple questions about video history, such as locating an event or selecting an option. VideoJev replaces this autoregressive decoding with a shared scoring head that evaluates temporal intervals and textual options directly against the video evidence. This approach achieves significant inference speedups—exceeding 100× on video QA—while improving accuracy over standard multimodal large language models.
Paper Primer
VideoJev treats video understanding as a structured decision task rather than a generative one. It uses a shared scoring head to map video and query representations to probabilistic judgments, allowing the model to output event spans or answer choices without token-by-token generation.
The core mechanism hinges on reusable visual computation: the model encodes the video prefix once and then processes multiple queries either sequentially (Cached inference) or in a single forward pass (Joint inference). This design: scores candidates directly, shares visual context across queries, and avoids autoregressive decoding overhead.
VideoJev significantly improves temporal grounding precision and QA accuracy compared to generative backbones.
VideoJev-4B and VideoJev-35B-A3B outperform their respective Qwen backbones on grounding mIoU by 6.7 and 10.8 points. Inference speedups reach 2.8–3.1× for grounding and exceed 100× for video QA tasks.
Direct decision learning provides an architectural advantage over autoregressive fine-tuning.
When trained on identical data, VideoJev-4B outperforms an autoregressive control model by 1.3 points in mean grounding mIoU and 1.9 points in mean QA score. The gains are consistent across seven benchmarks, confirming that direct supervision aligns the readout more effectively with the required task structure.
Why does this approach use direct scoring instead of the standard autoregressive generation used by most MLLMs?
Autoregressive generation is computationally expensive and often unnecessary for structured decisions like selecting an answer or identifying a timestamp. Direct scoring allows the model to output these decisions in a single forward pass, enabling efficient reuse of the visual context across multiple queries.
What is the scope of VideoJev — does it replace the need for reasoning models entirely?
VideoJev is designed for "System One" tasks—rapid, structured judgments. While it matches or exceeds "Thinking" (reasoning) models in speed and accuracy on many benchmarks, the authors suggest it is best used as a component in larger workflows where a "System Two" agent decomposes complex goals into these simpler, evaluative visual checks.
Introduction: The System One Paradigm
We frame the inefficiency of autoregressive video models and introduce VideoJev’s shared scoring approach.
Video agents must turn raw video streams into timely decisions, but existing models either waste computation by generating full text or rely on task‑specific heads. This creates a bottleneck when the same visual history must be consulted repeatedly for different queries.
System One treats a model’s output as a typed probabilistic judgment that downstream software can consume directly, avoiding step‑by‑step token generation.
VideoJev reduces redundant computation by using a shared scoring head that evaluates temporal intervals and textual options as candidate tokens, allowing the model to cache visual features across multiple queries. The scores directly yield event spans or answer choices, unifying temporal grounding and video question answering.
In Cached inference the video prefix is encoded once and reused for each arriving query, eliminating repeated visual processing. Joint inference processes multiple queries together in a single forward pass, scoring each candidate in parallel.
**Figure 1** VideoJev: structured visual decisions with shared computation. Top: temporal intervals and textual options share a candidate-scoring interface; *Cached inference* reuses the video prefix for arriving queries, while *Joint inference* scores each query group in one forward pass. Bottom: prediction quality versus model forward latency at both model scales, using the results in Tables 1 and 2. Left: mean grounding mIoU across three benchmarks and per-query latency on full Charades. Right: mean Video QA score across four benchmarks and per-question latency on full LVBench.
Shifting from autoregressive generation to direct candidate scoring cuts inference latency dramatically while preserving the semantic flexibility of Multimodal Large Language Models (MLLMs).
The VideoJev Architecture
VideoJev unifies temporal and option scoring while sharing video computation across queries.
The core obstacle in video understanding is the repeated cost of encoding the same video for each downstream query.
VideoJev replaces the language‑model’s final MLP with a single scorer that can evaluate both temporal intervals and textual options using the same visual features.
Compute $u = (3,4)/\sqrt{3^2+4^2}= (0.6,0.8)$.
Compute $v_1 = (1,0)/1 = (1,0)$, $v_2 = (0,1)/1 = (0,1)$, $v_3 = (2,2)/\sqrt{8}= (0.707,0.707)$.
Dot products: $u^{\top}v_1 = 0.6$, $u^{\top}v_2 = 0.8$, $u^{\top}v_3 = 0.6+0.8 \times 0.707 \approx 0.99$.
Logits: $s_1 = 10 \cdot 0.6 + b$, $s_2 = 10 \cdot 0.8 + b$, $s_3 = 10 \cdot 0.99 + b$ (with $b$ learned later).
Normalizing each projection forces the scorer to rely on angular similarity rather than vector magnitude, making the logits comparable across candidates of different scales.
How does this scoring differ from a standard classification head that directly maps $h_i$ to logits?
Standard heads treat each candidate independently and use raw linear scores, which can be dominated by vector length. VideoJev first normalizes both decision and candidate vectors, so the logit reflects pure directional alignment; the fixed scale 10 then ensures a well‑behaved softmax.
Because the video prefix is causal, its representation can be computed once and reused for every subsequent query, avoiding redundant video encoding.
Why not simply re‑encode the video for each query instead of caching?
Re‑encoding repeats the expensive transformer operations on every frame. Caching moves that cost to a single upfront pass, turning $O(N_{\text{frames}})$ work per query into $O(1)$, which dramatically reduces latency when $K$ queries share the same video.
Joint inference concatenates several queries (and their candidate lists) after a single video prefix, allowing one causal forward pass to produce scores for all queries at once.
What prevents joint inference from leaking information between queries?
Only the textual context (previous queries and candidates) is visible; the model never writes back predicted answers. Thus each query’s decision token sees the same video prefix and the same set of earlier texts, but not the ground‑truth answer of any other query.
**Figure 2.** VideoJev: direct candidate scoring with shared visual computation. (a) Temporal intervals, represented by timestamps, and textual options share an input format. (b) Cached inference prefills the video once and processes each query suffix separately; Joint inference processes each query group in one causal language-model forward pass. Both modes share the backbone and scorer. (c) Temporal scores yield an event span; option scores select an answer. Scores and the reference interval are illustrative.
**Figure 5. Qwen3.5-4B Direct inference.** (a) Queries branch independently from a shared video cache. (b) All option predictions are read in one causal forward. Both modes use batch size 1, restrict logits to valid option tokens, and leave answer slots empty; [readout] marks a logit readout, not inserted text.
Temporal Grounding and Supervision
Predict a continuous event span by scoring short video intervals and stitching the best run.
Think of the video as a road of short segments; the model lights up the segments that belong to the event and then picks the longest consecutive stretch of lit segments — like finding the longest run of green lights.
Subtract the log‑odds of the threshold: $\tilde{s}_i = s_i - \log\frac{\tau}{1-\tau}$ for every interval $i$.
Run Kadane’s algorithm on the adjusted scores $\tilde{s}_i$ to find the contiguous subarray $[a^{*},b^{*}]$ with the largest sum.
If the maximal sum is negative, set $a^{*}=b^{*}=\arg\max_i s_i$ (the strongest single interval).
Output the continuous prediction $[\ell_{a^{*}}, r_{b^{*}}]$ using the boundaries of the selected intervals.
Apply a lightweight post‑processing step to guarantee the interval is non‑empty and lies within the video duration.
Compute scores $s=[0.2,0.8,0.6,0.1]$ and probabilities $p_i=\sigma(s_i)\approx[0.55,0.69,0.65,0.52]$.
Adjusted scores with $\tau=0.25$: $\tilde{s}=s-\log\frac{0.25}{0.75}\approx[0.2+0.55,0.8+0.55,0.6+0.55,0.1+0.55]=[0.75,1.35,1.15,0.65]$.
Kadane selects the contiguous run $[a^{*},b^{*}]=[2,3]$ because $1.35+1.15$ is the maximal sum.
The predicted span is $[\ell_{2}, r_{3}] = [2,6]$, which fully contains the true event $[2.5,5.5]$.
This toy shows that even intervals with modest individual scores can be included if they form a longer contiguous block, illustrating why the decoder prefers a max‑sum run over isolated high‑score spikes.
Video QA reuses the same shared scoring head: for a question with $K$ answer options the head produces logits $s_i$, which are normalised with a softmax and trained with cross‑entropy, while a permutation step removes position bias during training.
How does this interval‑stitching decoder differ from the bounding‑box regression used in standard object detection?
Object detectors predict start and end coordinates directly, often with a separate regression head. Here we predict a binary membership score for each short interval and then infer the span by finding the highest‑scoring contiguous run (Kadane). This lets a single scoring head serve both grounding and QA, and it naturally adapts to events of arbitrary length without explicit length regression.
Experimental Results
VideoJev cuts inference time and raises temporal‑grounding quality.
Recall that VideoJev shares a scoring head across temporal intervals and textual options, letting it cache visual features for many queries.
VideoJev‑4B runs 56 % faster than the nearest open‑source baseline while delivering higher temporal‑grounding quality.
Latency 0.276 s versus 0.636 s (Molmo2‑4B) and 0.703 s (Qwen3‑VL‑4B); $mIoU$ 82.8 % versus 44.4 % and 71.8 % respectively.
VideoJev‑4B raises $mIoU$ by 9 % compared with the strongest open‑source competitor.
$mIoU$ 82.8 % versus 71.8 % (Qwen3‑VL‑4B) and 44.4 % (Molmo2‑4B) on the CharadesTL benchmark.
During inference the model can either generate answers token‑by‑token (autoregressive) or score all candidates in parallel; VideoJev opts for the latter to avoid repeated decoding.
How does “Autoregressive Control” differ from standard autoregressive decoding?
Standard autoregressive decoding generates each answer token sequentially, incurring a separate forward pass per token. VideoJev’s control scores all answer candidates in parallel after a single visual encoding, so the cost does not grow with answer length.
VideoJev consistently outperforms autoregressive baselines in both latency and accuracy.
**Figure 6.** Joint inference grounds six Charades queries in one forward; three readouts are shown in order. Both models match the one-second transition, while VideoJev-35B-A3B locates adjacent actions more precisely.
**Figure 7** Localizing a compound jump-rope event on ActivityNet Captions. VideoJev-35B-A3B more closely matches the annotated interval.
**Figure 3.** Multi-query efficiency at 4B. Mean forward time for 1–10 queries per video on Charades (a) and Video-Holmes [6] (b); shading shows 95% confidence intervals. Solid/dotted lines use Cached inference (dotted: multi-token generation); dashed lines use Joint inference. End labels pair ten-query latency with separately evaluated full-benchmark Charades mIoU or LVBench accuracy; quality settings follow the source evaluations in Appendix A.2.
Ablation Studies
Component removals reveal modest but consistent drops, confirming each design choice’s contribution.
We first report the baseline temporal‑grounding results (Table 1). VideoJev‑4B attains 65.3 % mean $mIoU$, a 6.7‑point gain over Qwen3.5‑4B, and improves the stricter R1@0.7 metric by up to 4.8 points.
VideoJev‑4B improves mean $mIoU$ by 6.7 points over the Qwen3.5‑4B baseline.
Table 1 shows 65.3 % versus 58.6 %.
VideoJev‑4B raises R1@0.7 by up to 4.8 points compared to the baselines.
Table 1 reports gains of 2.8–4.8 points across the three datasets.
Option‑selection results (Table 2) show that both VideoJev variants outperform their Instruct backbones by large margins while keeping inference fast.
VideoJev‑4B improves the four‑benchmark mean QA score by 8.0 points over its Instruct backbone.
Table 2 reports an 8.0‑point uplift.
VideoJev‑35B‑A3B improves the four‑benchmark mean QA score by 5.7 points over its Instruct backbone.
Table 2 reports a 5.7‑point uplift.
Joint inference yields a 129× speedup over the Thinking baseline for VideoJev‑4B on QA benchmarks.
Table 2 lists a 129× acceleration.
Joint inference yields a 100× speedup over the Thinking baseline for VideoJev‑35B‑A3B on QA benchmarks.
Table 2 lists a 100× acceleration.
Joint inference improves mean grounding by 0.8 points at the 4 B scale.
Table 3 compares VideoJev‑4B (baseline) to the AR control.
Joint inference improves mean QA by 0.5 points at the 4 B scale.
Table 3 shows a 0.5‑point gain.
Joint inference improves mean grounding by 0.3 points at the 35 B‑A3B scale.
Table 3 reports a 0.3‑point uplift.
Joint inference improves mean QA by 0.7 points at the 35 B‑A3B scale.
Table 3 reports a 0.7‑point uplift.
Two‑stage adaptation raises mean grounding $mIoU$ by 0.1 points compared to single‑stage training.
Table 8 reports a +0.1 point difference.
Two‑stage adaptation reduces mean QA by 0.4 points relative to single‑stage training.
Table 8 shows a –0.4 point delta.
Instead of forcing a hard match between predicted and ground‑truth intervals, the loss rewards partial overlap proportionally, encouraging the model to hone in on the correct temporal span.
How does Soft IoU differ from a hard IoU threshold used in earlier video‑grounding work?
Hard IoU treats any prediction below a fixed overlap as completely wrong, yielding zero gradient; Soft IoU supplies a smooth, non‑zero gradient even for low‑overlap predictions, allowing the model to learn from all examples rather than discarding near‑misses.
**Table.** Comparison of VideoJev-4B (baseline) and Qwen3.5-4B AR SFT across Temporal grounding and Video QA benchmarks.
**Table 3.** Isolating architectural gains with an autoregressive control. VideoJev-4B and an autoregressive Qwen3.5-4B are both fine-tuned in a single stage on exactly the same post-training data. † denotes the VideoJev-4B Single-stage (100%) variant from Table 8. The two models share the same training and evaluation settings.
**Table 4.** Cached inference versus Joint inference.
**Table 8.** Two-stage adaptation. The two-stage schedule first aligns the scorer with frozen features, then adapts the language backbone and scorer together. Table 8 shows broadly comparable performance to single-stage training: two-stage adaptation scores slightly higher on four of seven benchmarks, while single-stage training leads by only 0.1 point in mean grounding mIoU and 0.4 points in mean QA, mainly due to MLVU. Two-stage adaptation achieves this quality while updating the language backbone on only 70% of the videos; the remaining 30% use cheaper head-only training. We adopt this schedule for its lower cost and comparable performance; aligning the scorer first may also stabilize backbone adaptation.
Analysis and Calibration
We dissect how joint training, supervision tweaks, and adaptation schedules affect VideoJev’s limits.
Understanding where the shared scorer helps or hurts is essential for pushing VideoJev toward reliable deployment.
Joint training of temporal grounding and video QA lifts the mean grounding mIoU by 0.7 points.
Table 5 shows the baseline TG+QA reaches 60.1 mIoU versus 59.6 for TG‑only.
Adding Soft IoU supervision raises the mean grounding mIoU by 4.6 points while also nudging QA performance upward.
Table 6 reports 65.3 mIoU (+4.6) compared with the 60.7 baseline; QA improves from 73.9 to 74.3.
Removing the final‑layer MLP yields a modest 0.2‑point gain in both grounding and QA metrics.
Table 7 shows the baseline without the MLP reaches 60.1 mIoU and 62.8 QA, each 0.2 higher than with the MLP.
**Figure 4** Confidence calibration on Video QA. Points show mean confidence and observed accuracy in equal-width bins containing at least 20 questions; all bins contribute to ECE. The dashed diagonal denotes perfect calibration. MLVU uses seven-task-balanced weights; the other benchmarks weight questions equally. Curves use the reported per-benchmark predictions.
**Figure 12** An intent-inference disagreement on LVBench: VideoJev-4B interprets the interaction as courtesy, while VideoJev-35B-A3B selects the annotated arrest intent.
**Figure 13.** A shared failure on small-object counting in Video-MME. Both models miss the annotated answer of four books; the book detail is enlarged for display.
Calibration analysis pinpoints persistent struggles with small‑object recognition, guiding future improvements.
Related Work
We survey prior video MLLM, temporal grounding, and System One approaches relevant to VideoJev.
Recent multimodal large language models such as VideoChat, Video‑LLaMA, VideoChat2, VideoChat3, VideoChat‑Flash, Video‑o3, VideoLLM‑online, StreamingVLM, JoyAI‑VL‑Interaction, and OneStreamer extend video understanding by connecting visual encoders to language models and by introducing mechanisms like token compression, streaming caches, and joint perception‑memory‑response modeling. Complementary studies on vision‑language transfer (AWT) and test‑time prompt adaptation further improve efficiency, while benchmarks such as TVQA, NExT‑QA, EgoSchema, MVBench, and Video‑MME evaluate compositional, temporal, and long‑horizon video reasoning.
Temporal grounding approaches localize language‑described events via candidate matching, span prediction, moment‑set prediction, or clip‑level foreground, boundary, and saliency outputs, with methods like DualDETR jointly refining instance and boundary queries. Recent works such as TimeChat, VTimeLLM, LITA, Grounded‑VideoLLM, TimeSuite, TimeLens2, DisTime, and analyses of prefill attention further integrate temporal grounding into video MLLMs, while combined systems like GroundVQA and SEAL learn compact representations for joint grounding and question answering.
The System One paradigm, introduced by Jev and extended by this‑that‑model‑1.0, JevLite, Visual Jev, PixelJev, and FreeRet, frames visual tasks as typed probabilistic judgments over shared representations, enabling efficient multi‑choice evaluation. VideoJev adapts this paradigm to temporal decisions and repeated video queries, aligning multi‑query training with inference to reduce redundant computation. Related video studies also evaluate quality using metadata and detect anomalies from captions and summaries.
Supplementary Material
Supplementary details on evaluation protocol, latency, performance, and inference modes.
The evaluation protocol follows the paper’s main settings: video frames are sampled at 4 fps for grounding and up to 2 fps (max 2,048 frames) for QA, with each frame limited to 640 × 640 pixels. Generative baselines use the Instruct prompt unless explicitly labeled “Thinking”, while VideoJev defaults to Joint inference. All tasks run on idle H200 GPUs with batch size 1, measuring only the synchronized forward pass of visual encoding, prefilling, and decoding.
Latency is measured by summing the forward time of visual encoding, prefilling, and all decoding steps per query, then dividing by the number of queries. Video loading, preprocessing, and post‑processing are excluded, but truncated or invalid outputs are still counted. VideoJev encodes each video once and scores all queries in a single forward pass, whereas generative baselines reuse the visual encoding but decode each query independently.
Performance results cover temporal grounding on Charades, ActivityNet, and QVHighlights, and QA on LVBench. All methods use the same visual budget (4 fps, reduced frame area) and context limits (144 K tokens for grounding, 256 K tokens for QA). VideoJev applies a fixed $\tau$ = 0.25 and a smoothing window of 1 across all experiments, matching the Joint inference setup.
For autoregressive control, Table 3 compares single‑stage VideoJev‑4B with native Qwen3.5‑4B trained on identical data (404 k questions, 58 k videos). Both models train for one epoch with identical optimizer settings; only VideoJev freezes the video encoder and merger while Qwen retains its final MLP and tied language‑model head.
Inference modes are split into Cached inference, which encodes and prefixes the video once then processes each query from an independent cache, and Joint inference, which concatenates all queries after a single video prefix and reads predictions in one causal forward pass. Direct option prediction masks logits to valid option tokens and reads them either from the cache (Cached) or from the joint forward (Joint).
The qualitative benchmark section presents eight illustrative cases across seven datasets, comparing Joint inference outputs of VideoJev‑4B and VideoJev‑35B‑A3B. Ground‑truth intervals are shown in green; model predictions appear in brown (VideoJev‑4B) and red (VideoJev‑35B‑A3B). The examples highlight differences in temporal precision, compound event localization, and intent inference.