VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

Xinhao Li, Yuhan Zhu, Xiangyu Zeng, Yuhao Dong, Haoning Wu, Zhiqiu Zhang, Yuandong Yang, Changlian Ma, Qingyu Zhang, Yansong Shi, Xinyu Chen, Haoran Chen, Zizheng Huang, Jun Zhang, Kun Ouyang, Lin Sui, Ziang Yan, Yicheng Xu, Chenting Wang, Yinan He, Hongjie Zhang, Yi Wang, Yu Qiao, Yali Wang, Ziwei Liu, Kai Chen, Limin Wang

VideoChat3 is a fully open, 4B-parameter video MLLM that uses spatiotemporal tokenization and adaptive streaming to balance efficiency with generalist performance.

How can we build a fully open, efficient Video MLLM that handles both long-form video reasoning and real-time streaming interaction?

Existing video multimodal models often struggle to generalize across diverse video lengths and streaming interactions, while their high computational costs make them difficult to scale or reproduce. VideoChat3 addresses this by introducing an Inflated 3D Vision Transformer (I3D-ViT) that compresses video tokens spatiotemporally and an adaptive streaming controller that adjusts visual resolution based on the model's internal evidence state. The model achieves state-of-the-art performance among open-source models of its scale, while providing a fully reproducible foundation by releasing all training code, datasets, and weights.

Paper Primer

The core mechanism hinges on two innovations: I3D-ViT and adaptive streaming. I3D-ViT is a visual tokenizer that groups consecutive frames into chunks and applies 3D spatiotemporal attention to reduce redundancy before the tokens reach the language model. The adaptive streaming controller acts like a traffic light: it monitors incoming video at low resolution, switches to high resolution when it detects potential evidence, and triggers a response only when sufficient information is accumulated.

VideoChat3 outperforms larger open-source models in both general video understanding and computational efficiency.

Evaluations across general, long-form, and streaming benchmarks show superior performance compared to models like Qwen3-VL and Molmo2, despite using only 4B parameters. The model achieves this balance through a 16x spatiotemporal compression ratio and a dynamic pixel-quota policy that reduces compute for routine video intervals.

Why does the paper emphasize "fully open" status as a core contribution?

Many high-performing video models are closed-source or lack transparency regarding training data and recipes, which prevents the community from reproducing results or understanding the drivers of performance.

How does the model handle the trade-off between processing long videos and maintaining real-time streaming capabilities?

It uses a state-conditioned loop where the model predicts its own "response state" (Silence, Standby, or Response), which dynamically adjusts the visual input resolution for the next frame window to prioritize efficiency during routine moments.

Introduction and Motivation

We expose the core tension in video MLLMs and outline our open, efficient, generalist solution.

Video MLLMs must balance visual perception quality with computational efficiency, a tension that hampers real‑world deployment. Current open‑source models are limited by specialization to particular video settings, excessive compute cost, and incomplete reproducibility. VideoChat3 addresses these gaps with an I3D‑ViT encoder, Adaptive Frame Resolution, a large‑scale video data pipeline, and full open‑source release.

**Figure 1.** VideoChat3 achieves strong performance across diverse evaluation benchmarks, including temporal perception, long video understanding, and temporal grounding, while also supporting online proactive responses.

The core challenge is the trade‑off between visual quality and compute efficiency in video MLLMs.

VideoChat3 Architecture

We detail the I3D‑ViT encoder and Adaptive Frame Resolution that compress video for efficient LLM processing.

Video MLLMs must balance visual fidelity against token‑budget limits; naive frame‑wise image tokenization discards motion cues and wastes redundancy.

**Figure 2. VideoChat3 architecture with I3D-ViT.** VideoChat3 follows the classical ViT–MLP Projector–LLM architecture, with I3D-ViT enabling efficient video encoding before visual tokens are passed to the LLM. Specifically, spatial $2 \times 2$ merging and temporal $T$-frame pooling reduce the visual sequence length by approximately a factor of $4T$. Under the default setting of $T = 4$, this yields a $16 \times$ spatiotemporal compression ratio; for clarity, the figure illustrates the mechanism with $T = 2$.

I3D‑ViT inflates a pretrained image tokenizer into a spatiotemporal transformer, letting neighboring frames interact before tokenization.

Chunk 0 (frames 0‑1) is flattened into a token sequence of length $8 \times 8 \times 2=128$.

Spatiotemporal self‑attention mixes spatial and temporal information across the 128 tokens.

Temporal pooling averages the representations of frames 0 and 1, yielding a single token set of length $8 \times 8=64$.

Chunk 1 undergoes the same steps, producing another 64‑token set.

The two pooled token sets are concatenated, giving a final output of 128 tokens instead of the original 256 (4 × 8 × 8), a $2\times$ reduction per chunk and $4\times$ overall.

I3D‑ViT captures motion between adjacent frames inside each chunk, so the pooled tokens retain dynamic cues while halving the token count.

**Figure 3. Illustration of the Adaptive Frame Resolution.** State tokens control whether the next streaming window is encoded at a low or high pixel quota. In the soccer example, routine play is monitored at low cost; when players rush toward the goal, the Standby state enlarges the next window so the model can inspect whether the ball goes in before triggering Response.

The model monitors the stream at low cost and only upsamples frames when a state token signals possible relevance.

Window 1: LLM emits Silence → $b_2 = B_{\text{low}} = 16$ pixels for Window 2.

Window 2: LLM emits Standby → $b_3 = B_{\text{high}} = 64$ pixels for Window 3.

Window 3: LLM emits Response → answer generation starts; $b_4 = B_{\text{low}} = 16$ pixels for the following monitoring window.

The controller automatically upsamples only when a Standby cue appears, keeping most of the stream cheap while still capturing fine details when needed.

Deterministic controller mapping state token to pixel quota.

Data Construction and Synthesis

We build three complementary video corpora to balance breadth, depth, and streaming readiness.

Academic video corpora are reliable but provide only short, label‑only supervision, while long videos demand sparse, temporally distributed evidence. We therefore construct three complementary datasets that together cover semantic breadth, temporal depth, and streaming readiness.

We treat existing academic video sets as trustworthy semantic anchors and densify their sparse labels into evidence‑grounded instruction data.

Sample 1: the LLM expands “yes” to “Yes, the person lifts the box because the box is visible in frames 2‑3.”

Sample 2: the caption is rewritten as “The brown dog runs across the lawn from left to right between seconds 1‑4, chasing a ball.”

Sample 3: option B is converted to the full sentence “The car stops at the red light after traveling 15 m, as seen in frames 5‑7.”

The judge model checks each expanded answer against the original label; all three pass because the core decision (yes / dog runs / car stops) is unchanged.

Enhancement adds visual evidence and temporal cues without altering the original semantic target, giving the model richer generative supervision.

How does this differ from simply training on the original short answers?

Short answers give the model a single token to predict, offering no guidance on which visual elements to mention or how to structure the response. The enhanced version forces the model to generate a grounded narrative, teaching it to reference concrete video evidence.

**Figure 4.** Source Distribution of VideoChat3-Academic2M. VideoChat3-Academic2M contains 2.27M caption/QA instances from six academic sources, dominated by LLaVA-Video [26], Spoken-MIT [27], and Vript [28].

**Figure 5. Example of Annotation Enhancement from Concise Answers to Evidence-Grounded Responses.** A short-phrase answer is rewritten into a temporally grounded, evidence-rich response that explains the observed actions and supports the final answer with video-specific cues. This illustrates how annotation enhancement increases supervision density while preserving the original semantic label.

We synthesize long‑video supervision by breaking each video into coherent segments, annotating them, and re‑assembling the evidence into instruction data that requires cross‑segment reasoning.

PySceneDetect identifies two shot changes at 40 s and 80 s, yielding three segments.

The LLM annotates S1 as “A person walks into a kitchen and opens a fridge.”

S2 receives “The person chops vegetables, heats oil, and sautés the mixture.”

S3 is described as “The finished dish is plated on a plate and garnished.”

Prompting the LLM with all three descriptions produces a QA pair: Q = “What does the person do before plating?” A = “The person chops vegetables and sautés them (segments 2).”

Segment‑level evidence makes it possible to ask questions whose answers depend on events spread across the video, something a single‑clip annotation cannot capture.

Why not annotate the whole video directly instead of segmenting?

Annotating the full video forces the model to guess long‑range context and often leads to vague or incomplete descriptions. Segmentation isolates coherent visual units, enabling precise, timestamped evidence that can be recombined for cross‑segment reasoning.

**Figure 6. Source Distribution of VideoChat3-LV116K.** The panels summarize the collected long-video repository used to construct VideoChat3-LV116K, with 116.2K rows. The duration statistics reveal a clear temporal-scale gap: academic sources are mostly short clips, with mean durations from 3.0s to 59.0s, whereas our collected long-video shards average 156s to 1.3K seconds and extend to much longer maxima. This complementarity provides both reliable short-clip semantic anchors and long-range supervision for sparse evidence, cross-segment aggregation, and event-level reasoning.

**Figure 7** Long-Video Data Synthesis Pipeline for VideoChat3-LV116k. The pipeline converts raw long videos into structured supervision through candidate filtering, temporal boundary segmentation, segment-level annotation, quality examination, and full-video annotation assembly. Instead of annotating an entire long video directly, it builds a validated segment-level evidence ledger, making sparse long-range events easier to capture and reason over.

We turn offline video‑QA pairs into proactive streaming supervision by marking when sufficient visual evidence appears and inserting response‑state tokens that guide the model’s timing.

The VLM identifies the spill interval (6‑10 s) as the visual clue.

Clue verification confirms that cropping to 6‑10 s alone yields the correct answer.

Streaming sequence: windows 0‑5 s → , windows 6‑10 s → , windows 11‑30 s → “Yes”.

The model learns to emit the answer immediately after the clue interval ends, rather than waiting for the full video.

Response‑state tokens let the model act as soon as the necessary evidence is observed, which is essential for real‑time assistance.

Why introduce special tokens like and instead of letting the model decide when to answer?

Without explicit tokens the model may delay or answer prematurely, because it lacks a clear signal about evidence availability. The tokens encode the causal timing relationship between visual evidence and the answer, enforcing proactive behavior.

**Figure 8** Source composition and temporal coverage of VideoChat3-OL617K. *Left:* Distribution of 617,183 instances across 40 JSONL shards. StreamForest General and Streamo contribute 272,424 (44.1%) and 259,977 (42.1%) instances, respectively, while StreamForest Drive, Seeker, and Supplement provide the remaining data. *Right:* Minimum, mean, and maximum observed context spans for 438,902 records with timing information, shown on a logarithmic scale with zero-length spans placed at 1 second for visualization. Mean spans range from 12.7 seconds for StreamForest Drive to 153.5 seconds for Seeker, providing supervision across both short- and long-horizon causal streaming contexts. This diversity supports the evidence accumulation and response-timing behaviors used to construct proactive streaming QA supervision.

The multi‑source approach balances academic breadth with long‑video temporal structure, yielding dense, evidence‑grounded supervision across short, long, and streaming regimes.

Training Recipe

VideoChat3 follows a four-stage curriculum, evolving from visual tokenizer pre-training to long-form streaming adaptation.

The training recipe for VideoChat3 follows a four-stage curriculum, gradually transitioning from raw visual representation learning to specialized long-form streaming interaction. This staged approach ensures the model first masters visual semantics before attempting complex temporal reasoning or instruction following.

**Table 1.** Detailed configuration for each training stage of VideoChat3. Dataset items denote the total number of samples used across all sub-stages within each stage. The corresponding training curricula are described in the subsections below.

The model learns in four distinct phases: first grounding visual tokens in language, then aligning the full architecture, then tuning for general instruction following, and finally adapting to streaming temporal contexts.

To handle streaming video, the model uses three special tokens—</Silence>, </Standby>, and </Response>—to manage its internal state and decide when to output an answer based on incremental visual evidence.

**Figure 9. State-transition supervision mask for streaming training.** The output sequence consists of </Silence>, </Standby>, and </Response>. Blue arrows indicate the state-token positions whose losses are retained. We keep all *Transform* positions, where the target state changes between adjacent windows, because they define the temporal decision boundaries. From the remaining continuation positions, we randomly select an equal number of *Keep* positions to provide supervision for maintaining the current state.

General Video Understanding Results

VideoChat3 balances video quality and efficiency using I3D‑ViT and adaptive token allocation.

We assess VideoChat3 on four complementary aspects of video understanding—temporal perception, long‑video comprehension, complex reasoning, and temporal grounding—using a suite of established benchmarks.

VideoChat3‑4B improves the MMVU reasoning benchmark by +5.9 points over the next best fully open model.

MMVU overall score rises from 84.6 (Molmo2‑4B) to 90.5 (VideoChat3‑4B).

**Table.** Performance comparison of various video understanding models across four categories: Temporal Perception, Long Video, Reasoning, and Temporal Grounding.

Streaming Performance

VideoChat3‑4B sets new online video benchmarks, excelling in perception‑memory and proactive response.

VideoChat3‑4B outperforms all prior online video models on four of six perception‑and‑memory aggregates, achieving the highest ODVBench overall score.

Table 3 shows VideoChat3‑4B scoring 72.3 on ODVBench versus the next‑best 60.5 from StreamForest.

VideoChat3 enables proactive streaming responses without auxiliary modules.

Efficiency and Latency Analysis

VideoChat3 halves visual tokens and cuts latency up to 54 % for long videos.

VideoChat3 cuts total inference latency by up to ≈54 % for long videos compared to Qwen3‑VL.

Table 4 reports total latency 20.412 s for VideoChat3 versus 44.449 s for Qwen3‑VL at 2048 input frames.

Because VideoChat3’s encoder performs explicit spatio‑temporal compression, it emits only half as many visual tokens as Qwen3‑VL while incurring a modest encoder‑latency increase. The LLM’s compute scales with the token count squared, so halving the token sequence cuts FLOPs by more than 60 % and reduces GPU memory by 26 GB at 2048 frames, making long‑video inference tractable.

**Table 4.** Inference cost comparison on NVIDIA H200 using the HuggingFace Pipeline with FlashAttention-2.

Ablation Studies

Ablation studies quantify each component’s impact on VideoChat3’s performance.

We evaluate each design choice by isolating it on the OVO‑Timing benchmark and on a suite of video understanding tasks, reporting the effect of removal or alteration.

**Table (b)** Comparison on Video Benchmarks. Temporal-grounding columns TL share the TimeLens suite and differ by split.

I3D‑ViT processes four times as many frames as MoonViT thanks to its 4× temporal compression, enabling denser visual evidence without exceeding token limits.

**Table 1.** Comparison of different visual tokenizers on various video benchmarks.

**Table 7.** (a) Perception-window policy. (b) State-token supervision.

Dynamic perception‑window policy attains an OVO‑Timing F1 of 35.5, surpassing fixed‑budget baselines.

Table 7a shows the Dynamic row achieving F1 = 35.5, whereas Fixed low and Fixed high policies lag behind.

State‑transition mask yields a Precision of 33.1 and Recall of 97.9, outperforming both “All state tokens” and “Transitions only” variants.

Table 7b reports Precision = 33.1 and Recall = 97.9 for the mask variant, the highest among the three.

**Table 8.** Data for Stage2

**Table 9.** Ablation study on VideoChat3-OL617K pipeline.

VideoChat3‑OL617K increases OVO‑Timing F1 by +31.5 points (4.0 → 35.5), while preserving offline video QA performance.

Table 9 compares the Baseline and + VideoChat3‑OL617K rows, showing the F1 jump.

**Table.** Data for Stage3

Collectively, these ablations confirm that each component—temporal compression, dynamic windowing, state‑transition supervision, and the three data pipelines—contributes measurably to VideoChat3’s superior video understanding and streaming capabilities.

Qualitative Examples and Conclusion

Appendix details evaluation configurations and additional qualitative examples.

**Table.** Benchmark configurations for Temporal Sampling and Visual Budget.

**Table 12.** Evaluation configurations for online proactive-response benchmarks. “Low Tokens / Frame” and “High Tokens / Frame” denote the maximum per-frame visual-token budgets under low- and high-resolution perception, respectively. “High-Budget Frames after Standby” denotes the number of subsequent frames processed with the high token budget after a *Standby* prediction.

The table presents configuration parameters for two benchmarks: OVO-Timing and ProactiveVQA. It is divided into two main categories: "Temporal Sampling" (with sub-columns "Sampling Rate" and "Max. Frames") and "Adaptive Visual Allocation" (with sub-columns "Low Tokens / Frame", "High Tokens / Frame", and "High-Budget Frames after Standby").

**Figure 10.** Qualitative example of fine-grained video captioning. VideoChat3 describes the complete pole-vault sequence, covering the run-up, pole planting, takeoff, bar clearance, and landing.

**Figure 11.** Qualitative example of long-video question answering. VideoChat3 retrieves a specific visual detail from an extended video and correctly identifies the color of the floor microphone stand.

**Figure 12** Qualitative example of temporal video reasoning. VideoChat3 recognizes events distributed across the video and arranges them in chronological order.

**Figure 13.** Qualitative example of temporal video grounding. Given a language description, VideoChat3 accurately localizes the corresponding event, with only a minor boundary deviation from the ground-truth interval.

**Figure 14.** Qualitative example of online proactive response. During streaming perception, VideoChat3 remains silent when relevant evidence is absent and produces timely responses once informative visual content appears.

**Figure 15** Qualitative example of dense video captioning. VideoChat3 identifies multiple events throughout the video and generates temporally coherent descriptions that capture its evolving content.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers