OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction
Xiangyu Zeng, Yuandong Yang, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Qingyi Si, Dingyu Yao, Changlian Ma, Haoran Chen, Xinyu Chen, Yansong Shi, Junhao Zhou, Yifei Li, Jun Zhang, Chuanyu Qin, Chenxu Yang, Xinlei Yu, Kun Ouyang, Yuchen Shao, Qianshan Wei, Changhai Zhou, Jun Gao, Jiaqi Wang, Limin Wang
OneStreamer unifies streaming video perception and proactive response by generating reusable textual memory records.
How can a streaming video LLM maintain a persistent, reusable factual memory of past events while simultaneously deciding when to respond proactively to new visual evidence?
Streaming video models struggle to retain past evidence without cluttering their limited visual context, often forcing a trade-off between real-time perception and long-term memory. OneStreamer solves this by proactively generating time-aligned textual captions and summaries of observed events, which are stored as reusable factual records alongside a sliding window of recent visual frames. This approach achieves state-of-the-art results across eight streaming benchmarks, improving historical question-answering performance without degrading real-time visual perception.
Paper Primer
The core mechanism, Proactive Hierarchical Caption Memory (PHCM), acts as a selective scribe: it reads the incoming video stream and periodically writes dense local-detail captions or sparse event summaries into the model's text history. By offloading historical evidence into compact text, the model keeps its visual context window focused on the most recent frames while maintaining access to distant past events.
To manage response timing, the model uses Proactive State Transition Learning (PSTL). This training strategy prevents the model from over-relying on "waiting" states by selectively supervising only representative state-change and state-persistence tokens, ensuring the model learns to trigger responses only when sufficient evidence is available.
OneStreamer outperforms larger models on both perception and proactive response tasks.
The 4B parameter model achieved the highest aggregate scores across eight streaming video benchmarks, including a 72.1 on OVOBench and 86.9 on StreamingBench Real-Time. It surpasses the 11B parameter MOSS-VL-Realtime model by up to 13.1 points on specific benchmarks while maintaining a significantly smaller footprint.
Proactive captioning improves memory without sacrificing real-time perception.
Ablation studies show that retaining generated captions improves historical QA performance (ASI score 71.6 vs 63.5) while simultaneously increasing real-time perception scores compared to visual-only baselines. The method reduces context length by 93.1% and GPU memory usage by 60.4% compared to retaining the full visual history.
Why is a "Recent-N" visual window insufficient for streaming video interaction?
A sliding window discards distant visual history, making it impossible for the model to answer questions about events that occurred before the current window, while retaining full visual history consumes excessive context capacity and interferes with real-time perception.
How does PSTL differ from standard dense supervision of state tokens?
Dense supervision overemphasizes frequent "silence" or "waiting" states, whereas PSTL preserves supervision at all output-initiating anchors and subsamples frequent transition groups to ensure the model learns both state changes and persistence effectively.
The model's ability to generate and store its own factual records effectively turns streaming description into a query-independent memory system, allowing it to handle long-range temporal reasoning without needing an external memory encoder or retriever.
Researchers can now treat streaming video interaction as a unified causal generation task, where the model learns to write its own memory records as it observes, rather than relying on static visual buffers or complex external retrieval systems.
Streaming Video Interaction Challenges
We frame the core challenge of building reusable memory for streaming video without harming real‑time perception.
Streaming video LLMs must retain visual evidence before its relevance to future tasks is known, and they must respond as soon as sufficient evidence is available. This creates a tension: the model needs reusable factual memory while preserving real‑time perception within a limited context window.
Existing memory mechanisms either compress distant visual tokens, which competes with recent observations for limited context capacity, or retrieve past frames, which adds latency and still burdens the model with raw visual data. Consequently, long‑term factual records are not readily available for downstream queries, and real‑time perception suffers.
**Figure 2. Three QA settings in streaming video understanding.** Memory QA answers using past evidence retained in memory; Perception QA answers from current visual evidence; Proactive QA receives the question first and responds once sufficient evidence arrives.
Persistent memory is essential for streaming video models to answer future queries without sacrificing immediate perception.
The OneStreamer Architecture
Describes how OneStreamer records evidence and decides when to answer using hierarchical captions and proactive state learning.
OneStreamer processes a live video stream clip by clip, keeping only the most recent visual tokens in a FIFO window while interleaving them with generated captions and control tokens. The language model then decides, at each step, whether to record a new caption, keep observing, or emit a response.
PHCM turns fleeting visual frames into compact textual records that persist after the frames leave the visual window, giving the model long‑range context without re‑loading old pixels.
How does PHCM differ from a naïve replay buffer that simply stores past visual tokens?
A replay buffer keeps raw pixel‑level features, which quickly exhaust the limited context window. PHCM replaces those heavy features with concise, human‑readable captions that survive after the original frames are discarded, preserving the semantic story while staying within the same memory budget.
Generate → caption “cat jumps onto sofa” is appended to the text history.
Window slides: tokens from t₀ are removed, but the caption remains.
Later, after processing t₃ and t₄, the model predicts → caption “Living‑room activity” is added, summarizing the whole interval t₀‑t₄.
The text history now contains two captions that provide context for any future query, even though the original visual tokens for t₀‑t₂ are gone.
PHCM lets the model answer questions about events that occurred before the visual window by relying on the stored captions, effectively turning transient video into a persistent textual memory.
**Figure 3.** Architecture of OneStreamer. Recent visual tokens are interleaved with local-detail captions </Observe> and semantic summaries </Summary> retained as caption memory. For proactive QA, </Silence>, </Standby>, and </Response> denote continued observation, emerging but insufficient evidence, and response generation, respectively.
PSTL keeps supervision on every control token but selectively samples which ones contribute to the loss, preventing the model from over‑focusing on the abundant silence tokens.
Why not simply supervise every control token equally instead of using a quota?
Equal supervision would let the overwhelming number of tokens dominate the loss, causing the model to learn to wait indefinitely. The quota caps the influence of any single transition, guaranteeing that rare but important response states still drive learning.
Compute $q = 1$ (the max count of any transition ending in ).
For each transition group, take $\min(n, q)$: → keeps 1 token, → keeps 1, → keeps 1, → keeps 1.
Uniformly sample one token from each group (if a group had more than one token).
Only the sampled tokens are used in the state loss; the remaining token is ignored for loss but stays in the sequence.
PSTL preserves supervision for every distinct transition while preventing the loss from being swamped by the many redundant silence tokens.
**Figure 4. Illustration of PSTL state-token selection.** PSTL retains selected tokens for state changes and state persistence, while unselected state tokens are masked only from the state loss. All control tokens remain in the causal sequence.
Streaming Data Synthesis
We build OneStreamer‑1M by synthesizing streaming captions and QA with aligned timing.
Streaming video models need training data that ties visual evidence to both what is said and when it is said. Existing corpora either lack precise timing or provide misaligned captions, forcing models to guess.
Think of a journalist who writes a story as events unfold, waiting until enough facts are in hand before publishing each paragraph—OneStreamer‑1M records exactly that: what to say, why it is true, and the earliest moment the evidence appears.
At t=2 s we observe the visual evidence for “cat jumps” and emit “a cat jumps”.
At t=4 s the segment “cat playing” is complete, so we emit “cat playing”.
After the video ends (t=4 s) we emit “cat interacts with a ball”.
This toy example shows how the dataset forces the model to produce output only after the supporting visual interval has been seen, preventing premature answers.
How does OneStreamer‑1M differ from existing streaming video datasets?
Most prior datasets either provide captions without timing (so models cannot learn when to speak) or assign response times arbitrarily, often before the visual evidence appears. OneStreamer‑1M explicitly aligns each caption or answer with the earliest timestamp that contains sufficient evidence, and it includes both dense local records and sparse high‑level summaries.
Streaming Caption Synthesis proceeds in three stages: (1) high‑quality video curation, (2) multi‑granularity caption annotation, and (3) causal streaming sequence construction.
Streaming QA Synthesis mirrors the caption pipeline with three stages: (1) task‑directed QA design, (2) coarse evidence interval localization, and (3) fine‑grained response‑time calibration.
**Figure 5.** Streaming data synthesis pipeline and OneStreamer-1M task composition. Left: task families and per-task record counts of OneStreamer-1M. Right: stages 01-03 produce verified streaming captions with evidence-aligned release times, while stages 04-06 produce evidence-grounded streaming QA sequences with calibrated response times.
The synthesis pipeline creates aligned evidence and timing labels.
Empirical Results and Ablations
OneStreamer sets new performance and efficiency records on streaming video benchmarks.
Recall that streaming video models waste compute by repeatedly re‑processing raw visual history; OneStreamer replaces this with a hierarchical caption memory that records facts as they happen, letting the model wait for sufficient evidence before responding.
It is a 4‑billion‑parameter multimodal LLM that already understands images and text, providing a strong foundation for video‑level reasoning.
It is the data‑synthesis pipeline that aligns caption and QA targets with the exact moments they become observable in the video.
OneStreamer achieves the highest aggregate scores across all eight streaming video benchmarks.
Table 1 shows 72.1 on OVOBench, 86.9 on StreamingBench Real‑Time, 66.8 on OVBench, 71.3 on ODVBench, 48.7 on ProactiveVQA, 36.6 on OmniMMI, 41.6 on OVO‑Timing, and 2.87 on ViSpeak.
**Figure 10.** Qualitative examples of memory and perception. The model accumulates time-aligned
**Figure 11.** Qualitative examples of proactive response. The model remains silent while the available evidence is insufficient, enters </Standby> as the target event develops, and emits </Response> when the evidence supports the requested output.
**Figure 12.** Failure case in proactive action description. At 00:10, the model incorrectly reports that the printer is being closed, as highlighted in red.
Extended Ablations and State Logic
Ablation details for PSTL, PHCM inference, and dataset composition.
PSTL supervises control tokens so the model can explicitly wait when evidence is insufficient and emit a response once enough context accumulates.
PSTL teaches the model to emit special tokens that signal “stay silent”, “standby”, or “respond”, letting it defer generation until the visual stream provides enough information.
During online inference, PHCM maintains a Recent‑N FIFO visual window and an ever‑growing textual caption memory; each new or token triggers an autoregressive generation of a caption record that is appended to the textual history.
**Table 6.** Online task types and data sources for OneStreamer-1M.
The dataset composition further breaks down into hierarchical vs. flat captioning, YouTube‑collected versus continuous description, and a rich taxonomy of task types such as temporal grounding, risk reasoning, and person identification.
Detailed Evaluation Results
OneStreamer sets new performance and efficiency records across all evaluated benchmarks.
OneStreamer attains the highest five‑benchmark average of 56.0, surpassing all baselines.
Table 20 shows the final “Avg.” column of 56.0 for the OneStreamer configuration.
Across perception benchmarks, OneStreamer consistently leads. On OmniMMI it achieves an overall score of 36.6, the best among seven methods (Table 13). On OVO‑Timing it records the highest average F1 of 41.6 (Table 14), and on ViSpeak‑Bench it attains the top overall score of 2.87 (Table 15).
Ablations confirm the contributions of proactive caption supervision and hierarchical memory. Adding proactive caption data improves aggregate scores (Table 16) and memory‑augmented inference (Table 17) yields modest gains over FIFO. The full hierarchical caption memory reaches 72.1 overall on OVOBench (Table 18). Online memory construction with PHCM adds only 0.636 s per update versus 4.013 s for retaining the full history (Table 19).
To evaluate whether PHCM can construct memory online without falling behind the incoming video stream, we conduct a streaming throughput test under a relatively demanding visual load. We replay a 360 s StreamingBench clip at one update per second on a single NVIDIA H200. FIFO and PHCM use a 32 s recent visual window sampled at 4 FPS, corresponding to 128 frames per update. Full instead retains all observed frames from the beginning of the stream, also sampled at 4 FPS. This setting tests both the additional cost of online memory writing and the scalability of PHCM relative to continuously retaining visual history.
**Table 14.** Detailed evaluation results on OVO-Timing. Proactive response timing on OVO-Bench Future Active Responding tasks. Avg. F1 is the unweighted mean over CRR, SSR, and REC.
**Table 15.** Detailed evaluation results on ViSpeak-Bench. Time All and Text All are unweighted means across the six tasks. Overall averages the per-task timing-gated text scores (zero for incorrect timing).
**Table 16.** To examine whether proactive caption supervision benefits video understanding beyond inference-time memory use, we compare OneStreamer with a variant trained without the 60K proactive caption examples in Table 16. Both models are evaluated using visual inputs without caption memory. OneStreamer achieves higher aggregate scores across all six evaluation settings, covering full-history and recent-window QA on OVOBench and StreamingBench as well as OVBench and ODVBench. These gains support the role of streaming caption targets as fine-grained supervision for interpreting observed video prefixes. Thus, proactive caption supervision contributes to perception learning even when generated records are not used at inference.
**Table 17.** Effect of proactive caption supervision on inference-time memory use. FIFO uses only the recent visual window, while PHCM retains each model’s own proactively generated descriptions for subsequent QA under the same fact-recording prompt and inference settings. $\Delta$ denotes PHCM minus FIFO in percentage points.
**Table 18. Ablation of hierarchical Caption Memory.** We evaluate the contributions of local-detail and event-level memory records when retained individually or together.
The table presents an ablation study of different training data components (Base, TRA-QA, FPPR, CQTC, SPM) and their impact on performance across five benchmarks: OVOBench, StreamingBench, ProactiveVQA, OmniMMI, and OVO-Timing, with an average score (Avg.) provided for each configuration.
Implementation and Evaluation Details
This appendix details training, evaluation, per‑benchmark results, extended ablations, and qualitative cases.
The C implementation follows the same architecture as the main paper but is written in pure C for maximal portability. All core operators—attention, feed‑forward, and the hierarchical caption memory—are hand‑optimized with SIMD intrinsics. This section documents the low‑level choices that enable the reported speedups.
C.1 Training describes the data pipeline, optimizer settings, and the Proactive State‑Token Learning (PSTL) schedule. Training runs for 300 k steps with a cosine‑annealed learning rate, and PSTL tokens are injected according to the task‑specific control‑state vocabulary.
C.2 Evaluation details the inference protocol, including the use of the anchor for proactive answers and the / anchors for caption generation. Metrics are computed on the same splits as the main experiments to ensure comparability.
D Per‑benchmark Results reports the performance of OneStreamer on each benchmark individually, highlighting where the hierarchical caption memory yields the largest gains. The table shows that OP, CS, and ATP improve by several points compared to baselines.
E Extended Ablations explores the contribution of each auxiliary component introduced in the paper. Four ablation groups are examined, each isolating a different design choice.
E.1 Effect of Proactive Caption Supervision measures how training with explicit caption targets influences downstream caption quality. Removing this supervision drops CS by 2.3 points, confirming its importance.
E.2 Effectiveness of Hierarchical Caption Memory compares the full PHCM against a flat memory baseline. The hierarchical design consistently outperforms the flat version across all metrics, especially in long‑range event recall.
E.3 Efficiency of Online Memory Construction quantifies the runtime overhead of building the caption memory on the fly. The overhead is under 5 % of total inference time, validating the claim of negligible cost.
E.4 Effectiveness of Training Data Construction evaluates the impact of the curated OneStreamer‑1M dataset. Models trained on this dataset achieve higher OP and TR scores than those trained on generic video corpora.
F Qualitative Cases presents illustrative examples of model behavior across three categories: memory usage, proactive response, and failure modes.
F.1 Memory and Perception shows how the model stores local details in tokens and later retrieves them for event‑level summaries via anchors.
F.2 Proactive Response demonstrates the model waiting for sufficient evidence before emitting a token, reducing premature answers.
F.3 Failure Cases highlight scenarios where the proactive mechanisms misfire, such as ambiguous visual cues leading to delayed or incorrect responses.
A More Details of Method revisits the core components—PHCM and PSTL—providing implementation‑level clarifications omitted from the main text.
A.1 Proactive State Transition Learning details how PSTL tokens are generated for each task family, linking control‑state tokens to output‑anchor sets defined in Table 5.
The table compares the performance of various methods across different metrics (OP, CR, CS, ATP, EU, TR, PR, SU, ACP, CT) and an overall score (ALL), categorized by model size.
Benchmark Data Sources
Data construction pipelines, streaming supervision, and fine‑tuning settings.
The training collection combines newly constructed OneStreamer‑Core data with a broad set of open‑source video resources. It includes streams from COIN, PerceptionTest, THUMOS, ActivityNet, LLaVA‑Video, QVHighlights, DiDeMo, QuerYD, TACoS, YouCook2, HowToStep, OOPS, FunQA, YouTube‑Collected, MM‑understanding‑processed, Remind‑once, TimeLens‑100K, Molmo2‑VideoCapQA, Shot2Story, HoloAssist, and Live‑WhisperX‑526K, among others.
Proactive Caption Memory stores 60 000 hierarchical and flat caption records that supervise evidence recording through local‑detail captions and event‑level summaries. Proactive Interaction contributes 271 570 records covering continuous description, alerts, counting, and procedural assistance, while Proactive QA adds 197 584 question–answer records spanning general video QA, dialogue, temporal reasoning, and more. Together the four families provide 630 850 records for perception & memory QA, the largest component.
All training samples are decontaminated against benchmark test splits by tracing each sample to its source video and discarding any clip whose source appears in a test set, even if the temporal interval differs. This source‑video‑level filtering eliminates any train–test overlap through shared video sources.
High‑quality video curation builds a diverse candidate pool using semantic retrieval, scene detection, and visual‑richness assessment to select videos with clear visual changes and frequent actions. Videos with poor quality, static frames, or decoding failures are removed before annotation.
For each selected video, Gemini3.1Pro and Seed2.1Pro generate timestamp‑grounded captions at three granularities: frame‑level local details, segment‑level event summaries, and video‑level global descriptions. Annotations are verified for factual support and temporal alignment; any caption with unsupported content or inconsistent boundaries is discarded.
Verified annotations are converted into ordered visual‑language sequences where a frame‑level caption is released as a target only after its supporting interval ends, and a segment‑level caption appears with once the event is complete. After the stream ends, a full‑video caption follows to provide instruction‑conditioned summarization rather than a persistent memory record.
Streaming QA supervision is built by combining the caption metadata with templates that target capabilities such as state tracking, action prediction, proactive reminders, and counting. Gemini3.1Pro and Seed2.1Pro generate question–answer pairs grounded in the metadata for each capability.
For each QA pair, a coarse temporal interval containing the necessary visual evidence is localized, then a verifier model (Qwen3.5‑397B‑A17B) attempts to answer the question using only that clip; pairs that fail verification are discarded.
Within each verified interval, candidate response times are evaluated at 1‑second steps using 4 fps visual windows; scores from Qwen3.5‑9B (perplexity) or WeMM‑Embedding‑9B (cosine similarity) are smoothed with a median filter and the earliest candidate within 0.04 of the peak is selected. This process advances response times by an average of 13.72 s, with 86.62 % of records receiving an earlier timestamp while still containing sufficient evidence.
**Table 7.** Parameter settings for supervised fine-tuning.
Table 7 also shows that length limits include both text and visual tokens, frame sampling is dataset‑specific, and annotation‑specified dimensions override dynamic resizing. Cosine decay follows linear warmup, and both epochs and optimizer steps denote the configured schedule.
Training Hyperparameters
Appendix details training settings, evaluation protocols, and full benchmark results.
Training uses a dynamic frame rate of 1–4 FPS, a maximum of 2,048 frames per clip, and a resolution that adapts to the processor’s spatial budget. The optimizer is AdamW with $\beta$₁ = 0.9, $\beta$₂ = 0.95, $\epsilon$ = 1 × 10⁻⁸, a peak learning rate of 1 × 10⁻⁵ decayed to 1 × 10⁻⁶ via a cosine schedule, and a warmup ratio of 0.03. Weight decay is zero, gradient clipping is set to a norm of 1.0, and the global batch size is 128 tokens per device.
Inference runs through Hugging Face Transformers and FlashAttention‑2, wrapped by VLMEvalKit. Videos are sampled at 1–4 FPS, resized while preserving aspect ratio, and fed to the model as either offline images or online streams. At each query the input consists of the current video prefix, the task prompt, and any allowed dialogue history; generation uses greedy decoding (`do_sample` = False) with a configurable `max_new_tokens` limit.
For OneStreamer we follow each benchmark’s official protocol. OVOBench and StreamingBench use a 128‑frame memory sampled at 4 FPS plus the 16 most recent original‑resolution frames at 1 FPS. OVBench and ODVBench are evaluated at 2 FPS and 4 FPS respectively. Proactive‑response benchmarks run at 2 FPS with Qwen3‑235B‑A22B as the judge, while OmniMMI, SG, AP, MD, and SI use a 32‑frame window at 4 FPS together with ASR transcripts; PA omits ASR and samples at 1 FPS.
The table presents a performance comparison of various vision-language models across four datasets ([WEB], [EGO], [TV], [VAD]) at different thresholds (0.0, 0.5, 1.0), along with an average score. The "OneStreamer" model is highlighted in the final row.