DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding
Boyuan Sun, Deshui Miao, Wenzhao Gao, Shaoyong Jia, Shaohui Jiao, Chi Liu, Yan Shu, Derek Li, Bryan Dai, Shengsheng Qian, Qibin Hou
DeltaStreamer uses change-aware memory and query-driven replay to recover visual details omitted during continuous video observation.
How can an agentic system efficiently process streaming video to answer retrospective queries without losing critical details to KV-cache truncation?
Streaming video systems often struggle to balance token budgets with the need to capture event transitions, frequently omitting fine-grained visual details that only become relevant after a user query arrives. DeltaStreamer addresses this by recording explicit inter-frame changes alongside scene descriptions in a hierarchical memory, then using a query-driven planner to trigger targeted replay of historical frames when stored text lacks sufficient evidence. On the OVO-Bench benchmark, this approach achieves an overall score of 66.07%, outperforming representative streaming video agents.
Paper Primer
The system operates as an agent that continuously observes incoming frames, building a hierarchical directed acyclic graph (DAG) of textual memory. Before a query arrives, it stores structured observations consisting of scene descriptions and explicit "deltas" that capture changes between adjacent frames, ensuring event transitions are preserved rather than obscured by repetitive content.
When a query arrives, a planner generates a persistent evidence contract that guides memory retrieval and tool use. If the retrieved textual memory is insufficient to answer the query, the agent uses a targeted replay mechanism to revisit specific historical frames, recovering pixel-level visual details that were never encoded into the textual memory.
DeltaStreamer improves streaming video understanding by explicitly recording inter-frame changes and enabling on-demand visual replay.
Performance on OVO-Bench and StreamingBench benchmarks compared to existing streaming agents like EventMemAgent and STRIDE. Achieved an overall score of 66.07% on OVO-Bench, a +5.32 point gain over EventMemAgent.
Explicit change recording and hierarchical memory organization are critical for performance.
Ablation studies comparing 2F-VisualDelta (explicit changes) against 2F-NoDelta, and hierarchical memory against flat memory. Explicit visual deltas provided a +4.95 point gain when combined with planning, while hierarchical memory outperformed flat memory by 4.82 points.
Why is "delta-augmented" memory necessary if the model already sees the video frames?
Consecutive frames in a stream often contain highly similar content, which can lead models to produce repetitive descriptions that obscure event transitions. Explicitly recording changes as "deltas" ensures the system captures the progression of events rather than just the static state of the scene.
What happens if the agent's memory still lacks the specific detail needed to answer a question?
The agent uses a targeted replay mechanism to revisit the original historical frames. This allows the system to recover pixel-level visual evidence that was omitted during the initial observation phase, effectively grounding the answer in the raw video data.
The system assumes that streaming video understanding is an active information-gathering task rather than a passive encoding problem, shifting the focus from simple token compression to traceable, query-driven investigation.
For researchers building streaming agents, this paper demonstrates that memory should be structured around event transitions and query-driven retrieval rather than just maximizing the amount of compressed history retained.
Introduction and Motivation
Streaming video understanding must retain evolving details and retrieve them on demand.
Streaming video arrives continuously, so a system can only process frames it has already seen and must operate without knowledge of future content. Existing pipelines that compress or truncate the KV‑cache capture the current view but leave inter‑frame changes implicit, and once a frame is discarded the visual evidence it contained cannot be recovered for later questions. This gap motivates a memory that explicitly records visual deltas and a planner that can retrieve or replay those details on demand.
Retrospective Querying lets the system actively look back at earlier frames to fetch missing visual evidence once a query reveals a knowledge gap.
How does Retrospective Querying differ from merely reading a KV‑cache entry?
Reading a KV‑cache entry only returns whatever representation was stored at that time; Retrospective Querying can invoke a new perception step to fetch raw pixels or higher‑resolution crops, allowing the system to fill in details that were never encoded in the cache.
**Fig. 1** Comparison of streaming video understanding paradigms. Compression or truncation in KV-cache reuse and multi-turn dialogue can obscure transitions and discard details needed by later queries. DeltaStreamer records explicit changes in structured memory and uses query-driven planning and targeted replay to recover missing visual evidence.
The core trade‑off is between low streaming latency and the richness of historical detail that can be retained for future queries.
The DeltaStreamer Framework
DeltaStreamer stores only changes and uses a planner to retrieve needed visual details.
Streaming video systems must answer questions that arrive after the video has started, but they cannot keep every frame in memory. The resulting gap forces a design that preserves only the most informative information before the query and can later retrieve or reconstruct missing details.
Instead of storing every frame, the system records a concise scene description plus a short textual delta that captures what changed since the previous frame — like a version‑control log that only notes the edits.
$t=0$: $O_0 = (D_0=\text{"empty room"}, \Delta_0=\text{"initial frame"})$ (no previous frame).
$t=1$: $O_1 = (D_1=\text{"room with a red ball"}, \Delta_1=\text{"ball appears"})$.
$t=2$: $O_2 = (D_2=\text{"room with a red ball and a blue cube"}, \Delta_2=\text{"cube appears"})$.
Novelty scores $s(O_1)$ and $s(O_2)$ are high because the Jaccard similarity with $O_0$ is low, so both observations are stored.
Storing only the delta (“ball appears”, “cube appears”) plus the current description avoids keeping the full pixel data while still preserving the sequence of changes.
How does Delta‑Augmented Memory differ from simply saving every video frame?
Saving frames stores raw pixels, which quickly exceeds any memory budget. Delta‑Augmented Memory keeps a compact textual description and a short delta, reducing storage from megabytes per frame to a few words while still allowing reconstruction of the full scene by replaying the recorded changes.
The planner works like a GPS that, given a question, charts a route through the memory graph, decides whether to keep watching, and can trigger a targeted replay to fetch missing visual details.
Why not just perform a brute‑force search over all stored observations?
Brute‑force search would examine every node, incurring linear time in the number of observations. The agentic planner uses the salience scores and hierarchical pointers to prune irrelevant branches, achieving sub‑linear retrieval and allowing the system to act in real time.
DeltaStreamer organizes observations in a directed acyclic graph. Working memory $W$ holds the most recent observations, episodic memory $E$ aggregates related observations into summaries, and long‑term pins $L$ keep compact high‑level summaries linked to source episodes. An entity index $J_t$ maps object names to their observation timestamps, enabling fast lookup during back‑tracking.
**Fig. 2** Overview of DeltaStreamer. Before the query, adjacent-frame comparison produces delta-augmented observations stored in hierarchical memory. After the query, the planner guides evidence retrieval and tool use, while targeted replay recovers visual details for verification.
**Fig. 3** Illustrative delta comparison.
**Fig. 4** Qualitative examples of DeltaStreamer. The upper case recalls the action following the first block placement. The lower case recovers the missing wall letters from a historical crop.
OVO-Bench Performance
DeltaStreamer’s results on OVO‑Bench across perception and tracing tasks.
DeltaStreamer achieves the highest backward‑tracing average of $70.64$ on OVO‑Bench, beating the runner‑up by $0.92$ points.
Table 1 shows DeltaStreamer $70.64$ versus EventMemAgent $69.72$.
OVO‑Bench evaluates video models on two complementary abilities: real‑time visual perception (e.g., OCR, object‑joint‑recognition) and backward tracing (answering queries that require recalling earlier events).
**Figure 12.** OVO-Bench examples of temporal reasoning, attribute recall, and checking query premises.
**Figure 9.** What does the woman do after opening the jar?
**Figure 10.** What happens immediately after the view changes to a blank canvas?
DeltaStreamer’s edge in backward‑tracing tasks translates to more reliable temporal reasoning on real‑world video streams.
Benchmark Metrics and Data
Planner performance highlights a peak metric of 72.02 and average of 61.14 across queries.
The Query‑Driven Planner attains a maximum observed metric of 72.02.
The raw performance table shows 72.02 as the highest value among all reported metrics.
When a question arrives, the planner decides whether to pull past evidence, keep watching incoming frames, or invoke external tools, based on what the query actually needs.
How does this planner differ from a simple retrieval‑only approach?
A retrieval‑only system would just fetch past frames matching the query. The Query‑Driven Planner additionally decides whether to keep observing new frames or to replay historical frames, and it encodes those decisions in an evidence contract that guides tool use and dynamic memory updates.
StreamingBench Performance
DeltaStreamer tops StreamingBench with an 84.0 overall score, beating prior models by a large margin.
Streaming video systems must retain visual details over long horizons; DeltaStreamer stores explicit deltas and retrieves them only when needed, avoiding the loss incurred by naive truncation.
DeltaStreamer achieves the highest overall score of 84.0 on StreamingBench, surpassing the next best model by over 10 points.
Table 2 lists DeltaStreamer with an overall 84.0, while the closest competitor (STRIDE) reaches 73.08.
StreamingBench is a suite of video‑question tasks that probe a model’s ability to perceive frames in real time, combine information from many sources, and reason about context.
**Fig. 13** StreamingBench examples of action sequencing, scene changes, and visual reasoning.
**Table 2.** Evaluation results on StreamingBench [28].
**Table.** Comparison of metrics between OVO-Bench and StreamingBench.
Ablation Studies
Ablation results quantify each component’s contribution to the final performance.
This section isolates the impact of each architectural component by removing it and measuring the performance drop.
DeltaStreamer scores 66.07 on OVO‑Bench, a +5.32 improvement over EventMemAgent.
Table 1
On StreamingBench the system reaches 59.53, beating STRIDE by +1.95 points.
Table 2
Adding explicit visual deltas raises the score from 61.12 (2F‑NoDelta) to 66.07 (2F‑VisualDelta), a +4.95 gain.
Table 3
Hierarchical memory improves the OVO‑Bench score to 66.07, a +4.82 increase over flat memory’s 61.25.
Table 4
Including a DAG organization adds +3.34 to the score (66.07 vs 62.73 without DAG).
Entity indexing contributes +2.67 (66.07 vs 63.40 without index).
Switching from fixed grouping to BOCPD episode mode yields +2.30 (66.07 vs 63.77).
Plan generation alone reaches a score of 63.85.
Table 5(a)
Adding persistence raises the score further to 64.77.
When hierarchical memory is present, adding delta without planning improves the score by +0.93.
Table 5(b)
Middle recall boosts accuracy from 63.69 to 66.07, a +2.38 improvement.
Table 6
Removing the replay tool drops accuracy to 63.28, a -2.79 decrease.
**Fig. 14** Some examples of using replay or historical crops.
**Fig. 5** Visible memory (a) and tool use (b) on OVO-Bench and StreamingBench.
Implementation Details
Supplementary implementation, benchmark, and prompt details for DeltaStreamer.
This appendix first details the methodology: observation scoring (A.1), memory construction and rendering (A.2) with its episode‑construction sub‑process (A.2.1) and entity‑index handling (A.2.2).
Next, we describe the planner schema (A.3), the suite of evidence‑gathering tools (A.4), and the look‑back verification/replay logic that balances visual inspection cost against answer certainty (A.5).
Section B enumerates benchmark protocols, including the Omni‑Visual‑Observation Benchmark (OVO‑Bench) (B.1) and StreamingBench (B.2), with sample counts and task groupings.
Section C reports memory‑size growth and replay‑frequency statistics; Section D discusses limitations (D.1) and broader impacts (D.2) of the system.
Finally, Section E provides the prompt templates used for pre‑query observation, memory summarization, query‑driven planning, and evidence gathering, and Section F supplies additional qualitative examples.
Limitations and Discussion
Discusses system limits, ethical considerations, and the prompting schema used by DeltaStreamer.
DeltaStreamer’s design requires repeated model and tool invocations, which can increase inference latency. In practice, frame arrival and processing may be pipelined asynchronously, but the added calls still pose a latency overhead.
Compact visible memory reduces the footprint of stored frames, yet the system must retain records for each historical frame, incurring non‑trivial storage costs. Errors in observations or temporal anchors can propagate through retrieval, and agreement across memory layers does not guarantee perfect correctness.
Our evaluation uses a single shared backbone on two benchmarks, leaving open the effect of alternative backbones. The weaker performance on StreamingBench highlights the difficulty of precise response triggering, because retrospective evidence cannot compensate for a missed response window.
DeltaStreamer can improve accessibility tools and enable users to review extended activities by retrieving relevant historical evidence. However, continuous observation and structured replay expose personal routines and may enable non‑consensual monitoring.
Biases or errors in stored descriptions can misrepresent actions, leading to incorrect answers about people’s behavior. We therefore recommend informed consent, limited retention, strict access controls, and deletion mechanisms for both raw frames and derived memories.
Prompt Template E.1 Pre‑Query Observation defines a two‑line observation turn. The agent must report a full description of the current frame, list all actors, objects, actions, OCR text, and scene details, and then explicitly state the observable delta from the previous frame.
Template E.2 Memory Summarization and Online Recaps provides two summarization calls: an episodic summary (1–3 sentences) covering actors, action chain, final scene state, and salient changes; and a long‑term record (up to ~6 sentences) that enumerates every concrete fact from a run of observations.
Template E.3 Query‑Driven Planning instructs the agent to first restate the question’s need, point‑of‑view, evidence type, evidence locus, answerability, meta‑status, and strategy before any tool use. This structured plan ensures the agent reasons about where the required evidence resides and how to obtain it.
Template E.4 Evidence Gathering and Answering defines the tool‑call protocol, observation reporting, and response rules. The agent must emit a brief observation after each tool result, avoid unnecessary tool chaining, and only abstain when an “Unable to answer” option is explicitly provided.