LatentStream: Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

Hongyu Qu, Guangming Yao, Ling Xing, Xiaobin Hu, Rongxing Ding, Guibin Zhang, Fan Zhang, Yi Yuan, Xiangbo Shu, Shuicheng Yan

LatentStream replaces external memory retrieval with a progressive, self-optimizing latent working memory for streaming video.

How can we maintain a bounded, query-relevant memory for streaming video understanding without relying on expensive, static retrieval systems?

Streaming video models struggle to balance long-term history with bounded memory, often treating past frames as external, variable-length context that clutters the model's reasoning. LatentStream shifts this paradigm by internalizing retrieved visual evidence into a compact, fixed-length latent state that evolves iteratively through test-time optimization. This approach achieves state-of-the-art performance on streaming benchmarks, improving OVO-Bench scores by 10.2% while simultaneously reducing peak memory usage by nearly 30%.

Paper Primer

The core mechanism, Hierarchical Latent Memory Evolution, functions like a multi-stage research assistant: it assigns different groups of latent tokens to specific temporal scopes, iteratively retrieves relevant visual evidence from those scopes, and refines its internal state to maximize predictive confidence.

LatentStream significantly outperforms existing training-free streaming methods on OVO-Bench.

The 7B model achieved 64.2% accuracy, a 10.2% improvement over the baseline.

The framework improves computational efficiency during inference.

Peak memory usage dropped from 30.80 GB to 21.97 GB, and per-token decoding latency decreased by 51.0%.

Why is "internalizing" evidence into latent tokens better than just retrieving it as visual context?

Retrieving evidence as external context forces the model to process variable-length, potentially redundant visual data during every reasoning step. Internalizing this evidence into a fixed-length latent state allows the model to maintain a compact, evolving working memory that directly guides reasoning without the overhead of auxiliary visual inputs.

Does this method require retraining the underlying Video-LLM?

No. LatentStream is a training-free framework that keeps the backbone model frozen, using test-time optimization to refine latent memory tokens based on a hierarchical confidence reward.

Researchers can now treat streaming memory as an active, evolving latent state rather than a static external bank, enabling more efficient and accurate long-form video reasoning without parameter updates.

Introduction and Motivation

We expose the core gap in streaming video understanding and motivate LatentStream.

Streaming video understanding demands multimodal large language models (MLLMs) to ingest a continuous visual stream while respecting strict causality and a bounded memory budget.

Current methods compress past observations into an external memory bank and retrieve query‑relevant evidence as extra visual context, but this store‑and‑retrieve pipeline never internalizes the evidence into a compact, evolving latent state that can continuously guide reasoning.

**Figure 1.** Motivation of our LatentStream: paradigm comparison of memory-based and retrieval-based methods with our retrieve-and-internalize method LatentStream.

The essential move is replacing an external retrieval step with an internal latent evolution that continuously refines a compact memory.

Streaming Video Understanding Context

Related work surveys streaming video understanding and long‑term memory strategies.

Streaming video understanding differs from offline analysis by requiring causal, real‑time processing of continuously arriving frames and immediate query responses. Existing work falls into four groups: proactive interaction methods that predict when to answer, streaming memory methods that keep bounded historical context, streaming thinking methods that let reasoning evolve with new observations, and real‑time inference techniques that cut computation via selective model use or token reduction.

Long‑term memory management in streaming videos must retain useful history under strict memory and context limits, and prior approaches are grouped into hierarchical multi‑level memory, visual token compression and pruning, KV‑cache memory, and retrieval‑augmented memory. In contrast, LatentStream internalizes retrieved evidence into compact latent memory tokens that iteratively evolve, allowing historical information to directly influence ongoing reasoning.

Hierarchical Streaming Memory Framework

We detail the hierarchical streaming memory that continuously organizes visual tokens under a fixed budget.

Streaming video understanding is limited by a fixed token budget that forces a trade‑off between how much history can be stored and how accurately it can be retrieved. The core obstacle is that naïve buffers either discard useful evidence too early or overflow the budget, causing retrieval errors.

HSM treats the memory as a three‑tiered shelf: newest frames sit on the top shelf (short‑term), slightly older frames are shuffled to the middle shelf (mid‑term), and the oldest, most compressed frames rest on the bottom shelf (long‑term), all while never exceeding the total number of slots.

How does HSM differ from a simple FIFO queue that drops oldest tokens when full?

A FIFO queue discards tokens solely based on arrival order, ignoring redundancy or importance. HSM, by contrast, evaluates temporal and spatial importance scores and uses Jenks‑based adaptive consolidation to decide which tokens to drop, compress, or preserve, so the memory retains the most informative evidence rather than just the newest.

All six tokens are placed in $M_s$ (short‑term) because the buffer is empty.

When $t_7$ arrives, the oldest token $t_1$ is evaluated: its score $0.9$ exceeds the high‑threshold, so it is promoted to $M_m$ (mid‑term).

Tokens $t_2$ and $t_3$ have moderate scores; they are compressed into a single representative token $t_{2,3}$ and stored in $M_m$.

Tokens $t_4$, $t_5$, $t_6$ have low scores; they are dropped from $M_s$.

The mid‑term buffer now holds $[t_1, t_{2,3}]$, occupying 2 of its allocated slots; the remaining 4 slots stay in $M_s$ for new arrivals.

HSM preserves high‑importance recent evidence, merges moderate evidence, and discards low‑importance noise, all while respecting the fixed budget.

Think of the memory as a set of bins that automatically adjust their breakpoints to group tokens with similar importance, much like a librarian who rearranges shelves based on how often books are borrowed.

Why not use a fixed threshold (e.g., $s_i > 0.5$) instead of Jenks partitioning?

Fixed thresholds ignore the distribution of scores; when a video segment is highly redundant, many tokens may fall just below the threshold and be unnecessarily kept, wasting budget. Jenks adapts the breakpoints to the actual score distribution, guaranteeing that the three groups are optimally separated for the current content.

Tokens with scores $<0.4$ ($0.2$) are assigned to Drop.

Tokens with scores between $0.4$ and $0.75$ ($0.45,0.6$) are assigned to Compress.

Tokens with scores $\ge0.75$ ($0.8,0.95$) are assigned to Preserve.

Compress tokens are merged into a single representative token, reducing the mid‑term count by one.

Drop tokens are discarded, freeing space for new incoming tokens.

Jenks automatically creates thresholds that reflect the actual spread of importance, ensuring that only truly low‑importance tokens are dropped.

Receive new visual token $t$ from the encoder.

Insert $t$ into short‑term buffer $M_s$.

Compute temporal importance $s$ for all tokens in $M_s$.

Apply three‑class Jenks to $s$ to obtain Drop/Compress/Preserve groups.

Drop low‑importance tokens, compress the middle group into representative tokens, and promote Preserve tokens to $M_m$.

Compute spatial distances $d_{ij}$ among tokens in $M_m$.

Apply two‑class Jenks to $d_{ij}$ to decide which tokens to merge (low‑distance) or keep (high‑distance) for transition to $M_l$.

Maintain the invariant $|M_s|+|M_m|+|M_l| = B$ by discarding or merging as needed.

Jenks‑based adaptive consolidation pseudo‑code.

**Figure 2.** The overview of **LatentStream**. (a) Query-agnostic hierarchical streaming memory (HSM) organizes incoming visual tokens into short-, mid-, and long-term memories via Jenks-based adaptive consolidation. (b) Hierarchical latent memory evolution (HME) retrieves and internalizes historical evidence into compact latent memory via progressively expanding memory receptive fields. (c) Progressive confidence-guided latent memory optimization (PMO) refines the latent memory tokens with a hierarchical progression reward for streaming reasoning.

Latent Memory Evolution Mechanism

We evolve hierarchical latent tokens by repeatedly retrieving and integrating visual evidence.

The hierarchical memory bank stores a bounded summary of the video stream, but it is external and oblivious to the current query, so the model cannot internalize task‑relevant history.

We give the model mutable latent tokens that repeatedly pull in past visual evidence, blend it, and update themselves, turning an external dump into a compact, task‑aware internal state.

Iteration 0: bootstrap produces $Z^{s}(0),Z^{m}(0),Z^{l}(0)$ by attending to the query and the full memory under group‑specific masks.

Iteration 1, group $s$: compute cosine similarity of each $v_i\in C_s$ to the two short tokens, pick the top 2 (usually both $v_1,v_2$). $E^{s}(1)=\{v_1,v_2\}$.

Iteration 1, group $m$: similarity to $Z^{m}(0)$ over $C_m=\{v_1,v_2,v_3\}$; top 2 might be $v_2,v_3$. $E^{m}(1)=\{v_2,v_3\}$.

Iteration 1, group $l$: similarity over $C_l=\{v_1,v_2,v_3,v_4\}$; top 2 could be $v_3,v_4$. $E^{l}(1)=\{v_3,v_4\}$.

Compute reward $R(1)$; suppose it improves over $R(0)$. Merge each $E^{g}(1)$ with the previous empty pool, yielding the first latent evidence sets.

Even with tiny token counts, the hierarchical groups quickly gather complementary evidence: short tokens focus on recent frames, while long tokens already capture the most informative distant frames.

Bootstrap forward pass creates query‑conditioned initial tokens $Z(0)$.

For each group $g$, compute max cosine similarity $a^{g}_j(r)$ between current tokens $Z^{g}(r-1)$ and candidates $m_j\in C_g$.

Select the top $B$ candidates via $\text{TopK}_B$ to form evidence $E^{g}(r)$.

Insert $E^{g}(r-1)$ after $Z^{g}(r-1)$ and evaluate reward $R(r)$.

If $R(r) > R(r-1)$, merge new evidence with the previous pool; otherwise keep the old pool.

Update $Z^{g}(r)$ by back‑propagating the reward gradient, ready for the next retrieval round.

How does this differ from standard attention over a static memory bank?

Standard attention treats the memory as a fixed set of keys and computes a single weighted sum per query. Here the latent tokens themselves are mutable; they retrieve evidence, get updated by a reward signal, and only keep evidence that improves the progressive confidence reward, turning the memory into an evolving latent state rather than a static lookup.

Confidence-Guided Memory Optimization

We introduce a confidence-driven reward that steers latent memory to grow reliably.

Broadening retrieval across the three LMT groups brings richer historical context, but it also admits irrelevant evidence that can destabilize latent evolution. The frozen MLLM’s predictive uncertainty offers a built‑in signal to curb this drift.

The reward pushes the latent state to become more certain as it absorbs longer‑range evidence, while penalizing sudden confidence drops between successive groups.

How does this reward differ from a plain entropy regularizer applied to the final output?

Standard entropy regularization penalizes uncertainty only at the final prediction, ignoring how confidence should evolve across intermediate memory groups. Progressive Confidence Reward explicitly measures and rewards a monotonic confidence increase from short‑term to long‑term groups, shaping the entire latent evolution trajectory.

For group $s$, the top‑3 tokens are the first three entries; renormalized probabilities $\bar{p}_{s} = [0.6/0.9, 0.2/0.9, 0.1/0.9] \approx [0.667, 0.222, 0.111]$.

Entropy $E_s = -\frac{1}{\log 3}\big(0.667\log0.667 + 0.222\log0.222 + 0.111\log0.111\big) \approx 0.92$.

For group $m$, top‑3 tokens give $\bar{p}_{m} \approx [0.7/0.95, 0.15/0.95, 0.1/0.95] = [0.737, 0.158, 0.105]$, yielding $E_m \approx 0.78$.

For group $\ell$, top‑3 tokens give $\bar{p}_{\ell} \approx [0.85/0.95, 0.1/0.95, 0.04/0.95] = [0.895, 0.105, 0.042]$, yielding $E_\ell \approx 0.55$.

The progression $E_s > E_m > E_\ell$ holds, so the reward contributes positively without any penalty.

This toy example shows how the reward enforces a smooth confidence rise: early groups tolerate higher entropy, but as more history is incorporated the entropy must drop, guiding the latent memory to become increasingly certain.

**Table 1.** Comparison with state-of-the-art methods on OVO-Bench [30]. Best results among open-source models are in **bold**, and the best results among training-free methods are <u>underlined</u>. $^\dagger$ indicates the reproduced results.

Optimization Objective

Optimizes latent tokens with a confidence‑driven REINFORCE loop.

Streaming video models must juggle limited memory against the need to recall distant frames. The core obstacle is that static retrieval quickly degrades as more history accumulates, leaving the model uncertain about earlier content.

We treat the latent memory as a controllable policy: at each iteration we jitter the state, evaluate how the reward improves, and climb the gradient toward higher confidence.

Compute the reward: $R_{\text{prog}}(U)= (1-0.2)+(1-0.5)=1.3$ (no progressive terms because only one short‑term group).

Sample a perturbation $\xi\sim\mathcal{N}(0,0.01I)$; suppose $\xi = (0.08,\,-0.04)$, giving $U' = U + \xi$.

Evaluate $R_{\text{prog}}(U')$; the errors become $E_{g_1}(U')=0.18$, $E_{g_2}(U')=0.48$, so $R_{\text{prog}}(U')=1.34$.

Compute the REINFORCE gradient: $\frac{U'-U}{\sigma^{2}} = (8,-4)$, multiply by the reward $1.34$ → $(10.72,-5.36)$.

Update the latent state: $U \leftarrow U + 0.5 \times (10.72,-5.36) = U + (5.36,-2.68)$, increasing confidence for $g_1$ and decreasing it for $g_2$.

The perturbation‑reward loop pushes the latent memory toward states that raise the hierarchical reward, effectively sharpening confidence without touching any model parameters.

**Table 2.** Comparison with state-of-the-art methods on StreamingBench [25] and offline video benchmarks [13, 62, 43]. Best results among open-source models are in bold, and the best results among training-free methods are underlined. † indicates the reproduced results.

Experimental Results and Analysis

LatentStream delivers large accuracy gains while cutting memory and latency.

Recall that LatentStream replaces static retrieval with a progressive, confidence‑guided latent memory that continuously refines historical tokens.

LatentStream reaches 64.2 % overall on OVO‑Bench, a +10.2 % gain over the frozen Qwen2.5‑VL baseline.

Table 3 shows the full model achieving 64.2 % versus 54.0 % without the three core components.

**Figure 3.** Hyperparameter analysis on OVO-Bench [30]. (a) Effect of the number of evolution iterations $R$. (b) Effect of the candidate budget $B$. (c) Effect of Gaussian noise scale $\sigma$. "RT" / "BW" denote Real-Time Visual Perception and Backward Tracing; "Avg." is the mean of RT and BW.

**Table 3.** Impacts of core components on OVO-Bench and VideoMME.

**Table 5.** Ablation of progressive confidence-guided optimization on OVO-Bench [30] and VideoMME [13].

The experiments confirm that progressive latent memory not only outperforms static retrieval on streaming tasks but also generalizes to offline long‑video benchmarks, delivering a favorable accuracy‑efficiency trade‑off.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers