Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

Chang Nie, Jiaju Wei, Junlan Feng, Chaoyou Fu, Caifeng Shan

Light-Omni replaces iterative "detective-style" reasoning in video agents with a reflexive, dual-state memory architecture.

How can agentic video understanding models achieve real-time responsiveness without the latency penalty of iterative, "detective-style" reasoning?

Agentic video models currently rely on multi-step, "detective-style" reasoning to search and aggregate evidence, which introduces prohibitive latency and high computational costs when processing long-horizon streams. Light-Omni decouples this process into a dual-state design: a non-parametric global state that consolidates historical context via hierarchical merging, and a parametric latent state that enables reflexive action control and retrieval in a single forward pass. This approach achieves a 12.1× speedup and 2.6× memory reduction compared to existing reasoning-based agents, while simultaneously improving accuracy on long-video benchmarks.

Paper Primer

Existing video agents treat reasoning as a compensatory mechanism for poor retrieval, forcing the model to repeatedly plan, search, and cross-reference clues like a detective. This iterative workflow creates a bottleneck where the model spends more time deliberating than responding, making real-time interaction with hour-long video streams computationally infeasible.

Light-Omni is a reflexive framework: it uses a hierarchical "global state" to maintain a compact, resolution-decaying summary of past events, and a "latent state" to generate retrieval embeddings directly from the model's hidden features. By fusing these states, the model bypasses explicit multi-step planning, instead triggering actions and retrieval as an implicit reflex during a single forward pass.

Light-Omni significantly outperforms reasoning-based agents in both accuracy and efficiency.

Compared to the M3-Agent baseline, Light-Omni delivers a 2.4% accuracy gain, a 12.1× speedup, and a 2.6× reduction in GPU memory footprint. 20.5× speedup and 3.3× memory reduction over the base Qwen2.5-Omni-7B model.

The dual-state mechanism provides superior robustness against noisy, context-dependent queries.

Under audio-injected noise, Light-Omni's performance degradation is limited to 1.3%, compared to 3.7%–5.1% for standard RAG-based approaches. Higher Signal-to-Noise Ratio (1.352 vs. 1.201) in retrieval tasks.

Why does this approach prioritize "reflex" over the "reasoning" used by other agents?

The authors argue that heavy reasoning is primarily a workaround for the lack of a persistent global context. By building an explicit global state, the model can rely on reflexive retrieval rather than iterative, multi-step deliberation, which is the primary source of latency in current agentic systems.

Is this a new model architecture or a plug-in module?

It is a framework that functions as a universal memory module. It can be integrated into existing Multimodal Large Language Models (MLLMs) to enhance their long-term memory capabilities without requiring a complete architectural overhaul.

Light-Omni demonstrates that long-horizon video understanding can be achieved through compact, hierarchical memory consolidation rather than expensive iterative reasoning, enabling real-time agentic performance.

The Reflexive Paradigm Shift

We expose the inefficiency of iterative reasoning in video agents and motivate Light‑Omni’s reflexive design.

Current video agents treat each query as a detective case: they iteratively search, rewrite, and plan before acting, which balloons compute and latency. This workflow exists because models lack a persistent global context that would let them answer directly. Human cognition avoids such costly deliberation by maintaining a holistic mental map that triggers reflexive responses.

When a video agent cannot rely on an overarching memory of past observations, it compensates by chaining multiple reasoning steps—search, rewrite, plan—each incurring extra compute and latency.

**Figure 1.** Comparison of video understanding paradigms and overall performance. (Left) Recent video agents rely on “detective-style” iterative reasoning to bridge the semantic gap between query and offline-constructed memory representations, suffering from high latency and local vision. In contrast, Light-Omni leverages dual contextual states to maintain a global context and directly generate semantically aligned embeddings for evidence aggregation. By incorporating sleep-time memory consolidation, it enables seamless online interaction with both global vision and near-constant latency. (Right) Light-Omni achieves strong performance on long-video benchmarks while maintaining near-constant latency regardless of video duration.

The core shift is moving from costly detective‑style reasoning to instant reflexive actions powered by dual contextual states.

Video Understanding Paradigms

We survey omni‑modal, retrieval‑augmented, and agentic video models, highlighting their limits and key innovations.

Recent multimodal large language models (MLLMs) such as the Qwen‑Omni and Gemini series process interleaved video, audio, and text streams end‑to‑end, yet they still hit finite context windows and high compute costs on long‑horizon inputs. Retrieval‑augmented and agentic video approaches aim to extend temporal continuity, but each introduces its own trade‑offs.

Open‑source and proprietary models that ingest video, audio, and text jointly, using unified transformer backbones.

Proprietary omni‑modal models that similarly handle video, audio, and text, emphasizing high‑fidelity perception.

Methods that store compressed frame features or textual metadata (clip summaries, ASR transcripts) and retrieve relevant entries during inference.

Agentic video system that decomposes queries via multi‑turn reasoning and invokes external tools (ASR, search, grounding) to gather evidence.

Agent that extends OmniAgent with modality‑aware tool selection and deeper hierarchical planning for very long videos.

A streaming variant of video‑language models that processes each incoming frame immediately, maintaining a rolling hidden state instead of re‑encoding the whole history.

A high‑throughput video stream encoder that compresses raw frames into a lightweight token sequence using a fast convolutional front‑end.

A hierarchical memory dispatcher that routes incoming video tokens to either a short‑term buffer or a long‑term episodic store based on predicted relevance.

Dual-State Architecture

Light‑Omni’s dual contextual state design separates fast reflexes from slow memory consolidation.

Iterative reasoning pipelines in existing video agents incur prohibitive latency because each query must trigger a multi‑step deliberation over external tools. Light‑Omni eliminates that bottleneck by splitting the workload: a lightweight reflex path handles immediate responses, while a slower consolidation path updates a persistent memory without blocking the forward pass.

The model keeps two parallel states— a fast‑changing latent state for instant reflexes and a slowly‑evolving global state that aggregates long‑term knowledge—so each can operate at its natural timescale.

Compute $S_0$ by feeding $I_0$ into the MLLM; suppose $S_0 = (0.2,\,0.1)$.

Generate reflex action $A_0$ from $S_0$; the model outputs “say hello”.

Schedule a consolidation step: append the tuple $(I_0, S_0)$ to the memory buffer, yielding $M_0$.

At $t\!=\!1$, a new input $I_1$ (“ask weather”) arrives. The latent state $S_1$ is computed from $I_1$ and $S_0$, giving $S_1 = (0.3,\,0.4)$.

Reflex action $A_1$ becomes “report weather”. Meanwhile, the consolidation thread updates $S_g$ using $M_0$, producing an enriched global state that will inform future queries.

The reflex path never waits for the memory update; it always produces an action from the freshest latent state, guaranteeing low latency even as the memory grows.

**Figure 2.** Overall architecture of Light-Omni. It operates via dual contextual states augmented with long-term memory. (Top) Real-time Reflex ($arrow$): Light-Omni continuously processes streaming inputs to generate the latent state $S_l$ for instantaneous action control and context-aware retrieval. (Bottom) Sleep-time Consolidation ($arrow$): The system asynchronously manages long-term memory via CRUD (Create, Read, Update, Delete) operations and hierarchical aggregation to refresh the global state $S_g$. This coupled design ensures reflexive and lightweight video understanding.

By isolating fast reflexes from slow consolidation, Light‑Omni delivers near‑constant inference latency without sacrificing the richness of a multimodal long‑term memory.

Reflex and Consolidation Mechanisms

Dual-state design couples fast reflexes with a hierarchical global context.

Iterative reasoning over long video streams forces the model to recompute expensive attention for every frame, wasting compute and latency.

Like a biological reflex arc, the system processes each incoming frame on the fly and immediately produces a latent state $S_l$ that drives an action decision without any multi‑step deliberation.

Frame 1: concatenate $[S_g, I_1, P_{\text{soft}}]$, run $\pi_\theta$ → hidden $H_1$.

Extract $h_1^{\text{ret}}$ and $h_1^{\text{act}}$, compute $\text{ret}_1$ and $\text{act}_1$ via the heads.

Result: $\text{act}_1 = 1$ (trigger) and $\text{ret}_1 = [0.7,0.2]$.

Frame 2: repeat with $I_2$; $\text{act}_2 = 0$, $\text{ret}_2 = [0.4,0.5]$.

The reflex path produces a decision after a single forward pass, so latency grows only with the number of frames, not with the depth of reasoning.

How does this reflex path differ from the multi‑step reasoning pipelines used in prior video‑LLMs?

Prior pipelines repeatedly attend over the entire memory bank for each query, incurring $O(N)$ passes. The reflex mechanism collapses the whole pipeline into one pass by conditioning on a compact global state $S_g$ and using dedicated heads, eliminating the iterative loop.

Think of the global state $S_g$ as a hierarchical summary that keeps recent fine‑grained details while progressively compressing older events, much like a multi‑level notebook where older pages are merged into higher‑level outlines.

Since $N_0=3 \ge k+1$, merge the oldest $knodesm_1,m_2$ into a new level‑1 node $m_{1}^{\prime}$.

After merging, level‑0 retains $m_3$; level‑1 now holds $m_{1}^{\prime}$.

Future incoming node $m_4$ is added to level‑0, giving $N_0=2$ (no further merge).

The global state $S_g$ now consists of $[m_{1}^{\prime}, m_3, m_4]$, a compact three‑node representation.

The merge keeps the total node count at $k+1$ while automatically promoting older information to a coarser level.

Why not simply concatenate all episodic scripts into a single long vector?

Concatenation would grow linearly with video length, exploding memory and computation. Hierarchical merging caps the representation size and preserves a multi‑scale view, enabling fast lookup for the reflex head.

Light‑Omni also maintains two explicit memory stores: semantic memory $M_s$ holds distilled key‑value facts, while episodic memory $M_e$ records chronological omni‑modal scripts.

The user profile $M_p$ is refreshed only when the buffered topic set exceeds the capacity threshold $\tau_t$, preventing unnecessary updates.

Receive streaming input $I_t$.

Query the current global state $S_g$ (produced by hierarchical merging).

Append learnable soft prompts $P_{\text{soft}}$ to the concatenated sequence $[S_g, I_t, P_{\text{soft}}]$.

Run the backbone $\pi_\theta$ once to obtain hidden features $H_t$.

Decode $h_t^{\text{ret}}$ and $h_t^{\text{act}}$ from $H_t$.

Compute retrieval embedding $\text{ret}$ and binary action $\text{act}$ via the two heads.

Update $S_l$ with $\text{ret}$ and $\text{act}$; feed $\text{act}$ to the control module.

Asynchronously merge any newly created episodic nodes into $S_g$ according to the capacity factor $k$.

Latent‑state generation and action decision (single forward pass)

**(Retrieval Embedding)**

Learning proceeds with a multi‑LoRA adapter stack that isolates memorization, generation, and reaction pathways, preventing gradient interference.

Two engineering tricks further cut latency: (1) visual/audio features are cached after the first extraction, and (2) temporally redundant tokens are pruned based on cosine similarity.

Performance and Efficiency

Light‑Omni uses a dual‑state design to deliver fast reflexive responses while preserving long‑term context.

Light‑Omni uses a dual‑state design to deliver fast reflexive responses while preserving long‑term context. This section evaluates its accuracy, efficiency, and robustness across several video‑understanding benchmarks.

Light‑Omni outperforms strong baselines on long‑video benchmarks, achieving 58.0 % average accuracy while delivering a 20.5× inference speedup and a 3.3× memory‑footprint reduction versus the Qwen2.5‑Omni‑7B baseline.

Table 1 reports Light‑Omni’s accuracy, latency, and memory gains across VideoMME‑long and LVBench; Table 2 shows a total interactive response time of 2.28 s, over 8× faster than M3‑Agent.

**Figure 3.** Impact of efficiency optimization strategies. Feature caching and redundancy pruning cumulatively reduce inference latency by 42% without compromising accuracy.

**Table 1.** Performance and efficiency comparison on long-video benchmarks. We report accuracy (%), latency (s), and memory footprint (GB). Subscripts highlight the absolute accuracy gains and relative efficiency multiples (speedup/reduction) achieved by Light-Omni relative to their corresponding baseline models.

**Table 2.** Efficiency comparison on VideoMME-long. “Interactive Response” measures per-query latency, while “Memorization” denotes the offline processing cost for an entire video (avg. 2436s). Light-Omni achieves the best accuracy with over 8× speedup than iterative agents.

**Figure 4.** Comparison of Light-Omni and reasoning-based video agents. Light-Omni achieves the best overall performance across the benchmarks. Compared to the strong runner-up WorldMM-8B (which relies on GPT-5-mini for memory construction), Light-Omni not only yields a 1.1% higher overall accuracy but also delivers an over 8× speedup (see Table 2).

Light‑Omni achieves superior accuracy with an 8× speedup, making it a practical drop‑in memory module for video agents.

Memory System Components

Detailed memory design and its impact on performance.

Light‑Omni stores three complementary views of the multimodal stream: a user‑centric profile, distilled factual knowledge, and a chronological event log.

Removing the unified retrieval representation (Sₗ) drops LVBench accuracy by 0.64 %.

Table 4 shows the “Sₗ (ours)” variant achieves 49.90 % accuracy, a +0.64 % gain over the baseline text‑embedding retrieval.

Removing the global‑state module ($S_g$) reduces VideoMME‑long accuracy by 2.99 %.

Table 3 reports $S_g$ (ours) at 66.10 % (+2.99 % over the vanilla baseline).

**Figure 6.** Similarity matrices between queries and ground-truth memories. Compared to naive RAG, Light-Omni achieves a sharper diagonal and higher SNR, ensuring robust semantic alignment under noisy inputs.

**Figure 5.** Robustness evaluation under noisy queries on LVBench. Light-Omni shows superior robustness.

**Figure 7.** Accuracy vs. retrieval granularity. Performance stabilizes around $N_s = 12$ for semantic memory. However, for episodic memory, $N_e > 4$ introduces distracting noise on LVBench.

**Figure 9.** Near-constant response latency of Light-Omni across varying video durations.

**Figure 8.** Scalability of Light-Omni over extreme time horizons.

Data Synthesis Pipeline

We detail the pipeline that builds Light‑Omni’s training set and its adapter splits.

Existing open‑source video corpora focus on offline understanding and therefore miss the long‑horizon, interactive scenarios required for agentic video models.

To fill this gap we construct a bespoke dataset through a four‑stage pipeline.

First, public videos are harvested from platforms such as YouTube and randomly grouped to emulate prolonged, multi‑session user interactions.

Second, simulated user dialogues are injected at diverse temporal anchors; we employ CosyVoice to synthesize speech that blends with the original soundtrack while preserving ambient tones and background noise.

Third, a commercial large multimodal model (Gemini‑3‑Flash‑Preview) is prompted to act as a personalized assistant and produce ground‑truth responses given the full video‑plus‑dialogue context.

Finally, intermediate signals that capture memory construction and retrieval steps are extracted, aligning precisely with the pipeline illustrated in Fig. 2.

**Figure 10.** Distribution of the Light-Omni training dataset across the three adapters. The dataset is heavily weighted towards the Memory and Reaction adapters to cultivate agentic behaviors.

The full synthesis yields 46,166 training instances drawn from 2,118 distinct videos.

These instances are divided among three adapters: Memory Adapter (10,493 samples), Generation Adapter (4,785 samples), and Reaction Adapter (30,888 samples), each targeting a specific capability.

LoRA hyper‑parameters ($r$ = 64, $α$ = 128, dropout = 0.05) and learning schedules are tuned per adapter, but the key design choice is to allocate substantially more data to memory and reaction streams to cultivate agentic behavior while keeping generation data modest to avoid catastrophic forgetting.

Training and Runtime Pipeline

Details of adapter training and the Light‑Omni inference pipeline.

Training adapts the $\\text{Qwen2.5-Omni-7B}$ backbone via independent LoRA adapters, each fine‑tuned on a single node with eight $\\text{NVIDIA H800}$ GPUs.

Algorithm 1 – Light‑Omni runtime pipeline (simplified).

The “Real‑Time Reflex” loop processes each incoming audio segment, performs a single retrieval‑decode step, and immediately enqueues the result for later consolidation.

Queued items are consumed by the “Asynchronous Memory Consolidation” thread, which extracts topics, updates the user profile, and refreshes the global state without blocking the forward pass.

Statistical breakdown of the runtime workload shows that topic generation dominates (86.2 % of 10 493 operations), while memory and generation adapters handle the remaining streams.

Adapter‑specific usage: the memory adapter processes 4 380 open‑ended QA queries (91.5 % of its 4 785 calls), and the generation adapter splits its 30 888 calls among retrieval (50 %), speech (35 %) and search actions (15 %).

Extended Benchmarking

Light‑Omni outperforms prior video models across perception and tracing benchmarks.

Light‑Omni leads in 9 of the 13 evaluated metrics, establishing a clear advantage over prior video models.

Table 7 shows Light‑Omni achieving the highest scores on OCR, ACR, FPD, OJR, and ASI, among others.

**Table 1.** Performance comparison of different models on Real-Time Visual Perception and Backward Tracing tasks.

Limitations and Discussion

We discuss Light‑Omni’s practical impact, limitations, and broader implications for video understanding.

Light‑Omni replaces costly iterative reasoning with a dual‑state design that yields fast reflexive responses while preserving long‑term context.

The system runs three asynchronous processes: (1) continuous input preprocessing (speech activity detection, face recognition); (2) on speech end or a 30‑second silence, reflexive actions are generated and queued; (3) a background worker updates the memory store from the queue.

**Table 9.** Ablation study of Light-Omni components. We report Accuracy (%) and Latency (s) across three long-video benchmarks

**Table 9.** Ablation study of Light-Omni's core components.

On the OVO‑Bench online tasks, Light‑Omni attains 54.51 % average accuracy, beating open‑source online MLLMs such as Dispider (+3.05 %) and Flash‑VStream (+25.41 %).

Against leading baselines—including proprietary MLLMs, RAG‑based agents, and reasoning‑based agents—Light‑Omni improves average accuracy by +10.9 % over the Qwen2.5‑Omni‑7B baseline and yields the largest gains on HippoVlog (+13.0 % over M3‑Agent, +8.8 % over WorldMM‑8B).

**Figure 11.** Empirical analysis of performance and system latency across different compression ratios.

Redundancy pruning operates at two granularities: inter‑frame pruning removes temporally similar frames by cosine distance of mean embeddings, and intra‑frame pruning keeps the most salient spatial tokens. Frames deemed redundant are reduced to a 32‑token anchor set, and salient tokens are sampled with compression hyperparameter $r_{\text{comp}} = 0.25$.

To ensure the synthetic dataset reflects real‑world interaction, source videos span diverse domains and the generated dialogues mimic authentic acoustic tones and environmental noise, a design validated by strong results across long‑video benchmarks.

The broader impact claim is that effective context maintenance and retrieval—not heavy reasoning—are the true cornerstone of video understanding. Light‑Omni’s dual‑state architecture demonstrates that reflexive intelligence can surpass iterative reasoning, suggesting future systems will treat foundation models as unified memory‑retrieval engines.

Limitations remain: performance is bounded by the backbone model (Qwen2.5‑Omni‑7B trails newer Qwen3 series), though Light‑Omni still lifts Qwen3‑VL‑8B by 2.5 % on average. Offline memory construction stays computationally heavy, halved relative to M3‑Agent but still demanding for massive video streams.

System Prompt Configurations

System prompts that drive each stage of the Light‑Omni pipeline.

This section enumerates the concrete prompts that operationalize the dual‑state design across memory construction, response generation, profile updating, and global‑state consolidation.

Memory Construction Prompt

Response Generation Prompt

User Profile Updating Prompt

Hierarchical Consolidation of Global State Prompt

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers