ReflectWorld-MM: An Entity-Oriented Multimodal Memory System for Open-Ended Video Streams

Xiaokang Ma, Yifan Sun, Zhihong Jin, Jie Gu, Yudong Luo, Shenyi Shao, Chu Tang, Jingmin Chen, Li Pu

ReflectWorld-MM organizes streaming video into an entity-centric, hierarchical memory that persists across time.

How can an agent maintain a persistent, entity-oriented memory of an open-ended video stream to support long-term reasoning?

Current multimodal agents store video memory as flat feature banks or token caches, which degrade as streams grow and fail to track persistent entities across long durations. ReflectWorld-MM replaces these with an entity-oriented architecture that resolves visual inputs into persistent identities and stores them in a hierarchical, externalized database. This system achieves state-of-the-art accuracy across six long-video benchmarks, consistently answering queries from its structured memory rather than by re-processing raw video.

Paper Primer

The system operates as a continuous pipeline: a perception front-end resolves raw audiovisual streams into entity-linked observations, which are then written into three distinct memory stores. Episodic memory tracks events at entity, trace, and schema levels; semantic memory distills durable facts about entities; and procedural memory manages user-defined rules.

The core mechanism is context-enhanced perception: before interpreting a new segment, the model injects the current entity history and scene context into the prompt. This ensures the past actively shapes how the present is perceived, rather than serving only as a passive retrieval target.

ReflectWorld-MM outperforms existing memory-augmented agents on entity-sensitive benchmarks.

On the M3-bench-robot and M3-bench-web datasets, the system improves over the strongest prior baseline (M3-Agent) by 9.1 and 10.4 points, respectively. Significant gains in identity-tracking tasks, with the system maintaining high accuracy while requiring video fallback for less than 7% of queries.

The system effectively scales to long-duration video.

Compared to a frontier model (GPT-5) without dedicated memory, ReflectWorld-MM's performance advantage grows with video length, reaching a 9.0-point improvement on the LVBench benchmark.

Why is an entity-oriented approach necessary for long-term video memory?

Prior systems organize memory by frame or token, which is content-agnostic and fails to track who or what reappears over time. By anchoring memory to persistent entities, ReflectWorld-MM can accumulate longitudinal knowledge and revise facts about specific objects or people as new evidence arrives.

How does this system differ from standard retrieval-augmented generation (RAG) for video?

Unlike standard RAG, which retrieves from a static corpus, ReflectWorld-MM uses a perception front-end where the past participates in seeing the present. It also maintains an evolving semantic store that can update or delete facts, rather than treating memory as an append-only log.

The system's reliance on a deterministic resolver for identity and rule-matching is a critical design choice: it prevents noisy perception matches from corrupting the memory and ensures that every identity decision remains auditable.

ReflectWorld-MM demonstrates that persistent multimodal memory should be treated as an active, hierarchical system component that participates in perception, rather than a passive storage bank.

Introduction and Motivation

We expose why existing video agents fail to remember entities over long streams and introduce ReflectWorld‑MM.

Current multimodal agents with long‑term memory treat a video stream as a flat cache or as hidden state inside the model, storing raw frames or token embeddings without any higher‑level organization.

Because this cache is indexed by frame rather than by persistent entities, the systems cannot reliably track who or what reappears, and their performance collapses once the video length exceeds the distribution seen during training.

ReflectWorld‑MM (RW‑MM) tackles this gap by introducing an entity‑oriented multimodal memory that structures an open‑ended video stream into a hierarchy of episodic, semantic, and procedural memory components.

Assistants that continuously watch the world must retain knowledge about *entities*—people, objects, and actions—over arbitrarily long video streams, yet existing models only keep short‑term, frame‑level caches that forget or cannot link observations across time.

**Figure 1.** From egocentric video to entity-oriented multi-modal memory. A person wearing Meta glasses wanders an office, producing a live video stream. ReflectWorld-MM perceives the stream continuously and writes memory. The OpenClaw assistant can then answer the question about the water cup by retrieving the matching memory.

The shift from monolithic video processing to entity‑oriented memory enables assistants to retain and reason about persistent objects across arbitrarily long streams.

Related Work

Key prior systems for long‑video and agent memory are surveyed and positioned.

Long‑video models have added external memories, while language agents have built explicit recall stores; both lines inspire our entity‑oriented design.

MovieChat compresses an hour‑long video into a sparse memory by merging dense token embeddings, enabling a single‑pass dialogue over the whole clip.

MA (Memory‑Augmented LMM) builds a frame‑level memory bank that stores compressed visual embeddings, letting the language model retrieve any past frame without hitting the transformer context window.

Beyond these two, several works target streaming or real‑time video but keep memory inside the model or in a flat token cache.

Uses a compact feature cache that is read by the model at each timestep, enabling low‑latency inference on live streams.

Maintains a rolling token state that the language model consumes, allowing continuous video ingestion.

Offloads key–value caches to external storage and retrieves them on demand for streaming question answering.

At inference time, these systems pick a subset of frames most relevant to the current query and build a per‑video tree structure.

Language‑agent memory work focuses on textual recall, often modeling cognition‑inspired episodic, semantic, and procedural stores.

Provides an external tier where a language model can write and read information beyond its context window.

Maintains a stream of textual observations and performs periodic reflection to synthesize higher‑level summaries.

Agentic notes that write structured entries into an external memory, enabling targeted retrieval.

Extract‑and‑update pipeline that first extracts salient facts then updates a persistent store.

Imposes a forgetting curve on stored entries, gradually decaying older memories.

Multimodal agents that keep long‑term visual knowledge also exist, but they typically fuse all modalities into a single graph.

Stores an entity‑centric multimodal graph and retrieves it across multiple dialogue turns.

Builds multi‑scale episodic, semantic, and visual memories for long‑video reasoning.

Constructs a knowledge‑graph memory from text corpora and retrieves subgraphs for augmentation.

Temporal graph that models evolving entity relations over time.

Summarizes a document corpus into an entity graph and uses it for retrieval‑augmented generation.

Systems and datasets that aim for lifelong, in‑the‑wild perception from a first‑person viewpoint.

ReflectWorld‑MM unifies a multi‑scale episodic store, an editable entity‑centric semantic store, and a procedural store, while operating on open‑ended streams rather than fixed recordings.

System Architecture

We describe how ReflectWorld‑MM builds entity‑oriented memory online and enriches perception with hierarchical context.

Standard long‑video models forget who or what they have seen because they treat the stream as a flat sequence; ReflectWorld‑MM solves this by anchoring every observation to persistent entities and by feeding past context into perception as it happens.

Perception is performed not on raw frames alone but on a view that already knows which entities are present and what they have done before.

Detect produces two face crops; ReID scores them 0.92, above the match threshold.

Resolver assigns the same identifier $e_A$ to both crops.

Entity $e_A$’s episodic log now contains two events: “enter kitchen at $t_1$” and “enter kitchen at $t_2$”.

When processing the second segment, the prompt is prefixed with “Person $e_A$ previously entered the kitchen at $t_1$”.

Because the identifier persists, the model can answer “Did anyone enter the kitchen before?” with a single lookup instead of scanning all frames.

The next step is to keep this entity‑oriented view up‑to‑date as the stream flows, which is the purpose of the online memory construction algorithm.

Algorithm 1 – Online memory construction in ReflectWorld‑MM.

The system treats each entity as a mini‑database that is incrementally updated as new evidence arrives, while a short‑term buffer preserves the immediate narrative.

Segment 1: $W$ receives $o_1$, $M$ writes $o_1$ to $e$’s episodic log.

Segment 2: $W$ receives $o_2$, $M$ writes $o_2$; Count($e$)=2 ⇒ Consolidate($e$) merges $o_1$ and $o_2$ into a prototype $p_{12}$.

Segment 3: $W$ receives $o_3$, $M$ writes $o_3$; Count($e$)=3 triggers WriteSchema because $B=3$, producing a semantic summary “entity $e$ performed action X three times”.

Consolidation keeps the memory size linear in the number of distinct entities, not in the total number of frames.

The perception front‑end supplies the algorithm with entity‑resolved observations, turning raw audiovisual streams into the structured inputs required for memory updates.

Before any memory operation, the raw video‑audio stream is sliced into coherent segments, each enriched with entity identifiers and contextual prompts.

Detect finds $e_A$ in frames 1–2, then no detection in frames 3–4, and detection again in frame 5.

ReID scores the frame 5 face against $e_A$’s gallery with 0.88 similarity, above the match threshold.

Resolver assigns the same identifier $e_A$, and the working memory retains $e_A$ as “currently present” across the gap.

Prompt for frame 5 is prefixed with “Person $e_A$ re‑appears after a brief occlusion”, enabling the model to maintain continuity.

The three‑layer prompt lets the model treat the occluded interval as a seamless narrative rather than a disappearance.

**Figure 2.** System architecture of ReflectWorld-MM. The perception front-end converts an open-ended stream into entity-resolved observations, writes them into hierarchical episodic, semantic, and procedural memory, and feeds retrieved context back into perception; a QA agent queries the same memory through tools.

**Figure 1.** A diagram illustrating the ReflectWorld-MM process for answering a multi-choice video question. The figure is divided into three main sections: (1) The Question, (2) ReflectWorld-MM memory retrieval process, and (3) The Answer, which includes the final selection and supporting visual frames.

Hierarchical Memory Structure

Hierarchical memory lets the system keep persistent, entity‑aware facts over endless video.

Standard long‑video models forget who did what after a few minutes, because they try to cram the whole stream into a fixed context. The trick is to externalize memory and organize it hierarchically, so that raw observations, short‑term summaries, and long‑term storylines coexist without overwhelming the model.

Think of the memory as a three‑tier notebook: the first page logs every entity sighting, the next page writes a brief paragraph for each moment, and the final page drafts a chapter that stitches many paragraphs together.

How does this differ from a flat episodic buffer that simply appends observations?

In a flat buffer every new observation expands the searchable set, forcing the model to attend over the entire history. The hierarchical design caps the search space at the level most relevant to the query—traces for moment‑level questions, schemas for story‑level questions—so the model never scans more than a bounded subset.

Beyond raw events, the system keeps a compact fact sheet for each entity that is continuously refined as new evidence arrives.

Compute the increment: $(1 - w)\,\gamma = (1 - 0.6)\times0.2 = 0.08$.

Update the importance: $w \leftarrow 0.6 + 0.08 = 0.68$.

Because the updated $w$ is still below 1, the fact remains revisable by future evidence.

The importance score asymptotically approaches 1, so a fact that is repeatedly observed becomes hard to overturn, while newly added facts stay flexible.

Why not simply store a count of confirmations instead of the bounded $w$ update?

Counts grow without limit and cannot be compared across entities with different observation frequencies. The bounded $w$ normalizes confidence to a common scale, making ranking across facts robust to varying exposure.

User‑defined rules are stored as lightweight predicates that the system checks against incoming evidence, turning a match into a notification only after a deterministic resolver validates it.

How does this differ from a rule engine that fires immediately on detection?

Here the rule match is merely a signal; the resolver can suppress duplicate alerts, enforce cooldowns, or require additional context, whereas a naïve engine would fire on every raw detection.

The agent receives a query and selects the most relevant memory level (entity, trace, schema, semantic fact, or procedural rule).

It issues a tool call to the indexed database, filtering by shared entity identifiers and importance scores.

The returned subset is ranked, with higher $w$ facts boosted.

The agent incorporates the retrieved items into its prompt and generates the final answer.

**Figure 4.** Entity-centric semantic memory over a one-week vlog. Observations from different days are linked to the same person, so repeated food/cafe and exercise evidence strengthens persistent semantic facts.

**Figure 5.** Task-adaptive extraction in a coffee-making scene. Early coffee-making evidence sets the event context, which guides later perception of dripper setup, extraction, adjustment, and checking as one workflow.

**Figure 6.** Procedural memory for proactive notification. A stored dog-on-sofa rule, grounded visual evidence, and deterministic resolution together trigger the notification.

All memory items are persisted in an indexed database, so the store can grow without inflating the model’s context window. This contrasts with streaming‑cache systems like ReKV, which still tie state size to stream length.

Performance Evaluation

ReflectWorld‑MM outperforms all baselines on entity‑sensitive and general long‑video benchmarks.

Recall that ReflectWorld‑MM organizes open‑ended video streams into a hierarchical memory that preserves entity identity. The following results quantify how this design translates into performance.

ReflectWorld‑MM achieves the highest accuracy on all six evaluated benchmarks, surpassing the next‑best method by up to +10.4 points.

Table 1 and Table 2 show ReflectWorld‑MM leading each column.

A benchmark that asks “who did what” in long videos, requiring the model to retain and retrieve persistent entity identities.

**Table 2.** Results on the general long-video benchmarks (accuracy, %). Best in **bold**.

The table presents a comparison of various methods across three benchmarks: VideoMME-L, LVBench, and HippoVlog. The methods are categorized into Base models, Long-video LLMs, RAG-based methods, and Memory-based methods, with ReflectWorld-MM listed as the final entry.

ReflectWorld‑MM consistently outperforms baselines on entity‑sensitive tasks.

Memory Quality and Ablations

We quantify how each memory component affects answer efficiency and accuracy.

Answer efficiency—accuracy together with token count and video fallback—reveals whether stored memory actually serves the agent.

**Table 3.** Memory quality measured by answer efficiency. For each benchmark we report accuracy, average answer tokens per question (Tok.), and the percentage of questions that still require video fallback (Vid.). Higher is better for accuracy; lower is better for Tok. and Vid.

M3‑Agent never falls back (0.0 % fallback) but its accuracy is far lower, showing that compact traces miss essential evidence; WorldMM attains strong accuracy on VideoMME‑L yet requires video for 34.0 % of questions.

To isolate the contribution of each memory component we perform a fixed‑memory ablation: the full system builds memory once, then each variant disables one component at answer time.

Removing entity association reduces accuracy on M3‑bench‑robot from 37.4 % to 33.6 %.

Accuracy drops from 37.4 % to 33.6 % when the entity‑linked retrieval module is blocked.

Qualitative Analysis

Qualitative case studies and deployment details show how ReflectWorld‑MM writes and serves persistent memory.

Recall that ReflectWorld‑MM tackles the inability of standard long‑video models to keep persistent, entity‑aware memory by structuring open‑ended streams into episodic, semantic, and procedural stores.

**Figure 7.** The ReflectWorld-MM dashboard on a live interview. Left: the video source and a chat panel where an agent answers from memory (e.g., “Who is Ed Ludlow?”). Middle: per-segment moments with timestamps. Right: the entity panel for one person, with the evolving semantic memory, sightings, and recent activity written during perception.

The capture layer ingests streams from webcams, network cameras, HTTP feeds, or smartphones and forwards them to the memory backend without assuming any fixed video length.

All writes are stored in an indexed database external to the model, so any downstream agent can attach to the same long‑term store rather than each running its own isolated pipeline.

**Table 4.** Provenance of reported evaluation numbers. Unless specified here, baseline numbers are quoted from the corresponding original papers. MC denotes multiple-choice accuracy computed by option matching.

**Table 7.** Local perception utilities used by the deployment stack. Required modules are used by the default local perception profile; optional modules can be enabled or replaced.

Entity‑centric semantic memory: a week‑long vlog (Figure 4) links daily observations of the same person, allowing the semantic store to consolidate routine facts (e.g., coffee‑shop visits) rather than keeping isolated clips.

Task‑adaptive extraction: in a coffee‑making sequence (Figure 5) the system first recognizes the “coffee preparation” scene, then uses that context to interpret later frames as dripper setup, extraction, and brew checking, showing that perception adapts to accumulated scene knowledge.

Procedural memory and proactive response: a user‑defined rule “Notify me when the dog is on the sofa” triggers after the system observes the dog jump onto and lie on the sofa, converting visual evidence into a semantic signal that is matched against the stored rule with deduplication and cooldown.

**Figure 8.** Qualitative trace on VideoMME (long). The role question is answered entirely from memory written during perception—cockpit name–role bindings, the landing-responsibility evidence, and the operational context—which the agent retrieves and grounds in the landing event to answer “Barry, First Officer.”

**Figure 10.** Qualitative trace on M3-bench-robot. The tape-counting question cannot be answered from a coarse summary alone; the agent retrieves candidate tape-use windows, opens adjacent context, verifies the frames, and counts three distinct tape-use episodes.

**Figure 11.** Qualitative trace on EgoLife-QA. The agent retrieves candidate cart-pushing memories, selects the latest matching event, and grounds the selected event to Hema Supermarket using visual place evidence.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers