EM2Mem: Event-Centric Multimodal Memory for Large Language Models

Yijun Chen, Yaqi Zheng, Yanya Li, Boyi Xiao, Buqiang Xu, Shuofei Qiao, Jizhan Fang, Xinle Deng, Yunzhi Yao, Xuehai Wang, Liuxin Zhang, Hui Li, Huajun Chen, Shumin Deng

EM2Mem organizes long-video evidence into event-indexed memory cells to enable efficient, grounded reasoning.

How can we organize long-video data into event-centric memory cells to improve retrieval and reasoning accuracy for LLMs?

Existing long-video question answering systems retrieve isolated fragments like captions or frames, forcing language models to reconstruct temporal and cross-modal alignments during inference when context is limited. EM2Mem shifts this burden to memory construction by parsing videos into short temporal events and binding heterogeneous evidence—captions, transcripts, and structured metadata—into unified, event-indexed memory cells. This align-then-retrieve design improves average accuracy by up to 3.7 points across benchmarks while reducing per-query latency by 4.67× and total inference tokens by 63.66%.

Paper Primer

The core mechanism is an event-centric memory schema that treats 30-second video segments as "event anchors." Instead of storing raw, disconnected data, the system uses these anchors to index a multimodal record containing structured fields (actions, objects, scenes) and multi-scale temporal context views.

EM2Mem is like a library cataloger: it reads the entire book (the video) once to create a structured index of events and their relationships, so that when a user asks a question, the system retrieves a pre-assembled "dossier" of relevant evidence rather than searching for loose pages.

Construction-time unification outperforms retrieval-time fusion.

Aligning multimodal evidence into event cells before indexing preserves cross-modal dependencies that are often lost when modalities are retrieved independently. Structured event fields combined with construction-time unification achieve 71.2% accuracy, compared to 68.0% for retrieval-time fusion.

Event-indexed memory significantly improves inference efficiency.

By retrieving pre-aligned event cells, the model avoids expensive cross-modal alignment during the query phase. 4.67× reduction in per-query latency and 63.66% reduction in total inference tokens.

Why does this approach require more offline work than existing methods?

EM2Mem intentionally shifts computation from inference to memory construction to create coherent, generation-ready evidence units. This upfront cost is amortized over repeated queries, making it highly efficient for long-term video repositories or benchmark evaluations.

Does this method replace the need for visual input during answer generation?

No; while the event-centric memory provides the primary evidence, the system still attaches a small number of keyframes to the query-specific evidence view to allow the language model to perform lightweight visual verification.

For long-video QA, organizing memory around coherent events rather than modality-specific fragments enables more accurate, attributable, and computationally efficient reasoning.

Introduction and Motivation

We frame the gap that long‑video QA faces when retrieving isolated fragments instead of event‑centric evidence.

Multimodal memory enables long‑video QA, but current approaches retrieve isolated fragments (captions, frames, transcripts, etc.) that lack context and require the LLM to reconstruct cross‑modal and temporal alignments at inference time, limiting attribution. We introduce EM2Mem (Event‑Centric Multimodal Memory), which binds heterogeneous evidence to event anchors during memory construction, creating event‑indexed memory cells that unify multimodal records, temporal context, graph‑linked relations, and provenance. Across three benchmarks EM2Mem raises accuracy by up to 3.7 points, improves event‑level Top‑5 recall by 7 points, and cuts per‑query latency by 4.67× while reducing inference tokens by 63.66%.

Long‑video QA demands reasoning over evidence scattered across minutes or hours of visual observations, speech transcripts, OCR, and scene dynamics, which cannot be captured in a single context window. Fragment‑centric memories store modality‑specific fragments that are searchable but not generation‑ready, forcing a retrieve‑then‑align step where the LLM must stitch together missing cross‑modal and temporal bindings under a limited context budget. Inspired by event cognition, we reorganize multimodal memory around event anchors, aligning all evidence offline into event‑centric cells before retrieval.

**Figure 1.** EM$^2$Mem organizes multimodal evidence into event-centric memory cells, enabling grounded retrieval over events rather than isolated fragments.

Shifting from fragment‑based retrieval to event‑centric memory unifies multimodal evidence, delivering more accurate, efficient, and attributable long‑video QA.

Problem Formulation

Provides context on long‑video QA and existing multimodal memory approaches.

Given a long video with visual and textual evidence streams $V=\{V_{\text{vis}},V_{\text{text}}\}$ and a natural‑language question $q$, the system must produce an answer $\hat{y}$ that is explicitly grounded in the video’s evidence.

Prior multimodal memory systems break a long video into per‑modality chunks—visual clips or text excerpts—and store each chunk as a searchable summary and embedding.

The EM2Mem Framework

EM2Mem reorganizes long videos into event‑centric memory for efficient LLM reasoning.

EM2Mem follows an align‑then‑retrieve design: it first aligns heterogeneous evidence into event‑centric cells, then retrieves compact evidence for answering questions.

Instead of a flat list of frames, the video is split into short events, each becoming a memory cell that bundles local multimodal evidence and multi‑scale context.

Event 1: $e_1 = (1,\;0\text{s},\;30\text{s})$, $R_1$ contains a caption “person plants seeds”, a transcript excerpt, and keyframe $k_1$.

Event 2: $e_2 = (2,\;30\text{s},\;60\text{s})$, $R_2$ holds “watering the garden” and associated visuals.

Event 3: $e_3 = (3,\;60\text{s},\;90\text{s})$, $R_3$ records “harvest discussion”.

Each $\mu_i$ is formed by pairing $e_i$ with its $R_i$ and the set $C_i$ of context views (see next concept).

Grouping frames into event cells reduces the retrieval search space from thousands of frames to just three cells while preserving all local evidence.

How does an event‑centric memory cell differ from a traditional frame‑level memory?

Traditional memories treat each frame or caption as an independent entry, requiring the model to retrieve many fragments and then fuse them. An event‑centric cell aggregates all evidence for a short temporal segment, so retrieval picks a whole, context‑rich event in one step, dramatically cutting the number of look‑ups.

Each event also gets summaries at coarser time scales (e.g., 3 min, 10 min, 1 h) so the model can reason over longer intervals without scanning every event.

Block $h_1^{3\text{min}}$ aggregates $\text{utext}_{1,3\text{min}}$ = “planting and watering activities” and $\text{uvis}_{1,3\text{min}}$ = collage of keyframes from both events.

Metadata $m_{1,3\text{min}}$ merges actions {plant, water} and objects {seeds, hose}.

Both event cells $\mu_1$ and $\mu_2$ include $h_1^{3\text{min}}$ in their $C_i$ sets.

Even though the question may refer to “the first few minutes”, the model can answer using a single 3‑minute view instead of stitching together two separate events.

Why not just use a sliding‑window summary instead of explicit multi‑scale views?

Sliding windows produce overlapping, redundant summaries and lack a clear hierarchical relationship. Temporal context views are defined at distinct scales, each anchored to the exact set of events they cover, enabling the model to pick the most appropriate granularity without ambiguity.

Two lightweight graphs link events through shared entities and long‑term patterns, providing cross‑event clues while staying grounded in concrete video evidence.

How do these graphs differ from building a full‑blown graph neural network over all frames?

Full GNNs treat every frame as a node, leading to massive graphs that are expensive to run and hard to interpret. $G_E$ and $G_S$ operate on the much smaller set of event anchors and only add edges that are semantically meaningful, keeping computation cheap and explanations traceable.

Given a question $q$, compute relevance scores for each memory cell $\mu_i$ using signals from $R_i$, $C_i$, and graph neighbors.

Select a shortlist of high‑scoring cells via an LLM‑based selector.

Expand each selected cell by traversing $G_E$ and $G_S$ to gather linked entities, temporal transitions, and semantic facts.

Assemble a query‑specific evidence view $E_q$ that concatenates captions, transcripts, keyframes, context summaries, and retrieved graph facts.

Condition a large language model on $(q, E_q)$ to generate the final answer $\hat{y}$.

Retrieval pipeline – high‑level pseudocode.

**Figure 2.** Overview of EM$^2$Mem. EM$^2$Mem parses a long video into event anchors, constructs event-centric multimodal memory cells with local records $R_i$ and multi-scale context views $C_i$, and links them through episodic and semantic graphs $G_E$ and $G_S$. Given a question, lightweight retrieval selects and expands relevant event cells to compile evidence for answer generation.

Experimental Results

EM2Mem delivers up to 37.39× throughput boost and 2.0‑3.7‑point accuracy gains over prior memory baselines.

EM2Mem outperforms the strongest memory baseline WorldMM† by up to $3.7$ accuracy points and achieves a $37.39\times$ throughput improvement.

Table 1 shows average accuracies $66.0$ vs $64.0$ (EgoLifeQA) and $67.7$ vs $65.3$ (Ego‑R1 Bench); Table 2 reports the $37.39\times$ speedup.

**Table 1.** Performance comparison on EgoLifeQA and Ego-R1 Bench. WorldMM denotes the originally reported results. † denotes our reproduced results under the same evaluation setting as EM²Mem. For Ego-R1 Bench, we report the published WorldMM results because reproducing WorldMM requires rebuilding subject-level long-video memories for all six participants, which incurs substantial construction and inference cost. The corresponding entries are marked as “–”.

The table compares two memory organization methods, "4-channel RRF" and "Aligned-30s", across six metrics: $R@1$, $R@5$, $C@150$, $C@300$, $C@600$, and QA. The final row indicates the performance gain achieved by the Aligned-30s method over the 4-channel RRF method.

EM2Mem consistently raises accuracy while dramatically lowering inference cost across all evaluated long‑video QA benchmarks.

Ablation and Design Analysis

Ablation and analysis dissect how each EM2Mem component and design choice impacts long‑video QA performance.

Recall that EM2Mem reorganizes long‑video evidence into event‑centric memory cells, shifting multimodal alignment to an offline construction phase.

Removing temporal context views drops overall accuracy to 60.4%.

Full EM2Mem achieves 66.0%.

Removing semantic memory reduces accuracy to 61.4%.

Removing the episodic graph lowers accuracy to 61.6%.

A baseline that only uses a local 30‑second event record attains 64.0% accuracy.

Full EM2Mem reaches 66.0%.

Structured event fields combined with construction‑time unification achieve the highest accuracy, 71.2%.

Other representations (raw frames, flattened captions) top out at 68.0%.

Adding three keyframes after retrieval improves overall accuracy from 63.2% to 66.0%.

Figure 3d shows the upward trend.

**Figure 3.** Analysis of EM$^2$Mem design choices and efficiency. (a) Structured event fields with construction-time unification achieve the best accuracy. (b) Component-wise ablations show different contributions across question types. (c) EM$^2$Mem improves event-level recall as retrieval budget increases. (d) Visual keyframes improve overall accuracy, especially with three keyframes. (e) EM$^2$Mem amortizes its wall-clock overhead as the number of queries increases; dashed lines indicate wall-clock and token break-even points.

**Figure 4.** Event-level recall under different selector budgets.

Efficiency and Cost Analysis

We analyze EM2Mem’s limits: construction cost, amortization, ablations, alignment, granularity, and evidence unification.

Recall that EM2Mem moves multimodal alignment to an offline construction phase, building event‑centric memory cells that can be queried efficiently at inference time.

**Table 9.** Offline token cost of memory construction. Prompt and completion denote model input and output tokens. The construction cost is incurred once and amortized across repeated queries.

**Table 10.** End-to-end cost on EgoLifeQA. “Memory-Cons.” denotes offline memory construction, and “Inference” denotes QA inference. EM^2Mem incurs higher construction cost but substantially reduces inference cost, yielding a 3.03× end-to-end wall-clock speedup. The token-count break-even occurs after approximately 536 queries.

**Table 11.** Category-wise ablation on EgoLifeQA. Full denotes EM^2Mem. w/o TCV, w/o SM, and w/o EG remove temporal context views, semantic memory, and episodic graph, respectively. 30s uses only 30-second event records. Accuracy is reported in %.

**Table 14.** Anchor sensitivity on all 500 EgoLifeQA questions. $C@B$ denotes macro target-time coverage (%) under a $B$-second retrieved-video budget.

**Table 13.** Analysis of evidence interface and alignment timing on EgoLifeQA. Cap., Vis., and SER denote flat caption evidence, raw visual evidence, and structured event records, respectively. Late denotes retrieval-time alignment, while Early denotes construction-time alignment. EM^2Mem denotes our event-centric unified multimodal memory framework. All numbers denote accuracy (%).

**Table 15.** Concrete memory objects and query-time trace for the question “Who plans to grow flowers?” Scores in the ranking row are normalized coarse anchor scores.

Overall, the limits analysis shows that EM2Mem’s construction overhead is justified when memories are reused, and that each architectural component—temporal context views, semantic memory, and episodic graphs—contributes measurably to performance.

Related Work and Conclusion

We situate EM2Mem among prior work and summarize its contributions, limits, and ethical considerations.

Long video understanding has evolved from video‑text pretraining to video LLMs that integrate visual encoders with language models for instruction following and long‑form comprehension. Recent extensions address longer contexts via token compression, hierarchical memory, and temporal reasoning, yet scaling to hour‑long videos makes evidence sparse and temporally distant, motivating explicit organization mechanisms.

Memory‑augmented systems for LLMs store multimodal evidence such as captions, transcripts, keyframes, OCR outputs, and graph indices, retrieving relevant pieces before generation. Closely related egocentric and agentic approaches build hierarchical memories, episode summaries, or adaptive retrieval procedures to support long‑range reasoning.

WorldMM is a compact representation that captures the global multimodal context of a video by aggregating event‑level embeddings across modalities into a single memory tensor.

Limitations of EM2Mem stem from its structured memory design: converting visual evidence to textual fields loses fine‑grained details such as small objects or subtle color cues, and the align‑then‑retrieve paradigm still relies on selected keyframes for detailed visual reasoning.

Ethical considerations arise because long‑video QA can expose privacy‑sensitive routines and personal environments; while we use public benchmarks and do not collect new data, misuse of event‑centric memory could enable privacy leakage or biased profiling, so deployment must enforce consent, access control, and data minimization.

We thank the anonymous reviewers for their feedback and acknowledge support from national science projects, university funds, and industry research grants.

Dataset Details

Details the benchmark datasets, their statistics, and query‑type mappings.

This appendix enumerates the datasets used to evaluate EM2Mem and clarifies how their query types relate to the proposed event‑centric taxonomy.

**Table 7.** Summary of benchmark datasets used in our experiments.

**Table 8.** Mapping Ego-R1 Bench query types to the EgoLifeQA category taxonomy.

The license note reminds readers that all datasets are publicly released and must be obtained from their original sources; usage is limited to academic research under the respective licenses.

Implementation and Prompting

Implementation specifics for EM2Mem, baselines, prompts, and computational costs.

We evaluate EM2Mem against a wide spectrum of baselines spanning four categories: Base MLLMs, Long‑video MLLMs, RAG‑based pipelines, and Memory‑based long‑video reasoning methods.

Baseline models include GPT‑5, Gemini 2.5 Pro, Qwen3‑VL‑8B‑Instruct, VideoChat‑Flash, Time‑R1, Video‑RTS, LightRAG, HippoRAG, Video‑RAG, EgoRAG, Ego‑R1, HippoMM, M3‑Agent, and WorldMM.

For all baselines except WorldMM we reuse the results reported in the original WorldMM paper to keep dataset splits, metrics, and evaluation protocols identical.

WorldMM† is reproduced with the same backbone as EM2Mem: GPT‑5‑mini‑2025‑08‑07 for memory construction and GPT‑5‑2025‑08‑07 for retrieval and answer generation.

**Figure 6.** Prompt for constructing text-derived fields in Multimodal Event Records, part 1.

**Figure 8.** Prompt for constructing visual fields in Multimodal Event Records, part 1.

**Figure 10.** Prompt for constructing text summaries in Temporal Context Views, part 1.

Prompt for constructing visual summaries in Temporal Context Views, part 2.

During EM2Mem memory construction we invoke GPT‑5‑mini‑2025‑08‑07 to turn event anchors into multimodal records, then build Temporal Context Views at four temporal granularities.

Episodic triplets are harvested from both local records and context views; semantic triplets with similarity > 0.6 are consolidated by an LLM, and the top‑10 are kept for inference.

At inference time each query retrieves up to KE = 5 episodic candidates and KS = 8 semantic facts; scores combine a weight of 1.00 for the local record with scale weights 0.65, 0.45, and 0.30 for longer context windows, and a graph‑expansion decay of 0.60.

The LLM selector (GPT‑5‑2025‑08‑07) picks `K_sel` = 5 event anchors, and for each anchor we attach up to $K_V$ = 3 keyframes before feeding everything to the answer model.

All experiments run on two NVIDIA A100 40 GB GPUs with eight parallel workers for embedding‑index storage and retrieval.

Table 9 reports the offline token cost of constructing each memory component; Table 10 shows that EM2Mem adds 10 k s of construction time but cuts inference wall‑clock time from 229 k s to 6 k s, yielding a 3.03× overall speedup.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers