Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory
Minghao Guo, Qingyue Jiao, Xiujin Liu, Jianhong Pang, Ruimeng Yang, Naihao Xu, Liwei Che, Haochen Xue, Ziqi Wang, Xinying Cai, Jiaxuan Luo, Jiaxin Deng, Xuesong Ye, Jintao Qu, Wenyao Gao, Jingwen Gu, Chenyang Zhao, Shichun Liu, Guibin Zhang, Zefang Liu, Siyin Wang, Wujiang Xu, Danrui Li, Zeru Shi, Woongyeong Yeo, Shuang Liu, Wei-Chieh Huang, Weizhi Zhang, Tianyu Yang, Kangsan Kim, Yebowen Hu, Bowen Wen, Bowen Jiang, Haohao Su, Ming Yin, Sheldon Yu, Mian Zhang, Yuyan Chen, Hanrong Zhang, Kai Mei, Zihao Zhang, Mingyu Jin, Xi Zhu, Hengyi Wang, Tianxin Wei, Linyong Nan, Zhenwen Liang, Zhendong Chu, Ziqi Huang, Yongye Su, Sixun Dong, Shujian Liu, Kaiqu Liang, Jiaojiao Han, Zhaoye Fei, Kun Wang, Shihan Dou, Zhichun Guo, Xingjian Diao, Zeqi Gu
A two-axis analytical framework for multimodal agent memory, classifying systems by how they form representations and manage persistent state.
How do current multimodal AI agents store, manage, and evolve their long-term memory across different modalities and interaction settings?
Multimodal agents struggle to maintain continuity over long horizons because they cannot simply store every perceptual input, yet they must retain enough detail to support future reasoning and action. The authors introduce a two-axis framework: Axis I (Formation and Representation) characterizes how experience is encoded—ranging from raw source media to latent parameters—while Axis II (Memory Management) classifies systems by how they update memory, from static reuse to self-evolving policies. This taxonomy synthesizes 297 memory architectures, revealing that while dynamic memory updates are common, systems that evolve their own management policies through feedback remain rare.
Paper Primer
The paper addresses the "multimodal memory lifecycle" problem: how to bridge the gap between high-dimensional, heterogeneous observations (video, audio, 3D space) and the need for persistent, reusable state. Unlike text-only agents, multimodal systems face a fundamental trade-off between preserving raw perceptual fidelity and maintaining a compact, editable memory that doesn't overwhelm the model's context window.
The core contribution is a unified taxonomy that maps memory systems onto two coupled axes. Formation (Axis I) determines the "what" and "how" of encoding, while Management (Axis II) defines the "when" and "where" of persistent adaptation: T1 (Static) reuses fixed memory, T2 (Dynamic) updates state via a stable policy, and T3 (Self-Evolving) uses feedback to improve the management policy itself.
The field is shifting from static retrieval toward dynamic, experience-driven memory management.
A corpus analysis of 297 memory architectures shows a high prevalence of dynamic memory updates (T2) compared to static systems (T1). The authors identify 82 benchmark records across modalities, highlighting that while dynamic updates are now standard, self-evolving (T3) systems remain a significant frontier.
Why is this framework necessary for multimodal agents specifically?
Multimodal observations—such as video streams or 3D spatial maps—cannot be losslessly reduced to text. The framework is required to navigate the trade-offs between storage cost, recoverability of source evidence, and compatibility with downstream reasoning.
What is the primary distinction between T2 (Dynamic) and T3 (Self-Evolving) memory?
T2 systems update the memory content (e.g., merging entries or forgetting stale data) using a fixed, pre-defined policy. T3 systems go further by using feedback to update the policy itself, allowing the agent to learn how to better write, maintain, or retrieve memories over time.
Researchers should move beyond simple retrieval-augmented generation (RAG) and focus on "lifecycle-aware" design, where memory formation is explicitly coupled with management policies that can evolve through deployment-time feedback.
Abstract
We propose a two‑axis taxonomy for multimodal agent memory and outline its design tiers.
Multimodal agents need persistent memory to operate over long horizons, yet existing systems lack a unified taxonomy. We introduce a two‑axis analytical framework—Axis I (Formation and Representation) describing what information is stored and how it is encoded, and Axis II (Memory Management) describing how memory adapts with experience. Axis II distinguishes three tiers: T1 Static Memory (fixed reuse), T2 Dynamic Memory (policy‑driven updates), and T3 Self‑Evolving Memory (policy that itself evolves).
This figure lists 37 institutions and organizations, each represented by a numbered superscript and their respective logo followed by their name.
The Challenge of Multimodal Memory
We argue that multimodal agents need persistent memory and a two‑axis taxonomy to guide design.
Large foundation models such as LLMs and MLLMs now drive AI agents that observe, reason, and act across multimodal environments. As interactions unfold over time, agents must retain information beyond the immediate context, making persistent memory a core component.
Memory lets multimodal agents maintain continuity, personalize behavior, and refine decisions by preserving experience across interactions.
**Figure 1.** Two complementary axes of multimodal agent memory. Axis I, Formation and Representation, characterizes the memory representation supplied to downstream computation. Axis II, Memory Management, characterizes the scope of persistent change in memory state and management policy.
The shift from one‑shot reasoning to continuous interaction loops drives the need for persistent, adaptable memory.
Foundations of Agent Interaction
Background surveys multimodal memory approaches, highlighting representation trade‑offs and management tiers.
Multimodal agents balance rich perceptual fidelity against storage cost; static memories preserve detail, dynamic memories trade compactness for adaptability, and self‑evolving memories aim to improve both over time.
Static memory stores fixed representations without modifying content during deployment. Methods such as MemoryBank, PIXELRAG, and VisRAG retain raw perceptual evidence or indexed tokens, enabling direct retrieval but incurring high storage overhead.
Dynamic memory updates its contents as agents interact, supporting revision, consolidation, and forgetting. Approaches like DynaMem, MovieChat, and StreamChat modify stored entries to reflect new observations, trading stability for adaptability.
Self‑evolving memory not only updates stored items but also evolves the policy governing writes and retrieval. Systems such as H2‑EMV, RRM, and ABot‑AgentOS learn how to manage their memory jointly with the agent’s task objectives.
These methods retain modality‑native evidence such as raw images, video frames, or 3D maps, allowing agents to query the original perceptual data when needed. They excel at preserving detail but incur high storage and retrieval costs.
Explicit representations translate perceptual inputs into textual or structured symbols such as captions, scene graphs, or event records. This facilitates efficient search and manipulation while abstracting away low‑level detail.
Latent approaches embed experiences into learned vector spaces or directly update model parameters, offering compact, differentiable memory that integrates tightly with the agent’s policy but can be opaque to inspection.
Hybrid systems combine multiple representation types, linking compact abstractions to recoverable source evidence. They aim to balance fidelity and efficiency by composing explicit records with latent embeddings.
**Figure 2.** Publication landscape of multimodal agent memory. Representative methods across image, video, GUI, and embodied settings from 2023 to 2026, organized by memory-management tier: T1 Static Memory, T2 Dynamic Memory, and T3 Self-Evolving Memory.
**Figure 3.** Cumulative growth of multimodal-agent memory architectures differs across modalities. Cumulative architecture counts in the taxonomy-current corpus are plotted by publication year from 2023 to 2026 for embodied, video, image, GUI, and audio/speech modalities. Embodied and video memory architectures show the steepest growth by 2026, followed by image and GUI systems, whereas audio/speech memory remains comparatively sparse. The shaded interval highlights the recent growth period after 2025.
**Figure 6.** Representations in multimodal agent memory. Multimodal experience may be retained close to the source modality, transformed into explicit textual or structured records, or internalized as latent states or parametric memory. We classify each memory path by the representation ultimately supplied to downstream computation, rather than by formats used only for storage or indexing. These families expose recurring trade-offs among recoverability, interpretability, compactness, and tight integration with the agent policy. Hybrid architectures combine multiple representations, often linking compact abstractions to recoverable source evidence.
Axis I: Formation and Representation
Mapping how multimodal agents form memory and what they store for future reasoning.
Across the surveyed systems, the field converges on three representation families—source‑specific, cross‑represented, and internalized latent—each trading off recoverability, compactness, and integration with the agent policy.
This axis groups memory architectures by the form of representation they expose to downstream reasoning or action, rather than by storage format alone.
How does internalized latent memory differ from modality‑specific encodings?
Modality‑specific encodings are stored as feature vectors that the downstream model reads directly; internalized latent memory, by contrast, lives as hidden states or key‑value caches that are updated during inference and never appear as explicit inputs, making it harder to inspect but cheaper to reuse.
These representations keep the original perceptual modality (images, video frames, audio clips) or its learned feature encoding as the memory unit. They enable the agent to retrieve raw evidence or modality‑native tokens during inference, preserving fine‑grained detail at the expense of storage and processing cost.
This family transforms multimodal observations into textual or graph‑based records. Natural‑language captions, event summaries, and scene graphs become the memory units that downstream models consume directly, enabling inspection, editing, and semantic retrieval.
Memory is embedded directly into the model’s computation either as persistent hidden states / key‑value caches (latent) or as weight updates / adapters (parametric). The agent accesses past experience without explicit retrieval, achieving high compactness.
**Figure 4:** The multimodal memory lifecycle illustrated through a household-robot example. The same experience forms alternative memory representations, managed as Static, Dynamic, or Self-Evolving Memory, and later accessed and used for action.
**Figure 5.** Taxonomy of multimodal agent memory along two complementary axes: Formation and Representation, and Memory Management. T1 Static Memory stores and reuses fixed memory; T2 Dynamic Memory updates memory through a stable policy; T3 Self-Evolving Memory updates memory through an evolving policy. Representative systems may incorporate multiple mechanisms across memory lifecycle stages and therefore appear in multiple categories.
Axis II: Memory Management
Memory Management tiers trade stability for adaptability, revealing a design gap in policy learning.
Static memory is cheap but immutable; dynamic memory adds controlled updates; self‑evolving memory closes the loop by learning its own update rules.
Memory Management decides whether stored knowledge stays fixed, is periodically updated, or even learns how to update itself.
How does “static” differ from “dynamic” if both store the same data?
Static memory never changes after the initial encoding, so any new observation is ignored; dynamic memory applies a stable policy that can insert, modify, or delete entries during later interactions.
Static memory stores a fixed representation that is created once and then reused across tasks. It is ideal for slowly changing domains such as medical knowledge bases, long videos, or GUI environments, where the cost of initial encoding is amortized over many queries. The tier excels at providing consistent behavior but cannot incorporate corrections or new evidence after formation, so any omissions or errors persist.
Dynamic memory applies a stable management policy that can modify the stored state during ongoing interaction. The policy may be hand‑crafted or learned offline, but it does not evolve at inference time. Five fundamental operations—Update, Consolidation, Revision, Reorganization, and Forgetting—cover the full lifecycle of a mutable store. Each operation is characterized by how it handles heterogeneous multimodal representations (local, coordinated, or fused) and by the computational mechanism (hard‑coded heuristics, prompted LLMs, trained modules, or iterative optimization).
Self‑evolving memory closes the feedback loop: experience not only updates the stored state but also adapts the policy that governs future updates, maintenance, and retrieval. Policy evolution can occur in three sub‑domains: write‑policy (what to admit and how to encode it), maintenance‑policy (what to retain, abstract, or discard), and retrieval‑policy (how to formulate queries and prioritize candidates). The evolving policy may be represented by learned parameters, symbolic rules, utility functions, or executable programs, and it persists across deployments.
**Figure 7.** Progression across the three memory-management tiers. T1 Static Memory stores and reuses fixed memory. T2 Dynamic Memory updates memory through a stable policy. T3 Self-Evolving Memory updates memory through an evolving policy. The tiers therefore represent an increasing scope of change, from memory access to memory-state updates and, finally, policy evolution.
Table 1 collects the tier‑wise scope, core requirement, and representational bottlenecks, providing a compact reference for the three memory‑management categories.
Static memory is a read‑only store: once the data is encoded, the system can only retrieve it.
Why would a system choose an immutable store when the world changes?
When the underlying knowledge changes rarely (e.g., medical guidelines) the amortized cost of a single heavy encoding outweighs the occasional need to update, and immutability guarantees consistent behavior.
Dynamic memory maintains a mutable state that a stable policy can update, consolidate, revise, reorganize, or forget.
How does a “stable” policy differ from a learned one?
A stable policy is learned offline and then frozen; it cannot adapt to new feedback during deployment, unlike the evolving policies of T3.
Self‑evolving memory not only changes its stored state but also continuously refines the very rules that govern those changes.
What prevents a self‑evolving policy from drifting into harmful behavior?
The paper emphasizes traceability and validation: policy updates are logged, and downstream utility checks ensure that revisions improve task performance before being persisted.
Cross-Axis Analysis and System Design
We map how representation choices shape memory‑management tiers and reveal dominant design patterns.
Across 297 surveyed architectures, dynamic update policies (T2) dominate, hybrid representations are the most frequent design choice, and only explicit memory artifacts support deployment‑time policy evolution.
This family stores raw modality observations (e.g., images, video frames, audio clips) unchanged, preserving the original evidence for later verification. Because the stored items are immutable, editability is limited to appending or replacing whole observations, not correcting semantic errors within them.
Explicit memory separates facts, events, entities, and attributes into distinct, addressable units. Each unit can be directly edited, versioned, and linked to provenance records, making it well‑suited for precise updates and verification.
Information is encoded in model parameters, hidden states, or learned adapters. The representation is distributed, making targeted edits difficult and verification reliant on behavioral tests rather than direct inspection.
Hybrid systems combine source‑preserving artifacts with explicit, cross‑represented units, aiming to balance fidelity of raw evidence with the editability of structured records. The dual pathways enable both verification against original media and targeted semantic updates.
Cross‑Axis Interaction captures how a representation choice (Axis I) directly shapes the feasibility of memory‑management operations (Axis II).
How does Cross‑Axis Interaction differ from simply saying that representation and management are separate concerns?
It emphasizes a causal link: the chosen representation limits what management operations are possible (e.g., editability or verification), so the two axes cannot be optimized in isolation.
**Figure 8.** Mosaic plot of representation types across management tiers for 297 multimodal memory architectures. Rectangle area and labels show counts. Colours encode Pearson residuals: blue indicates fewer architectures than expected, yellow indicates indicates near expectation, and peach indicates more than expected.
**Figure 10.** Multimodal-agent memory architectures concentrate in T2 Dynamic Memory across modalities. Cumulative counts of architectures in the taxonomy-current corpus show rapid growth after 2025, with most systems classified as T2 Dynamic Memory rather than T1 Static Memory or T3 Self-Evolving Memory. The matrix summarizes architecture-level incidence across memory-management tier and representation family for all modalities. Each dot denotes one architecture, positioned horizontally by publication year; architectures published before 2023 are grouped near the approximate 2023 position, marked as ~'23. Irregular envelopes summarize the density of architectures in each cell, and cell labels report counts. Representation-family counts are non-exclusive, so an architecture may appear in multiple columns when it combines source-preserving evidence, cross-represented explicit records, latent or parametric state, or hybrid multi-path memory.
**Figure 11:** Systems realization and lifecycle of multimodal agent memory. A conceptual organization of the mechanisms that connect logical memory with runtime execution and physical storage, rather than an architecture implemented in full by any single analyzed system. The **control plane** (§6.4) covers logical-memory and execution-state lookup, reuse validation, and resource-aware admission and scheduling. The **cross-layer runtime** (§6.5) connects memory to computation through materialization and connects execution history to retained memory through write-back. The **physical memory hierarchy** (§6.2, §6.3) covers residency, replication, transfer, compression, and reconstruction across accelerator HBM, host DRAM, local SSDs, and remote resources. **Consistency and coordination** (§6.6) concern identity, visibility, provenance, invalidation, and isolation across these paths; the guarantees of individual mechanisms remain scoped to the artifacts they manage.
Evaluating Multimodal Memory Systems
Maps benchmark coverage, evaluation dimensions, and identifies gaps in multimodal memory assessment.
The survey reveals that Dynamic Memory (T2) is the dominant architecture across image, video, GUI, and embodied benchmarks, while Static (T1) and Self‑Evolving (T3) memories remain sparsely represented.
Static Memory stores a fixed representation that is written once during training and never updated at inference time. Surveyed systems in this tier typically embed a frozen encoder output or a pre‑computed cache and rely on it for downstream tasks without runtime modification.
Dynamic Memory continuously updates its internal state during interaction, allowing new observations to be written and old ones to be revised. The majority of surveyed architectures adopt this tier, enabling agents to maintain coherence across turns, sessions, or frames.
Self‑Evolving Memory not only updates at runtime but also adapts its update policy based on deployment feedback, effectively learning how to manage memory itself. This tier is rare in current literature, appearing only in a handful of recent works.
**Figure 9.** T2 Dynamic Memory predominates across image, video, GUI, and embodied memory systems in our taxonomy. Architecture-level incidence is shown for image, video, GUI, and embodied memory systems in the taxonomy-current corpus. Within each modality panel, rows correspond to memory-management tiers and columns correspond to representation families: source-preserving modality-native memory, cross-represented explicit memory, latent or parametric memory, and hybrid multi-path memory. Each dot denotes one architecture, positioned horizontally by publication year from approximately 2023 to 2026; architectures published before 2023 are grouped near the ~'23 position. Irregular envelopes summarize the density of architectures in each cell, and cell labels report counts. Representation-family counts are non-exclusive, so an architecture may be counted in multiple columns when it combines multiple memory representations.
Benchmark Landscape and Capability Coverage
Survey of multimodal agent benchmarks reveals a recall‑centric focus and a lack of memory‑grounded action evaluation.
Across multimodal agent benchmarks, evaluation overwhelmingly focuses on recall‑oriented tasks, while memory‑grounded action and conflict resolution are sparsely measured.
Image benchmarks treat memory as a collection of visual episodes linked to dialogue, time, or personal context. They primarily evaluate question answering and retrieval, with occasional generation that uses images directly. Most resources replace raw images with captions for QA, reserving the visual modality for generation or fine‑grained evidence tests. Supervision is largely answer‑ and evidence‑based; provenance, state updates, and action labels appear only in a minority of datasets.
Video benchmarks span three regimes: fixed‑recording lifelog recall, causal streaming evaluation, and future‑oriented interactive assistance. All rely on question answering or retrieval, but streaming settings introduce incremental evidence arrival. Multi‑view benchmarks require alignment across agents or cameras, testing distributed evidence fusion. Only a few tasks assess whether memory influences subsequent actions, leaving dynamic memory‑management largely unevaluated.
Audio benchmarks focus on long‑horizon spoken dialogue and streaming audiovisual experience. The dominant protocol is answer‑, evidence‑, and timestamp‑based supervision. ChatMem‑Bench / VoiceMem provide extensive speech‑grounded QA with speaker and acoustic cues, while other datasets (e.g., EgoMem) combine voice with visual identity. Fixed‑recording audio tasks (PlanRAG‑Audio, LongAudioQA) assess static retrieval, largely falling into T1 static memory. Interactive ASR benchmarks probe state repair but do not evaluate persistent memory across sessions.
GUI benchmarks assess cross‑step state retention, cross‑session experience reuse, and action‑outcome tracking within mobile or desktop interfaces. Tasks span direct evaluation of GUI‑state tracking (GS), action‑outcome (AO), and cross‑session reuse (XR). Benchmarks such as MemGUI‑Bench and MementoGUI‑Bench provide hierarchical metrics for fidelity, learning, and efficiency. Personalization (KnowU‑Bench) and controlled synthetic suites (Memory‑World, DataScope) add preference inference and deterministic evaluation, yet conflict resolution and policy evolution are weakly covered.
Embodied benchmarks test agents that navigate, map, and manipulate partially observable environments. They are organized into navigation/mapping, manipulation, and long‑horizon interaction regimes. Tasks require spatial memory (SM), object‑state tracking (OS), and task‑state progress (TS). Datasets such as FindingDory, 3DMem‑Bench, and SpaMEM evaluate spatial persistence and object histories, while MEMORA‑Bench and APRS probe planning and perception under uncertainty. Manipulation suites (LIBERO‑Mem, MIKASA‑Robo, RoboMME) demand non‑Markovian object histories. Long‑session benchmarks (WorldLines, WorldMemArena, RoboMME‑Interference) begin to assess dynamic memory updates, yet conflict resolution, unanswerable queries, and lifelong policy evolution remain scarce.
Future Directions and Open Challenges
We outline open challenges in multimodal memory formation, management, and deployment.
The preceding sections identified three coupled concerns: what experience is retained and how it is represented, how memory is maintained as new evidence arrives, and how logical operations are realized under finite resources. Their interaction grows more consequential over long trajectories, changing environments, and shared deployments. Accordingly we distinguish challenges in formation and representation, challenges in memory management, and tensions between the two, before discussing deployment‑oriented evaluation and infrastructure problems.
Recoverable representation requires moving from lossy compression toward representations that preserve links to supporting evidence. Current systems often lack a deliberately carried provenance bundle, so defining a minimal set of fields that identify the source, production method, and supporting observations is an open direction.
Semantic addressability asks whether specific content can be identified and changed within a memory representation. Latent and parametric memories lack explicit units, making updates risky because unintended information may be altered; achieving identifiable units without sacrificing compactness remains a key open problem.
Maintaining up‑to‑date memory must go beyond merely recording changes; the system needs to know whether an older state is still valid, superseded, or historical. Multimodal streams can present conflicting observations, so policies should explicitly model status flags such as current, historical, uncertain, or superseded.