Memory for Large Language Model Agents: A Systematic Evaluation of Agent Memory from a Data Management Perspective

Wei Zhou, Xuanhe Zhou, Shaokun Han, Hongming Xu, Guoliang Li, Zhiyu Li, Feiyu Xiong, Fan Wu

A systematic evaluation of agent memory systems, decomposing them into four data-management modules to identify architectural trade-offs.

How do different LLM agent memory architectures compare in terms of retrieval fidelity, long-horizon stability, and operational cost when viewed as data management systems?

Agent memory systems are currently evaluated as monolithic black boxes, making it impossible to determine why specific architectures fail under complex, long-horizon tasks. The authors decompose these systems into four distinct data-management modules—representation, extraction, retrieval, and maintenance—and evaluate 12 representative architectures across five benchmark workloads. No single architecture dominates; effectiveness depends on aligning the memory structure with the specific workload bottleneck, such as temporal reasoning or stateful execution.

Paper Primer

The paper formalizes agent memory as a four-module system: Representation (logical/physical storage), Extraction (transforming raw input into primitives), Retrieval (routing and indexing), and Maintenance (lifecycle governance). This framework allows for fine-grained ablation, isolating whether a failure stems from poor semantic compression, stale updates, or inefficient retrieval routing.

Memory architecture effectiveness is workload-dependent rather than universal.

Composite hybrid systems excel in conversational QA, while graph-based methods outperform others in single-hop factual recall but struggle with temporal reasoning. Performance gaps between architectures are significant, with structure-aware systems like Zep reaching 48.0 LLM Judge Accuracy on long-memory tasks, while others fail to maintain temporal consistency.

Retrieval fidelity degrades sharply with temporal distance in similarity-based systems.

Ablation studies show that while flat embedding-based retrieval is effective for recent context, it fails as the evidence-to-query distance increases, whereas hierarchical or graph-based structures maintain higher Recall@10. Flat retrieval baselines drop sharply after the shortest evidence-distance bin, while structured systems like A-MEM maintain significantly higher recall at larger retrieval budgets.

Why is a data-management perspective necessary for evaluating agent memory?

Existing benchmarks treat memory as a black box, measuring only end-to-end success (e.g., F1 scores). This obscures critical system-level failures like factual contradictions, stale data, and high latency, which are only visible when analyzing the underlying storage and maintenance lifecycle.

How do agent memory workloads differ from traditional database workloads?

Unlike traditional databases that rely on exact predicates, agent memory requires semantic matching, handles continuous and often contradictory updates, and must support heterogeneous access patterns ranging from long-context synthesis to structured fact lookup.

The authors identify that "hallucinations of the past" occur primarily in systems lacking explicit lifecycle management, where append-only stores return stale facts because they fail to resolve contradictions or perform targeted overwrites.

For developers, this paper shifts the focus from "which model is best" to "which memory structure matches the workload." Future agent-native systems should prioritize structured evidence organization over simple flat-embedding caches to ensure long-horizon stability.

The Evolution of Agent Memory

We frame agent memory as a data‑management stack and outline its four functional modules.

Memory for large language model (LLM) agents has moved from simple retrieval buffers to full‑blown data‑management stacks, yet evaluations still treat the memory layer as a monolithic black box. This mismatch hides critical trade‑offs in cost, robustness, and update handling. Think of the evolution like a single bookshelf expanding into a multi‑floor library with cataloguing, shelving, and circulation systems.

Agent memory must provide persistent storage, efficient extraction, accurate retrieval, and disciplined maintenance so that long‑horizon agents can reason over evolving information without overloading the LLM’s context window.

**Figure 1.** Typical Execution Workflows of Agent Memory.

Viewing memory as a data‑management stack clarifies where bottlenecks arise and guides systematic design.

Defining Agent Memory Scope

Defines agent memory as a data‑management object and outlines its four lifecycle modules.

We treat agent memory as a standalone data‑management object rather than merely an algorithmic component of the LLM pipeline. This perspective lets us examine how memory is represented, stored, retrieved, updated, and maintained under realistic agent workloads.

Memory for an LLM agent spans several categories. Along the temporal axis, short‑term memory holds the volatile state of an ongoing session, while long‑term memory persists across sessions. Functionally, long‑term memory includes episodic records, semantic facts, procedural strategies, and user preferences.

Formally, we denote the agent’s persistent memory as $M$, a data‑management object that accumulates state beyond a single inference step and is accessible to the agent during future reasoning and action.

To operationalize $M$, we define an agent memory system $M_{\text{sys}} = \langle R, S, Q, U\rangle$, a tuple of four modules governing the memory lifecycle. $R$ handles logical representation and physical storage; $S$ extracts primitives from heterogeneous inputs; $Q$ retrieves relevant subsets based on a query; $U$ maintains entries through conflict resolution, capacity management, and semantic consolidation.

Unlike Retrieval‑Augmented Generation (RAG), which is a stateless, read‑only fetch of a static corpus for a single generation step, an agent memory system is persistent, updatable, and governs the full long‑term lifecycle of agent‑specific state. Context engineering similarly focuses on curating the LLM’s immediate context window, whereas our system manages state across many interactions.

Agent memory workloads also diverge from traditional database OLTP/OLAP workloads. Access patterns are semantic, often expressed in natural language, requiring approximate matching rather than exact predicate evaluation. Memory entries evolve continuously, accommodating uncertain and contradictory observations, and workloads combine long‑context synthesis, episodic recall, structured lookup, temporal reasoning, and streaming updates within a single architecture.

Taxonomy of Memory Components

Maps memory representation styles and their trade‑offs.

The section classifies how agents encode their knowledge and where that data lives, exposing the core trade‑offs that shape downstream reasoning.

Agent memory systems fall into three high‑level camps—flat token sequences, structured graphs/trees, and heterogeneous composites—each balancing expressiveness, retrieval granularity, and storage cost.

Why isn’t a flat token list sufficient for all reasoning tasks?

Because without explicit relational links the system cannot efficiently answer queries that require traversing entity relationships or hierarchical summaries; graph or tree structures encode those links directly, enabling constant‑time edge look‑ups.

Memory representation defines the logical model that maps raw observations into a form the agent can query and manipulate, while storage decides where that model lives physically.

What would break if a graph‑based system stored its nodes in a flat vector cache?

The adjacency information would be lost, so relational queries would devolve into costly linear scans, negating the primary advantage of the graph topology.

Flat, one‑dimensional memory either as discrete text tokens or continuous latent vectors.

Memory organized as nodes and edges (graphs) or hierarchical parent‑child trees.

Memory objects that bundle text, metadata, embeddings, and graph links into a single container.

**Figure 2.** Memory Representation Methods.

**Table 1.** Taxonomy and Characteristics of Agent Memory Systems.

Memory Extraction Strategies

Defines extraction pipelines and contrasts three extraction approaches.

Memory extraction is the step that turns raw interaction traces into structured primitives ready for storage.

Extraction pulls the useful nuggets out of a raw conversation, turning a stream of tokens into discrete facts or embeddings that the memory system can later retrieve.

How does Memory Extraction differ from Memory Retrieval?

Extraction creates the stored representation from raw inputs; retrieval only reads an already‑stored representation. Extraction decides *what* to store and *how* to structure it, while retrieval decides *which* stored item to surface given a query.

Stores the raw token stream directly, without any parsing or structuring.

Distills unstructured input into independent factual statements or dense embeddings without a predefined schema.

Uses an LLM to fill a rigid, predefined schema (e.g., JSON or graph triples), producing strictly typed records.

**Figure 4.** Memory Extraction Methods.

With extraction defined and the three families compared, the next stage is to retrieve the stored primitives efficiently.

Retrieval and Query Routing

This section maps five retrieval paradigms, their trade‑offs, and how agents route queries.

Memory Retrieval is the agent’s query engine: given a current request, it selects and returns the most relevant stored facts so the reasoning module can use them.

How does Autonomous Agentic Routing differ from the other retrieval paradigms?

Unlike index‑driven methods that scan a pre‑built structure, Autonomous Agentic Routing lets the LLM itself generate a plan—issuing function calls or expanding the query—so the retrieval process is guided dynamically rather than being a fixed lookup.

Uses the transformer’s own self‑attention over the KV cache as a retrieval mechanism, avoiding any external database access.

Maps queries and memories into a continuous latent space and performs K‑Nearest‑Neighbor search using cosine similarity.

Retrieves information by walking explicit edges in a knowledge graph, extracting semantically linked clusters.

Delegates retrieval planning to the LLM itself, which can emit tool‑calls or generate refined queries.

Combines multiple retrieval engines in a pipeline, either sequentially pruning candidates or in parallel before fusion.

**Figure 5.** Memory Retrieval Methods.

Memory Maintenance and Updates

We categorize memory‑maintenance strategies and compare their update style, eviction trigger, overhead, and fidelity.

Memory Maintenance is the set of operations that keep an agent’s stored facts coherent, bounded, and useful over time — it decides what stays, what is merged, and what is discarded. It is like a librarian who not only shelves books but also archives old editions, merges duplicate copies, and periodically reorganizes the catalog.

How does Memory Maintenance differ from a simple cache eviction policy?

Cache eviction only drops items based on a fixed rule (e.g., LRU) and discards history. Memory Maintenance combines logical versioning, semantic merging, and learned scoring, so facts can be retained in a compressed form or updated rather than outright lost.

Uses append‑only logs and timestamps to mark facts as valid or obsolete, preserving a full history.

Physically drops or overwrites data once storage limits are reached, using deterministic or score‑based policies.

Leverages an LLM to merge redundant observations into dense summaries and to issue explicit CRUD commands.

Runs heavy neural updates as asynchronous background processes, fine‑tuning model parameters from stored interactions.

**Figure 6.** Memory Maintenance Methods.

End-to-End System Assessment

We rank memory systems by how well they preserve task‑critical evidence across benchmarks.

Recall that the paper treats agent memory as a stack of four functional components to locate bottlenecks. Here we compare twelve representative systems on three end‑to‑end workloads to see which design choices actually move the needle.

MemoChat attains the highest Task Success Rate on DB‑Bench, reaching 55.40 %.

MemoChat’s 55.40 % success surpasses the next best Long Context (48.20 % EM) and all other evaluated systems.

Simple retrieval or embedding‑based approaches that provide a static context without structured filtering.

Methods that preserve the order of tokens and apply simple sliding‑window or recurrent filters.

Systems that organize memory into graphs or temporal trees, enabling relation‑aware retrieval.

Approaches that combine coarse‑to‑fine filtering with trace‑preserving mechanisms.

**Figure 7.** Effectiveness of Memory Systems over LoCoMo, MemoryAgentBench (LongMemEval), LifeLongAgentBench (DB-Bench).

Overall, Structural Topological and Multi‑Paradigm Hybrid systems rank highest, indicating that preserving the right evidence at the appropriate abstraction level is more decisive than raw token matching.

Retrieval Fidelity and Robustness

Evaluates how memory systems retrieve evidence and stay robust to updates.

This section quantifies how well each memory system surfaces the exact evidence a query needs and how stable that behavior remains after facts are revised or the underlying language model changes.

Extends context window by concatenating past tokens; treats memory as a flat sequence.

Retrieves dense embeddings from a static index; no explicit structure beyond similarity scores.

Agent‑level controller that decides which memory slots to query based on learned policies.

Chat‑style memory that appends each turn to a growing transcript and attends over the whole history.

Graph‑organized memory where nodes represent entities and edges encode temporal relations.

Hierarchical tree where leaves store raw snippets and internal nodes aggregate evidence summaries.

Hybrid approach that first filters with a dense index then refines using a lightweight graph.

Lightweight memory that stores only high‑frequency facts in a compact cache.

Flat memory that stores raw snippets without any linking or hierarchy.

Hybrid filtered memory that first selects a coarse evidence set then applies a fine‑grained scorer.

Operates like an operating system for memory: allocates, deallocates, and version‑controls facts.

Structured evidence space that links related snippets via learned relations.

**Figure 10.** (a) Context-length robustness on LongBench; (b) Session-history growth on LongMemEval; (c) Temporal evidence-distance drift on LoCoMo.

Stability and Operational Cost

Maps memory system families onto cost‑utility trade‑offs and highlights their disagreements.

We now view the evaluated memory systems as four camps on a cost‑utility map. The map makes explicit where the families agree and where they diverge on latency, construction overhead, and overall utility.

Flat, long‑context prompting without any structural indexing or consolidation.

Systems that extend the context window (e.g., LongCtx, SimpleMem) but keep a flat representation.

Graph‑ or relation‑aware memories (Cognee, Zep) that explicitly link entities across time.

Hybrid pipelines that first locate the relevant session (coarse) then resolve details (fine), e.g., MemOS, MemoryOS.

Component-Level Ablation Study

We isolate each memory component to see which design choices drive overall performance.

The paper’s core premise treats agent memory as a four‑stage data‑management stack; here we vary one stage at a time to pinpoint its impact on end‑to‑end results.

MemOS Fast Memorize attains the highest Answer F1 (40.8) on LoCoMo among representation variants.

Table 3 shows MemOS Fast Memorize reaching 40.8 Ans. F1 versus 39.7 for the next best LightMem variant.

A‑MEM with Hybrid‑Balanced fusion achieves the top Answer F1 (24.6) on LoCoMo among retrieval variants.

Table 5 reports 24.6 Ans. F1 for Hybrid‑Balanced, outperforming Hybrid‑Sparse‑Leaning (23.0).

MemoryOS Conservative‑Merge improves Answer F1 to 23.5 on LoCoMo, surpassing the default setting.

Table 6 (referenced in the text) shows an increase from 23.2 to 23.5 Ans. F1 when using Conservative‑Merge.

**Figure 9.** Ablation of LLM Backbones.

**Figure 12.** Ablation of Maintenance Strategies.

Tables 2–5 collectively confirm the three findings: (1) coarse, coverage‑preserving extraction (Table 4) stabilizes downstream reasoning; (2) modest hybrid fusion with lightweight planning (Table 5) yields the best retrieval quality; (3) balanced maintenance—conservative merging without delayed flushing—maximizes long‑horizon consistency (Table 6, Figure 12).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers