Metis: Memory Foundation Model

Zeyu Zhang, Ziliang Guo, Yihang Sun, Xichong Zhang, Xixuan Hao, Zehao Lin, Yang Zhang, Xiaoyan Zhao, Tong Shen, Bo Tang, Zhi-Qin John Xu, Junchi Yan, Haofen Wang, Xu Chen, Feiyu Xiong, Zhiyu Li, Tat-Seng Chua

Metis internalizes memory into foundation models as a dynamic, parametric state updated via autonomous forward computation.

How can we transform LLMs from stateless predictors into stateful learners by integrating memory storage and retrieval directly into the model's internal forward pass?

AI agents currently rely on external memory modules like Retrieval-Augmented Generation (RAG), which decouple memory from the backbone and prevent end-to-end optimization. Metis replaces these external modules with a native memory architecture: it compresses historical information into a dynamic, parametric memory state that is updated and accessed autonomously during the model's forward pass. This approach eliminates the need for external retrieval and concatenation, allowing the model to maintain statefulness across multi-step interactions while remaining fully differentiable.

Paper Primer

Metis integrates memory into the Transformer backbone using Metis blocks, which consist of a local memory block (the dynamic state) and a hyper memory block (the static guidance). The model performs memory storage and utilization as native operations: it selects important hidden states to update the dense memory network and uses memory attention to fuse this stored information with standard causal attention.

Metis achieves native memory capability without increasing inference latency relative to external memory systems.

The memory storage and utilization branches are decoupled from the original attention path, allowing them to execute in parallel during the forward pass. Layer-level latency is determined by the maximum of the parallel branches rather than their sum.

Why move memory inside the model instead of using standard RAG?

External memory is decoupled from the backbone, making end-to-end optimization impossible because gradients cannot propagate through discrete retrieval operations. Metis internalizes these procedures into a continuous function space, enabling data-driven post-training and eliminating the overhead of external storage and prefilling.

How does the model handle the "time-streaming" nature of online information?

Metis treats memory as a prediction problem where the model learns to store information based on its predicted future utility. It uses specific optimization objectives—memory reconstruction and memory operation—to train the model to autonomously remember, forget, and update its internal state during inference.

The Memory Bottleneck in AI Agents

We expose why external memory hampers Transformers and motivate native memory as a unified alternative.

External memory modules are attached to Transformers, but this design introduces several fundamental issues.

Standard Transformers keep memory in a separate cache, which forces a split between storage and reasoning; integrating memory directly into the model’s forward pass removes that split and enables end‑to‑end learning.

**Figure 1.** From external memory to native memory.

Metis embodies this native‑memory approach: a dedicated block stores a compact state and updates it during the forward pass, eliminating any external cache.

Training Metis relies on two objectives: a memory‑reconstruction loss that forces the state to retain useful information, and a memory‑operation loss that teaches the model to perform remembering, forgetting, and updating autonomously.

Even with these advances, Metis shows two notable shortcomings: performance drops on very long‑term tasks because information must be compressed into a fixed‑size state, and occasional semantic blending leads to information confusion.

The key insight is that moving memory from an external cache into the model’s own parameters—native memory—unifies architecture, optimization, and efficiency.

Defining Native Memory

Native memory embeds a dynamic parameter state into the model, enabling persistent, instruction‑driven updates across interaction steps.

The preceding section highlighted the external cache that grows with each interaction, creating a memory bottleneck that scales linearly with dialogue length.

Think of the model as carrying a mutable notebook: a small set of its own parameters is rewritten on‑the‑fly, so every forward pass can read and write from this internal sheet.

Step 1: Initialize $M_1 = (0.0, 0.0)$ and $\Phi = (0.5, -0.3, 0.2, 0.1)$.

Process instruction $X_1$ → forward pass produces an intermediate activation $a_1$ that is added to $M_1$, yielding $M_2 = (0.2, -0.1)$.

Step 2: Generate response token $y_{1,1}$ using $\theta_2 = \Phi \cup M_2$; the dynamic part now biases the attention scores toward patterns seen in $X_1$.

Instruction $X_2$ arrives; the forward pass again updates $M_2$ → $M_3 = (0.35, -0.05)$.

Step 3: The model now answers a query that requires knowledge from both $X_1$ and $X_2$, leveraging $M_3$ without any external cache.

The update is a simple additive shift of the dynamic slice, yet because $M_t$ is used in every linear layer, even a tiny change propagates throughout the network, enabling persistent state.

How does native memory differ from the traditional key‑value cache used in standard Transformers?

The KV cache stores discrete token‑level vectors outside the model and is only read during attention; native memory folds a mutable parameter slice into the model itself, so reads and writes happen as part of every matrix multiplication, making the memory continuous and jointly trainable.

Beyond the technical contrast, native memory aligns with lifelong‑learning goals: the model can accumulate interaction‑specific knowledge without re‑initializing, and the same mechanism scales as agents evolve toward more autonomous reasoning.

The Metis Architecture

Metis introduces a fixed-size internal memory block that updates natively during forward passes.

Standard Transformers rely on an ever‑growing external cache; Metis instead embeds a fixed‑size, persistent state that evolves during the forward pass.

The Metis block equips each Transformer layer with two complementary memories—a fast‑changing local matrix and a slow‑changing hyper‑parameter matrix—so the model can read and write state without ever leaving the layer.

Compute importance scores $p_t$ from the aggregated vector; suppose $p_t = [0.4, 0.3, 0.2, 0.1]$.

Select the top‑2 tokens (positions 1 and 2), so $L'_t=2$ and $\Phi_t$ extracts rows 1 and 2.

Project the selected hidden states to keys and values: $\tilde K_t = [[1,0],[0,1]]$, $\tilde V_t = [[2,1],[1,2]]$.

Update the local memory: $M_{t+1}=0.5\,M_t + \frac{0.5}{2}\frac{\tilde K_t^\top\tilde V_t}{\sqrt{2}} = 0.5\,M_t + \frac{0.25}{\sqrt{2}}\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix}$.

Compute queries $\tilde Q_t$ (identity), normalize with $S_t$ (initially $[1,1]$), and obtain memory attention $\tilde A_t = \operatorname{diag}(\tilde Q_t S_t^{-1}\tilde Q_t M_{t+1})$, which yields the diagonal of $M_{t+1}$.

The update blends a fresh outer‑product of selected tokens with the existing memory, and the diagonal readout ensures each token receives a scalar memory score that can be merged with causal attention.

Native memory storage step

Native memory utilization step

**Figure 1.** Overview of the Metis framework. (a) Multi-step interaction process. (b) Integration of Metis blocks into a causal language model. (c) Detailed architecture of the Metis block, including original attention, memory attention, native storage updates, and hyper/local memory components.

Theoretical Insights on Memory Procedures

We augment each layer with a virtual memory prefix and blend its attention via a learned mask.

Standard Transformers cannot directly reuse information from earlier steps without growing an external cache, which hampers truly stateful inference.

We prepend a tiny “sticky‑note” prefix $P^{(l)}_{t}$ to the layer input, letting the model glance at a compact summary of past steps while processing the current tokens.

Form $\tilde{H}^{(l)}_{t}$ by concatenating the prefix vector $p$ and the two token vectors $h_{1},h_{2}$ → shape $(3\times4)$.

Project to queries, keys, values: $\hat{Q},\hat{K},\hat{V}$ each become $3\times d_{k}$ matrices (here $d_{k}=2$ for illustration).

Build the mask: the $3\times3$ matrix has $-\infty$ in the upper‑right $1\times2$ block, zeros elsewhere, allowing $p$ to attend to $h_{1},h_{2}$ but not vice‑versa.

Apply Softmax* to $\hat{Q}\hat{K}^{\top}/\sqrt{d_{k}}+$mask, obtaining attention weights that give the prefix a 0.3 share of the probability mass.

Blend with $\gamma$: final output for the two regular tokens = $0.5\times$(standard attention) $+$ $0.5\times$(value‑only term), yielding a representation that incorporates the prefix while still respecting causal constraints.

The prefix acts as a compact, learnable summary of all earlier steps; $\gamma$ lets the model dial its influence up or down without changing the architecture.

Theoretical Error Analysis

We dissect the sources of error in the compressed memory attention mechanism.

Standard Transformers accumulate all past KV pairs, while the hyper‑memory block compresses them into a fixed‑size matrix; we now examine how this compression introduces error.

For compactness we introduce two intermediate quantities that appear repeatedly in the summations.

Applying a first‑order Taylor expansion yields the same expression, revealing three distinct error sources: $\epsilon_{1}$ (attention error from irrelevant information), $\epsilon_{2}$ and $\epsilon_{3}$ (structural errors from global normalization).

Each error term contains at least one factor of &#x3Ctilde;$Q^{(t)}_j$(l) with j≠c; when the similarity between the query at step c and other steps is low, the corresponding error magnitude is expected to be small.

Data Construction for Native Memory

We synthesize primary and auxiliary corpora to teach Metis native‑memory operations.

Metis requires a training corpus that explicitly exercises its native‑memory operators. Without such data the model would have no supervised signal to learn storing, updating, forgetting, or reflecting information across time.

We need temporally ordered, state‑consistent sequences so the model learns to store and retrieve information at the right step — like laying a chain of dominoes where each piece must fall before the next can be knocked down.

How does this differ from ordinary data augmentation used in language‑model pre‑training?

Standard augmentation merely perturbs surface text; our pipeline creates *state‑consistent* multi‑turn interactions that explicitly train the model to write to, read from, and modify a persistent memory slot across time steps.

Turn 1 (explicit) writes the fact into memory.

Turn 2 (distractor) is processed but does not modify the memory slot.

Turn 3 (implicit) reinforces the same fact without an explicit command.

Turn 4 (query) asks “What is the capital of France?” and the model must retrieve the stored fact.

This sequence shows that the model must keep the fact alive across a non‑memory turn, proving that the memory slot truly persists.

The table categorizes operations based on interaction streaming patterns and their corresponding source benchmarks. The operations listed are Remember, Update, Forget, and Reflect, each associated with specific information flow sequences and sets of benchmark references.

Primary data alone does not stress the model with interference or memory‑pollution scenarios that occur in real conversations, so we augment it with auxiliary data targeting those edge cases.

We sprinkle challenging patterns over the clean streams so the model learns to focus on the intended memory signal, much like adding noisy background music to a rehearsal so the performer learns to keep the melody clear.

Why not simply increase the size of the primary data instead of building auxiliary patterns?

Scaling up the clean primary set would mostly add more of the same easy‑mode examples; the auxiliary patterns deliberately introduce interference and irrelevant dialogue, which the model would never see if we only enlarged the primary corpus.

Turn 1 stores fact 1 in memory slot A.

Turn 2 stores fact 2 in memory slot B.

Turn 3 issues the joint query, requiring the model to retrieve both slots simultaneously.

The correct answer concatenates “Paris, Berlin”.

This example demonstrates that the model must keep distinct entries separate and retrieve them together, exposing a potential binding error that would be invisible in single‑fact streams.

**Table 3.** Summary of the auxiliary data. Each subtype composes facts or dialogues into a complex interaction pattern, where the final answers remain consistent with the intended memory state.

Training Objectives

Training objectives shape native memory via reconstruction, operation, and a curriculum sampler.

Mid‑training, Metis must learn to store information persistently while still following instructions, a tension that standard Transformers cannot resolve because their memory grows externally.

The sampler gradually shifts training focus from easy storage examples to harder long‑range and regularization cases, letting the model first learn to write into memory before being asked to manipulate it.

Epoch 0: `w_rec`(0)=0.8, `w_op`(0)=0.2 → $\pi_{r}$ec=0.8/(0.8+0.2)=0.8, $\pi_{o}$p=0.2.

Epoch 2: linear factor = 2/4=0.5 → `w_rec`=0.8+(0.2‑0.8)*0.5=0.5, `w_op`=0.2+(0.8‑0.2)*0.5=0.5 → $\pi_{r}$ec=$\pi_{o}$p=0.5.

Epoch 4: factor=1 → `w_rec`=0.2, `w_op`=0.8 → $\pi_{r}$ec=0.2/(0.2+0.8)=0.2, $\pi_{o}$p=0.8.

The schedule forces the model to see many reconstruction examples early, then gradually exposes it to more operation examples, matching the intuition that writing must be mastered before updating.

How does this sampler differ from a simple fixed‑weight curriculum?

Unlike a static schedule, the sampler linearly anneals each subset’s weight, so the probability of drawing a step changes continuously from its start value to its end value, providing a smooth curriculum rather than a hard switch.

With the curriculum in place, three concrete objectives drive the native‑memory parameters: reconstruction, operation, and regularization.

Metis learns to store a passage verbatim by being asked later to reproduce it, forcing the internal state to retain the exact content.

Step 1 stores the embedding of “cat — sat — mat” into the native‑memory state.

Step 2 generates token 1 “cat”; the loss contribution is $-\log P(\text{cat}|…)$.

Step 2 generates token 2 “sat”; loss adds $-\log P(\text{sat}|…)$.

Step 2 generates token 3 “mat”; loss adds $-\log P(\text{mat}|…)$.

The average loss ℓ = (‑log p₁ ‑ log p₂ ‑ log p₃)/3 is minimized when each probability is 1, i.e., perfect reconstruction.

Because the target is identical to the stored reference, the model cannot rely on the instruction to guess; it must preserve the exact token sequence in its internal state.

Why not train directly on the downstream task instead of adding a separate reconstruction step?

The reconstruction step gives a strong, early signal that forces the internal state to become a reliable container; without it, the model would receive only weak indirect gradients from downstream tasks, making it hard to shape the memory representation.

Beyond copying, Metis must learn to modify its internal state according to explicit or implicit commands—updating, forgetting, or reflecting information as directed.

How does the model know whether a step is an update versus a forget operation?

The instruction $X_t$ encodes the intent: explicit samples use verbs like “update” or “forget”, while implicit samples rely on contextual cues. The target $Y_t$ is constructed to reflect the intended state change, so the loss penalizes any mismatch.

The curriculum sampler lets Metis first master lossless storage before being challenged with realistic memory operations.

Experimental Setup

Metis training combines two loss terms and evaluates on four benchmark datasets.

The training objective consists of two distinct loss terms that together supervise both operation execution and regularization.

Equations (16) and (17) define $L_{\text{op}}$ and $L_{\text{reg}}$ respectively, covering all supervised steps.

The operation loss $L_{\text{op}}$ directly supervises each memory read, write, or forget step by matching the model’s prediction to the target instruction.

The regularization loss $L_{\text{reg}}$ addresses two realistic failure modes: interference, where similar facts collide, and memory pollution, where irrelevant facts leak into responses.

We evaluate on the MemOps benchmark (Full, Gold, and a held‑out Test split) and on four QA datasets—LoCoMo (Gold), SQuAD, HotpotQA, and LongMemEval—using the original metrics reported by each benchmark.

Performance on Memory Tasks

Metis outperforms all baselines on memory tasks, especially without context.

Recall that Metis replaces the external cache with a fixed‑size native memory that updates during the forward pass, enabling truly stateful inference.

Metis‑27B attains the highest average score on the MemOps (Gold) benchmark in the no‑context setting.

Table 5 shows an average of $93.44$ for Metis‑27B, surpassing the next‑best Metis‑9B ($90.78$).

Metis‑27B also leads on memory‑based QA under no‑context, achieving the best average performance.

Table 6 reports an average of $66.54$ for Metis‑27B, ahead of Metis‑9B ($63.45$) and all baselines.

**Table 1.** Performance comparison of different models on MemOps (Gold) and Metis Test Set across Full Context, Partial Context, and No Context settings.

**Table 5.** Performance comparison of different models on MemOps (Gold) and NextMem (Contextual Generation) benchmarks under Full, Partial, and No Context settings.

Out-of-Distribution Performance

Metis transfers its native memory ability to out‑of‑distribution benchmarks, beating baselines on ATM‑Bench.

Removing adaptive aggregation (w/o SA) collapses performance, dropping the overall average by 68.63 %.

Table 7 reports a $\Delta$Avg of ‑68.63 % for the w/o SA variant relative to the full model.

Metis‑9B outperforms the strongest baseline by 14.96 points on ATM‑Bench (Gold).

Table 8 shows Metis‑9B scoring 44.40 while the best baseline (Temp‑LoRA‑27B) scores 29.44.

On MemDaily, Metis remains competitive but does not consistently lead; Temp‑LoRA‑27B still holds the top spot on several categories such as Aggreg. and Cond.

**Table.** ATM-Bench, and retrieval-target messages for MemDaily. ATM-Bench scores list-recall, number, and open-ended questions using Jaccard similarity, post-processed exact match, and an LLM judge, respectively. MemDaily reports deterministic single-choice accuracy for all six question types.

**Table 8.** Results on OOD memory benchmarks. The best and unique second-best scores are bolded and underlined, respectively. The average score is calculated according to the official category counts.

Long-term Memory Capacity

We probe how much information Metis retains across single updates and long interaction trajectories.

Step‑level capacity measures the maximum number of tokens a model can encode in a single memory update, while trajectory‑level capacity measures how many successive update steps the model can retain information over.

To evaluate these capacities we built a synthetic dataset of 20 fictional users, each defined over a shared schema of 40 distinct persona domains (e.g., demographics, education, relationships). For each user the domains are shuffled with a fixed seed, expanded into biography‑style sentences, then collapsed into concise first‑person statements; the statements form an ordered trajectory that is probed by queries.

Step‑level capacity is tested by resetting the memory state for every user at step $t$, concatenating the first $t$ statements into a single update, and then querying the model about the first, middle, and last facts of that update. Because the full history is re‑encoded from scratch each time, this isolates the amount of information a single update can handle as the input length grows.

Metis maintains high accuracy when the single update contains only a few words, but its accuracy falls sharply as the input length exceeds several hundred words. The decline is most pronounced for the first fact, while middle and last facts show larger fluctuations. In contrast, the full‑context baseline retains substantially higher accuracy across all positions, especially for the first fact.

**Figure.** Accuracy comparison across different steps (First, Middle, Last) for Metis and Full Context methods, plotted against Word Count (left) and Group Count (right).

Trajectory‑level capacity is measured by resetting the memory only once per user, then feeding the 40 statements in consecutive groups of $g=5$ statements. After each group‑level update the model is queried about the first, middle, and most‑recent facts, allowing us to track how information degrades as the number of update steps grows.

Performance again declines with longer trajectories: the accuracy of the first fact drops almost monotonically, indicating that early information is progressively weakened. Middle and recent facts remain unstable, suggesting that each new update introduces interference throughout the entire memory state rather than simply overwriting the oldest entries.

Backbone Capability Preservation

We evaluate Metas‑4B versus Qwen3.5‑4B on general benchmarks in empty and memory‑filled states.

We assess Metas‑4B against Qwen3.5‑4B on four general benchmarks in two settings: the Initial Stage ($t=1$) with an empty memory, and the Active Stage ($t>1$) after storing irrelevant messages.

**Table 9.** Performance comparison between Qwen3.5-4B and Metis-4B across different benchmarks in the Initial and Active stages.

Storage Efficiency via Decomposition

Ablation shows low‑rank memory compression recovers performance at modest rank.

We evaluate how compressing the Metis memory state with low‑rank decomposition impacts the four benchmarks.

We approximate the full memory matrix by keeping only its top singular directions, discarding the rest.

How does this differ from simply reducing the dimensionality of each memory token?

Dimensionality reduction treats each token independently, breaking the cross‑token correlations captured by the full matrix. Low‑rank decomposition operates on the entire matrix, preserving those correlations while still discarding redundant directions.

**Figure 4** Results of low-rank decomposition under different ranks of memory states.

Table 10 aggregates these trends across the four benchmarks, revealing dataset‑specific sensitivities to compression.

At rank 1 the overall score collapses to 14.43, a 43.5 % drop relative to the full‑rank model.

Table 10 reports 14.43 (43.5 %) for the overall metric when $k\!=\!1$.

At rank 64 the overall score reaches 33.10, recovering 99.9 % of the full‑rank performance.

Table 10 shows 33.10 (99.9 %) for $k\!=\!64$.

Increasing the rank beyond 64 yields at most a 0.5 % improvement, indicating negligible benefit.

Overall scores rise from 33.10 at $k\!=\!64$ to 33.26 at $k\!=\!128$ (≈0.5 % gain) and then fluctuate.

Related Work and Case Studies

We situate Metis among external‑memory baselines and related paradigms.

Recall that Metis replaces the external cache of standard Transformers with a fixed‑size internal state that updates during the forward pass. Related work on external‑memory baselines falls into three families: textual, latent, and parametric approaches.

External‑memory baselines keep information outside the model’s parameters and retrieve it on demand, in contrast to Metis’s native internal state.

**Figure 5.** Case studies of Metis on different conversational scenarios.

Roadmap and Implementation Details

Appendix details the roadmap, experiments, and efficiency analysis for memory foundation models.

Figure 6 visualises a five‑level roadmap for memory foundation models, progressing from a basic stateful capability to self‑evolving intelligence.

Level I replaces the static predictor with a model that carries a mutable internal state $M_t$, so each output $Y_t$ depends on both the current input $X_t$ and the previous state.

Level II endows the model with a learned memory lifecycle: it must decide what to remember, update, consolidate, or forget, turning memory operations into trainable components.

Level III upgrades memory from a passive store to an experience‑driven learning mechanism, allowing interaction histories to shape representations and behaviours across tasks.

Level IV introduces persistent cognitive structures that continuously model world, user, task, and self, enabling coherent planning and personalization over long horizons.

Level V aspires to open‑ended capability evolution: the model abstracts valuable experiences into new knowledge structures and learning strategies, feeding them back into future adaptation.

The outlook emphasises that memory should be seen as a paradigm shift, turning foundation models from stateless predictors into self‑evolving learners.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers